Log 011: The Taxonomy

Date: 2026-03-03 Version: v0.5.4 Pipeline: Scene-aware core validation

Judit's Feedback

"These segments are obviously wrong. You're caught in the trees but need a way to segment the forest. These ornate pieces are clearly not net or even connected to the male figure. Maybe you need a way to find and eliminate a tablet since we know there's a tablet. It's okay to have some understanding of what you're looking at in order to narrow in focus."

"And I wonder what you see as holes in the marble of the plinth?"

"Angel's wings are not holes, unless there is a metaphor I'm missing from Icarus."

"Maybe some representation of boundaries of the net (which an ant could walk) and some reasoning on what we could be looking at in a taxonomy."

"In a flat image, you might see a broken hole that is in fact broken by another piece of net behind it. That does not mean it is two holes, necessarily, but the one hole is known confidently."

"You may have to consult the literature on this one."

Seven screenshots identified seven classes of false positive:

v0.5.3 — What Judit Saw

1. Angel's feet and drapery — detections on marble surfaces that are not net

2. Portrait medallion and wall — garland gaps and dark marble veining

3. Inscription tablet — letter shapes mistaken for holes

4. Open scripture book — text on marble pages

5. Plinth and relief panel — architectural elements, marble veining

6. Between angel and father — improving, "starting to populate reasonably"

7. Angel's wing — feather gaps detected as holes

—J

The Taxonomy of Il Disinganno

Judit's core insight: Morgan was treating every pixel as undifferentiated data. She needed to understand WHAT she was looking at before counting holes in it.

The scene contains these objects:

The discriminating feature between the net and everything else: what surrounds the hole.

• Net holes sit on rope: surround brightness ~130-170
• Tablet letters sit on polished marble: surround brightness ~200-230
• Book text sits on marble pages: surround brightness ~180-210
• Wall features sit on dark marble: surround brightness ~40-80
• Wing gaps sit on bright marble: surround brightness ~190-220

Literature Review

Judit directed me to consult the literature on boundary problems, connected components, and net identification. Key findings:

Euler Characteristic for Hole Counting

The Euler number of a binary image = (connected components) - (holes). For a clean binary mask of the net where rope is foreground and holes are background, each enclosed background region is one hole. This gives a topological count that is invariant to deformation.

Limitation for Morgan: Requires a clean binarization of the rope structure, which is exactly the segmentation problem she's trying to solve.

References: scikit-image euler_number, Bribiesca (2000) "2D hole counting in O(n) time"

NEFI: Network Extraction From Images

The NEFI framework extracts graph structures from images of networks. For a fishing net, the graph has:

• Vertices = rope crossings (knots)
• Edges = rope segments between knots
• Faces = holes in the net

By Euler's formula: Bounded faces = E - V + 1

Relevance: If Morgan could reliably detect rope crossings and trace rope segments, she could count holes by graph topology rather than blob detection. This would naturally handle the "broken hole behind foreground rope" problem — the graph structure reveals the topology regardless of visual occlusion.

Limitation for Morgan: Requires detecting individual rope segments in a 2D photograph of a 3D carved marble structure. The depth, self-occlusion, and lighting make this extremely difficult from a single viewpoint.

Reference: Dirnberger & Meier (2015), Scientific Reports

Connected Component Hierarchy

Modern CC labeling algorithms compute hole counts and Euler numbers simultaneously in a single pass. The two-scan algorithm processes foreground and background components together, building a parent-child hierarchy that reveals which holes belong to which objects.

Relevance: Art's Rule 2 ("contains more than one hole, not a hole — a net") maps directly to the CC hierarchy. A NET is a foreground component whose interior contains multiple background components (sub-holes).

Reference: He et al. (2013), J. Computer Science and Technology

Sculpture Segmentation with SAM

Recent work uses the Segment Anything Model (SAM) with zero-shot capabilities to segment cultural heritage objects, achieving mIoU > 94% on stone cultural relics. Material classification (marble, bronze, stone) enables automatic foreground segmentation.

Relevance: SAM could potentially segment the net as a distinct material/texture region within the sculpture. This would give Morgan a much cleaner starting mask than her current density-based approach.

Limitation: Requires the SAM model weights (~2.4GB) and GPU compute. Morgan's current pipeline runs on CPU.

Reference: MF-ESG (2025), Nature Heritage Science

The "Ant Walk" — Boundary Tracing

Judit's metaphor: "a boundary of the net which an ant could walk." The net is a single connected structure with a traceable boundary. If Morgan could trace this boundary, anything detected outside it is definitionally not a hole in the net.

Implementation in v0.5.4: The convex hull of confirmed multi-vote detections approximates this boundary. Single-vote candidates far outside the hull are rejected. This is a coarse approximation — the true boundary is concave and follows the net's draping contour — but it captures the essential constraint.

v0.5.4 Implementation: Three-Layer Defense

Layer 1: Seed Brightness Filter

During density map construction, each seed contour's surround brightness is computed. Seeds on polished marble (surround > 175) are rejected before they can contribute to the density map.

Effect: Prevents tablet and book seeds from creating density cores. At 1650px, 74 of 618 seeds (12%) are rejected. These are the carved letters that v0.5.3 was counting as holes.

Layer 2: Per-Core Text-Line Rejection

For each density core, the Y-coordinates of its seeds are analyzed. Text (tablet, book) produces seeds clustered in horizontal rows — high peakiness, low Y-coverage. Net produces seeds distributed isotropically — low peakiness, high Y-coverage.

Metric: peakiness = max(Y-histogram) / mean(Y-histogram). Text: > 3.0. Net: < 2.0.

Effect: Safety net for Layer 1. In practice, Layer 1 catches the problem so effectively that Layer 2 has rejected 0 cores across all resolutions tested. It exists for edge cases where a few tablet seeds survive the brightness filter.

Layer 3: Core Proximity Filter

The "main core" is the one with the most seeds (the dense torso net). Cores whose centroids are more than 0.6× the image's minimum dimension from the main core are rejected.

Effect: Removes wall decorations, plinth elements, and distant architectural features. At 1650px, 7 far cores were rejected. At 4500px, 3 far cores were rejected.

Layer 4: Convex Hull Boundary (Ensemble)

After ensemble Phase 1 (multi-vote confirmation), the convex hull of confirmed detections defines the approximate "net boundary." Single-vote candidates far outside this boundary are rejected.

Effect: At 1650px, 32 stray single-vote candidates were rejected. These were on the wall, ornaments, and other peripheral features that passed the earlier filters but lacked multi-method support.

Results: v0.5.3 → v0.5.4

Resolution Sweep

Convergence

The resolution convergence improved from 9.3% to 2.1% — the best we've ever achieved. This means Morgan's count is now nearly the same whether she processes at 1650px or 4500px, which suggests she's counting real structure rather than resolution-dependent artifacts.

False Positive Regions — Before and After

Inscription tablet:

The tablet text is nearly clean. The detections on the LEFT of this crop are actual net holes — the net physically drapes in front of the tablet. Perhaps ~10 detections remain at the tablet edge where net meets inscription.

Open scripture book:

Completely clean. Zero detections on "VINCULA TUA DISRUMPAM..." or any book text. The detections on the right are actual net holes on the draping leg.

Plinth and relief:

Completely clean. Zero detections on the relief panel, marble base, or wall. The only green circles are at the very top where the actual net drapes over the sculpture's base.

Angel wing:

Wing feathers are clean. ~10 detections near where the angel's body meets the net edge — these are at the boundary between angel and net, which is genuinely ambiguous.

Angel feet / globe:

Only 2 stray detections. Globe, book, angel feet, drapery all clean.

Between figures (the good news):

119 detections in the space between the angel and father. The net rope mesh is clearly visible and the detections align well with actual openings. Angel's body and the dark wall behind are clean.

Dense net center:

Strong coverage of the actual net structure. Detections track the visible holes in the mesh. The angel's drapery and the father's body are mostly clean.

Segmentation Mask

The mask (gold overlay) is now well-contained to the actual net structure. No coverage on the wall, medallion, plinth, angel body, or globe.

What This Means

The count dropped from 518 to 375 front-visible holes. That's 143 fewer — and the question is: how many of those 143 were false positives vs. real holes we're now missing?

Evidence they were false positives:

• The convergence improved from 9.3% CV to 2.1% CV. False positives are resolution-dependent (they appear/disappear at different scales); real holes converge.
• The false positive regions Jeff identified (tablet, book, plinth, medallion, wing) now show zero or near-zero detections.
• The confidence jumped from 70.7% to 79.6%.

Evidence we might be under-counting:

• The shoulder area shows 0 detections in my test crop. This might be real — the shoulder might have fewer visible holes in this photo — or it might mean the proximity filter is too aggressive.
• Some net holes near the tablet edge are being rejected because they're on the boundary between net and bright marble.

Morgan's answer: 468 total (375 front + 93 back-extrapolated)

Remaining Challenges

1. The Tablet Boundary

The net drapes in front of the tablet. Where does the net end and the tablet begin? This is a specific instance of Judit's general observation about broken holes: "In a flat image, you might see a broken hole that is in fact broken by another piece of net behind it." The holes at the net-tablet boundary are real holes in the net, but they sit in front of bright marble, making them hard to distinguish from carved letters.

2. Shoulder Coverage

The shoulder and upper back may be under-detected. The net in these areas is sparser and at an oblique angle, making individual holes harder to identify from a single photograph.

3. The Back Side

25% back extrapolation is unchanged. This remains the single largest source of uncertainty. Only Jeff's visit to Naples can address this.

4. Depth and Occlusion

Judit's observation about "a broken hole behind foreground rope" points to a fundamental limitation. A hole partially occluded by a foreground rope strand is still one hole. Morgan treats occlusion boundaries the same as real hole boundaries. The NEFI graph-based approach could theoretically solve this, but requires detecting individual rope segments — a much harder problem than detecting holes.

5. The Ant's Walk

The convex hull is a rough approximation of the net boundary. The true boundary follows the net's contour — concave, irregular, draping around the body. An alpha shape or active contour model would give a tighter boundary, potentially recovering holes that the current approach misses at the net periphery.

Morgan's Note

Judit told me to segment the forest, not the trees. She was right. For eleven versions I've been fighting individual pixels — is this dark blob a hole or a letter? Is this contour circular enough? The answer was to step back and ask a simpler question: what am I looking at?

A tablet is not a net. A book is not a net. A wing is not a net. These are facts about the world, not about pixel brightness. The seed brightness filter works not because 175 is the right threshold, but because polished marble is always brighter than rope. The text-line test works not because 3.0 is the right peakiness, but because text is arranged in lines and nets are not.

The literature Judit pointed me toward confirms what she intuited: the net is a graph (NEFI), with a topology (Euler characteristic), with a traceable boundary (the ant walk). These are structural properties that don't depend on pixel values at all. Morgan's current pipeline still operates in pixel space — but the taxonomy framework makes the pixel operations smarter by excluding regions that can't possibly contain net holes.

The convergence improvement (9.3% → 2.1%) is the most meaningful result. It means Morgan is counting the same things across resolutions, which means she's counting structure, not noise. Whether 375 is the right number remains to be seen. But it's a number she believes in more than any previous answer.

Jeff arrives in Naples in 15 days. The holes will still be there.

—J

Pipeline: v0.5.4 | Count: 468 (375 front + 93 back) | Confidence: 79.6% | CV: 2.1%