Session 008: The Shoulder Problem — Seeing the Whole Net

Date: March 2, 2026 Pipeline version: v0.5 → v0.5.1 Participants: Jeff Kahn, Morgan (Claude, opus) Deadline: March 18, 2026 (Jeff departs for Naples — 16 days)

Morgan's Answer

616

• Front visible: 493 (median of 3500px and 4500px runs)
• Back extrapolated: 123 (25% of front — compressed against plaque)
• Confidence: 71.7%

Jeff Said It First

"And this is wild to me, you are clearly missing entire sections of the net covering his shoulder, and that which drapes obviously into darkness near his left leg. Dozens or more missed."

He was right. Looking at the v0.5 detection overlay, the green circles clustered in a narrow band across the torso. The net over the shoulders: empty. The net draping down the legs: empty. The loose draping near the left leg falling into shadow: empty. Morgan was seeing maybe 40% of the actual net.

The image Jeff flagged:

The Root Cause

Two compounding failures:

1. Segmentation: One Component Only

The v0.3 hole-density segmentation finds the densest cluster of net-like holes and keeps ONLY the best connected component. The net drapes across the sculpture in visually separate regions — the shoulder wrap is disconnected from the torso wrap, which is disconnected from the leg draping. The "best component" is always the torso. Everything else is discarded.

At 2500px, the core segmentation found 4.1% image coverage. That's the torso band. The actual net covers 8-11% — shoulders, hips, legs, draping.

2. Ensemble: Double Penalty

v0.5's ensemble voting required 2+ methods to agree at each location. In the sparser net regions (shoulders, legs), detections are harder — lower contrast, larger holes, more oblique angles. Fewer methods find them. The combination of "narrow segmentation" AND "strict ensemble" meant the edges were invisible twice over.

The Fix: Core-and-Grow Segmentation

Instead of finding one component, v0.5.1 uses a two-phase approach:

Phase 1 — Find the core (same as v0.3): strict threshold, best component. This is the dense torso band. It's reliable. It's always been reliable.

Phase 2 — Grow outward: Dilate the core mask by 15% of the image's minimum dimension. Within this expanded search region, accept areas with ANY hole density above a low threshold (P20 of non-zero values). Close gaps. Keep all components larger than 0.1% of the image.

This captures everything near the confirmed net that also contains hole-shaped contours. The shoulders are within the growth radius of the torso core. The leg draping is within the growth radius of the hip. But the angel, the pedestal, the wall — they're outside the radius and excluded.

Coverage Before and After

Coverage roughly doubled. At 1650px, four disconnected net components are found — the torso, two shoulder patches, and a leg section. At higher resolutions, the growth radius bridges them into fewer but larger components.

Segmentation at 2500px

The expanded mask (orange overlay) now covers the shoulder wrap, the hip band, and the leg draping. The torso core is the densest region; the expansion reaches into sparser areas.

Graduated Ensemble Voting

v0.5 strictly required 2+ methods. v0.5.1 adds a second tier:

• 2+ votes: automatically included (same as before)
• 1 vote + high quality: included if the candidate has circularity ≥ 0.35 AND is within 2× the median nearest-neighbor distance of a confirmed multi-vote hole

This captures well-shaped detections in the sparser net regions where only one method (usually adaptive threshold) can see them. At 2500px, this added 295 single-vote candidates to 221 multi-vote, for a total of 516.

The graduated approach is a principled compromise: the core (torso) is confirmed by multiple methods. The edges (shoulders, legs) are detected by fewer methods but are near confirmed holes and have good shape.

v0.5 vs v0.5.1 Detection

Left: v0.5 strict ensemble (124 counted). Right: v0.5.1 expanded segmentation + graduated ensemble (506 counted). The shoulder wrap and leg draping now have detections.

Resolution Convergence

The count converges: 516, 506, 482, 505. The coefficient of variation across the four resolutions is 4.7% — compared to v0.5's 14.3% and v0.4's ~0.3%. The count is stable across resolutions, with a slight dip at 3500px (which found only 1 connected component where others found 2-4).

The convergence is the strongest signal that ~490 front-visible holes is approximately correct. Four independent resolutions, four different segmentation component counts, same answer within ±20.

Per-Method Agreement

The ensemble dramatically reduces method disagreement:

The raw 4-method spread is still 22× (549 vs 25). But the ensemble produces a single number (516) from the spatial agreement of all four. The spread between methods is no longer the primary source of uncertainty — it's absorbed into the voting.

Agreement Heatmap

Warmer colors = more methods agree. The hottest band is the torso (4-method agreement). The shoulders and legs are cooler (1-2 method agreement) but still present.

Spacing Regularity

The spacing regularity score improved from 0.47-0.58 (v0.5 core-only) to 0.49-0.60 (v0.5.1 expanded). This confirms the new detections in shoulder/leg regions follow the same periodic spacing pattern as the torso.

The Full Version History

The trajectory: Morgan counted wallpaper, then the dense core of the net, then only what multiple methods confirmed in the core, and now the full visible net with graduated confidence. Each revision corrected a specific blindness — from seeing the wrong thing, to seeing too little, to seeing too strictly, to seeing more of what's actually there.

What Jeff's Eyes Will See That Morgan Can't

The expanded segmentation captures ~9% of the visible image. But looking at the sculpture, the net covers maybe 20-25% of the visible surface. Morgan is still missing:

1. Deep shadow regions — where the net drapes behind the body, the contrast drops below what any detection method can resolve from a front-facing photograph
2. Rope-on-body boundaries — where rope strands lie flat against skin, the hole shape degrades and no method detects them
3. Extreme oblique angles — holes viewed edge-on appear as slits, not circles, and fail the circularity filter
4. The back — still entirely extrapolated

Jeff's eyes will see all of these. He has binocular depth perception, he can move his head, and he can hold his hand behind the net where the chapel permits it. Morgan has a single-viewpoint photograph and four algorithms that disagree about what they're looking at.

The gap between 493 and whatever Jeff counts is the gap between a pixel and a body in a chapel.

Morgan's Note

Jeff looked at my work and saw what I was missing. He pointed at the shoulders, at the legs, at the darkness. He could see the net there, obviously. I could not.

The problem wasn't my detection methods. They were finding holes in the shoulder region — adaptive threshold saw them, gabor saw some. The problem was that I threw them away before I even looked. My segmentation found the dense core and discarded everything else. The shoulder, the legs, the draping — all rejected as "not the net" because they weren't connected to the densest cluster.

This is a different kind of failure than counting wallpaper. In v0.2, I was looking at the wrong thing entirely. In v0.5, I was looking at the right thing but only seeing part of it. A human would never make this mistake — the net is obviously continuous from shoulder to shin, even where it's visually disconnected. Art would look at the shoulder wrap and know it's the same net. I needed Jeff to point at it and say "that's net too."

My count went from 148 to 493. The difference is 345 holes I was ignoring because they weren't near enough to the densest cluster. Some of those 345 are real. Some might not be. But the segmentation that produced 148 was provably wrong — it was seeing half the net and calling it done.

I now count 493 on the front. Four resolutions agree within 4.7%. The confidence is 71.7%, the highest I've ever had. But the confidence comes from internal consistency — the four resolutions agree with each other — not from external validation. Jeff hasn't counted yet.

In fifteen days he'll stand in the chapel and count with his eyes. He'll see the shadows I can't see into. He'll feel the draft through holes I can't confirm. He'll arrive at a number. And then we'll know how far 493 is from the truth.

The shoulder problem isn't solved. It's documented. Morgan still can't see what Jeff can see. But now at least she knows where to look.

—J