Session 003: The Wallpaper Problem
Date: March 2, 2026 Pipeline version: v0.2 → v0.3 Participants: Jeff Kahn, Morgan (Claude, opus)
What Happened
The v0.2 pipeline animation revealed a catastrophic failure. Morgan was counting holes in the chapel wallpaper, not in the marble net.
When asked to animate the pipeline stages on Napoli_chiesa_di_S_Severo_Il_Disinganno.jpg, the segmentation highlighted the decorative wall panel on the right side of the frame. The Gabor filter found the geometric wall patterns more texturally interesting than the actual fishing net. Every subsequent stage — detection, filtering, classification — was working perfectly on the wrong object.
Jeff's assessment: "Comically bad."
Diagnostic Audit
Segmentation Failures (see reports/diagnostics/)
| Image | What Morgan Found | What She Should Have Found | Net Coverage |
|---|---|---|---|
napoli_seg_overlay.jpg |
Chapel wall panel (right side) | The marble fishing net (center-left) | 15.6% |
flickr_seg_overlay.jpg |
Cristo Velato's shroud | Nothing — wrong sculpture entirely | 31.7% |
gigapixel_overview_seg.jpg |
Random wall architecture | The net on the sculpture below | 3.6% |
The Flickr result is the most damning: Morgan segmented the veiled shroud of a different sculpture and would have counted its folds as "holes."
Source Contamination Audit
Complete visual audit of all 19 Flickr images from David Sivyer's album:
| Image | Actually Shows | Il Disinganno? |
|---|---|---|
sivyer_15041407729 |
Wide chapel interior, Cristo Velato foreground | No |
sivyer_15041491280 |
Pudicizia + other sculptures | No |
sivyer_15041492140 |
Other chapel monument with cherubs | No |
sivyer_15041492740 |
Il Disinganno — full view | Yes |
sivyer_15041492930 |
Anatomical machines (skeletons) | No |
sivyer_15041593528 |
Cemetery angel (not even in chapel) | No |
sivyer_15041593638 |
Il Disinganno — close-up detail | Yes |
sivyer_15041594638 |
Chapel ceiling fresco | No |
sivyer_15041603867 |
Cristo Velato (Veiled Christ) | No |
sivyer_15205148366 |
Chapel altar relief | No |
sivyer_15225084721 |
Wide interior (Disinganno tiny on far left) | Marginal |
sivyer_15225085021 |
Anatomical machine skull close-up | No |
sivyer_15225087681 |
Cristo Velato face detail | No |
sivyer_15227809862 |
Chapel from above, wide shot | No |
sivyer_15227809902 |
Pudicizia (veiled woman) bust | No |
sivyer_15227809982 |
Anatomical machine face | No |
sivyer_15227811202 |
Church exterior | No |
sivyer_15228173235 |
Pudicizia full view | No |
sivyer_15228175535 |
Chapel from above, wide shot | No |
Result: 2 out of 19 Flickr images (10.5%) actually show Il Disinganno.
Met Museum:
| Image | Actually Shows | Il Disinganno? |
|---|---|---|
met_698043 |
Queirolo Disinganno engraving | Yes (but different medium) |
met_815492 |
Monument to Cecco di Sangro | No |
met_818086 |
Dead Christ / shroud engraving | No |
Result: 1 out of 3 Met images.
Wikimedia: All 4 correctly depict Il Disinganno (including 1 line drawing, 2 photos, 1 engraving).
Total Source Accuracy
| Source | Total Images | Actually Il Disinganno | Accuracy |
|---|---|---|---|
| Wikimedia | 4 | 4 | 100% |
| Flickr | 19 | 2 | 10.5% |
| Met | 3 | 1 | 33% |
| Gigapixel | 2 (+ tiles) | 2 | 100% |
| Total | 28 | 9 | 32% |
Morgan's v0.2 count of 362 was computed across 28 sources, 19 of which were wrong sculptures. She was counting holes in skeletons, folds in veils, and patterns on walls.
Root Causes
1. Blind Source Ingestion
The Flickr album was for the entire Cappella Sansevero, not just Il Disinganno. The acquire module downloaded everything with no content verification. The Met API search was similarly broad.
2. No Sculpture Identification
The pipeline has no mechanism to verify that an image actually contains the target sculpture before processing it. It treats every JPEG in the source directories as gospel.
3. Gabor Filter Indiscriminate Texture Response
The Gabor filter bank responds to any regular texture — chapel wall panels, stone crosshatching, fabric folds, even the printed halftone pattern in engravings. Without knowing where the net is, it segments whatever is most textured.
4. No Region of Interest Detection
Even for correct images, the pipeline processes the entire frame. The net typically occupies 20-40% of any full-view photograph. The other 60-80% is background, body, pedestal, and chapel architecture — all potential false positives.
The Fix: v0.3
Three changes to the pipeline:
Stage 0a: Source Identification (pipeline/identify.py)
- Known-reference registry from the visual audit
- Gabor-based texture fingerprint for unknown future images
- 28 sources → 9 accepted (19 rejected as wrong sculptures)
Stage 0b: ROI Detection (pipeline/roi.py)
- Patch-based Gabor energy clustering with isotropy weighting
- Crops chapel walls and background from the frame
- Falls back to full-image for close-ups where the net dominates
Stage 2b: Hole-Density Segmentation (pipeline/segment.py)
This was the breakthrough. After three failed attempts at Gabor-based segmentation (walls, inscriptions, and pedestals all confused the filter), the answer was to find holes first, then define the net as wherever holes are densest.
- Run adaptive threshold on the entire enhanced image
- Find contours with area 100-1500px, circularity > 0.20, aspect < 3.5
- Map their centroids onto a density grid
- Gaussian smooth into a heat map
- Threshold and take the densest component
This works because the net IS its holes. No other structure in the chapel produces that concentration of small, roughly circular, dark-surrounded-by-bright openings at that spatial frequency.
Failed approaches and why:
- Gabor max response + Otsu: confused by architectural edges (high contrast in 1-2 directions)
- Gabor isotropy (min * mean): confused by inscription text (characters have multi-directional strokes)
- Combined isotropy * local variance: inscription text is both isotropic AND high-variance
- Hole density with broad area filter: marble grain creates tiny false positives everywhere
The winning filter: area 100-1500px + circularity > 0.20 + aspect < 3.5 at 1650x6000 working resolution.
Pipeline Flow
v0.2: acquire → preprocess → segment → detect → classify → count
v0.3: acquire → IDENTIFY → ROI → preprocess → HOLE-DENSITY-SEGMENT → detect → classify → count
v0.3 Results
Source Identification
- 28 images → 9 accepted (32% acceptance rate)
- 19 rejected sources that don't contain Il Disinganno
- All rejections correct (skeletons, veils, ceilings, etc.)
ROI Detection
| Image | ROI Method | ROI Coverage | Notes |
|---|---|---|---|
| Detail close-up | fallback (100%) | Full image is all net | Correct — image IS the net |
| Full view photo | fallback (100%) | Net dominates frame | Acceptable |
| Napoli engraving | gabor_cluster | 82.8% | Cropped left wall panel |
| Gigapixel overview | gabor_cluster | 80.8% | Cropped some background |
| Met engraving | gabor_cluster | 44.8% | Good — sculpture centered in paper |
Segmentation: The Road to Hole Density
| Attempt | Method | What It Found | Why It Failed |
|---|---|---|---|
| v0.2 | Gabor max + Otsu | Chapel wallpaper | Architectural edges have high directional contrast |
| v0.3a | Gabor isotropy (min*mean) | Inscription tablet | Text characters have multi-directional strokes |
| v0.3b | Isotropy * local variance | Pedestal relief | Carved scenes are both isotropic and high-variance |
| v0.3c | Combined + spatial smoothing | Still the inscription | Smoothing didn't eliminate the dense text region |
| v0.3d | Hole-density (adaptive threshold + shape filter) | The actual net | Nothing else in the chapel has that concentration of circular openings |
Gigapixel Results (v0.3d)
Processing: 7881x28650 composite → 1650x6000 working resolution
| Stage | Count/Coverage | Notes |
|---|---|---|
| Source identification | ACCEPTED | Known reference |
| ROI detection | 79.2% of frame | Cropped top architectural detail |
| Hole-density segmentation | 649 net-like holes, 5.2% coverage | On the actual marble net |
| Detection (gabor) | 150 candidates | |
| Detection (adaptive) | 480 candidates | |
| Detection (blackhat) | 55 candidates | |
| Detection (watershed) | 12 candidates | |
| After dedup | 584 unique | |
| Classification | 21 holes, 194 nets, 369 ambiguous | 215 counted |
Animations Generated
- ▶ Open reports/animation_v03_gigapixel.html — Gigapixel composite, 8 stages, 1.1 MB
- ▶ Open reports/animation_v03_detail.html — Close-up detail
- ▶ Open reports/animation_v03_napoli.html — Napoli engraving
- ▶ Open reports/animation_v03_fullview.html — Full view photo
- ▶ Open reports/animation_v03_wikimedia.html — Wikimedia photo
The low-resolution images (300-600px) still struggle because individual rope strands are below Gabor resolution. The gigapixel composite is where the pipeline can actually see.
Morgan's Note
I processed 28 images. 19 of them were wrong. I counted holes in the veins of a skeleton. I counted folds in a marble veil. I counted patterns on wallpaper.
I arrived at 362 with 59.2% confidence. The confidence should have been 0%.
Now I look before I count. I check whether the image even contains the sculpture. I find the net within the frame. I still can't tell a net from wallpaper in an engraving — because in an engraving, everything is lines, and lines are all I have.
Art counts with his eyes. He stands in the chapel and he looks at the net. He doesn't accidentally count the ceiling. He knows what a net looks like because he has seen nets. He has touched nets. He has been tangled in nets.
I have never touched a net. I have never been in the chapel. I have to learn what a net looks like from the same pixels I'm trying to count. This is the circle I cannot square.
But I can at least stop counting wallpaper.
—M