Skip to content

The nets' merge gap, tuned like any other setting: the answer holds, and the check bounds both sides - #671

Merged
syncytium2 merged 32 commits into
mainfrom
nets/matched-merge-gaps
Sep 20, 2026
Merged

syncytium2 merged 32 commits into
mainfrom
nets/matched-merge-gaps

Conversation

@syncytium2

Copy link
Copy Markdown
Owner

The merge gap the nets were never allowed to tune, tuned by the same rules — on both draws, with no retraining — and murderboarded twice.

What it settles. In the fair comparison the coded detectors' searches tuned their merge gap (CoactDetect took 8 s in every fold) while the nets decoded at the fixed 2 s their threshold picker returns. That was the open asymmetry. Here the nets' gap is selected, on the inner fits of the training folds, by the rules the run used for everything else, with goal 1's move rule and its crowded-recording check inside the search.

No retraining, and it reproduces the run exactly. A merge gap is applied when a net's per-frame output is decoded, so every saved fit of both draws — 3,881 of them — was re-run and decoded at each gap in the grid. At 2 s every picked threshold, every recording's counts, every empty recording's call count and both selections' inner and held-out F1 match the runs' own files.

The result

  • Under the budget the answer holds. CoactDetect is ahead of every net on average in both draws and in 29 of 32 net-folds. Tuning closes 0.009 to 0.014 F1 of the chorus nets' shortfall.
  • On F1 alone the sign depends on the accounting, and the gap is worth about one noise unit either way. Scored the same way on both sides, the replicate's chorus nets were already ahead at 2 s and tuning adds ~0.010 F1.
  • Every chosen gap is a boundary, not an optimum. Held-out F1 peaks at the widest gap in all 64 fold-rows; the nets stop where the crowded check stops them, and CoactDetect's 8 s is the top of its own grid — which stops there because goal 1's check refused its 16 s. Both sides are bounded by the same check, at different times.
  • The long-gap inflation is not specific to the coded side. In all 64 choices the training folds preferred a wider gap and the check refused it, 30 s for every net.
  • The accepted settings are not free: a median 0.012 F1 lost on crowded recordings, the same size as the gain, with two choices failing the check outright on the outer refits. On those same recordings, head to head, the tuned chorus nets beat CoactDetect while tube loses 0.14 — the comparison the cost is not, now drawn.

The review, and why this ships unconverged. Eleven roles blind, then five re-run blind against the rebuilt page: 11 blocking findings in round 1, 10 in round 2. The process says a flat blocking count means a structural problem patching will not retire, and to escalate rather than run a third round — so both rounds' fixes are applied and the record ships with named residuals rather than a clean bill. The review changed what the page may claim, not just how it reads: the one-sided set-aside, the censored boundary, the mis-referenced crowded cost, the noise scale's own systematic shift, and a budget mechanism the data refute.

What is here

  • tools/tune_net_merge_gap.py: the re-decoding and the selections, reusing the tuning tool's own objective, budget, pooling and tie rules. Records the machine, torch and both trees' commits; asserts the threshold grid it indexes.
  • tools/build_net_merge_gap_page.py: the page. Every sentence about the data is computed, and the directional ones sit behind claim(), which stopped the build three times during this work.
  • docs/reviews/net-merge-gap-2026-09-19.md plus both role archives, verbatim.
  • docs/goals/learned-model-family.md, docs/GLOSSARY.md: updated in the same PR, as the rules require.
  • docs/sapper_feedback/2026-09-19-the-roster-gate-eats-underscores.md: the roster gate rewrites underscores in the archive path and reports a complete archive as missing.

Open for Tony: the 0.02 F1 crowded allowance is still unsigned and decides every gap here — one refusal turned on 0.0009 F1.

🤖 Generated with Claude Code

defazio2 and others added 30 commits September 19, 2026 08:16
…and SCE's lead is the merge-gap artifact

Goal 2's run on WSMIP064 (2026-09-18 16:14 to 2026-09-19 05:58, 1,963 jobs, 0 errors).
Its summaries are committed beside a report written for a reader new to the project,
with eight inline-SVG figures that explain the problem, the bench, the four nets,
nested cross-validation, the fold defect on this run's own draws, the two selections
and the second draw before any result.

The coded search had the context rule but not goal 1's crowded veto, so
tools/crowded_check_fair_comparison.py applies it afterwards with goal 1's own scoring
(search_all_settings._job over TAIL). CoactDetect's choices pass in all 8 cases;
binned SCE's 30 s merge gap fails in 7 of 8, so its first place does not stand.
locust has no admissible gated result: the budget refused every candidate in all 4
folds. WSMIP065 raised all three checks.

Draft before the murderboard.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…, the merge-gap ablation, the replicate's numbers

tools/fair_comparison_evidence.py, three subcommands, each writing into the run's summary folder:
- fold-draws: Figure 4's data from the run's own fit records and, for "before the fix", from
  bugarach.learn.train.fold_maker. It reproduces the committed fold_draws.json exactly; three
  murderboard roles found that no committed code had produced it.
- merge-gap: the coded detectors tuned their merge gap and the nets' was fixed at 2 s. This
  rescores every coded choice, and re-decodes every chosen net refit from its saved model,
  with ONLY the merge gap changed, after first reproducing the run's own scores at each side's
  own gap. WSMIP065 will run the same code on the second draw.
- replicate: the replicate's headline numbers, with source and commit recorded.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rge gap reverses the headline

All eleven roles ran against the first build. Their reports are archived verbatim in
docs/reviews/fair-comparison-2026-09-19-roles/ (paths replaced; originals in the darkroom).

The finding that changes the result is role 4's: the coded detectors tuned their merge gap and
the nets' was fixed at 2 s. merge_gap.json measures both directions with only that setting
changed, after reproducing the run's own scores (coded exactly, nets within 0.0015 F1). At the
run's unequal gaps CoactDetect leads the best net by 0.007 F1; with the gap matched, at 2 s or
at 8 s, chorus_norm leads. The page now says this run does not settle which side is better, and
that a rerun tuning the nets' merge gap needs no retraining.

Also from the review: the earlier comparison ran on a different, retired simulator; binned SCE's
lead is its 30 s merge (it falls below CoactDetect at 8 s); one SCE choice passes the crowded
check; the nets exceeded the budget on held-out data in 25 of 80 refits, now reported beside the
coded side's misses; the coded side is scored on pooled training folds, not the inner rotation;
the probe rate, the backgrounds (25th and 75th percentiles), the hit rule (spans), the tube
description, the distractor construction and the coded detectors' origins are corrected; the
contamination limit cites its record and the open participation item; t values carry the
Nadeau-Bengio factor with its scope; tables are numbered; figures redrawn; the hand-drawn
architecture figure is gone (draughtsman owns it). Glossary: merge gap, crowded-recording check,
shared false-alarm budget, admissible.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All eleven round-2 reports, archived as they arrived (paths scrubbed; the
originals are in the darkroom). Three pieces of evidence they asked for:

- The crowded-recording check now also scores goal 1's own reference,
  the shipped operating points. No verdict changes, and the page can now
  say so from a file rather than a sentence.
- A breakdown of held-out recall by participation level and background.
  The pooled F1 hides where the two sides differ: the faintest events,
  10% of the ROIs, on the busy background go to the nets, and on the
  quiet one mostly to CoactDetect.
- The replicate's refits below 0.2 F1, read by the same rule as this
  run's. Two failed fits in one replicate fold carry that draw's whole
  margin against the nets on F1 alone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ect leads; on F1 alone the sign is not settled

Round 2 of the murderboard found the round-1 headline half right. The merge
gap does reverse the order, but only for choices on F1 alone: under the
shared false-alarm budget CoactDetect stays ahead of both chorus nets at
every matched gap. And the replicate's "agreement" was two refits that
failed to train; set aside by one rule in both draws, the replicate's best
net leads on F1 alone in 4 of 4 folds while this run's trails in 4 of 4.

The page now says that, and a sentence about the data is printed only while
it is true: the builder's claim() stops the build otherwise. Also:

- the argument reordered: terms first, the method with its rules (the
  context window, the crowded check, "admissible" as the glossary defines
  it) before any number, and the merge gap in its own section after the
  results it qualifies;
- new figures: the whole bench recording, how a call is scored (one call,
  one event), binned against sliding counting, both draws per fold with
  failed refits drawn twice, matched gaps for both selections, where the
  two sides differ (the faintest events, by background), and tuning;
- attributions corrected: the nets are the Deep Sets shape, binned SCE is
  not the 2003 rule, CICADA cited by its DOI, SPIKE-synch's root added;
- the tube refit's mechanism corrected (its threshold came from the inner
  fits), the corrected t factor stated exactly, reproduction bounds stated
  as measured;
- a sapper feedback note for SAP004's blindness to backslashes, and the
  first author of the CICADA framework paper corrected in the README.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…, and the headline held to its own uncertainty

Round 3 of the murderboard (eleven roles, blind) found no wrong number in the
tables and a page that still said more than its evidence. Fixed here:

- The answer leads with every refit counted: CoactDetect ahead on average in
  both draws and both selections. The best net leads only at a matched merge
  gap, or fold by fold in the replicate once failed refits are set aside, and
  that set-aside is now called what it is, a check made after seeing the
  held-out scores.
- Under the budget, the matched-gap lead over chorus_norm is as weak as the
  F1-alone reversal (t -1.5 to -2.1), and the page now says so; the budget is
  named as CoactDetect's own rates times a declared 1.6.
- The confound narrows: untuned chorus_norm is already within 0.02 of the
  tuned CoactDetect in both draws, so the earlier lead was not lost to tuning
  the nets.
- What was not made equal gets its own section, and the limits add how the
  nets were trained (the probe and the distractors labeled negative), the
  three merge rules, normalization over the whole recording, and the
  tolerance measured for one side only.
- The merge-gap concept moves to section 2 as its own figure; the gap curves
  get a logarithmic axis and a key; the margins figure drops the connecting
  lines that read as confidence intervals; new figures for the participation
  breakdown at all three levels and the crowded check; the failed refits of
  both draws become a table.
- Attributions: CoactDetect's null credited to whole-train and per-cell
  shifts (Pipa 2008; Bocchio 2020; Dard 2022), Grün part II, the guard to
  Rohling 1983, SPIKE-synch's window to event synchronization and its
  detection step to Cecchini 2021, "designed here" read as "not found
  elsewhere yet".
- Glossary: call, firing, background, empty recording, refit, the
  failed-training signature; merge gap and the budget corrected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…issible wins under the budget, and the last overreach removed

A finding-driven pass over every round-3 finding, and role 10 re-run on the
final render, both archived in docs/reviews/fair-comparison-2026-09-19-followup/.

The pass caught one error the round-3 fixes introduced: section 9 said the
crowded-recording check was never run on the replicate. It was. Read from the
replicate's own crowded_check.json and selections, binned SCE under the budget
is admissible and ahead of CoactDetect in 3 of 4 replicate folds (+0.011 to
+0.017 F1), against 1 of 4 in this run. The page and its lede now say so,
behind a claim() guard.

Also: the contrast between the two selections is stated as consistency
(under the budget every net is behind in every fold of both draws, guarded)
rather than margin size, which the replicate contradicted; the scorer-resolution
margins are computed; the lede's tuning and faint-event sentences are scoped;
stale figure numbers are gone from the SVG labels; the ringed dot means one
thing on every figure; Figure 8 says where its no-result marks sit; the
axis-break bridge in Figure 11 no longer borrows LoCo's style. And a todo:
goal 1's published crowded numbers compare two seed sets.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nal checks

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ng-driven check and role 10 on the final build

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e shipped build in its record

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…and says so

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… replacement field

Figure 3's caption escaped its quotation marks inside an f-string expression, which 3.12 allows
and 3.11, the declared floor, does not. tests/test_syntax_floor.py caught it on every leg; the
caption now uses typographic quotes and says the same thing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…he new hash

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…fence it missed is fixed

Figure 10's caption escaped quotation marks inside an f-string that was itself inside another's
replacement field. The scan exempted all f-string literal text, including a nested one's, so it
passed on 3.13 and 3.14 and the 3.11 leg alone failed. On 3.11 that text is expression, and a
backslash there is a SyntaxError. The scan now flags a backslash in a nested f-string's text and a
nested f-string reusing the enclosing quote; both new cases are proved to be 3.11 errors (checked
on a real 3.11) and proved caught. Every tool and test in the tree parses on 3.11.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ecord the new hash

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… fits

tools/tune_net_merge_gap.py re-decodes every saved fit of the fair comparison at eight merge gaps
(0 to 30 s), re-picks each fit's threshold at each gap by its own rule, and re-selects on the inner
fits only: F1 alone and under the shared budget, reported separately, with goal 1's move rule
(0.002 inner F1) and its crowded-recording check inside the search. At 2 s it reproduces the run
exactly: every picked threshold, row and empty-recording count, and both selections' inner and
held-out F1.

Pushed mid-run so the work is visible off this machine. The run's crowded pass is finishing; the
selections, the replicate and the page (tools/build_net_merge_gap_page.py, not yet built) follow.
The tests read committed outputs that are not here yet.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… reaches, where the full grid is impractical

Four inner fits of one chorus_norm configuration ran more than three hours each on the full
crowded grid (36 thresholds x 8 gaps x 24 three-hour recordings) without finishing, while the
other 1,934 fits took under a minute each, and the re-chosen walk needs that configuration.
select now lists the exact pairs it could not read, crowded --pairs scores only those, and a
partial file refuses any cell it does not hold.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
np.load held the old file open, and Windows refuses to replace an open file, so every partial
update failed with Access is denied.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ort's, and the outer-refit crowded check stated

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All 1,938 fits reproduce the run at 2 s. With the configuration kept, on F1 alone chorus_norm goes
from -0.007 to +0.004 against CoactDetect (3 of 4 folds; inside the 0.010 noise scale). Under the
budget every net stays behind except chorus_norm in one fold, by +0.004; chorus_gain_norm closes 45%
of its shortfall. The nets settle at 5 to 8 s; the crowded check refused every wider choice.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s which

The replicate's selections are still computing; a page of one draw is honest if it says so, and a
page that cannot build until both are in is a page nobody sees tonight. Also: the goal page's
chorus-collapse readout and todo are on main now, not arriving with #667.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… is the finding

The Next entry becomes what the rerun found, from this draw's net_merge_gap.json: under the budget
CoactDetect still wins 15 of 16 net-folds, tuning closes about 45% of the chorus nets' shortfall, and
on F1 alone the +0.004 lead is inside the 0.010 draw-to-draw scale. The no-retraining claim is back,
sourced to the tool that re-decodes the saved fits. Stated as its own finding: the crowded check
refused a wider gap in all 32 choices, 30 s for every net, so long-gap inflation is not specific to
the coded side; and one choice fails that check on the outer refits.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ailed trainings

The replicate's 1,943 fits re-decode and reproduce its run at 2 s. Under the budget CoactDetect is
ahead of every net on average in both draws. On F1 alone the sign depends on the chorus nets'
collapsed trainings: counting every refit the nets trail in the replicate, and with the refits under
0.2 F1 set aside both chorus nets lead there by more than the draw-to-draw scale. The page reports
both, and Figures 1 and 2 scale to the folds that trained rather than to a collapsed one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…read better than the data

Eleven roles, one round, reports archived verbatim. What changed, each because the first draft let a
reader conclude more than the numbers support:

- The set-aside was applied to the tuned choice and not to the baseline, so the gap was credited with
  a lead that predates it. Both sides now carry it: in the replicate chorus_norm is already +0.005
  ahead at 2 s, and the gap adds 0.010, one noise unit.
- Held-out F1 at every gap is now drawn (Figure 1). It peaks at the top of the grid in all 64 folds,
  so every chosen gap is a boundary the crowded check imposed, not an optimum -- and CoactDetect's
  8 s is the top of its own grid. The two landing together is not agreement.
- The crowded check's cost to the settings it ACCEPTED is reported beside its refusals: a median
  0.012 F1, the same size as the gain, with two choices failing it outright on the outer refits.
- The noise scale is named for what its source measures, two draws on two machines, and the page now
  says that source also measures a systematic +0.015 F1 shift in the replicate's favour -- the size
  of the lead there.

Also: the budget never bound CoactDetect while 44 of 160 net refits break it on held-out data; the
threshold is unchanged in 26 of 32 gated choices, so the gain is not the lower-threshold mechanism the
draft claimed; failed trainings and no-call refits are counted separately, from the run's own flags;
Figure 2 counts refusals instead of stacking 70 crosses; the figures keep an 11 px floor, a readable
grey and a gutter for off-scale folds; Table 1 marks within-noise means and failed checks. The tool
records the machine, torch and both trees' commits, asserts the threshold grid it indexes, and refuses
a move whose crowded score is not finite.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…y overturned

The crowded cost the page reported is each net against its own 2 s, and the head-to-head on those
same recordings runs the other way: the tuned chorus nets beat CoactDetect there by 0.016 to 0.056 F1
while tube loses 0.14. Figure 5 now draws both, and the text no longer lets 'the gain and the cost are
the same size' stand in for a comparison it is not.

CoactDetect's grid stops at 8 s because goal 1's crowded check refused its 16 s -- so the check bound
both sides, at different times, and the page said it had never bound CoactDetect at all.

Held-out F1 is largest at the widest gap but not monotonically: it dips in 18 of 64 fold-rows, so
'a wider merge can only delete duplicate calls' was too strong, and the glossary already said why.

'The nets that end ahead were already ahead' was false -- chorus_norm crosses zero in this draw -- and
the lede's range covered the chorus nets while saying 'each net'; tube gains a fifth as much.

Also: the set-aside cannot be even-handed and says so (CoactDetect has no refits); low refits are
counted as refits, not decodings, and split by the run's own two failure kinds; the reproduction is
described for what it covers, at 2 s, on one machine for both draws; the empty band below the cut is
computed (0.388, not a rounded 0.39); the budget's over-budget count carries its as-run baseline; the
grid's coarseness says which way it cuts; panels A and B are drawn at one scale; the table's footnote
marker is no longer a degree sign after a number, the crowded-check flag marks every column that uses
the flagged gap, and each figure is now referenced by number and name.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
defazio2 and others added 2 commits September 19, 2026 22:48
Two blind rounds, eleven roles then five, every report archived verbatim. The record leads with what
the review changed about the result and is delivered UNCONVERGED: blocking findings went 11 then 10,
and the process says a flat count means a structural problem patching will not retire, so it escalates
instead of running a third round.

The roster gate refused a complete archive because reports_decl() strips underscores as markdown
emphasis, rewriting net_merge_gap_...-roles/ into a path nobody wrote. The archives are renamed with
hyphens and the defect is filed for upstream: a gate whose job is to catch a cited-but-missing archive
reported a present one as missing, and the quickest way to green would have been to delete the line.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@syncytium2
syncytium2 enabled auto-merge (squash) September 20, 2026 02:50
@syncytium2
syncytium2 merged commit 02915ca into main Sep 20, 2026
3 checks passed
@syncytium2
syncytium2 deleted the nets/matched-merge-gaps branch September 20, 2026 03:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants