The nets' merge gap, tuned like any other setting: the answer holds, and the check bounds both sides - #671
Merged
Conversation
…and SCE's lead is the merge-gap artifact Goal 2's run on WSMIP064 (2026-09-18 16:14 to 2026-09-19 05:58, 1,963 jobs, 0 errors). Its summaries are committed beside a report written for a reader new to the project, with eight inline-SVG figures that explain the problem, the bench, the four nets, nested cross-validation, the fold defect on this run's own draws, the two selections and the second draw before any result. The coded search had the context rule but not goal 1's crowded veto, so tools/crowded_check_fair_comparison.py applies it afterwards with goal 1's own scoring (search_all_settings._job over TAIL). CoactDetect's choices pass in all 8 cases; binned SCE's 30 s merge gap fails in 7 of 8, so its first place does not stand. locust has no admissible gated result: the budget refused every candidate in all 4 folds. WSMIP065 raised all three checks. Draft before the murderboard. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…, the merge-gap ablation, the replicate's numbers tools/fair_comparison_evidence.py, three subcommands, each writing into the run's summary folder: - fold-draws: Figure 4's data from the run's own fit records and, for "before the fix", from bugarach.learn.train.fold_maker. It reproduces the committed fold_draws.json exactly; three murderboard roles found that no committed code had produced it. - merge-gap: the coded detectors tuned their merge gap and the nets' was fixed at 2 s. This rescores every coded choice, and re-decodes every chosen net refit from its saved model, with ONLY the merge gap changed, after first reproducing the run's own scores at each side's own gap. WSMIP065 will run the same code on the second draw. - replicate: the replicate's headline numbers, with source and commit recorded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rge gap reverses the headline All eleven roles ran against the first build. Their reports are archived verbatim in docs/reviews/fair-comparison-2026-09-19-roles/ (paths replaced; originals in the darkroom). The finding that changes the result is role 4's: the coded detectors tuned their merge gap and the nets' was fixed at 2 s. merge_gap.json measures both directions with only that setting changed, after reproducing the run's own scores (coded exactly, nets within 0.0015 F1). At the run's unequal gaps CoactDetect leads the best net by 0.007 F1; with the gap matched, at 2 s or at 8 s, chorus_norm leads. The page now says this run does not settle which side is better, and that a rerun tuning the nets' merge gap needs no retraining. Also from the review: the earlier comparison ran on a different, retired simulator; binned SCE's lead is its 30 s merge (it falls below CoactDetect at 8 s); one SCE choice passes the crowded check; the nets exceeded the budget on held-out data in 25 of 80 refits, now reported beside the coded side's misses; the coded side is scored on pooled training folds, not the inner rotation; the probe rate, the backgrounds (25th and 75th percentiles), the hit rule (spans), the tube description, the distractor construction and the coded detectors' origins are corrected; the contamination limit cites its record and the open participation item; t values carry the Nadeau-Bengio factor with its scope; tables are numbered; figures redrawn; the hand-drawn architecture figure is gone (draughtsman owns it). Glossary: merge gap, crowded-recording check, shared false-alarm budget, admissible. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All eleven round-2 reports, archived as they arrived (paths scrubbed; the originals are in the darkroom). Three pieces of evidence they asked for: - The crowded-recording check now also scores goal 1's own reference, the shipped operating points. No verdict changes, and the page can now say so from a file rather than a sentence. - A breakdown of held-out recall by participation level and background. The pooled F1 hides where the two sides differ: the faintest events, 10% of the ROIs, on the busy background go to the nets, and on the quiet one mostly to CoactDetect. - The replicate's refits below 0.2 F1, read by the same rule as this run's. Two failed fits in one replicate fold carry that draw's whole margin against the nets on F1 alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ect leads; on F1 alone the sign is not settled Round 2 of the murderboard found the round-1 headline half right. The merge gap does reverse the order, but only for choices on F1 alone: under the shared false-alarm budget CoactDetect stays ahead of both chorus nets at every matched gap. And the replicate's "agreement" was two refits that failed to train; set aside by one rule in both draws, the replicate's best net leads on F1 alone in 4 of 4 folds while this run's trails in 4 of 4. The page now says that, and a sentence about the data is printed only while it is true: the builder's claim() stops the build otherwise. Also: - the argument reordered: terms first, the method with its rules (the context window, the crowded check, "admissible" as the glossary defines it) before any number, and the merge gap in its own section after the results it qualifies; - new figures: the whole bench recording, how a call is scored (one call, one event), binned against sliding counting, both draws per fold with failed refits drawn twice, matched gaps for both selections, where the two sides differ (the faintest events, by background), and tuning; - attributions corrected: the nets are the Deep Sets shape, binned SCE is not the 2003 rule, CICADA cited by its DOI, SPIKE-synch's root added; - the tube refit's mechanism corrected (its threshold came from the inner fits), the corrected t factor stated exactly, reproduction bounds stated as measured; - a sapper feedback note for SAP004's blindness to backslashes, and the first author of the CICADA framework paper corrected in the README. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…, and the headline held to its own uncertainty Round 3 of the murderboard (eleven roles, blind) found no wrong number in the tables and a page that still said more than its evidence. Fixed here: - The answer leads with every refit counted: CoactDetect ahead on average in both draws and both selections. The best net leads only at a matched merge gap, or fold by fold in the replicate once failed refits are set aside, and that set-aside is now called what it is, a check made after seeing the held-out scores. - Under the budget, the matched-gap lead over chorus_norm is as weak as the F1-alone reversal (t -1.5 to -2.1), and the page now says so; the budget is named as CoactDetect's own rates times a declared 1.6. - The confound narrows: untuned chorus_norm is already within 0.02 of the tuned CoactDetect in both draws, so the earlier lead was not lost to tuning the nets. - What was not made equal gets its own section, and the limits add how the nets were trained (the probe and the distractors labeled negative), the three merge rules, normalization over the whole recording, and the tolerance measured for one side only. - The merge-gap concept moves to section 2 as its own figure; the gap curves get a logarithmic axis and a key; the margins figure drops the connecting lines that read as confidence intervals; new figures for the participation breakdown at all three levels and the crowded check; the failed refits of both draws become a table. - Attributions: CoactDetect's null credited to whole-train and per-cell shifts (Pipa 2008; Bocchio 2020; Dard 2022), Grün part II, the guard to Rohling 1983, SPIKE-synch's window to event synchronization and its detection step to Cecchini 2021, "designed here" read as "not found elsewhere yet". - Glossary: call, firing, background, empty recording, refit, the failed-training signature; merge gap and the budget corrected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…issible wins under the budget, and the last overreach removed A finding-driven pass over every round-3 finding, and role 10 re-run on the final render, both archived in docs/reviews/fair-comparison-2026-09-19-followup/. The pass caught one error the round-3 fixes introduced: section 9 said the crowded-recording check was never run on the replicate. It was. Read from the replicate's own crowded_check.json and selections, binned SCE under the budget is admissible and ahead of CoactDetect in 3 of 4 replicate folds (+0.011 to +0.017 F1), against 1 of 4 in this run. The page and its lede now say so, behind a claim() guard. Also: the contrast between the two selections is stated as consistency (under the budget every net is behind in every fold of both draws, guarded) rather than margin size, which the replicate contradicted; the scorer-resolution margins are computed; the lede's tuning and faint-event sentences are scoped; stale figure numbers are gone from the SVG labels; the ringed dot means one thing on every figure; Figure 8 says where its no-result marks sit; the axis-break bridge in Figure 11 no longer borrows LoCo's style. And a todo: goal 1's published crowded numbers compare two seed sets. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nal checks Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ng-driven check and role 10 on the final build Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e shipped build in its record Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…and says so Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… replacement field Figure 3's caption escaped its quotation marks inside an f-string expression, which 3.12 allows and 3.11, the declared floor, does not. tests/test_syntax_floor.py caught it on every leg; the caption now uses typographic quotes and says the same thing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…he new hash Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…fence it missed is fixed Figure 10's caption escaped quotation marks inside an f-string that was itself inside another's replacement field. The scan exempted all f-string literal text, including a nested one's, so it passed on 3.13 and 3.14 and the 3.11 leg alone failed. On 3.11 that text is expression, and a backslash there is a SyntaxError. The scan now flags a backslash in a nested f-string's text and a nested f-string reusing the enclosing quote; both new cases are proved to be 3.11 errors (checked on a real 3.11) and proved caught. Every tool and test in the tree parses on 3.11. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ecord the new hash Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… fits tools/tune_net_merge_gap.py re-decodes every saved fit of the fair comparison at eight merge gaps (0 to 30 s), re-picks each fit's threshold at each gap by its own rule, and re-selects on the inner fits only: F1 alone and under the shared budget, reported separately, with goal 1's move rule (0.002 inner F1) and its crowded-recording check inside the search. At 2 s it reproduces the run exactly: every picked threshold, row and empty-recording count, and both selections' inner and held-out F1. Pushed mid-run so the work is visible off this machine. The run's crowded pass is finishing; the selections, the replicate and the page (tools/build_net_merge_gap_page.py, not yet built) follow. The tests read committed outputs that are not here yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… reaches, where the full grid is impractical Four inner fits of one chorus_norm configuration ran more than three hours each on the full crowded grid (36 thresholds x 8 gaps x 24 three-hour recordings) without finishing, while the other 1,934 fits took under a minute each, and the re-chosen walk needs that configuration. select now lists the exact pairs it could not read, crowded --pairs scores only those, and a partial file refuses any cell it does not hold. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
np.load held the old file open, and Windows refuses to replace an open file, so every partial update failed with Access is denied. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ort's, and the outer-refit crowded check stated Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All 1,938 fits reproduce the run at 2 s. With the configuration kept, on F1 alone chorus_norm goes from -0.007 to +0.004 against CoactDetect (3 of 4 folds; inside the 0.010 noise scale). Under the budget every net stays behind except chorus_norm in one fold, by +0.004; chorus_gain_norm closes 45% of its shortfall. The nets settle at 5 to 8 s; the crowded check refused every wider choice. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s which The replicate's selections are still computing; a page of one draw is honest if it says so, and a page that cannot build until both are in is a page nobody sees tonight. Also: the goal page's chorus-collapse readout and todo are on main now, not arriving with #667. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… is the finding The Next entry becomes what the rerun found, from this draw's net_merge_gap.json: under the budget CoactDetect still wins 15 of 16 net-folds, tuning closes about 45% of the chorus nets' shortfall, and on F1 alone the +0.004 lead is inside the 0.010 draw-to-draw scale. The no-retraining claim is back, sourced to the tool that re-decodes the saved fits. Stated as its own finding: the crowded check refused a wider gap in all 32 choices, 30 s for every net, so long-gap inflation is not specific to the coded side; and one choice fails that check on the outer refits. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ailed trainings The replicate's 1,943 fits re-decode and reproduce its run at 2 s. Under the budget CoactDetect is ahead of every net on average in both draws. On F1 alone the sign depends on the chorus nets' collapsed trainings: counting every refit the nets trail in the replicate, and with the refits under 0.2 F1 set aside both chorus nets lead there by more than the draw-to-draw scale. The page reports both, and Figures 1 and 2 scale to the folds that trained rather than to a collapsed one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…read better than the data Eleven roles, one round, reports archived verbatim. What changed, each because the first draft let a reader conclude more than the numbers support: - The set-aside was applied to the tuned choice and not to the baseline, so the gap was credited with a lead that predates it. Both sides now carry it: in the replicate chorus_norm is already +0.005 ahead at 2 s, and the gap adds 0.010, one noise unit. - Held-out F1 at every gap is now drawn (Figure 1). It peaks at the top of the grid in all 64 folds, so every chosen gap is a boundary the crowded check imposed, not an optimum -- and CoactDetect's 8 s is the top of its own grid. The two landing together is not agreement. - The crowded check's cost to the settings it ACCEPTED is reported beside its refusals: a median 0.012 F1, the same size as the gain, with two choices failing it outright on the outer refits. - The noise scale is named for what its source measures, two draws on two machines, and the page now says that source also measures a systematic +0.015 F1 shift in the replicate's favour -- the size of the lead there. Also: the budget never bound CoactDetect while 44 of 160 net refits break it on held-out data; the threshold is unchanged in 26 of 32 gated choices, so the gain is not the lower-threshold mechanism the draft claimed; failed trainings and no-call refits are counted separately, from the run's own flags; Figure 2 counts refusals instead of stacking 70 crosses; the figures keep an 11 px floor, a readable grey and a gutter for off-scale folds; Table 1 marks within-noise means and failed checks. The tool records the machine, torch and both trees' commits, asserts the threshold grid it indexes, and refuses a move whose crowded score is not finite. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…y overturned The crowded cost the page reported is each net against its own 2 s, and the head-to-head on those same recordings runs the other way: the tuned chorus nets beat CoactDetect there by 0.016 to 0.056 F1 while tube loses 0.14. Figure 5 now draws both, and the text no longer lets 'the gain and the cost are the same size' stand in for a comparison it is not. CoactDetect's grid stops at 8 s because goal 1's crowded check refused its 16 s -- so the check bound both sides, at different times, and the page said it had never bound CoactDetect at all. Held-out F1 is largest at the widest gap but not monotonically: it dips in 18 of 64 fold-rows, so 'a wider merge can only delete duplicate calls' was too strong, and the glossary already said why. 'The nets that end ahead were already ahead' was false -- chorus_norm crosses zero in this draw -- and the lede's range covered the chorus nets while saying 'each net'; tube gains a fifth as much. Also: the set-aside cannot be even-handed and says so (CoactDetect has no refits); low refits are counted as refits, not decodings, and split by the run's own two failure kinds; the reproduction is described for what it covers, at 2 s, on one machine for both draws; the empty band below the cut is computed (0.388, not a rounded 0.39); the budget's over-budget count carries its as-run baseline; the grid's coarseness says which way it cuts; panels A and B are drawn at one scale; the table's footnote marker is no longer a degree sign after a number, the crowded-check flag marks every column that uses the flagged gap, and each figure is now referenced by number and name. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two blind rounds, eleven roles then five, every report archived verbatim. The record leads with what the review changed about the result and is delivered UNCONVERGED: blocking findings went 11 then 10, and the process says a flat count means a structural problem patching will not retire, so it escalates instead of running a third round. The roster gate refused a complete archive because reports_decl() strips underscores as markdown emphasis, rewriting net_merge_gap_...-roles/ into a path nobody wrote. The archives are renamed with hyphens and the defect is filed for upstream: a gate whose job is to catch a cited-but-missing archive reported a present one as missing, and the quickest way to green would have been to delete the line. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The merge gap the nets were never allowed to tune, tuned by the same rules — on both draws, with no retraining — and murderboarded twice.
What it settles. In the fair comparison the coded detectors' searches tuned their merge gap (CoactDetect took 8 s in every fold) while the nets decoded at the fixed 2 s their threshold picker returns. That was the open asymmetry. Here the nets' gap is selected, on the inner fits of the training folds, by the rules the run used for everything else, with goal 1's move rule and its crowded-recording check inside the search.
No retraining, and it reproduces the run exactly. A merge gap is applied when a net's per-frame output is decoded, so every saved fit of both draws — 3,881 of them — was re-run and decoded at each gap in the grid. At 2 s every picked threshold, every recording's counts, every empty recording's call count and both selections' inner and held-out F1 match the runs' own files.
The result
The review, and why this ships unconverged. Eleven roles blind, then five re-run blind against the rebuilt page: 11 blocking findings in round 1, 10 in round 2. The process says a flat blocking count means a structural problem patching will not retire, and to escalate rather than run a third round — so both rounds' fixes are applied and the record ships with named residuals rather than a clean bill. The review changed what the page may claim, not just how it reads: the one-sided set-aside, the censored boundary, the mis-referenced crowded cost, the noise scale's own systematic shift, and a budget mechanism the data refute.
What is here
tools/tune_net_merge_gap.py: the re-decoding and the selections, reusing the tuning tool's own objective, budget, pooling and tie rules. Records the machine, torch and both trees' commits; asserts the threshold grid it indexes.tools/build_net_merge_gap_page.py: the page. Every sentence about the data is computed, and the directional ones sit behindclaim(), which stopped the build three times during this work.docs/reviews/net-merge-gap-2026-09-19.mdplus both role archives, verbatim.docs/goals/learned-model-family.md,docs/GLOSSARY.md: updated in the same PR, as the rules require.docs/sapper_feedback/2026-09-19-the-roster-gate-eats-underscores.md: the roster gate rewrites underscores in the archive path and reports a complete archive as missing.Open for Tony: the 0.02 F1 crowded allowance is still unsigned and decides every gap here — one refusal turned on 0.0009 F1.
🤖 Generated with Claude Code