Skip to content

fix(b1-gate): refuse --held-out outright; the ADR-0028 set is spent - #312

Merged
Telschow merged 2 commits into
mainfrom
fix/b1-gate-refuse-held-out
Nov 7, 2026
Merged

Telschow merged 2 commits into
mainfrom
fix/b1-gate-refuse-held-out

Conversation

@Telschow

@Telschow Telschow commented Nov 7, 2026

Copy link
Copy Markdown
Owner

Follow-up to #309. Maintainer decision (2026-11-07): the held-out set of ADR-0028 (seeds 100 to 124) was evaluated once (ADR-0036) and is spent, so navkit gate b1 no longer runs it.

Changes

  • navkit gate b1 --held-out exits 2 with the reason and a pointer to navkit gate candidate (ADR-0030). The flag is kept only so the refusal is explicit.
  • run_gate and check_seeds have no held-out mode: development seeds only.
  • tests/test_seed_policy.py: b1_gate.py moves from the gate list to the guarded list. The candidate gate is now the one procedure that may use held-out seeds.
  • seed_policy docstring, navkit gate help and a CI comment are updated to match.
  • ADR-0028 gets a dated note. Its decision section is unchanged.
  • The recorded held-out result still renders from docs/data/b1_gate_heldout.csv (scripts/render_b1_heldout.py, which runs no case).

Evidence

  • Test-first. Every refusal test stops at a monkeypatched sentinel. On the old code, run_gate(100-124, held_out=True) got past the guard and hit the sentinel, so the red phase ran no held-out seed.
  • Full suite: 2787 passed, 36 skipped, 2 xfailed (local, -n auto). ruff, format, mypy and make docs-fast are clean.
  • No number, table, CSV or golden changes. No seed of 100 or more was executed.

🤖 Generated with Claude Code

The held-out set 100-124 was evaluated once (ADR-0036). Maintainer decision,
2026-11-07: `navkit gate b1` takes development seeds only. The flag stays so
the refusal names the reason and points at `navkit gate candidate`.
run_gate and check_seeds lose their held_out mode. The seed-flag inventory
moves b1_gate.py from the gate list to the guarded list. ADR-0028 gets a
dated note; its decision section is unchanged.

Test-first. On the old code the new run_gate test got past the guard on
100-124 and was stopped by the monkeypatched sentinel, so no held-out seed
ran in the red phase.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread tests/test_b1_gate.py Fixed
…a removed keyword

CodeQL py/call/wrong-named-argument flagged the deliberate held_out=True call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Telschow
Telschow merged commit 8a00486 into main Nov 7, 2026
30 checks passed
@Telschow
Telschow deleted the fix/b1-gate-refuse-held-out branch November 7, 2026 17:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants