Skip to content

Hardening: held-out seed guards, local build hygiene, claims matched to evidence - #309

Merged
Telschow merged 4 commits into
mainfrom
claude/contested-nav-research-hardening
Nov 7, 2026
Merged

Telschow merged 4 commits into
mainfrom
claude/contested-nav-research-hardening

Conversation

@Telschow

@Telschow Telschow commented Nov 7, 2026

Copy link
Copy Markdown
Owner

Post-remediation hardening from the 2026-11-07 review of bee838d. Part of #250.

What changes

Held-out seeds (fix, test-first)

  • navkit gate b1 accepted seeds from 125 (the ADR-0030 blocks of the candidate gate) without a flag, and --held-out accepted any seeds. The parser and run_gate now share b1_gate.check_seeds: development seeds, or exactly 100 to 124 with --held-out.
  • navkit run --config used a YAML's own stream seeds unchecked when --seed was absent. run_case now refuses held-out configured seeds when no seed replaces them. navkit run checks every selected case before the first one runs.
  • Every new refusal test stops at a monkeypatched sentinel before any run, so a red phase cannot draw a held-out seed. The test that asserted 125-149 is accepted now asserts the refusal.

Local build and automation hygiene

  • .dockerignore keeps .env files, node_modules and the Sandcastle logs and worktrees out of the build context. The Dockerfile ends with COPY . .. A test checks the patterns.
  • Sandcastle builds the branch name from a numeric issue id instead of taking it from the planner's output.
  • The merger prompt no longer says main rejects its pushes. It doesn't today (see below).

Claims that the evidence did not support

  • EuRoC and TUM VI: "Not verified here" is replaced by the rights statement and licence the pages already quote. The README licence section points to ADR-0042 and Decide whether EuRoC-derived rows may be published #278, and says no rights-holder answer is recorded.
  • FAQ: per-frame rows (vo_errors.csv) are committed, not only aggregates.
  • Site footer: the MIT licence is the code's.
  • README "How this was built": pull requests through CI are the practice, not an enforced rule.
  • UrbanNav: the gate section no longer names multipath as a measured cause. The urbannav_imposed docstring says it writes a CSV with --csv.
  • ADR-0010 amendment: the streams-only sentence covers the three vision cases and the two metrics it was read on. The decision section is unchanged.

No number, table, generated CSV or golden snapshot changes.

Evidence

  • Full suite: 2787 passed, 36 skipped, 2 xfailed (pytest -n auto, local, macOS, at 320e1ec).
  • ruff, ruff format, mypy (87 files) and make docs-fast (763 passed) are clean.
  • Seeds run in this work: development seeds 0 to 99 only, in an exploratory shared-stream measurement written to scratch and not committed. No seed of 100 or more was executed.

Not in this PR (maintainer decisions)

  • The Main ruleset targets refs/heads/Main, so main enforces only deletion and non-fast-forward. Repository settings: for the maintainer.
  • CLAUDE.md:31 still says the sandbox token cannot push to main.
  • Whether gate b1 --held-out should be refused outright now that the set is spent.
  • EuRoC rows: Decide whether EuRoC-derived rows may be published #278.

🤖 Generated with Claude Code

Telschow and others added 4 commits November 7, 2026 11:05
…anches from the issue id

.dockerignore did not exclude .sandcastle/.env, and the Dockerfile ends with COPY . .,
so a local build copied the sandbox tokens into a layer. The Sandcastle planner's
free-text branch went into shell blocks of the review prompt; it is now built from
a numeric issue id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- EuRoC and TUM VI: replace "Not verified here" with the rights statement and
  licence the pages already quote; point the README licence section at ADR-0042
  and #278, say some files also hold TUM VI rows and no rights-holder answer is
  recorded.
- FAQ: per-frame rows are committed (vo_errors.csv), not only aggregates.
- Site footer: the MIT licence is the code's; EuRoC-derived results are
  non-commercial.
- README: PRs through CI are the practice, not an enforced rule; some commits
  went to main directly and Sandcastle merges its own PRs.
- UrbanNav: the gate section no longer names multipath as the measured cause.
- urbannav_imposed docstring: it does write a CSV with --csv.
- ADR-0010 amendment: the streams-only comparison sentence covers the three
  vision cases and the two metrics it was read on.

No number, table, CSV or golden is changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… pushes

The Main ruleset targets refs/heads/Main, so main enforces only deletion and
non-fast-forward, and the sandbox token is an admin token. The prompt's guard
stays; only the false reason is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… config seeds are checked

- `navkit gate b1` accepted seeds from 125, the ADR-0030 blocks of the candidate
  gate, without any flag, and `--held-out` accepted any seeds. Both the parser
  and run_gate now go through b1_gate.check_seeds: development seeds, or exactly
  the ADR-0028 set with --held-out.
- `navkit run --config` used the YAML's own GNSS, vision and camera-drop seeds
  unchecked when --seed was absent. run_case now refuses held-out configured
  seeds when no seed replaces them, and navkit run checks every selected case
  before the first one runs.

Test-first. Every new refusal test stops at a monkeypatched sentinel before any
run, so a red phase cannot draw a held-out seed. The test that asserted 125-149
is accepted without the flag now asserts the refusal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Telschow
Telschow marked this pull request as ready for review November 7, 2026 14:48
@Telschow
Telschow merged commit f7ddbd0 into main Nov 7, 2026
30 checks passed
@Telschow
Telschow deleted the claude/contested-nav-research-hardening branch November 7, 2026 15:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant