Skip to content

test(seed-policy): inventory every seed flag; guard the shared study runner - #305

Merged
Telschow merged 2 commits into
mainfrom
test/seed-flag-inventory
Nov 7, 2026
Merged

Telschow merged 2 commits into
mainfrom
test/seed-flag-inventory

Conversation

@Telschow

@Telschow Telschow commented Nov 7, 2026

Copy link
Copy Markdown
Owner

Follow-up to the post-remediation audit (CN-AUD-02: every seed-taking path needs a refusal).

  • New test scans src/navkit for --...seed... arguments and fails if a module is neither listed as guarded nor as a gate. It also checks the list for stale entries.
  • It found one gap: the shared study runner took --seeds for Flags(seeds=True) without the policy check. No study sets that flag, so no held-out seed was ever reachable. The runner now calls require_development before any case runs.
  • Refusal tests for --growth-seeds, --fault-seeds, --ensemble-seeds and the runner. No seed of 100 or more is run.

Not changed: the two gate procedures, which may use held-out seeds by design.

🤖 Generated with Claude Code

…runner's --seeds

The inventory test fails when a module adds a seed flag without a listed refusal case. It found one gap:
report/runner.py accepted --seeds for a study with Flags(seeds=True) without the policy check. No study sets
that flag today, so nothing ran. The runner now refuses seeds from 100 before any case runs.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Comment thread tests/test_seed_policy.py
import re
from pathlib import Path

import navkit
@Telschow
Telschow merged commit bee838d into main Nov 7, 2026
30 checks passed
@Telschow
Telschow deleted the test/seed-flag-inventory branch November 7, 2026 09:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants