feat: Phase F3 declared snapshot/restore capability with measured-fork branching and honest refusal - #68
feat: Phase F3 declared snapshot/restore capability with measured-fork branching and honest refusal#68txmed82 wants to merge 3 commits into
Conversation
…k branching and honest refusal - BranchingSupport ladder (unspecified < declared < prefix_replay < measured_fork; not_supported is a refusal state, never a requestable mechanism) on WorldCapabilities, riding the same positional task-vs-adapter cross-check as every other gate via gates(). - measure_branch_support: two independent forked rollouts of one seed grade the ladder; measured_fork only from byte-equal canonical digests, prefix_replay only within an explicit finite tolerance, otherwise not_supported with the first difference. - branch_from: every arm re-runs the shared prefix on a fresh instance; a prefix that fails to replay is refused with zero candidate comparisons - a seed alone must not be asserted to restore a full mutable state. BranchEvidence keeps claimed separate from support so a loud declaration cannot overwrite a weak measurement. - require_branch_support: fail-closed gate; a bare declaration may drive arms but can never certify equality; lumen-gym declares prefix_replay (the one mechanism the pinned stack proves), no built-in claims measured_fork.
|
Droid encountered an error —— View job |
|
@droid review |
|
Droid encountered an error —— View job |
|
@droid review |
|
Droid encountered an error —— View job |
…rtification hole _rollout discarded both observations, so a world whose nondeterminism lives only in the policy-visible stream digested byte-equal and was certified measured_fork; two different candidate plans whose reward/info are action-independent constants could also digest equal, so a branch comparison could claim candidate divergence the policy could refute. The reset record and every step entry now carry jsonable(observation), and _first_difference names the diverging channel so a refusal says which stream failed to replay. Regression envs pin all three holes: observation-only jitter cannot certify a fork (and is refused outright as an arm replay), and identical plans on worlds that differ only in observation must digest apart. Verified counterfactually: with the observation stripped from the trace each new test fails, with it recorded each passes.
|
@droid review |
|
Droid encountered an error —— View job |
|
@droid review |
|
Droid finished @txmed82's task —— View job Droid review complete The branching ladder, fail-closed gate, and task-vs-adapter cross-check are carefully designed and well tested, with strong honest-refusal semantics. One high-confidence correctness bug: the tolerance branch in Posted 1 inline comment. |
| candidate_diverged=False, | ||
| max_float_delta=0.0, | ||
| ) | ||
| if delta <= tolerance and len(first) == len(second): |
There was a problem hiding this comment.
[P1] Zero/finite tolerance excuses categorical and structural divergence, granting prefix_replay to worlds whose twins differ in non-float channels
_compare_fork_pair grades PREFIX_REPLAY whenever delta <= tolerance and trace lengths match, but _float_delta returns 0.0 for every non-numeric leaf: strings, bools (explicitly skipped), mismatched dict keys (only left.keys() & right.keys() is walked), and list elements past the shorter length all contribute nothing. So with the default tolerance=0.0 on both measure_branch_support and branch_from, two twin rollouts that diverge purely categorically — e.g. an engine that stamps a run id or timestamp string into info/reset_info, or flips terminated on the final step so lengths stay equal — produce delta == 0.0 <= 0.0 and are graded prefix_replay with prefix_equal=True, and the recorded first_difference even claims the digests 'differ within tolerance 0'. This directly contradicts _float_delta's own stated contract ('a structural difference is caught by the digest comparison, which is stricter than any tolerance can be') — the digest mismatch is detected but then explicitly excused by the tolerance branch. The trigger is realistic (any engine logging non-numeric run metadata) and the observable result is wrong: measure_branch_support certifies prefix replay for a world whose twins categorically diverged, one rung above the not_supported it should report, which then lets require_branch_support(FORK_EQUALITY_MECHANISM)-adjacent consumers trust a replay guarantee the engine never provided. The tolerance check should only excuse differences that are actually float deltas; a byte-different pair with any non-numeric divergence should fall through to not_supported regardless of tolerance.
What
Phase F3 from
docs/NEXT_STATUS.md: a declared snapshot/restore capability for simulator branching with honest branch-equivalence refusal. The kernel ships no verifiedsnapshot/restoreprimitive (the bridges'snapshot()methods are read-only reports, not restorable checkpoints), so this PR:BranchingSupportladder (unspecified < declared < prefix_replay < measured_fork;not_supportedis a refusal state, never a requestable mechanism) onWorldCapabilities, riding the same positional task-vs-adapter cross-check as every other eligibility gate viagates()— a task can't grant itself a fork mechanism the installed adapter withholds;measure_branch_support: two independent forked rollouts of one seed grade the ladder —measured_forkonly from byte-equal canonical digests,prefix_replayonly inside an explicit finite tolerance, otherwisenot_supportedwith the first difference recorded.BranchEvidencekeepsclaimedseparate fromsupportso a loud declaration can't overwrite a weak measurement;branch_from: every arm re-runs the shared prefix on a fresh instance and is rolled twice; a prefix that fails to replay is refused with zero candidate comparisons — a seed alone must not be asserted to restore a full mutable state;require_branch_support: fail-closed gate — a baredeclaredclaim may drive arms (DRIVING_MECHANISM) but can never certify equality (FORK_EQUALITY_MECHANISM), which needs the measured rung;lumen-gymdeclaresprefix_replay(the one mechanism the pinned stack proves,tests/test_lumen_branch.py); no built-in claimsmeasured_fork— that rung is earned per adapter at run time.Tests
tests/test_branching.py(new): deterministic scripted fakes (no external stack) pin the four contract rules — declared-but-unmeasured rejection, measurement-earned acceptance, prefix-divergence honest refusal (zero arms emitted), candidate-divergence detection (candidate_diverged=Falseon identical plans means the intervention measured nothing), numpy-action content digests, tolerance-rung grading, fail-closed gate matrix, and the task-vs-adapter cross-check.All existing capability/conformance/bridge suites pass unchanged:
ruff check+ruff format --checkclean,mypyclean (180 files), 248 tests green acrosstest_branching / test_world_kinds / test_simulation_bridges / test_conformance / test_lumen_branch / test_multi_modality_contracts / test_eval_run / test_eval_p3 / test_eval_v03 / test_eval_contracts.Out of scope (stated, not silently skipped)
conformance.pygains no new check here (no Tier change); F4 (closed-loop simulator branching) and themeasured_forkdeclaration for non-Lumen bridges stay open until their forks are actually measured;LUMEN_REPLAYstaysunspecified— its adapter semantics aren't verified in-repo, and declaring an unproven fork mechanism is exactly what this PR refuses.Roadmap:
docs/NEXT_STATUS.mdF row (F3), plan §F lines 422-427.