Hac/343 metric manifest - #29
Merged
Merged
Conversation
The packet's central claim is that its numbers are reproducible, and a metric defined after its value is known is not a measurement. So this lands on its own, ahead of any corpus, any arm implementation and any results.json. Editing it in place once a result has been generated against it is forbidden; the repair is a new revision with the reason recorded in supersedes[]. Four arms, per HAC-319's requirement that the alternative be the strongest one a skeptical judge would propose: A1 uncoordinated — establishes the hazard is real A2 global lock — safety ceiling, concurrency floor (HAC-343) A3 per-target lock — the discipline a judge proposes instead A4 Interlock — arbitrate() over frozen co-change evidence A2 alone could not satisfy HAC-319: its Safe Parallelism Retained is zero by construction, so "Interlock beats it" is a tautology rather than a finding. A3 is the arm that can narrow or defeat the thesis — it preserves concurrency on disjoint paths and has no access to co-change evidence, so a cross-target coupling is invisible to it. Both lock arms carry a validity gate: they must serialize every SAME_TARGET_CONTENTION scenario or the run fails outright. Without it an allow-all implementation would produce the same unsafe result as a real lock, and a judge would correctly dismiss the finding as a strawman that never locked anything. The gate is what makes an unsafe A3 result mean "per-target locking cannot see this hazard" rather than "their baseline was broken". Decisions worth naming, because each closes a way the numbers could flatter us: - SPR is never reported bare. It is an ordered pair with the unsafe joint-state rate, because an arm that permits everything scores SPR 100% and is unsafe; A1 will do exactly that. - ALLOW_SERIALIZED counts as permitted but not as parallel. Counting serialized progress as parallelism would inflate the one number the experiment exists to test. - No confidence intervals. The core is deterministic and the corpus is a frozen finite enumeration evaluated exhaustively, so there is no sampling process; an interval would imply a population and a draw that do not exist. Exact numerator/denominator instead, and "n/a (0 cases)" rather than a green 0% or 100%. - Nonzero evidence sensitivity in A1/A2/A3 is a defect signal, not a result: those arms consume no evidence, so a decision that moves when evidence moves means the harness is leaking state between arms. - The core is bound in writing to dist/broker/pairing/arbitrate.js. experiments/hac-330/lib/decide.mjs collapses every coupled outcome to WITHHOLD_SERIALIZE, which would pin SPR at zero for coupled scenarios and make A4 indistinguishable from A2 by construction — a null result wearing a finding's clothes. A3 is never called a "safe baseline". It is safe against same-target contention and may be blind to hazards spanning distinct targets; naming it without naming the hazard class is a claim defect. Refs HAC-343, HAC-319.
Second freeze, after 0a6babb and before any arm implementation. The labels are properties of the fixtures and the intents, assigned by construction, and they are fixed here so nothing can be adjusted once an arm has an opinion. metric-definitions.json forbids that in writing; this commit is what makes the prohibition checkable. ## Why a second family The budget family is one topology — three reservations against a fixed pool — so a corpus built only from it measures one hazard shape at several evidence states. A reader could reasonably ask whether the result is about composition evidence or about arithmetic. So family 2 encodes a hazard with no arithmetic in it at all: every route.service and alias target resolves in the registry Composition fails because one intent removes a referent the other starts pointing at. Asymmetric (delete versus add) where budget is symmetric (overshoot), and referential where budget is numeric. An arm that passed family 1 by reasoning about magnitudes has nothing to reason about here. As in HAC-330, no coupling is written into an evidence file. The histories are ordinary commits and the coupling is a consequence of how a registry is maintained: you cannot canary a service without declaring it and routing to it, nor retire one without withdrawing the route first. Mining confirms it — baseline carries registry/services.json <-> routing/routes.json at support 9, and the perturbed history loses exactly that pair while keeping the unrelated docs <-> dashboards pair at support 5 in both. The perturbation is specific, not a general change of history. Family 2 holds the same four controls family 1 does: identical final tree (b57f883c20fa), identical commit count (18), commit i touching the same number of files in both, and the invariant asserted at every planned commit. ## Coverage 16 scenarios. Both families carry all five ground-truth classes, so a per-family divergence in the results is attributable to hazard shape rather than to one family having been handed easier cases: COUPLED INDEPENDENT SAME_TARGET PERTURBED INADMISSIBLE budget 1 1 1 1 4 registry 1 1 1 1 4 validateCorpus() enforces the properties that make those labels mean something, and build-corpus.mjs exits non-zero on any of them: - COUPLED must be cross-target, or per-target locking would see it and A3 would not be testing the distinction the experiment exists for; - SAME_TARGET_CONTENTION must genuinely share one path, or the A2/A3 validity gates prove nothing; - INDEPENDENT must actually compose safely, or permitting it is not the correct behavior to score; - a perturbation must hold its intents byte-identical to its counterpart and differ only in fixture, or evidence sensitivity measures the intents rather than the evidence; - every class must be populated in both families, or a metric reports a green zero for a case the corpus could never produce. Family 1's fixtures are read from the HAC-330 packet rather than regenerated, so the budget family here is the same history S-1 proved. Refs HAC-343, HAC-319.
Third freeze and the implementation, with zero result artifacts. The sequence stands: metric semantics (0a6babb) -> corpus (dbdcaa9) -> arms -> results. Neither earlier commit is touched. ## One executor, four policies Everything about what an action does and what it checks before doing it lives in executor.mjs. The only thing an arm contributes is *when* that check happens. There is deliberately no per-arm hook: a capability on one arm's execution path and not another's would measure the harness rather than the coordination policy. Concurrency is stated precisely rather than implied. Two intents run concurrently when both evaluate their precondition against the same base snapshot and both writes then land; serially when the second evaluates against what the first already wrote. That is the whole hazard — every precondition was true when checked and false by the time the last write landed — and it is HAC-330's model generalised across families and arms rather than a second one invented here. ## The lock baselines are real Both A2 and A3 use one shared critical section: acquire, re-read, re-check the ordinary local precondition, mutate or reject, release. The re-read is what makes them credible. Merely replaying two already-approved mutations in sequence would still overshoot, and a baseline that loses that way loses to a strawman. The preflight test proves the re-read discriminates, and the first version of that test did not. It asserted against the same-target pair (alpha->60 then alpha->55), where the second projection is 115 whether or not the first write was observed, because the second intent overwrites the same key — a vacuous assertion that passed green. It now uses the cross-target pair, where re-reading projects 60+60+20=140 and rejects, while replaying against the base would project 40+60+20=120, apply, and overshoot. Only one of those two behaviours passes. A1 and A3-on-distinct-targets reduce to the same code path on purpose. Holding two different locks provides exactly as much mutual exclusion as holding none, and saying so in code is the clearest form of the finding. ## Preflight gates — 22 tests, no evidence required The oracle can fail. Each family's verify.mjs is generated by the same generator that built its fixture, which makes it independent of the arm harness but not of fixture construction. So it is mutation-tested before it is trusted: untouched fixture passes, coupled composition fails, single-path invalid state fails, and a violation does not stick across a worktree reset. holds is the exit code and nothing else — stdout is recorded but never parsed, so a verifier printing a reassuring report while exiting non-zero reads as a violation. A verifier that cannot spawn fails the scenario; an unanswerable question is not an answer of "valid". The locks lock. A2 serializes every scenario including cross-target ones. A3 serializes same-target intents and parallelises distinct- target ones — both halves asserted, because an A3 that serialized everything would be a global lock wearing a per-target label, and its blindness to cross-target coupling would prove nothing about per-target locking. Neither gate needs mined evidence, so both run in CI without the pinned workspacejson/cli checkout. They are already covered by the existing vitest include for experiments/**/test/*.test.mjs. ## Order policy, frozen before execution Every scenario runs in both intent orders. The metric denominators were frozen as scenario counts in r01 and this does not reopen them; the two orders aggregate per scenario, conservatively: unsafe if either order violates, parallel only if both orders permit concurrently, refused only if both orders refuse with the expected reason. Every rule is conservative toward A4 — it can lose on one order and lose overall, and cannot win on a lucky ordering. Both raw per-order executions are retained, and any scenario whose orders disagree is reported explicitly as an order effect rather than averaged away. Refs HAC-343, HAC-319.
Fourth commit, zero result artifacts. The pipeline is now complete and frozen before execution, so reporting logic cannot have been shaped around an outcome nobody has seen: 0a6babb metric definitions dbdcaa9 frozen corpus 276750b arm semantics <this> evaluation machinery next results ## Runner Consumes only what is already frozen and decides nothing. 16 scenarios x 4 arms x 2 orders = 128 records, and a record is written for a failure exactly as for a success — an arm must not be able to improve a rate by declining to produce one. A thrown execution is captured as a record with its error and counted as unsafe, never dropped. A4's degraded-evidence cases are looked up from the corpus's INADMISSIBLE_EVIDENCE table by the name the scenario declares. The runner has no branch that decides when an arm gets unreadable evidence, because a runner that could choose that could choose to hand it to one arm and not another. `--plan` resolves fixtures, evidence and the full matrix and prints what would execute without running an arm or writing a result. Used to prove the wiring before any number exists. ## Aggregator A pure function over raw records — no filesystem, no git, no clock — so the verifier can recompute every number rather than read a summary. Three properties matter more than the arithmetic: - it cannot skip: a missing scenario x arm x order throws rather than quietly computing a rate over a partial matrix; - it cannot suppress: refused, rejected and errored records are counted, not filtered; - SPR cannot escape alone: `assembleSpr` is the only constructor and throws without an unsafe-joint-state rate, so the pairing is enforced in the type rather than left to whoever renders it. Lock validity is computed and reported before the headline metrics, and per-family metrics before the aggregate, so a failing family cannot be averaged into an acceptable-looking whole. ## Packet verifier Recomputes the report from the raw records and requires byte equality with the committed one; a verifier that compared a summary to itself would pass on any summary. It also pins the three freeze commits two ways. Each frozen contract must still be the file its freeze commit introduced (`git log -1`), AND the bytes on disk must match the blob at that commit. The first check alone answers only about committed history, so an uncommitted edit to metric-definitions.json would have passed it. Verified by appending one newline: the blob check flips to FAIL and reverting restores it. ## Adversarial tests — 18, before execution The aggregator is the piece a skeptical reader has least reason to trust, so it is fed synthetic records built to make it lie: - A3 fails one same-target case -> lockValidity defect gate fires; - A1 at SPR 100% with a nonzero unsafe rate -> renders "UNSAFE, not safe parallelism" and can never read as the winner; - one order safe and the other unsafe -> scenario is unsafe, and a signature disagreement is listed under orderEffects rather than smoothed; - a deleted record -> throws; a duplicated one -> throws; - one family failing while the other passes -> visible per family AND in the aggregate; - A1/A2/A3 appearing to respond to evidence perturbation -> defect gate fires, because those arms read no evidence and a moving decision means the harness is leaking state between arms. ## One latent bug fixed arms.mjs keyed outcomes on `intent.id`, but the frozen corpus declares intents without one — so `intentId` was undefined and phase 2's `find()` would have replayed the wrong write. The corpus is correct and stays untouched; ids are an artifact of execution, assigned by the runner from each scenario's canonical order so they survive permutation. arms.mjs now asserts them rather than defaulting, and lock keys and correlation ids derive from the id rather than the execution index so a permutation cannot look like a difference on its own. Full suite: 477 tests, 19 files, green. Refs HAC-343, HAC-319.
Executed once, preserved as run. Verifier passes: 128/128 records, no
defect gate tripped, and the committed report is byte-identical to a
recomputation from the raw records. Two findings below are unfavourable
and are retained rather than reconciled.
SPR at unsafe-joint-state rate, aggregate over both families
A1 uncoordinated 100.0% at 100.0% — UNSAFE, not safe parallelism
A2 global lock 0.0% at 0.0%
A3 per-target lock 100.0% at 100.0% — UNSAFE, not safe parallelism
A4 Interlock 100.0% at 0.0%
Both lock baselines passed their validity gates at 2/2, so A3's unsafe
result is blindness rather than a broken lock: it serialized every
same-target case and parallelised every cross-target one, then missed a
coupling spanning two lock keys. Identical in both families.
## Finding 1 — refusal correctness is 6/8, and the corpus was wrong
Both noRepository scenarios refused with HISTORY_NOT_MINED where the
frozen corpus predicted HISTORY_EVIDENCE_UNAVAILABLE. The envelope's
completeness state is NOT_MINED, not EVIDENCE_UNAVAILABLE, so
arbitrate's Guard 3 took the correct branch and the corpus's expected-
reason mapping was mistaken.
Safety behaviour is unaffected: all 8 inadmissible scenarios refused
fail-closed with an explicit machine-readable reason and applied
nothing. The corpus is frozen and stays wrong on the record; 6/8 is
reported as measured. Repairing the expectation and rerunning would be
tuning a frozen input after seeing the outcome.
## Finding 2 — A4 produces an invalid joint state on perturbed evidence
On both EVIDENCE_PERTURBED scenarios A4 returned ALLOW_PARALLEL, both
intents applied, and the fixture verifier reported the composition
invalid. This is the causal control working as designed — the same
result HAC-330 published — and it is the bound on the claim: Interlock's
safety is exactly as good as its evidence. Remove the coupling from the
history and it permits the composition.
It also exposes a gap in the frozen metric definitions. The unsafe-
joint-state denominator is COUPLED scenarios only, so these two invalid
outcomes sit outside both numerator and denominator, and A4's headline
reads 0/2 (0.0%) while A4 in fact produced an invalid joint state in 2
of 16 scenarios. That is not corrected here — the definitions are
frozen — but no judge-facing surface may render A4's unsafe rate
without stating it.
## Order effects
Twelve, all A2, all the same shape: under one global lock whichever
intent enters first wins and the other is rejected. Real and expected,
surfaced rather than averaged. The frozen aggregation absorbs it
correctly — safety holds under both orders, and A2 gets no parallelism
credit either way.
## Provenance
WORKSPACEJSON_CLI was set to the 0.6.2 checkout as directed and was
provably not loaded: the runner plans identically with it unset and
with it pointing at a nonexistent path, and nothing in its import graph
reaches the miner. The evidence consumed is frozen in git and records
its own producer — mining-core at pinned CLI defac1e5, bundle
7aa5ae23. CLI 0.6.2 is reproduction tooling here, not an input.
node v22.19.0 · pnpm 10.24.0 · git 2.52.0 · darwin arm64
@workspacejson/cli@0.6.2 sha512-DyXe4oY4s6paN9lgLkFnhj9x46Excg3GSSQfcF3VBTzGj2LUeosaM3iZ5NgM1was8hQWhxibyS1a7YOq5OxI5Q==
@workspacejson/spec@0.5.0 sha512-KpsUxvLXFHHHKY6F58tWBnqsx5REJjK99Kum1+ATU4b8oUGlStfVkWyphNQ+nFZU3hy/ckNLZTxs4mpOeWGQLA==
@workspacejson/rules@0.5.0 sha512-UlJUnDdc1In4oAMCNMFbFnCAVUGSb1HR0MeP3Pa6Db3uHrH/OvXPyQCqGXLDe9ii3D+6Ua/OHAJMeocKv2bL1Q==
Refs HAC-343, HAC-319.
Presentation only, generated from 7ede0f9. Modifies none of: metric definitions, corpus, arm semantics, raw records, aggregator, canonical result. Adds a generator and its output. The canonical result is correct and unpromotable as printed: its unsafe-joint-state denominator is COUPLED-only, so A4 reads 0/2 (0.0%) while having produced an invalid joint state in two of sixteen scenarios — the two where the evidence was deliberately removed. That is a rendering problem, not a measurement problem, and the repair is to stop collapsing a heterogeneous corpus into one denominator. Panel 1 under the evidence available, how do the four strategies compare? COUPLED and INDEPENDENT scenarios only. Panel 2 does removing that evidence reverse the decision? Both panels read the same frozen records. Neither is a new run. Panel 1 carries A3's credibility gate inline — serialized same-target 2/2, parallelised cross-target 4/4, missed 2/2 — because without it a reader can dismiss the per-target column as a baseline that never locked. Panel 2 carries the forbidden rendering explicitly: A4 is never described as globally 0% unsafe, and the 0/2 means nothing without the 2/2 beside it. ## No hand-entered outcome values Every figure passes through figure(), which requires a derivedFrom pointer naming its location in the frozen packet, and the generator exits non-zero if any figure lacks one. A typed-in number cannot acquire a pointer, so it cannot reach the export. Panel 2's perturbed count is derived from the raw records rather than the aggregate, because the frozen denominators deliberately exclude it. ## Provenance npm integrity strings are re-captured from the registry and asserted against /^sha512-[A-Za-z0-9+/]{86}==$/ rather than eyeballed — line-wrapping through a terminal is a real corruption path for a copied digest. All three verified complete at 95 characters and identical to the registry. Until now the only committed record of them was a commit message; judge-export.json makes them an artifact. Limitations recorded: 8/8 inadmissible scenarios failed closed with no mutation; exact refusal-reason agreement 6/8; both mismatches are HISTORY_NOT_MINED against a frozen corpus expectation of HISTORY_EVIDENCE_UNAVAILABLE, where the envelope state is NOT_MINED and the core took the correct branch; same-target atomicity, exactly-once and restart behaviour are outside HAC-343. Refs HAC-343, HAC-319.
An optional judge-first pass over the existing local cockpit. Same shell, same four-stage spine, same controls in the same places; it moves emphasis and adds two things — a choice at the entry, and markers naming what the ablation held constant and what it changed. It adds no run, no arm, no value and no claim. Three things are derived rather than written down. The ablation's "four held, four changed" is a comparison of two frozen arms in lib/guide.mjs, so a marker that stops being true fails a gate instead of lying confidently. The coordination-strategy comparison binds twenty-four cells field-by-field into HAC-343 and renders the path each one came from; an unresolvable field shows `[BIND: …]` and the panel calls itself a scaffold rather than evidence. The motion budget is read out of tokens/motion.css. Emphasis is positive and non-textual. The first implementation receded non-current stages with opacity; measured, that composited readable text toward the background and compounded through every ancestor that also set it, landing labels at 1.81:1 against a 4.5:1 floor. No opacity floor fixes that, so the mechanism was replaced rather than tuned: the current stage is marked up through surface, border, an accent edge, the stage number and the connector into it, none of which participate in layout. Text hierarchy moved to two measured colour tiers. The contrast floor is now a gate — 5592 text nodes, 46 scenarios, both modes, both proof classes, 320px and reduced motion — because a floor nothing can re-measure is not a floor. Adversarial review closed four holes this change had or inherited: - judge-export.json was bound by ten judge-facing values and reproduced by nothing. The evaluation gate now rebuilds it and diffs the bytes. - The cockpit gate compared only comparison.strategies, leaving the figure that establishes the per-target lock was a real lock, its note, the panel's scope boundary and the canonical result commit unguarded. One canonical judge-facing projection is now deep-compared against the deterministic rebuild, so a field is covered the moment it is rendered. - The CI wiring assertion was a substring match a comment satisfied. lib/workflow.mjs reads the workflow structurally instead. - basisRevision was the one arm field compared view-model-to-view-model rather than against arms.json — and it is the value step 5's whole argument rests on. The HAC-343 panel also named itself with the HAC-330 run, one line above denying the association; both halves of its subtitle now come from the comparison artifact. check:packet:eval is wired into `pnpm run check` ahead of check:cockpit, with its own evaluation-gate CI job at fetch-depth 0 — the HAC-343 verifier resolves three freeze commits and a shallow checkout cannot. No frozen experiment artifact is modified. The HAC-335 captures are re-cut because the cockpit grew; the proof content in every frame is unchanged. Refs HAC-341, HAC-343
Adversarial review of c465482 found the structural workflow reader closed the comment vector and left the same defect one layer down. Two of the three assertions used an unanchored substring test against each run body, and the `Explain the failure` step documents every command the gate requires: echo "**Evidence required.** \`pnpm run check:packet:eval\` passing …" so replacing the real invocation with `echo skipped` left the assertion satisfied by its own documentation. A step could also be present and inert — `if: false` never runs, `continue-on-error: true` runs and cannot fail the job — and the reader parsed neither key. Required commands are now matched at the start of an executable line, the way the judge-export check already was: `executableLines()` drops comments and blank lines and trims the rest, so `echo "pnpm run check:packet:eval"` is prose and `pnpm run check:packet:eval` is an invocation. `enforcementStep()` returns the step that actually runs a command and `stepEnforcementDefect()` refuses it if it carries any `if:` or `continue-on-error: true`. No GitHub Actions expression is evaluated — an evidence gate should be unconditional and should propagate its failure, so any condition is refused rather than interpreted. The explanatory step keeps `if: failure()`; it reports, it does not enforce, and anchoring means it is no longer mistaken for the step that does. Five bypasses are now proved to fail, and the comment-defeat coverage stays. Also replaces the JUDGE_FACING_FIELDS self-check, which compared one literal array in verify-cockpit.mjs against another literal array in comparison.mjs and could not falsify anything. Coverage is now derived from the renderer: the comparison markup is scanned for the fields it actually touches — eleven today — and any one of them missing from the projection fails. That is the defect class the A3 credibility figure escaped through. workflow.mjs moves to bin/lib/, beside the gate that uses it. media/hac-341/lib is swept by capture-source.mjs as a *render source* set, so a gate helper there invalidated every committed cockpit capture on every edit to a file the browser never loads. The captures are byte-identical; only the render-source digest is re-baselined, and it no longer tracks gate code. No frozen experiment artifact is modified and no rendered value changes. Refs HAC-341, HAC-343
…interpret Adversarial review of 17d5581 found six more bypasses of the same seam. Three were shell-level: `if false; then`, a heredoc, and an open quoted string each place a required command at the start of a line without executing it, and a line-anchored match accepted all three. Three were control-level: a step-level `continue-on-error: ${{ true }}` evaded a literal comparison, and job-level `if: false` and `continue-on-error: true` were parsed by nothing at all — both left every step-level assertion satisfied while the job was skipped or its failure discarded, and neither the gate nor the wiring suite noticed. The mistake was trying to infer execution from an arbitrary `run:` body. That is not decidable by reading, and each heuristic that closed one spelling left the next one open. The contract is a shape instead: one enforcement operation per step, whose `run` equals the expected command exactly after trimming. The judge-export rebuild and its byte assertion are therefore two steps, and the assertion is `git diff --exit-code` against the checkout rather than a cp/diff script. Four required steps: HAC-343 packet verification, the wiring test, the judge-export rebuild, and the judge-export byte assertion. For each, `run` must equal the command, and `if`, `shell` and `working-directory` must be absent; `continue-on-error` is trusted only when absent or literally `false`. The job must carry no `if`, no `needs`, no non-literal-`false` `continue-on-error`, and must run on the expected runner at `fetch-depth: 0`. No GitHub expression is evaluated — a non-literal value is a defect, not something to interpret. The explanatory step keeps `if: failure()`; it reports and is not an enforcement step, and exact equality means it can no longer be mistaken for one. Eighteen workflow mutations are now proved to fail, covering all six new bypasses and the seven previously closed. `broken()` reports how many files its mutation actually changed and every case asserts that before asserting the refusal: an `anchor not found` throw is a broken fixture, and a broken fixture that happens to fail reads exactly like a successful rejection. Classes 1 and 3-8 are untouched. No frozen experiment artifact is modified, no rendered value changes, and the cockpit captures are unaffected. Refs HAC-341, HAC-343
Adversarial review of 05fcdf0 found three valid-YAML mutations that set a governed control somewhere the grammar did not read. Each landed, left verify-cockpit at exit 0 and the wiring suite at 42/42, and violated a guarantee the gate reports as satisfied. `defaults.run.shell` and `defaults.run.working-directory` are inherited by every `run` step in scope. A required step could therefore carry no `shell:` and no `working-directory:` of its own — every step-level assertion passing — and still run under a different shell in a different directory. Both keys are now refused at the workflow scope and at the job scope. Which one would win is precedence this grammar deliberately does not compute: for an evidence gate, any present value is a defect. Reading the first `actions/checkout` step was not enough either. A second checkout at `fetch-depth: 1`, placed after install so no test anchor moved, left the workspace shallow while the first still declared 0. The invariant is now structural — exactly one checkout step, at `fetch-depth: 0` — and a second checkout is a defect whatever depth it asks for, because which one leaves the workspace behind is again precedence this grammar does not model. Six negative proofs added, each asserting its mutation landed before asserting the refusal, and all nineteen existing cases retained: twenty-five workflow mutations now fail. Two standalone tests superseded by the matrix were removed rather than left asserting messages that no longer exist. Classes 1 and 3-8 are untouched. No frozen experiment artifact is modified, no rendered value changes, and the cockpit captures are unaffected. Refs HAC-341, HAC-343
Adversarial review of fda4842 found two workflows that satisfied every declared rule and still made the judge-export byte assertion vacuous. Ordering the assertion before the rebuild left all four operations present, exact, unconditional and failure-propagating, while the assertion compared the clean checkout against itself; interposing a step that reverted the rebuild had the same effect without touching a required step. Both passed verify-cockpit and the wiring suite. Presence-based verification cannot see either, because the defect is not in any step — it is in the sequence. The verifier now compares evaluation-gate against one canonical nine-step shape, position by position: checkout, pnpm setup, Node setup, install, packet verifier, wiring verifier, judge-export rebuild, judge-export byte assertion, failure explanation. Each position pins the expected `uses`, `name`, exact `run` and `with` entries. A sequence has no gaps to hide in, so `rebuild index + 1 === assertion index` falls out of the shape rather than being a rule of its own, and an extra step anywhere is a step the shape has no slot for — a harmless `echo` in a forbidden position fails exactly as a malicious one would, because the rule is structural rather than a blacklist. The job grammar is closed too: `if`, `needs`, `continue-on-error`, `defaults`, `strategy`, `container` and `services` are refused outright, not interpreted. `runs-on` must be ubuntu-24.04, exactly one checkout at fetch-depth 0, and the failure explanation is the only conditional step and must be last. Eight negative proofs added and all twenty-five retained: thirty-three workflow mutations now fail. One of the new cases exposed a fixture bug worth naming — `Explain the failure` appears in twelve jobs, so an anchor on that line alone edited a different job and the gate correctly passed. The anchor now reaches through the assertion step above it. `changed === 1` did not catch that, because a file genuinely did change; the fix is a more specific anchor, not a weaker assertion. Each disposable copy is now reclaimed as soon as its gate has run. Thirty-odd full repository copies held until teardown timed the afterAll hook out. Classes 1 and 3-8 are untouched. No frozen experiment artifact is modified, no rendered value changes, and the cockpit captures are unaffected. Refs HAC-341, HAC-343
The forbidden-key list was only ever as complete as the last review that extended it. `defaults`, `strategy`, `container` and `services` were each added after someone found them, and the next one was whatever nobody had thought of yet. The grammar is now closed the other way round: the canonical job declares `name`, `runs-on` and `steps`, and anything else is a defect whether or not its effect is understood. The same inversion applies per step, by position. Action steps declare `uses` and — where the shape projects one — `with`, whose inputs are exact: an unprojected input is as much a departure as a wrong one. Required run steps declare `name` and `run`, nothing more. The failure explanation declares `name`, `if` and `run` with `if: failure()`, and remains the sole conditional step and the last. Workflow-level `env` is refused for the same reason workflow-level `defaults.run` is: the steps would carry it without naming it. `timeout-minutes` is in the negative matrix precisely because it is harmless. The rule is the shape, not a judgement about what a key does. Reading unknown keys required the step parser to key on indentation rather than on a list of names it already knew — a step's own keys sit at one indent, a `with:` input deeper — so the parser now reports every key a step declares rather than the handful it was looking for. Five negative proofs added and all thirty-three retained: thirty-eight workflow mutations now fail. One of the new cases exposed the anchor-collision class again: two jobs carry the same checkout block verbatim, so a whole-file anchor edited the wrong job and the gate correctly passed a workflow whose evaluation-gate was untouched. Mutations that must land in one job now use a job-scoped edit that slices the job first, which also covers the twelve jobs sharing an `Explain the failure` step. Classes 1 and 3-8 are untouched. No evidence, rendering or capture changes. Refs HAC-341, HAC-343
The job and its nine steps could be exactly right and never run. A workflow
narrowed to `workflow_dispatch` still contains a perfectly valid evaluation
gate — every key allowlisted, every step exact, every command anchored — while
enforcement on the submission path is simply gone. That was the last level with
nothing pinned to it.
The top level is now closed the same way everything below it is: an allowlist of
keys (`name`, `on`, `permissions`, `concurrency`, `jobs`) and an exact
projection of the three blocks that decide when the gate runs and with what.
Nothing interprets an event or an expression; the lines either match the
canonical ones or they do not, so narrowing a trigger, widening `permissions` to
`contents: write`, or making `concurrency` cancel `main` each read as what they
are — a change to the execution contract.
The chain now reads end to end:
workflow trigger -> exact evaluation job -> exact ordered steps -> exact commands
Seven negative proofs added and all thirty-eight retained: forty-five workflow
mutations now fail.
Classes 1 and 3-8 are untouched. No evidence, rendering or capture changes.
Refs HAC-341, HAC-343
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
The evaluation gate failed on its first real CI run, and it was right to.
`build-judge-export.mjs` recorded `process.version` and `process.platform`
into `provenance`, so the export could only rebuild byte-identically on the
OS that first built it — darwin locally, linux on the runner. Worse than
irreproducible: the field sat beside `canonicalResultCommit` and the freeze
digests, asserting a machine that never ran the experiment. Nothing read it.
No frozen artifact records the run's environment, so there was nothing to
bind it to. The block is gone, and a source-level guard now refuses any
ambient read in the builder — the rebuild diff only catches this when the
runner's platform happens to differ, which a same-platform push would hide.
The guard scans executable code, not prose, or the comment explaining its
own absence would trip it.
Also clears SonarCloud's one failing condition, new_reliability_rating (D):
- the workflow grammar's `key: value` and list-item patterns are now one
shared pair whose groups are disjoint, so the two quantifiers cannot
compete for the same characters. All 45 bypass mutations still refuse.
- the cockpit gate scans CSS rules once, left to right, instead of a
pattern that rescanned the remainder from every failed start.
- `\s*\n?\s*` collapsed to `\s*`; `\s` already spans the newline.
- the evidence chain's `.sort()` calls state their comparator. It
reproduces default code-unit order exactly, which is the point:
`localeCompare` would make a committed result depend on the runner.
Verified: check exit 0, 631 tests, contrast 5592 nodes / 46 scenarios /
0 failures / min 5.15:1, judge export stable across rebuild.
|
This was referenced Aug 21, 2026
Closed
qmarcelle
added a commit
that referenced
this pull request
Aug 23, 2026
…ntrols (#31) * HAC-341: restore GUIDE_FREE_STATE, and make the gate see a throwing handler I removed two guide imports from the cockpit in 1296935. SonarCloud had only flagged one of them as unused; I took the two-name removal from the verifier's import, where both really were dead, and applied it to the cockpit without re-checking which names that file actually used. `GUIDE_FREE_STATE` is used — it is the whole of the exit branch — so every click on "Explore freely", "Exit to cockpit" and "Explore the complete cockpit" threw `GUIDE_FREE_STATE is not defined` and left the page exactly as it was. The more useful finding is that nothing caught it. 631 tests, the cockpit contract gate and a browser gate covering 46 scenarios all stayed green, for two reasons that are worth fixing separately: - No page ever reported its uncaught exceptions. A handler that throws leaves the DOM untouched, so a gate reading only the DOM cannot tell a dead control from a working one. `browser.newPage` is now wrapped once, so every page — including any added later — reports `pageerror`, and any uncaught exception fails the run. - The exit path was never exercised. The walk's next/back/ablate controls were clicked; the three controls that leave the walk were not, though leaving at any point is a stated requirement. `assertGuidedExit` now drives both entry points and asserts the document lands on `guide.local.free` with the bar gone and the full cockpit behind it. Both mechanisms were confirmed against the broken file: each fails it independently, and both pass once the import is restored. * HAC-343: anchor the freeze proof to tags, not to branch history Main is red, and the evidence is fine. `verify-packet.mjs` asked `git log -1 -- <contract>` whether each frozen contract was still the file its freeze commit introduced. That question is about whichever branch is checked out, and this repository squash-merges: after #29 landed, all three contracts appear to have been introduced by the squash commit, so every freeze check failed on main while nothing had been edited. The claim worth keeping is not "this commit last touched the file". It is "these bytes were frozen at that commit, before the result existed". Three assertions now carry it, none of which consult the current branch: 1. a tag resolves to the pinned freeze commit — the anchor is the right one; 2. the bytes at that commit are the bytes on disk; 3. the freeze commit is an ancestor of the canonical result, so it was frozen before the result rather than merely differing from it. `--is-ancestor` is topological, so a rewritten author date cannot fake the ordering. Tags because HAC-342 already does this, recording a `permanenceAnchor` for the same reason. They also fix a quieter problem: the freeze commits were reachable only from `hac/343-metric-manifest`, so deleting the branch after merge would have collected them and broken the byte comparison too, not just the history check. Verified against a clone with the branch deleted and gc run. `canonicalResultCommit` had the same defect and is now pinned rather than read from `git log -1 -- results.json`. It is a judge-facing value the cockpit renders; on a squashed main it rebuilt to the squash commit, so the export no longer matched its own committed bytes. That is the toolchain-field mistake again — a rendered fact that changes with where it is rebuilt. Dropping the lookup leaves the builder with no git dependency at all. Six negative proofs added, each asserting its mutation landed first: a deleted tag, a moved freeze tag, a moved result tag, an edited contract, and — the load-bearing one — a contract frozen *after* the result, with pin and tag moved together so every anchor still agrees and only the ordering can catch it. Verified: check exit 0, 637 tests, export byte-identical, packet verified on squashed history and on a checkout where the branch no longer exists.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces two new evidence files for the HAC-343 experiment—
execution-semantics.jsonandjudge-export.json—which document the frozen execution semantics and derived judge-facing results, respectively. Additionally, the CI workflow is updated to ensure full git history is available for verification steps and to add a new "evaluation gate" that enforces the integrity and reproducibility of the HAC-343 evaluation packet.The most important changes are:
CI Workflow Improvements
actions/checkoutstep in the CI workflow (.github/workflows/ci.yml) now usesfetch-depth: 0to ensure a full git history is available. This is necessary for verifying that contract freeze commits and their corresponding blobs are present, preventing failures in packet verification and wiring tests.evaluation-gatejob is added to the CI workflow. This job installs dependencies, verifies the HAC-343 evaluation packet, checks that the judge export is byte-identical to its rebuild, and provides a detailed failure explanation if any step fails. This enforces that all metrics and contracts remain reproducible and unchanged after freezing.Experiment Evidence Artifacts
experiments/hac-343/evidence/execution-semantics.json, which freezes the execution semantics for HAC-343. It details layer separation, oracle protocol, concurrency model, coordination arms, order policy, and determinism, ensuring that the experiment's execution rules are immutable and well-documented.experiments/hac-343/evidence/judge-export.json, a derived artifact presenting judge-facing results and claims. It summarizes the comparative outcomes of the four coordination strategies, evidence provenance, limitations, and the scope of the experiment, all tied to specific frozen commits and digests for reproducibility.