Skip to content

Await Arch checks and carry provable catalog curation in ledger appends - #377

Merged
MaxGhenis merged 7 commits into
mainfrom
fix/resolver-await-arch-checks
Oct 20, 2026
Merged

MaxGhenis merged 7 commits into
mainfrom
fix/resolver-await-arch-checks

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Review rounds 1 to 5 are done. Round 5, a confirmation at these heads, is APPROVE WITH NITS with no P1 or P2. Companion PR: PolicyEngine/chronicle#372, which should land first.

CI note: main's site test (forecast-catalog.test.ts, observation IDs reused by correction rows) was fixed by #378 (merged 670b069), so build-and-test can go green here.

The gap

Branch protection on Chronicle's codex/thesis-ledger-facts requires three checks: Append gate, Trusted base append gate and Arch checks. REQUIRED_APPEND_GATES named only the first two. The resolver's token is an admin and enforce_admins is off, so await_append_gates passing was the whole merge condition.

I checked all 28 resolver append PRs, 2026-07-10 to 10-19, against the check runs on their heads, bases and merge commits:

count
merged with Arch checks red on the proposal head 17
turned a green branch red 8 (#151 #156 #162 #170 #191 #303 #361 #369)
merged onto an already-red branch 9
  • Red stretches: 35 h, 28 h, 4 h, 83 h, 12 days (08-23 to 09-04), 13.5 days (09-29 to 10-12), 29 h and 8 h. That is about 30 of the last 72 days.
  • August's failures were a stale catalog, which in-proposal regeneration has since fixed.
  • Every failure after that came from two curation tests in tests/test_build_series_catalog.py:
    • the period-strip audit: a concept's first strip in a new identifier shape, or the old exact spelling list;
    • test_identity_uuid_map_matches_reviewed_anchor, which moves on every first-observation mint or placeholder enrichment.

The trade-off

(a) Await Arch checks and refuse on red. On its own, this deadlocks.

  • Curation cannot land before the rows. Chronicle reports a reviewed strip that is not yet live as an error ("reviewed strip … is gone"), and the anchor pins the post-append digest.
  • Registered windows fail closed once they close (FIRST-PRINT WINDOW MISSED (refusing): USAspending snapshots, A-19, IRS 1304, SBA).
  • So every first-of-shape row of a windowed target would be refused until its window closed, and could then never land.

(b) Keep landing and alert. This keeps data on time, but still merges past a required check and leaves the branch red until someone curates.

(c) This PR: proposals carry the curation the resolver can prove, and Arch checks is awaited.

  • Every red verdict in the census was curation that could be proven mechanically. The curator sessions in Pin that the catalog preflight judges each trial by its own rows #363 and Resolution loop failed 2026-10-18 #371 checked exactly two things: the strip is the row's own period spelled directly, and the anchor delta is same-UUID placeholder enrichments. Both checks are mechanical.
  • catalog_curation stages the base with the exact catalog and registry the proposal ships (from _prepare_release_files) and probes the base's own catalog code. It never regenerates: Chronicle mints new identities under random UUIDs, so a second generator run would describe a different catalog.
  • It records a first strip as reviewed only when all of these hold:
    • its only kind is derived;
    • its shape has one period slot;
    • only rows in this append carry the identifier it spells;
    • every such row's registration contract names the row's dataPointId and a series that the period slot directly follows. The segment is then the period by the dataPointId grammar the resolver itself parses (parse_ref_period).
  • It moves the identity anchor only by fresh-UUID mints and same-UUID placeholder enrichments of this append's own concepts. The move is recorded in the fixture, and Chronicle re-derives it (identity_anchor_problems, chronicle#372).
  • It records a shape only when every first spelling of that shape is provable, because Chronicle reviews a (concept, field, shape) as one decision.
  • It then dry-runs the curated state with the base's own audit. Curation may only remove its own reports: a report still standing on a recorded strip, a new report, or a cleared report it did not answer drops all curation rather than shipping it. Whatever is still red is reported.
  • It never raises; any failure means no curation (today's behaviour) plus a note.

What a red Arch checks does now: ARCH_CHECKS_POLICY, for Max to rule

This PR changes nothing about what lands or when. After this PR, Arch checks goes red only for curation the resolver cannot prove (an overlap first strip, a new suspect segment, an unregistered row), for an already-red base, or for a genuine test failure.

  • land (default, current behaviour): merge on the two append gates; print ARCH CHECKS FAILED AT MERGE; open or comment on a Chronicle issue titled "Ledger branch red after a resolver append", listing what the dry run expects a curator to do. The issue is also filed when Arch checks has not reported within the 20-minute grace but the dry run predicts red.
    • A failed status read while waiting on Arch checks never refuses a proposal whose required gates already passed.
  • refuse (option a for that residue): Arch checks becomes a required gate. A red verdict closes the proposal and nothing appends. Rows that need curation wait, and a windowed one can miss its window.

This choice moves data timing, so it is queued as a decision for Max rather than decided here. Flipping it is a one-constant change, and the tests already cover both settings.

Evidence: replaying the incidents through the real code

catalog_curation was run on the real Chronicle bases, with chronicle#372 overlaid, the real appended rows and the real Thesis registration contracts:

append strips the resolver writes anchor move dry-run residual human curation
#361 (base 8c4a0f86) 5 × fy2026 56bb47bc…→b06bf665…, 5 enrichments none #363: identical
#369 (base 7adcc72f) 5 × fy2026 b06bf665…→d5b88530…, 5 enrichments none #371: identical
#368 (green) none none none none needed
#364 (onto red dd929cba) none: the base's own rows are left to a curator none the 5 inherited strips + stale anchor reported, not curated

So both outages this week would have merged green, with the curation in the same commit.

Invariants (Hypothesis plus examples)

  • Merge: for any check-run listing under either policy, await_append_gates returns only when every required gate's state is passed. It raises a rejection exactly when a required gate failed, and the advisory verdict it reports is never upgraded. Under refuse, a red or missing Arch checks never merges.
  • Strip partition and soundness: each first strip is either recorded or refused with a reason, and nothing else is touched. Everything recorded meets the proof above.
  • Anchor: a sanctioned append yields sorted events that undo exactly to the base digest. Eight kinds of other binding change are refused: remint, entity change on a full identity, concept change, unadmitted geography, placeholder mint, dropped identity, foreign concept and UUID reuse.
  • Fixture writer: round-trips, is a fixpoint, matches a stdlib reference layout, and reproduces Chronicle's fixture byte for byte.
  • Through propose_ledger_append: the fixtures in the proposal commit describe exactly the catalog that commit ships, even for an append that mints under a random UUID (the round-1 P2).
  • End to end against a fake Chronicle base: both fixtures are written and the dry run is green. A rejecting audit ships nothing. An unregistered strip is left to a curator, as is an already-stale anchor. A base without fixtures is a no-op, and a probe crash is survivable. The staged code never sees GH_TOKEN or the producer signing key.

Verification

Review round 1 (in-session Opus)

  • It found no path that merges past a required gate.
  • Fixed: curation from a second generator run (P2, the random-UUID mint); shape-level sibling clearance; the claimed match; advisory read failures; issue filing on a missing verdict; catalog_curation raising; issue pagination; repeated catalog rows; a non-regular fixture entry aborting the fetch.
  • Left to thesis#337: the allowlisted environment for the existing generator runs.
  • ruff: clean on the changed lines.

Review round 2 (hard tier, served by Opus 5.5 while the Codex lanes were held): APPROVE WITH NITS

  • P2-1, append gates vs tests/fixtures/**: checked, no change needed. The base's own check_thesis_facts_append.py (7adcc72f, receipt 0.6.2) accepts a commit equal to the real chronicle#369 proposal plus both curated fixtures: thesis-facts append check OK: 250 rows, immutable prefix 128, +5 appended vs base, release 27. receipt refuses only data mixed with gate code.
  • P2-2: a stateful Hypothesis test over sequences of listings and failed reads, covering multi-page listings, required re-runs mid-wait, and grace and budget expiry. Every merge must rest on a passing latest read, whose unbroken run of passing reads holds a confirmed pass. Mutation-checked: deleting either confirmation reset fails it, or fails the new deterministic re-run example.
  • P3-6: a required gate read as anything but passed ends the run of passing reads, so a fresh confirmation and grace are needed. Once a run holds a confirmed pass, a failed or unconfirmed read during the advisory wait merges, as the code did before advisory gates existed.
  • P3-5: the dry run judges strips and anchor separately and drops only the part it rejects, then re-probes what is left. A report counts against a recorded strip if it names that strip's field and shape, or names no shape at all.
  • P3-4: the strip proof now requires, in every field, that the dataPointId segment after the registered series is the stripped spelling.
  • P3-2 and P3-3: a mint is refused when it repeats an identity already bound, or takes a UUID the base registry holds. The probe also runs Chronicle's continuity check, so the dry run predicts the CI step.
  • P3-7: before each proposal, the resolver reads Arch checks on the ledger branch head and files a red head as an issue. A red that merged without a verdict is caught on the next run.
  • P3-9: under "refuse", Arch checks counts as passed only once both its push run and its pull-request run have completed.
  • P3-10: Actions' OIDC and runtime credentials and the GITHUB_ENV/GITHUB_PATH/GITHUB_OUTPUT files join the probe's denylist. thesis#337's allowlist is still the real fix.
  • P3-11 and P3-12: the anchor mutations are parametrized, so each one must be exercised; the mint-collision and registry-revive cases are reachable; the residual names a moved anchor once. The cross-repo check stays the manual replay.
  • Local runs: tests/test_resolve_pending.py plus tests/test_ledger_catalog_curation.py, 233 passed.

Review round 3 (hard tier, Opus 5.5): APPROVE WITH NITS

  • P3-A: under "land", a green push run alone could end the advisory wait before the pull-request run had checked anchor continuity. Arch checks now needs both runs for a verdict under either policy.
  • P2-A: the stateful gate-wait property now reaches a merge in about 22% of examples, up from about 0.5%, so its oracle runs. Deterministic tests pin each mutant the review named, and all four are killed:
    • a pass confirmed across a broken run;
    • a skipped twin counted as a run;
    • "land" not merging on an earlier confirmation (P3-C);
    • the grace not restarting after a required re-run.
  • P3-D: report_red_ledger_base now has a deadline on its read.
  • P3-E: the gate replay was re-run at these heads, recording the candidate's changed paths and the gate script's hash. The check passed.
  • Nits: may_merge is now confirmed_once, and all three return paths go through it. The unreachable mint-reuse branch is removed.

Review rounds 4 and 5 (hard tier, Opus 5.5): APPROVE WITH NITS

  • Round 4: the earlier-confirmation test now reaches unconfirmed passing reads, and fails for a may_merge that needs the latest read confirmed. The stateful property uses production's min_runs, and a follow-on test requires it to reach merges under both policies. A gate held by min_runs is described as "N of M runs completed".
  • Round 5 confirmed those fixes at 8195ec8, with no P1 or P2. tests/test_resolve_pending.py plus tests/test_ledger_catalog_curation.py pass at 8195ec8 (239; output in ~/reviews/thesis-arch-gate-2026-10/pytest-8195ec86.txt).
  • Known nits, not done:
    • the merge-floor test can skip silently under pytest-xdist --dist load;
    • the floor is statistical (about a 1-in-10⁵ chance of a spurious red);
    • the served bound could be tighter;
    • a failed advisory verdict also shows "(1 of 2 runs completed)";
    • issue filing has no timeout.

A third live case

chronicle#373 (2026-10-19 19:49Z) merged past a red Arch checks: statcan CPI took its first 2026_09 strip in a new shape. Replaying it through this PR's catalog_curation (registration found, strip derived alone) records exactly that strip, with an empty residual. With this PR it would have merged green. PolicyEngine/chronicle#374 curates it by hand.

🤖 Generated with Claude Code

Branch protection on the Chronicle ledger branch requires Append gate,
Trusted base append gate and Arch checks, but the resolver's admin token
merged on the first two alone. 17 of the 28 resolver appends from
2026-07-10 to 2026-10-19 merged with Arch checks red, and 8 of them turned
a green branch red, holding every other pull request into it until a
curation PR landed (about 30 of the last 72 days). Since August every such
failure came from two curation tests: a concept's first period strip in a
new identifier shape, and the identity anchor moving. Neither can be
curated before the rows exist, so refusing the append alone would deadlock
windowed first prints.

The proposal now carries the curation the resolver can prove
(catalog_curation): a first strip that is derived alone and sits in the
dataPointId segment right after the row's registered series, and an
identity-anchor move made only of fresh-UUID mints and same-UUID
placeholder enrichments, recorded for Chronicle to re-derive
(chronicle#372). The curated state is dry-run with the base's own audit
before anything visible happens; curation the audit rejects is dropped.
Replaying chronicle#361 and #369 reproduces the human curation in #363 and
#371 exactly, with nothing left red.

The resolver also awaits Arch checks. ARCH_CHECKS_POLICY decides what a red
verdict does: "land" (default, the old behaviour) merges and files the
curation work as a Chronicle issue; "refuse" makes Arch checks a required
gate. That choice changes when rows needing uncurated work land, so it is
queued for Max.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments
Project Deployment Actions Updated
thesis-api Ignored Ignored Preview Oct 20, 2026 3:49pm UTC
thesis-forecasts Ignored Ignored Preview Oct 20, 2026 3:49pm UTC

Request Review

MaxGhenis and others added 6 commits October 19, 2026 13:35
Review round 1 (in-session Opus), each finding checked before fixing:

- P2: Chronicle mints new identities under random UUIDs, so curation
  computed in its own generator run never matched the catalog
  _prepare_release_files shipped, and every append that minted lost its
  curation. catalog_curation now stages the exact catalog and registry the
  proposal ships and never regenerates. The fake Chronicle module mints
  uuid4 like the real one; a new end-to-end test through
  propose_ledger_append checks the shipped fixtures describe the shipped
  catalog for a fresh mint.
- P3: Chronicle reviews a (concept, field, shape) as one decision, so
  recording one spelling could clear an unproven sibling. A shape is now
  recorded only when all its first strips are provable, and the dry run
  requires curation to remove only its own reports (none new, none still
  standing on a recorded strip by spelling and concept).
- P3: a failed status read while waiting on the advisory Arch checks no
  longer closes a proposal whose required gates passed.
- P3: a missing Arch verdict is filed when the dry run predicts red.
- P3: catalog_curation never raises (any failure means no curation and a
  note); a non-regular fixture entry no longer aborts the tree fetch; a
  repeated catalog row is refused; the curation issue is found across
  pages; the existing generator runs also drop credentials from their
  environment (hygiene, not isolation: same user).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
thesis#337 gives the two existing generator runs an allowlisted
environment; changing the same calls here would only collide with it. The
new curation probe keeps its own denylist until it can take that allowlist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 2 (subfleet --task review --tier hard, served by Opus 5.5 with
the Codex lanes held): APPROVE WITH NITS, no merge past a required gate.

- P2-1 (append gates and tests/fixtures): checked, no change needed. The
  base's own check_thesis_facts_append.py (7adcc72f, receipt 0.6.2) accepts
  the replayed chronicle#369 proposal carrying both curated fixtures:
  "thesis-facts append check OK: 250 rows ... +5 appended vs base,
  release 27". receipt refuses only data mixed with gate code.
- P2-2: a stateful Hypothesis test over sequences of listings and failed
  reads (multi-page listings, required re-runs, grace and budget expiry):
  every return rests on a passing latest read whose unbroken run of
  passing reads holds a confirmed pass. Deleting either confirmation reset
  fails it or the new deterministic re-run example.
- P3-6: a required gate read as anything but passed now ends the current
  run of passing reads (a fresh confirmation and grace are needed); once a
  run holds a confirmed pass, a failed or unconfirmed read during the
  advisory wait merges as the code before advisory gates would have.
- P3-5: the dry run judges strips and anchor apart and drops only the
  rejected part (re-probing what is left); reports belong to a recorded
  strip by field and shape, and a report naming no shape counts as its.
- P3-4: the strip proof requires the dataPointId segment after the
  registered series to be the stripped spelling for every field.
- P3-2/P3-3: identity_anchor_move refuses one identity under two UUIDs and
  a mint of a UUID the base registry holds; the probe also runs Chronicle's
  continuity check, so the dry run predicts what CI checks.
- P3-7: report_red_ledger_base reads Arch checks on the ledger branch head
  before each proposal and files a red head, so a red that merged without a
  verdict is caught on the next run (never holds the append).
- P3-10: Actions OIDC/runtime credentials and the GITHUB_ENV/PATH/OUTPUT
  files join the probe's denylist (thesis#337's allowlist still to come).
- P3-11/P3-12: anchor mutations parametrized (each must be exercised),
  reachable mint-collision and registry-revive cases, a residual that names
  a moved anchor once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 2, P3-9: Arch checks runs once for the proposal branch's push
and once for its pull request. The push run can finish green before the
pull request run is even listed, and with chronicle#372 only the pull
request run checks the anchor move against the base for a new branch. So
when Arch checks is a required gate, await_append_gates now holds its pass
until two runs have completed (min_runs). The gate fake carries both runs,
as a real proposal does, and the stateful gate-wait property models the
rule under "refuse".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 3 (hard tier, Opus 5.5): APPROVE WITH NITS.

- P3-A: chronicle#372 leaves a new branch's anchor continuity to the pull
  request run, so under "land" a green push run alone could end the
  advisory wait and the merge would go ahead with continuity unchecked.
  Arch checks now needs both runs for a verdict under either policy.
- P2-A: the stateful gate-wait property almost never reached a return
  (about 0.5% of listings passed every required gate). Its listings now
  pass about half the time, Arch checks takes the shapes the twin rule
  must tell apart, and 22% of examples end in a merge the oracle checks.
  Deterministic tests pin each mutant the review named: a pass confirmed
  across a broken run, a skipped twin counted as a run, "land" not
  merging on an earlier confirmation (P3-C), and the grace not restarting
  after a required re-run. All four mutants are killed.
- P3-D: report_red_ledger_base bounds its read.
- Nits: may_merge is confirmed_once (one flag, three return paths); the
  unreachable mint-reuse branch is gone.
- P3-E: the gate replay was re-run at these heads with the candidate's
  changed paths and the gate script's identity recorded
  (~/reviews/thesis-arch-gate-2026-10/gate-replay-369.out): check OK.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 4 (hard tier, Opus 5.5): APPROVE WITH NITS, no P1/P2.

- P3-1: the P3-C test returned on its confirmed read before serving any
  unconfirmed pass. It now serves a confirmed pass, then passing reads
  with a new attempt id each time, and ends on a failed read or on the
  deadline; a may_merge that needs the latest read confirmed fails both.
- P3-2: the stateful gate-wait property runs with production's min_runs
  under both policies, and a following test requires a share of its
  examples to end in merges under each policy.
- P3-3: test_resolve_pending.py passes at this head (239 with the
  curation tests).
- Nits: a gate held by min_runs is described as "N of M runs completed";
  the torn-read advisory verdict is commented; a dead duplicate check in
  identity_anchor_move is gone. Both incident replays still match
  #363/#371 with nothing left red, and the base gate accepts the #369
  replay (~/reviews/thesis-arch-gate-2026-10/gate-replay-369.out).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis marked this pull request as ready for review October 20, 2026 15:56
@MaxGhenis
MaxGhenis merged commit 86a6397 into main Oct 20, 2026
11 checks passed
@MaxGhenis
MaxGhenis deleted the fix/resolver-await-arch-checks branch October 20, 2026 18:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant