Repository navigation
Conversation
scripts/household_set.py samples households from the dataset the installed policyengine.py bundle certifies, renames them mc_000 onward, computes every output on the installed policyengine-us with the reference conventions, and writes a board in the frozen layout with batches. On the frozen board's own households it reproduces all 1,935 scored 2.41.0 references exactly. The tool pilot takes --board: the engine and fix pins come from the board's last engine_upgrade revision, and a household set's arms are scored against the same model's no-tools run (--baseline-dir). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
scripts/batch_judge.py writes one arm's answers on a batch as a dashboard payload (each model a board model), so consensus-flags and the two-stage adversary run unchanged: two of three models agreeing off the reference flag a cell. The payload carries what the models saw: the v2-pilot household prompt (checked against each record's sent prompt hash), the v2-pilot definitions, and the condition they answered under. Derivations are pruned engine traces on the household set's own reference system. The adversary takes a payload's outputDefinitions and answerCondition; the default renders both stage headers byte for byte as before. Dry run on today's frozen-board no-tools answers (v2 wording): 23 of 1,935 scored cells flagged, 12 of them among the 2026-10-05 adversary's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Max, 2026-10-22: "in case youre running any claude api stuff please use the key on my clipboard". With JUDGE_API_KEY_SECRET naming an agent-secret entry, the judges bill that key and nothing else: it refuses a lane config, a lane token, AUDIT_ACCOUNT or the desktop opt-in alongside it; every call runs in a fresh empty config directory removed at exit; the key joins only the claude calls' allowlisted environment; and check-login requires a first-party api_key login from ANTHROPIC_API_KEY in that directory. Sidecars record the mode and the secret's name, never the key. The lane and desktop paths are unchanged. Why now: the desktop login's CLI session had expired, and the Subfleet lanes were closed, so the batch judges had no login to run on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The adversary prompt tells a judge to exclude policybench.org, policyengine.org and GitHub on every search. Codex's search tool takes no blocked_domains, so GPT-6.1 Sol did it with -site: operators, and the audit rejected those stage-1 outputs as searches naming PolicyEngine (rehearsal, 2026-10-22: 2 of 3). searched_for() drops a -site: exclusion of a blocked domain before the contamination check, in both runners' audits. A query that still names the engine or the benchmark (beside exclusions, under site:, -inurl:, or a look-alike domain) is still rejected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Max, 2026-10-23: "we can still test before that but i'd like v2 to be clean". household_set.py build takes --purpose v2 (the default) or test. A v2 set needs the installed policyengine.py bundle to certify its own dataset for the installed engine; anything else is refused unless --purpose test. A test set may name a published dataset (--dataset-repo/--dataset-file/ --dataset-revision); it records the release manifest's build engine and compatibility claim, purpose test and certified_pair false. Engines after policyengine-us 2.2.1 ignore Populace's filing_status_input and derive filing status from the household; the build records each sampled household whose stated status differs from the engine's (its scenario and reference carry the engine's). The sampler takes the dataset as an option; the default call is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Max, 2026-10-23, on what test batch 0 showed about the calculator ("yes go"):
- Names: a name written with spaces (as the prompt labels its facts) or a
benchmark output key (free_school_meals_eligible, head_medicaid_eligible)
is read as its engine variable, and the result's notes say how. Every model
had a call refused for a guessed name in all 10 households.
- Placement: an input set on another entity moves to the one unit that holds
the people it was set for (a group left out becomes one unit of everyone, as
the engine reads it); otherwise it stays an error.
- Overrides: a contribution set as the realized amount is read as its
_desired input (Sol and Opus put "traditional 401k contributions desired"
into traditional_401k_contributions in 6 households, moving tax), and an
input the engine computes and no stated fact sets is flagged as replacing
its formula (Sol set rent, not pre_subsidy_rent).
- Explanations: explain (up to 3 variables) returns the computation steps and
the parameters read, with values and cited sources. Opus overrode a correct
free-meals answer in Louisiana, which pays the reduced-price copay.
Tool version 1 (pe, web+pe) is byte for byte as before. The notices and tool
description now state the board's engine (a household set on 2.60.4 had been
told 2.41.0); the frozen board's text is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Max, 2026-10-23: "thinking we might want to do all these tests with a cheap model until we're super ready for v2 with the web and tool experiments". batch_judge.py payload takes --models (the frontier three by default). The payload's household prompt is the first tool-contract model's, checked against that model's records, and the payload records which models it holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The calculator's six features (names, placement, overrides, explain, concise, flat) are each on or off, so a study can change one at a time. "pe2" keeps tool version 2 (the first four, its notice byte for byte), "pe3" offers all six, and any other set runs as "pe:" plus its features, e.g. "pe:names+flat". - concise: an explanation gives the value, its direct inputs and up to five parameters unless `detail` is "full". - flat: the household is described as a state, people with a role (head, spouse or dependent) and facts, and household facts; the tool builds the units the benchmark's own situations have and places each household fact on its entity. Tests: the v2 notice is reproduced from its features; each feature adds only its own behavior; flat builds the benchmark's units and refuses what it cannot place; and on the installed engine eight frozen-board households described flat compute every board output exactly as Scenario.to_pe_household does. Five mutants of the flat and arm logic are all caught. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prompts label some facts in words rather than as the variable name with spaces: employment_income is "gross wages and salaries", hours_worked_last_week "usual weekly hours worked", has_esi "has employer-sponsored insurance" (policybench.prompts.INPUT_LABEL_OVERRIDES). In the 20-household test set nearly every household has at least one; hours appear in 17, employer coverage and premiums in 11 each. Tool version 2's names feature cannot read these, and its notice says the facts carry the variable names. A Flash-Lite smoke test of the flat format met three such errors and fell back to setting employer_sponsored_insurance_premiums to 0 and weekly_hours_worked, which the SNAP work rules do not read. The new labels feature reads a fact under its prompt label (case and punctuation aside, renames applied) and, as the reference situations do, also sets stated usual weekly hours as weekly_hours_worked_before_lsr (PE_INPUT_ALIASES). pe3 now offers all seven features. pe2 is unchanged and keeps its notice as it ran; every other feature set's notice says only what the tool reads. Tests: label reading, the alias (and leaving a set value alone), pe2 unchanged, the variable-name claim only in pe2's notice across every feature set, every prompt label reading back to its engine name, and a differential: twelve frozen-board households described flat with each fact as its prompt writes it compute every board output as the benchmark does. Three mutants (no alias, no renames, strict label matching) are caught. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The v2-pilot prompt relabels two facts itself ("home energy assistance
received" for spm_unit_energy_subsidy, "employer-paid health insurance
premiums ..." for employer_sponsored_insurance_premiums); the labels
feature now reads them, derived from V2_PILOT_CONVENTIONS so the two
stay in step. In the flat format a person given no name is named by
role (head, spouse, dependent_1, ...): Flash-Lite's first flat call in
two smoke households left the names out.
The label differential now writes each fact as the v2-pilot prompt
does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two of three Flash-Lite smoke households gave the head's rent as a household fact. pre_subsidy_rent is a person variable in policyengine-us, so the flat format refused it. Under the placement feature a person fact given for a household of one now goes to that person, as placement moves a person variable off a one-person unit; with two or more people it is still refused, now saying to give it under the person whose facts list it. The test registry's entities now match the engine's (pre_subsidy_rent is a person variable). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A differential against 5bbd571 (the commit the test batches ran) found version 2 requests carrying detail "full"; only execute_request reads it, through .get, so results were unchanged, but the request now matches exactly. 4,000 generated calls through versions 1 and 2, the notices, descriptions, schemas and 15 model profiles are identical to 5bbd571. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
11 of 12 tasks
GPT-6.1 Sol's review (REQUEST_CHANGES, 7 findings) at 9c43f4f: 1. A model's arguments could turn on features its arm does not offer: the worker read "__tool_features__" (and, as at 5bbd571, "__tool_version__") from the arguments dict. A pool now sends features in a tuple frame no JSON can be, the worker validates them (names, concise beside explain), and keys like "__tool_version__" in a model's arguments are refused as unknown arguments. No run record ever carried them. 2. A resume could mix treatments. Each household record now keeps its tool features and calculator schema hash; a resume compares features, notice and schema, and refuses a non-legacy record without them. 3. Labels missed what the prompt renders. The label index is now built from the prompt's own renderer (_format_input_line, then v2_pilot_text) for every promptable, labeled and reference-stated input: fallback labels, the Marketplace plan line, "state", partnership or S-corp income. Facts the reference drops (PE_DROPPED_INPUTS) are ignored with a note. A label that names two of the engine's inputs is not read. 4. Flat refuses person keys other than name, role and facts. 5. Flat stores household facts under the name written, so normalization compares them once normalized; roles set the relationship inputs and a stated one that contradicts its role is refused. 6. The flat example uses employment_income; placement replaces the wrong-entity error sentence; "full" detail is the trace, capped. 7. Diagnostics read flat people by their default names. Evidence: 210 passed, 4 frozen-engine skips. A labels-only differential builds 30 frozen households from their prompts' facts and matches the reference situation on every output, and so do all 100 frozen and all 20 test-set households in a one-off run, with or without take-up inputs. pe and pe2 are identical to 5bbd571 over 4,000 calls and 2,000 pool messages. 11 mutants of the fixes are caught. An end-to-end Flash-Lite pe3 smoke run had 0 tool errors on 3 households. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- flat: a household fact repeated with a malformed value was dropped without a word. _same_value now reports the repeat's validation problems, and only a repeat validated as equal is dropped quietly. A malformed first value is reported once, by normal validation, and never as a conflict. - The fallback record for a household the harness failed on carried no treatment, so one failure blocked resuming any custom arm. The record is built by harness_error_record, which now includes the notice hash and profile_treatment. resume_treatment_problems checks only the records a resume keeps, since a rerunnable record is replaced under the current profile. pe, web+pe and pe2 still behave exactly as at 5bbd571: 4,000 calls and 2,000 pool messages match. 5 mutants of these fixes were all caught. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…core A rerun is not sure to replace a rerunnable record: a budget stop, lost workers or a narrower --scenarios leave it in place, and scoring read whatever was there. treatment_problems now checks every record in the folder, rerunnable ones included, and command_score refuses a folder that mixes treatments, rebuilding the notices on the board's engine. Runs written before 6e1578f (test batch 0's frontier calculator arms, b6d835a) were told policyengine-us 2.41.0 while computing on 2.60.4. A folder written wholly so is one treatment and scores with a note; a mix of the two texts is refused, and a resume into it stays refused, since its new records would state the board's engine. All 39 run folders score (628 records); the six frontier calculator folders take the note. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
) Reviewed head 6b00a6f (GPT-6.1 Sol r4 AGREE); full suite at that head: 218 passed, 4 skipped, ruff clean.
… ceiling scripts/confirmatory_run.py runs the frozen pre-registration's run procedure on the treatment harness (6b00a6f's tool_pilot_argon.py, unchanged) and writes the run records the confirmatory report checks: - two lanes per model, households alternating in draw order, each household's arms in its drawn order, then reruns in passes; - one ToolPool per model with no features of its own, each call framed with its arm's features; - admission against each model's cap and reserve, waiting while a pass's reruns are undecided; - the hard ceiling over every model's spend, through the harness's Spend; - the failure classes and actions, with operator pauses, harness and inconclusive stops, and replaced attempts saved before their reruns; - the 30-second host log and the guard's and power log's lines. Tests follow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
30 of 33 tasks
Unit tests for the streams, arm strings and features, the classes and actions, the host evidence and the ceiling. Fake runs (a fake provider, pool and host log, on a fake board, with the windows scaled down) plant reruns, host trouble, an operator pause, a harness stop, the inconclusive stop, a refusal, the hard ceiling, a raising attempt, pool restarts and an interrupt. run_problems recomputes every rule of the run procedure from the files the runner wrote, and a property test checks it over random failures and spend. Two tests run the harness's own conversation under the runner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Record signals and wake a pipe watcher with a handler that takes no lock and raises nothing. The watcher sets interruption under admission and announces admitted work after releasing the lock. Archive moves and attempts commit under the same lock before interruption; admitted attempts finish and save outside it. Track worker began and done events independently of Thread identity and join bookkeeping. Escaping exceptions retain handlers and the invocation lock through completion, allow ten seconds for uncertain startup, tolerate repeated wait exceptions and preserve the original exception. Late targets return before file access. Apply completion tracking and safe late entry to the watcher as well. A caught model exception marks that model error and leaves global interruption unset. A second signal exits with 128 plus its number and preserves saved finals, archives and their recorded spend. Every unsaved attempt then under way is lost with its spend. A signal before handler restoration requires graceful exit 1 even when all workers completed. Run signal and cleanup regressions in bounded subprocesses, synchronize through events and use a thirty-second forced-exit deadline. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pipe Until the handlers are restored, a signal still writes to the watcher pipe; closing it first could let that write land in a file that reuses the descriptor. Note that an exception another handler raises before the exceptional wait begins cannot be shielded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The admission probe's control thread sent the second SIGINT and then at once raised "forced exit returned". The handler runs on the main thread, so on a loaded host the control thread could reach its own os._exit(99) first; one of three suite runs at 6597adb failed that way, with the forced message printed after the control thread's traceback. The thread now sleeps 20 s, inside the probe's 30 s execution deadline, before it reports a forced exit that never came. Test only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Exercise real signals after descriptor reuse and watcher shutdown. Require both handlers to restore before either descriptor closes. Pin late first SIGINT to exit 1 with completed finals, an unset interrupt and no announcement. Exercise watcher start failures before and after native startup and target entry beyond startup grace. Require the original start error to propagate, both handlers and descriptors to be released, and delayed targets to avoid closed descriptors and finish. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Describe the handler recording a signal and waking the watcher, with allocation failure as its exception limit. Describe the watcher setting interruption while running and a first signal recorded after shutdown forcing exit 1 without setting interruption or announcing it. State that guarded waits retry exceptions raised within them. Explain that another handler raising in an unguarded instruction before a guarded call, in cleanup bookkeeping or in finally can replace the original exception and skip waits. Keep executable behavior unchanged. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Catch recursion failures while reading saved finals, archived attempts, baseline exclusions and the CLI design so deeply nested JSON becomes a refusal naming the file. Include the baseline path in its validation errors, and require the CLI design to be a JSON object before stage initialization. Valid stage decisions stay the same. Saved records and baseline exclusions must refuse decoder recursion failures with a path-bearing StateError. The CLI must refuse unreadable, undecodable and non-object designs with their path and exit 1 before initializing a stage. Invalid UTF-8 must remain a decoding refusal. Synthetic regressions exercise every reader in a subprocess with a timeout and require the path before provider calls; final and archive cases also check preflight. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Check board inputs as strict UTF-8 and required JSON objects before delegating to the pinned harness. Attribute metadata engine checks, CSV parsing, exclusion validation and scenario reconstruction failures to their source files. Preserve absent optional exclusions and return the harness's loaded board for usable inputs. Saved finals and archived attempts must refuse numeric cost overflow with a path-bearing StateError at startup and preflight. Nested fixtures must control the child's recursion limit, derive a deeper payload and require the decoder's RecursionError as the refusal's cause. Allocation exhaustion continues to propagate. Provider-free regressions cover each board decoder, both cost boundaries and valid board results. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Give the CLI option test a real small board so it exercises the runner's input boundary and unchanged harness loader. Reference agreement failures must name both the reference CSV and exclusions JSON because either input can make that validation unusable. Preserve the checkpoint branch and its existing commits. The delivery branch retains two commits while including this fixture update and the paired reference paths. The CLI dry run must parse all model options and name its owed work before provider calls. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hash the bytes used by every board input check and guard the unchanged harness loader. Exclusions existence failures must name the exclusions file. Delegated I/O failures must name their filename or the board folder. Other delegated failures must name files whose checked bytes changed, or the board folder when no change is established, while preserving the original cause. Saved finals and archives must contain an object totals with a present numeric cost_usd. Booleans and other nonnumbers are refused. Costs must convert to finite nonnegative floats. Startup and calculator preflight use this reader; calculator attempt_cost and valid spend arithmetic stay unchanged. Provider-free regressions record 65 failures and 49 passes against 6928b06 before these fixes; final verification follows. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Check every saved final and archive against the pinned none-arm treatment before providers or calculator admission. Model, arm, prompt version, wording, notice, explicit null calculator fields, board outputs and provider turns must match. A completed reply or nonnull answer requires its contract, sent prompt and rebuilt hashes. Empty failed attempts may omit that evidence; any evidence present must match. Cheap finals must be in the draw and retain a predictions object. Preflight loads the checked board lazily to apply these same rules. Final link numbers must be consecutive integers excluding booleans. Archived prefixes retain the report equality rule. Known link fields must match while extra keys and an absent status matching null remain permitted, and earlier prefixes are preserved. Archive classification failures must name the archive. Valid numeric spend, rerun eligibility, interrupted-archive recovery, locking and worker execution keep their existing rules. Keep the minimal state-machine fixtures explicit about their injected treatment validator. Add real-board fixtures with harness-built evidence, conditional failure controls and exact archive/spend resume checks. Record 164 agreement failures and 32 controls against the original reader before these fixes. The final selection passes 315 cases, including legacy assertions updated for the new refusal reasons. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s accept 0300257 checks each saved no-tools record against the pinned none arm, and a cheap model's final must hold a predictions object. complete_preflight_design still wrote bare finals, so every preflight refused each model's first household and five tests never reached their planted fault. The fixture now builds each final through the harness with the records tests' full_record and writes the board files the no-tools loader checks. The harness and ceiling faults use the form a failed call saves: no answer and no prompt evidence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Record provider-free regressions for denied history enumeration, saved-evidence metadata failures, hash and decoder buffer disagreement, and board engine failures through calculator preflight and main. Each unusable file or folder must produce a refusal naming its path before providers. Baseline and eligible-pool hashes must bind the bytes decoded, and main must print all collected problems and return 1. The original production code fails all 38 regression cases. Two controls establish that runner startup, its stage check, and the separate report accept object-form variables with the same saved spend. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Enumerate saved histories and calculator evidence explicitly. Missing optional folders retain empty inventories; covered listing and stat failures must name their paths. Preserve sorted JSON histories, saved spend and attempt decisions. Guard exclusions metadata, model discovery, archive overwrite checks, summaries, calculator records and host logs, and the preregistration hash read. Read baseline exclusions and the eligible pool once each. Their hashes must bind the exact buffer decoded with strict UTF-8. Guard both calculator engine reads with the metadata sidecar path. Preflight retains its collected problems, and main prints every problem and returns 1. Valid board engine checks and provider execution retain their existing paths. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Explicitly list each candidate calculator stage before checking its literal households child. A covered failure listing that parent must become a preflight problem naming the stage folder. Preserve the same household path, record filter and early refusal once saved calculator records are found. The final regression run exposed this missing parent listing after the first implementation: 39 cases passed and the denied-stage-listing case still failed. Keep the original failing regression and verify the corrected inventory and preflight paths. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The three test modules added for this round take descriptive names: test_confirmatory_unreadable_evidence.py holds the folder, listing and stat refusals; test_confirmatory_input_reads.py holds the single-buffer hash and decode checks for the baseline exclusions and the eligible pool, and the saved-variables control shared with the report; test_confirmatory_board_engine.py holds the board-engine refusals through preflight and main. The control's imported report module drops the round number from its name too. No test or production code changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… per record Runner review r16 found two gaps. A no-tools model folder, a calculator arm or a calculator stage that a listing returned was dropped when its stat failed with ENOENT: a dangling link (an optional model's folder on an unmounted volume) or an entry removed after the listing. The report refuses such an entry. listed_is_dir now refuses every stat failure of a listed entry, at startup, in stage_problems and in the preflight. A missing root, households folder or attempts folder, which the code names itself, still reads as empty. history decoded each saved record with one read and hashed it with another, and Stage.archive hashed the runner's final again before the move. read_record now returns the digest of the bytes it decoded; an archive's link carries that digest, and the history keeps each final's digest (the bytes read at startup, or the bytes write_json wrote). Before the move, archive refuses a final whose bytes no longer hash to it. history now lists a model's finals before its attempts folder, as the report does, so a households folder that cannot be listed or searched is the folder both programs name (report review r15 finding 4). New tests: a dangling or vanished model entry (startup and preflight), a dangling or vanished calculator arm or stage (preflight), a file at or a stat failure on an optional model's households or attempts folder (startup and preflight), a second read of an archive or a final that differs from the hashed bytes, an archive link equal to the digest its final was read from, and finals changed after the runner read or wrote them. The board engine test now requires the preflight guard's own line and every saved-treatment line beside it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In suites r10, three of 0ae6df5's new cases failed in all three runs: - test_archive_refuses_a_final_changed_after_the_runner_wrote_it: Stage.archive created the attempts folder before it checked the target and the final's digest, so a refused move left an empty attempts folder in a history that had none. Both refusals now come first, and the folder exists only when the move proceeds. - test_saved_record_hash_and_decoder_share_one_buffer[archive] and [final]: the test hashed the files itself after installing the read counter, so its own read counted as a second runner read. It now hashes them before counting. The runner read each record once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… files
Review runner-r17 (sol61, REQUEST_CHANGES) found two gaps between the
runner and the report and three smaller items. This commit fixes each.
- The calculator preflight now refuses what the report refuses.
stage_problems refuses any model's archive that no final lists, and
any final whose action is rerun or pause-rerun. That covers optional
models, and cheap-model archives outside the draw. Startup recovery
still resumes both states.
- Saved records now go through the report's descriptor guard.
read_regular opens with O_NONBLOCK and refuses a descriptor that is
not a regular file, then reads through that descriptor. Before this, a
FIFO with no writer could hang startup while it held the invocation
lock. The archive move's digest checks use the same reader.
- The archive move hashes the moved bytes again after the rename. If a
write landed between the first hash and the rename, it moves the bytes
back and refuses. The docstring states the limit that remains: the
runner cannot see a write to the same inode during either hash or
after the second. Every later reader hashes each archive against its
link.
- New tests:
- a FIFO or device in place of a final or an archive, at startup and
preflight, bounded by a writer that releases a blocked open after
5 s;
- unfinished optional and outside-draw histories;
- a write between the digest check and the rename;
- a target collision that keeps both files;
- a digest refusal that leaves no attempts folder, and the missing
folder after a target-stat refusal;
- the changed-written-final case across every cheap model, which pins
the other models' continuation.
- The optional no-reply control now expects the preflight to refuse its
final, which owes a rerun.
- The read-counting helper also counts read_regular. It patches that
reader only where it exists, so the same cases run against c2ca2dc.
Against c2ca2dc, 14 of the new cases fail: 8 special-file, 1 move-back
and 5 unfinished-history. The collision, folder and continuation cases
pin existing behaviour.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI runs `uv run ruff check .` and `ruff format --check .` with the locked ruff 0.15.2. CI has not run on this draft. Run locally on c2ca2dc's tree, both fail on files this PR adds: four files needed formatting, and four lines ran past 88 characters. This commit formats those four files and rewraps the one string the formatter could not split. Every changed file parses to the same AST as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A mutant check of 274e1c7 found one fault that the tests missed. Its digest read in Stage.archive used the blocking sha256_file. The new case replaces a prepared final with a FIFO after startup. The archive move must refuse it at once by name, within the 5 s bound, and leave no attempts folder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he abandon suites-r12's first runner pass at 0193811 failed test_an_attempt_past_its_deadline_is_abandoned_and_stops_the_model with 0.02 == 0.01, and the next two passes held 730 passed. The test gave each turn 0.2 s against a 0.3 s deadline, so its second turn spent at 0.4 s, and it expected the waiting thread to abandon the attempt inside that 0.1 s. A wake 0.1 s late put the second turn's cost in the attempt's record, which is the spend so far that the runner's rule asks for. The runner was right and the test's expectation assumed a wake time. A plugin that delays the abandon by 0.25 s reproduces the same assertion at the same line. The first turn now spends at once and the second waits for the abandon itself, through a new `hold` key in FakeProvider. The deadline rises to 1 s and the other model's attempts to 0.2 s, which leaves 0.8 s of room against that model reaching its own deadline. The assertions are unchanged. The test passed 25 of 25 runs alone and 25 of 25 with the 0.25 s delay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eline rules Answers review runner-r18. stage_problems dropped the cost history returned and never compared it with spent_before, and the preflight never read the baseline. Calculator spending could therefore start from an understated ceiling, or on a baseline the report refuses, and the report would then withhold the results. stage_problems now sums every model's validated cost and refuses a spent_before below that sum by more than MONEY_TOLERANCE, the report's value. It refuses an unusable value by the report's money rule: finite and nonnegative, with numeric text and booleans counting as numbers there too. check_baseline is the startup check moved unchanged, so startup and the preflight share one rule. Tests: a full preflight against spend in a final, an archive and an optional model; each unusable spent_before; and each baseline fault. The second hash after a rename gets its own FIFO case, because the earlier case attacks the first hash. Two execution cases fail the second hash and the move back, then check that the other models finish, the lock is released and startup recovers the archive. The deadline test now records the join's timeout: its hold on the second turn hides when the abandon came, so a half deadline passed it. The ceiling test waits for the crossing where it slept 0.3 s against an unbounded competitor. The archive docstring now limits later digest checks to surviving links: after an interrupt, startup rebuilds the last archive's digest from the bytes then present. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Answers review runner-r19 and item 3 of confirmatory-report-r18. The preflight added each cost to a running total, and the report grouped its total by model, so the two programs rounded differently. A spent_before at the tolerance passed one and failed the other, and $2 of saved spend could round away beside a $1e16 cost. history now lists each saved cost, and exact_spend takes their correctly rounded sum, the same in any order and grouping. The report takes the same sum. An intermediate overflow means a total past every float, which the comparison refuses. Startup's spend uses the same sum. Tests: every order of two sets of costs at the tolerance, costs a running total rounds away, and a total past every float; a spent_before given as numeric text or a boolean, which counts as its number in both programs; and a design without spent_before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review runner-r20 found that restoring a running total at startup alone would pass every test. A new case saves costs whose running total stays under the cap and whose exact sum reaches it, and asserts that no owed attempt starts and that the invocation lock comes free. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s release Review runner-r21 found two gaps in the startup test. Rounding each model's costs first escaped it, so a second test saves costs whose subtotals fall one float under the cap and whose exact sum reaches it. The test's lock assertion read bookkeeping, so both tests now take the lock from a second open. `exact_spend`'s docstring limits the claim to a flat list of costs, in any order, and to histories that pass validation. Production code changes only in that docstring. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review runner-r22 found that startup's overflow rule had no test. Replacing `exact_spend(costs)` with `math.fsum(costs)` at startup alone would have raised a raw OverflowError and passed every test, because the existing overflow cases reach the preflight or refuse one invalid cost. A new test saves two valid 1.5e308 finals and requires that startup takes infinity for the total, stops every owed attempt and frees the lock. The helper takes the expected total as an argument. Production code doesn't change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI's test job failed 22 confirmatory tests on Linux at 3846666; they had passed on the Mac that ran every earlier suite. Fourteen reached a preflight check that compares the installed policyengine-us with the board's engine, and the lockfile installs 2.41.0 where the fixture board names 2.60.4. Eight reached host_power, which calls macOS's pmset. World already fakes the board's engine, so it now fakes the installed version and the power reading too. Its tests no longer read either from the host. A test that wants a different reading still sets it after World, as the lid-closed test does. The eight stubs in the unreadable-evidence tests and the ones in pool_design and engine_fault repeat World's and go. Production code does not change. A pytest plugin that hides pmset and reports 2.41.0 reproduced CI's 22 failures on this Mac at 3846666, by name. With this commit all 772 runner tests pass under that plugin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
World now reports the installed engine and the power reading, so no test could see the checks that read them, and no earlier test had held those checks with the real values either. An installed engine other than the board's must now refuse in the preflight twice, in its own check and through the no-tools treatment check, and again in main's final check, which must keep the preflight's problems. Each test requires that no provider is called. host_power gets its first test, on the four mixes of power source and lid state, with the text pmset and ioreg print on this Mac. Production code does not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP. This builds PolicyBench v2's head-to-head on fresh households from the latest Microcosm build, run in small batches with AI judges between them. Max's direction (2026-10-22):
In this PR so far
scripts/household_set.pybuilds a household set in the frozen board's layout:mc_000onward.latest_final.py, pins checked).The set stays outside the repository until a release, so a web arm can't find its answers.
Check of the reference path.
check-frozenruns the builder on the frozen board's own 100 households with policyengine-us 2.41.0. It reproduces all 1,935 scored references within 1e-6.tool_pilot_argon.py --board DIR:engine_upgraderevision.--baseline-dir), which must use the same prompt version.Progress (2026-10-22)
Batch judge. The batch judge, item 1 of the plan below, is built:
scripts/batch_judge.py: payload and derivations.outputDefinitionsandanswerCondition.JUDGE_API_KEY_SECRET), in a fresh config directory, refused alongside any login.Dry run. On the frozen board's no-tools v2 answers it flagged 23 of 1,935 cells at 2 of 3 models.
Rehearsal (two frozen households, 2.41.0): one full batch ran end to end, covering 12 model-arm runs, scoring, the trigger, the blind two-stage judges (6 valid verdicts) and collection.
Engine upgrade, frozen board on 2.58.0 with latest_final: 6 of 1,984 outputs move.
Judges (Max: "should we use gpt as judge for this?"):
2026-10-23:
v2 waits for a clean 2.60.x bundle (Max).
household_set.py build --purpose v2refuses anything but a certified pair. The Microcosm recertification is tracked in Recertify a published US release for a newer policyengine-us without a rebuild microcosm#1366.Test sets (
--purpose test) may name a published dataset. Test batch 0 ran 10 households from the Route A build 0fb05b6 on 2.60.4, for $17.06.Calculator tool version 2 (the
pe2arm, 6e1578f)._desiredand flags inputs that replace a formula.Tool version 1 is unchanged.
Update (2026-10-23 evening). The interface tests run on cheap models (Haiku 5.5, GPT-6 Luna, Gemini 3.1 Flash-Lite) until v2 is ready. Cheap batch 0 is done; batch 1 waits for a host reservation. Tool v3 (independent calculator features for an ablation pilot, and a finding that the prompts label some facts in words that tool v2 cannot read) is in Tool v3: calculator features one at a time, for the ablation pilot #243, stacked on this branch. It merges here after cheap batch 1, which runs from this branch at 5bbd571.
Plan (to update as it lands)
consensus-flagswith a three-model threshold: 2 of 3 models agreeing on the same wrong answer.adversary-preparewith the v2-pilot prompt and definitions the models saw.adversary-collect.Invariants
referencesrecomputes a set and must match it exactly.check-frozen, and a slow test on three households).🤖 Generated with Claude Code