Move the engine's random draws into the data build #2874

Activity

1. MaxGhenis commented on Nov 5, 2026

   

   MaxGhenis

   on Nov 5, 2026

   CollaboratorAuthor

   More actions

   Picking this up for the UK hub. Reply here if you're mid-change and I'll stop.

   ---

   Generated by Claude Code
2. MaxGhenis commented on Nov 5, 2026

   

   MaxGhenis

   on Nov 5, 2026

   CollaboratorAuthor

   More actions

   Ready for hub review

   Head 5a784a7e926d4bc5b1471770ae67d4f96f139140 on claude/deterministic-engine-phase-1a-s1-j31my4 (the harness pinned this branch name). Draft PR #2877, based on #2873's head 6f4d3313, so it contains #2873's two commits and lands after it. CI on the head: all 20 check runs succeeded.

   The work record above holds the drift table, the full results, the merge-tree results, #2308's test results, the rule-by-rule status, the ledger counts, what stage 2 needs and every judgment call. In short:

   Drift. Every core citation in the in-scope rules reads the same at 3.33.3 and 3.34.1. A script matched each cited block verbatim, and an AST comparison found none of the functions the rules rest on changed. From 3.34.1, Holder.set_input also drops calculated values over an input's own period (#593). That drops more stale caches and changes no rule.

   Findings and resolutions

   • utils/supplied_inputs.py: the three columns are registered; _supplied_value_storage, stored_supplied_value and _counted_input count a key only where core reads it; latest_supplied_input_period ranks with _latest_input_key; set_model_value refuses REQUIRED_STORED_INPUTS.
   • utils/stored_draws.py: the registry, input_source, check_draw_answer, the calculation-depth context variable, the setter rules, check_boolean_input and QUALIFIED_CORE_VERSIONS.
   • simulation.py: draw dispatch in calculate (after the lifecycle and stale-record cleanups), _calculate_source_backed_draw, the calculate_add/calculate_divide/calculate_output overrides, set_input checks, and the Core-route checks in build_from_dataset.
   • data/core_route_checks.py: the identity and supplied-period checks.
   • Consumers: scenarios/uc_reform.py (new_claimants_by_draw), scenarios/pip_reform.py (phased_out_by_draw), data/economic_assumptions.py (stored_student_loan_take_up, assign_new_borrowers).
   • Mutation survivors on the first pass: an unreachable _check_schema branch (removed) and two raises without tests (unregistered name, source changed mid-read): tests added, now killed.

   Tests.

   • The 17 required files, my 6 new files and 5 other core-Dataset-route files: 1069 passed, 0 failed on core 3.34.1, and 1069 passed, 0 failed on 3.33.3 (version printed).
   • code_health: 3556 passed.
   • Mutations: 25 of 25 killed. Revert checks: the old key rule fails 3 tests and the old ranking fails 2.
   • Ruff and git diff --check: clean.

   Merge-tree. Clean against main (d4655eb5) and #2816. #2265 and #2308 conflict in simulation.py and supplied_inputs.py (plus test_input_records.py for #2265), but they already conflict with main in those same files.

   #2308's tests at my head. 38 passed and 18 failed:

   • 3 failures are the plan's intended differences: an input on disk under a memory cache, and a branch cache over the branch's own input.
   • 15 failures come from that file's fixtures against today's registry: it writes 100.0 into boolean, enum and monthly variables, and it compares its own list of variables with the registry.

   Ledger. None of pe-uk's own Core-route fixtures is refused (0 for every reason and entity); policyengine.py is left to the hub.

   Guard entries. No guard entry changed, and RATCHET_CEILING is still 21.

   Not built / skipped. These qualification cases weren't exercised separately: a custom set_input handler, #571 witnesses (the tests check that draws refuse ADD and DIVIDE), a legacy dump without provenance, and memory_config as a disk route (three other disk routes are tested). The "two processes agree" determinism case has no subprocess test. No test needed survey data. No skips, xfails or allowlist changes.

   Judgment calls (details in the record):

   • the UC draw's default is 1.0;
   • a situation's PIP draw is used;
   • a situation's UC draw is ignored (protected amount, as today);
   • the fallback streams are drawn lazily (same values);
   • the rate parameter is gone from the student loan helper, and an AST test pins it;
   • ADD/DIVIDE/output refusals apply only in simulations built from data, while the setter refusals apply to every simulation;
   • Add inputs for all TAXBEN benefits #6's validation refuses floats, strings, None, NaN and NA.

   ---

   Generated by Claude Code