Repository navigation
benchmarks: tooling for the post1 rerun of the paper's embedded cells (build-only mode, wheel label, one wheel per cell, embedded-only stages) - #584
Draft
tae898 wants to merge 1 commit into
Conversation
… (build-only mode, wheel label, one wheel per cell, embedded-only stages) (#583) The post1 rows override the 26.11.1 rows of the cells that are rerun, all repetitions of a cell from one wheel (DECISIONS #297, #298). The harness had no way to stop a cell after build(), rows did not say which wheel file they ran on, the generated stages mix embedded and served arms, and nothing refused a cell with repetitions from two wheels. All of it is additive and off by default, because comparator stages pull main at every start and a mid-run change splits rows: no variable set means no behaviour change. - post1_rows.py (new): build-only helpers for the lanes, the opt-in wheel stamp, resolve() (one wheel per cell, problems instead of a silent mix), strip (rollback). - runner.py: --build-only (refused unless the results file ends _post<N>.jsonl), BENCH_BUILD_ONLY forwarded when set, stamp_row() before the row is written. - l1_tpc.py, l2_graph.py, e2_hybrid.py, l3d_dense.py: one guarded hook after build() and before any timed work. - land_stage.py: --post1-reps N (default 0) pulls the post1 rows file and lands the resolved rows; build-only rows are refused as cells. - engine_pin.py: an X.Y.Z.postN pin accepts the X.Y.Z engine jar (the check refused every post1 cell). - make_post1_rerun_stages.py (new): embedded-only stages on top of make_2611_stages.stage_script, cell list, dry run with an estimate, selftest, driver. - POST1-RERUN.md, CAMPAIGN.md paragraph, test_post1_rerun.py. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
5 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tooling for the post1 rerun of the paper's embedded cells (CAMPAIGN section 7 row 93 is the plan, #581 and PR #582 the docs row; I do not repeat it here). Draft: I am not merging anything, and nothing was run on the mini or in docker.
Closes #583
What it adds. A build-only mode (
runner.py --build-only, one guarded hook afterbuild()in each ofl1_tpc.py,l2_graph.py,e2_hybrid.py,l3d_dense.py; the row carriesrow_kind = "build_only"), a wheel label on the rows of a stage that opts in (wheel_label,wheel_name,wheel_sha256), a per-cell choice of one wheel at landing (land_stage.py --post1-reps N), an embedded-only stage generator with a dry run, an estimate and a selftest (make_post1_rerun_stages.py), the procedure (POST1-RERUN.md) and a rollback by wheel label (post1_rows.py strip). A build-only row may replace onlybuild_s,ingest_s,index_s,index_before_load,setup_s,gav_build_sand the load-transport facts; every timed figure of the cell stays at its 26.11.1 value, and the build window's memory figures sit beside the full row's asbuild_window_*.Why the changed files are inert for the live pass. Comparator stages pull main at every start, so each edit has to be a no-op unless asked:
runner.pyimport post1_rows,--build-only(default off),BENCH_BUILD_ONLYforwarded only when set,post1_rows.stamp_row(row)before the row is writtenstamp_rowreturns the row untouched (noBENCH_WHEEL_LABEL, noBENCH_BUILD_ONLY)l1_tpc.py,l2_graph.py,e2_hybrid.py,l3d_dense.pyif os.environ.get("BENCH_BUILD_ONLY") == "1":and a lazy import inside itland_stage.py--post1-reps(default 0), and a refusal of rows that carryrow_kind == "build_only"row_kindengine_pin.pyjar_pin(), used in the two engine-jar comparisons.postNCAMPAIGN.mdmake_2611_stages.pyis not changed (its--checkstill exits 0), sostage_drafts_2611.sha256holds.One finding.
engine_pin.verifydemands that the engine jar version equals the pin, and the pin of a post1 wheel is26.11.1.post1while its jar is26.11.1: withoutjar_pinthe runner refuses every post1 cell.Dependency on #554. The native chain needs the four-arm generator of #554 (
CHAIN_BUILD, the-nativeimage tag); the generator refuses a native chain without it. I applied this branch onto the #554 head (git apply --3way): seven files apply cleanly,engine_pin.pyconflicts in the two jar comparisons (same lines), and the same two-line change resolves it there. With that resolved,test_post1_rerun.pypasses and a native and a JVM chain generate (44 scripts, allbash -nclean) on the #554 head too.Checked.
compile()of every changed file,pyflakes(no new warning),uv run --no-project python test_post1_rerun.py(resolve rules, the stamp, rollback, the generator: no served arm in anyBACKENDS,--build-onlyand the wheel exports in every stage, the post1 rows file and rep state, no overlay in a build-only stage, a split cell list covers exactly the wanted cells, a drifted anchor is refused),bash -non every generated script and on the driver, the runner's refusal of--build-onlywith the pass's own rows file (run as a plain argument error).Not checked. No lane ran with the hook, no docker cell, no mini. The freeze's handling of the added columns and of two
engine_versionstrings in one freeze (the one-version gate of the four-arm freeze) is for the landing's dry run.verify_pair_c25.shmay compare the wheel version with the jar version and was not read for.postN. Thebuild_window_*copy matches field names by prefix (peak_,mem_,rss,cpu_,disk_,io_,heap,jvm_,gc_) and was not compared with real rows. The hours behind the dry-run estimate come from the scope note and the pass-1build_smedians and are estimates, not measurements.🤖 Generated with Claude Code