benchmarks: tooling for the post1 rerun of the paper's embedded cells (build-only mode, wheel label, one wheel per cell, embedded-only stages)

benchmarks/experiments/CAMPAIGN.md

@@ -242,6 +242,8 @@ The detail behind item 8:
 **ONE LINE OF DEVELOPMENT.** The instrument is developed on `main`, never on a long-lived branch (BUGS F65): mini pulls `main` at every stage start, so the instrument is landed on `main` before the first cell and never during the run, and a change that must not reach a running campaign waits for the next one. Anything else lands on `main`, behind a flag if it must, or waits. A later instrument is added the same way: `BENCH_INSTRUMENT` (`bench_common.INSTRUMENT`, default `2026-11`) names it, and code for it asks `instrument_at_least(...)` (`test_instrument_selector.py`).
 
 **The next campaign's stages are generated by `make_2611_stages.py`.** One template carries every guard the campaign learned (cap table, shadowed definitions, budgets against the cap, whole-cell cost, images present and current, comparator digests frozen at generation, the JPype pin read out of the image, the pinned wheel by sha256, the cap rule, the ALL-DONE trap); the instrument, `REPS` and the banner are its parameters. The generator selects the multi-model subset, narrowed by a profile (next paragraph). The subset is `ENGINES` in the file: ArcadeDB (embedded and served), SurrealDB (embedded and served), ArangoDB, Memgraph, PostgreSQL with pgvector, AGE and TimescaleDB (and plain PostgreSQL where a lane uses it), MongoDB with MongoDB Search, DuckDB with VSS and DuckPGQ, SQLite with sqlite-vec, the composed Qdrant plus Neo4j stack, Neo4j and Qdrant on their own (graph and dense; sparse on the second list), and LadybugDB on every lane it supports. Every registered arm is either in the subset or in `OUTSIDE`, so a newly registered arm forces a decision (`--check`). The core lanes are graph (interactive and analytics), cross-model, documents and dense; under `--profile full`, `--second-list` adds lifecycle, then sparse and time series, after them. TimescaleDB runs whenever the time-series lane does (so not under the default profile); `--without-timescaledb` leaves it out.
+**The wheels-only post1 re-measure has its own tooling (POST1-RERUN.md; the plan is row 93, issue #581).** `make_post1_rerun_stages.py` writes embedded-only stages on top of this generator, with `runner.py --build-only` cells where only the load figures can change, the post1 wheel pinned by name and sha256, and rows in `runs_page_<pin>_2611_post1.jsonl`; `land_stage.py --post1-reps N` lands, per cell, the rows of one wheel and refuses a cell that would mix wheels. None of it is active for a stage of the live pass: every new flag and variable is off by default.
+
 **THE PAPER PROFILE (the owner, 2026-10-25: the paper's experiments must finish by 2026-11-10; amended 2026-11-03, DECISIONS #232 and #235: the paper's numbers land about 2026-11-15, and the submission is 2026-11-24 for the SIGMOD 2027 industrial track or 2026-12-01 for the ICDE 2027 industry track, the fallback).** `--profile paper-2611` is the default and narrows the subset in the open (`PROFILES`, `DROP_WHY` in the file). (1) Core lanes only: graph, cross-model, documents and dense. The second list (lifecycle, sparse, time series) is out of scope for the paper, for the comparators and for the ArcadeDB chain, and `--second-list` is refused unless `--profile full` is given. (2) Neo4j and Memgraph stay in the graph lane (and cross-model where they are); their dense-lane cells (`neo4j_dense`, `neo4j_dense_int8`, `memgraph_dense`, `memgraph_dense_int8`) are out. The composed embedded stack (SQLite + LanceDB + LadybugDB in one process, `composed_embedded_sqlite_lance_ladybug`, the owner, 2026-10-25) is a Tier 1 cross-model arm: engine `composed-embedded`, right after the served composed stack, six lane cells per pass (hybrid in both durability classes and atomicity, at e2 and e2_500k), on the lane `e2` only; it runs in the dense image, so rebuild `dbbench:dense` (`./build_images.sh dense`) before the first pass. (3) The dense lane is fp32 only, for every engine: no `_int8` and no `_pq` arm, the comparators' and ArcadeDB's alike (`arangodb_dense_int8`, `arcadedb_dense_embedded_int8`, `arcadedb_dense_server_int8`, `mongodb_dense_int8`, `qdrant_dense_int8`, `sqlite_vec_dense_int8`, and the PQ arms when `BENCH_DENSE_PQ=1`). Quantized arms are not like-for-like across engines, and quantization is out of the paper's scope: fp32 for every engine, ArcadeDB's own INT8 and BINARY out by the owner's rule, jvector PQ, BQ and NVQ studied after the paper. ArcadeDB keeps its fp32 dense arms, embedded and served. (4) The target is N=3 (`TARGET_REPS`; `--project` prices 3 repetitions by default). The chain keeps the passes 4 and 5 after pass 3, in #503's order, so they run only if time is left; stopping after the first six groups is a complete result. Since 2026-11-03 the order on the bench host is: pass 3 (qC48), then the re-runs (qC14 at turbo off, qC15..qC23 at the 10 us idle hold, DECISIONS #234), then the four-arm ArcadeDB pass on the released 26.11.1 wheels, then passes 4 and 5 (qC49..qC80, which wait for `PASS45.go`). Amended 2026-11-05: if the 26.11.1 wheels are on PyPI and the four-arm runner is deployed when `REORDER reruns finished` is logged in the host's `STATUS.txt`, the four-arm pass (n=3 for the four ArcadeDB arms) starts first, and the window re-runs of qC14..qC17 follow it, still before `PASS45.go`; otherwise the order above stands. So the paper's comparators are N=3 for sure and N=5 where passes 4 and 5 finish before the tables freeze (DECISIONS #232, amendment of 2026-11-03). `--check` prints every dropped cell with its reason and count (lane cells per pass: 148 second-list, 8 Neo4j and Memgraph dense, 12 quantized dense) and proves that none is scheduled, in both chains whatever the arguments select (`profile_problems`); `--check --drafts DIR` lints written scripts for the same. `--profile full` is the whole subset as it was before the profile.
 Every generated stage exports `BENCH_INSTRUMENT=2026-11` before its first Python call, reads the selection back from `bench_common` and aborts if the tree does not select it, writes `runs_page_<pin>_2611.jsonl` (and overlay directories with the same suffix) so one rows file never holds two instruments, and fixes `REPS` at generation (`--reps`, default 5, at most 5; the target is 3). The generator re-executes itself with the variable set and refuses a different value in the environment. `land_stage.py` lands exactly these files (DECISIONS #196; the 2026-10-25 change that closed the gap between the pull and the freeze): it pulls `runs_page_<pin>_2611.jsonl` and the overlay directories with the same suffix, and `screen_pulled_rows` refuses the file before anything is merged unless every row is stamped `2026-11` and names the pin's engine commit, so a retired 2026-10 file or a file that mixes instruments never reaches `results/runs.jsonl`. The first landing meets no store (the October one is retired) and `merge_campaign.py` creates it. `export_web` no longer demands the `e4decomp_<pin>` artifacts of a landing scoped away from e4, because the chain has no e4 stage and `--only-lanes l2` would otherwise wait for one. A rehearsal restores what it wrote, the site payload (removed when the run created it) and the PAGE-SPEC.md block included. Failed cells (`rc != 0` or an error) merge as failures and are not frozen; censored queries keep their flag and publish no number. No stage script calls `land_stage.py` (the generated `qC` scripts only name it in a comment about the rows file); a landing is run by hand on the laptop.
 The comparators are the default mode and run on the release pair already installed (`ARCADEDB_WHEEL`, `ARCADEDB_SERVER_IMAGE`, `ARCADEDB_ENGINE_COMMIT`): `python3 make_2611_stages.py --after <last item of the queue on the machine> --status-file STATUS.txt --id-prefix qS --out DIR`. `--after` is required, with no default (a marker name, or `none`), and the first stage waits for `grep -qx "<marker> ALL-DONE"`, each later one for the stage before it. The marker must be the LAST item still to run, one that writes its line whether it runs or is skipped: a marker already in the machine's STATUS.txt makes the first stage start at once, beside the running jobs (2026-10-23: `qMA06 ALL-DONE`, written by a skipped study). So `--status-file` is required with a marker (a copy of that STATUS.txt, or `none` when there is no queue) and the generator refuses a marker the file already holds; `--check --drafts DIR --status-file STATUS.txt` makes the same test on drafts that exist. The subset is emitted in ENGINE TIERS in the user's exact pass order (2026-10-24; `--tiers`, on by default, `--no-tiers` for the plain rep-major chain). N defaults to 5, the target is 3 (the paper profile), and the order is `schedule_order`: n=1 T1, n=2 T1, n=3 T1, n=1 T2, n=2 T2, n=3 T2, n=4 T1, n=4 T2, n=5 T1, n=5 T2. Every prefix of it is a complete set of equal replicates for the tiers covered so far (each tier has run passes 1..k, none skipped, and a pass is whole before the next starts); `--reps 3` stops after the first six groups, which is the chain of Tier 1 passes 1-3 and then Tier 2 passes 1-3 as before. Tier 1 is ArcadeDB (the four 26.11.1 arms, when `--arcadedb-release` adds them), SurrealDB, ArangoDB, PostgreSQL with pgvector and AGE, MongoDB with Search, the composed Qdrant plus Neo4j stack, and the embedded references SQLite, LadybugDB and DuckDB; Tier 2 is Memgraph, Neo4j and Qdrant on their own, Redis 8, CrateDB and TimescaleDB (`TIER_ENGINES`, with `ARM_TIER` for TimescaleDB, an arm of the `postgresql` engine that the user lists in Tier 2). Every stage holds the arms of one tier, and stage ids stay one run of consecutive numbers (a stage that mixed both tiers is split in two, so the stage count grows, and each stage repeats its guards and image checks; past 99 stages the ids take three digits, so `qS100` cannot sort before `qS11`). `--check` prints the schedule and proves the order and the prefixes (the plan collapsed into (tier, pass) groups equals `schedule_order`, each group holds the whole pass), and `--check --drafts DIR` lints a draft folder for the same. An arm with no tier runs after Tier 2 in each block. The ArcadeDB arms are Tier 1 but not in this chain: DBBench keeps one ArcadeDB version per host, so they run only once the benchmark pin is 26.11.1 (next paragraph). Inside a tier the order is the one below: a pass is pass k of every stage of the tier (core lanes, then the second list), and each stage runs ONE repetition (`--only-reps $PASS`). With `--no-tiers` the chain is pass 1 of every stage, then pass 2, up to pass N. If time runs out, every engine of a tier has the same n; repetitions days apart show drift. A stage whose environment declares fewer reps (`REPS=k`) is left out of the later passes. Inside a lane the engines run in `ENGINE_PRIORITY` (SurrealDB, ArangoDB, PostgreSQL, MongoDB, Redis 8, composed Qdrant+Neo4j, Neo4j, Qdrant, DuckDB, SQLite, Memgraph, LadybugDB, CrateDB; ArcadeDB's arms lead it when they join; Redis and CrateDB are skipped until their adapters are in `ENGINES`). The rep-1-first rule and the cap rule hold across passes: a pass above 1 runs a cell only when rep 1 of it was clean, and only a timeout of rep 1 censors the larger sizes. WHICH REPS EXIST is `~/rep-state-2026-11-<pin>.tsv` on the host (one `label, rep, ok|fail` line per finished repetition; the rows themselves carry the rep number in the runner's results file): stop after any pass, and the next pass, or a restarted stage, picks up from it with no regeneration; N can go to 3 or stop at 2. Delete that file to start a chain over. The ArcadeDB pass is `--arcadedb-release X.Y.Z` with `--engine-commit`, `--image-digest sha256:...`, `--wheel-name`, `--wheel-sha256` and `--wheel-url`; it refuses unless all five are given, because the release may not exist when the drafts are made. It writes `<prefix>00`, the pair install (official image by digest, wheel by sha256, `build_matched_pair.sh`, `verify_pair_c25.sh`), and ArcadeDB's lane stages behind it. THE ARCADEDB GATE: the ArcadeDB chain is its own chain (Tier 1 only, passes 1..N in order), generated with `--after` the last comparator stage, so the comparators run the schedule first. `<prefix>00` waits after its `--after` for the marker file `$HOME/conf/go-arcadedb-2611` on the host (it never creates it; it sits before the ALL-DONE trap, so a killed waiter hands nothing on), and every ArcadeDB lane stage aborts without it. The user or the agent touches that file when `CURRENT_PIN` moves to 26.11.1, and not before; `--check --drafts` lints that the gate is in the pair install and in every ArcadeDB stage and in no comparator stage.

benchmarks/experiments/POST1-RERUN.md

@@ -0,0 +1,49 @@
+# Re-measuring paper cells on a wheels-only post1
+
+The procedure for rerunning the paper's ArcadeDB embedded cells on `26.11.1.post1`, the wheels-only release whose engine is byte-identical to 26.11.1 (DECISIONS #297). The plan and the open question of which cells qualify are CAMPAIGN section 7 row 93 (issue #581). This file is the how. Nothing in it runs by itself, and nothing in it touches the live pass: the rerun writes its own rows file and its own rep state, from its own worktrees.
+
+## What was built
+
+| piece | file | what it does |
+|---|---|---|
+| build-only mode | `runner.py --build-only`, a hook in `l1_tpc.py`, `l2_graph.py`, `e2_hybrid.py`, `l3d_dense.py`, `post1_rows.py` | A cell stops right after `build()` (load and index, before any timed work) and writes a row with `row_kind = "build_only"`. The runner refuses the flag unless the results file is named `runs_page_<pin>_2611_post<N>.jsonl`, so such a row cannot land in the pass's own rows file. |
+| wheel label | `post1_rows.stamp_row`, called by the runner | A stage that exports `BENCH_WHEEL_LABEL`, `BENCH_WHEEL_NAME` and `BENCH_WHEEL_SHA256` gets `wheel_label`, `wheel_name` and `wheel_sha256` on every row. A row without them is labelled by its `engine_version`. The build is `engine_backend` (`native` or `jvm`), as before. |
+| the choice of one wheel per cell | `post1_rows.resolve`, used by `land_stage.py --post1-reps N` | Per in-process ArcadeDB cell, the rows of ONE wheel: the newest wheel that has rows for the cell, only when it is complete at N repetitions. A cell that would mix wheels, or whose newer wheel is short, is a problem and the landing exits nonzero before anything is merged. |
+| stage generator | `make_post1_rerun_stages.py` | Writes the wheel stage and the lane stages of the rerun, embedded arms only, on top of `make_2611_stages.stage_script` (so the TEMP GATE, the guards, the cap rule and the rep gate are the live chain's own), plus `ORDER.txt` and the sequential driver `run_post1.sh`. |
+| tests | `test_post1_rerun.py` | Pure Python, no docker, no timing. `uv run --no-project python test_post1_rerun.py`. |
+
+## What a build-only row may replace
+
+Only these fields of the same cell and repetition, and only when the glue change does not touch a query path (DECISIONS #299: the ingest changes #318, #324 and #325; not the read-path changes #321 and #323, which need a `full` cell):
+
+`build_s`, `ingest_s`, `index_s`, `index_before_load`, `setup_s`, `gav_build_s`, and the load-transport facts `lineitem_buckets`, `async_writers`, `async_sync`, `load_call_rows`, `columnar_insert`.
+
+Every timed workload figure of the cell stays at its 26.11.1 value. The memory, disk and cpu figures of the build window are carried beside the full row's own as `build_window_*` and never over them, because the full row's figures cover the whole container life. The merged row says where each half came from: `timed_wheel`, `timed_ts_utc`, `build_figures_wheel`, `build_figures_wheel_sha256`, `build_figures_run_id`, `build_figures_ts_utc`, `build_figures_replaced`. Its `ts_utc` is the newer of the two, because the freeze keeps the newest row per cell key.
+
+A cell list entry has one of three modes: `build-only`, `full` (a whole cell, with its overlay) or `skip`. The default list is every embedded cell of the paper profile (22 jobs per pass per chain): lane cells `build-only`, the two multipass overlays `skip` (an overlay times queries, there is nothing to stop after).
+
+## The order of the real rerun
+
+1. **Preconditions.** The live n=3 pass and the PG-wire catch-up have ended; `bench/arcadedb-import-26111` (the four-arm generator, PR #554) is merged, because the native chain needs it (`make_post1_rerun_stages.py` refuses a native chain without it); the post1 wheels are released (tag and PyPI) and the byte-identity guard of DECISIONS #297 passed; the user said go. Nobody syncs the mini during the live pass: a dirty or moved tree aborts its queues.
+2. **Pin check.** The engine commit is the pass's (`ENGINE_COMMIT` of the pins file), the wheel label is `26.11.1.post1`, and each wheel's file name carries that version. The stage reads the version out of the built image and refuses on a difference, so a wheel whose metadata says another version cannot run (the candidate JVM wheel of the glue A/B is one: its metadata says 26.11.1). `engine_pin.jar_pin` makes the engine-jar check accept the `X.Y.Z` jar under an `X.Y.Z.postN` pin.
+3. **Cell list.** `python3 make_post1_rerun_stages.py --write-cells post1_cells.tsv`, then edit the mode column from the measured A/B (never by judgment, DECISIONS #298).
+4. **Dry run, laptop.** `python3 make_post1_rerun_stages.py --dry-run --cells post1_cells.tsv --hours HOURS.tsv --reps 3` prints the stage list and an ESTIMATE per chain. Generate the scripts with the real pins, `bash -n` is part of the selftest. Run ONE build-only cell on the laptop with a synced wheel into a scratch results file (`runner.py --lanes l1tpc --workloads oltp --scale tpch1 --backends arcadedb_embedded --build-only --results-file runs_page_dry_2611_post1.jsonl`) and read the row back: `row_kind`, `wheel_sha256`, `build_s`, and no timed figure. The laptop verifies, the mini measures.
+5. **Mini, only on the go.** Worktrees at the harness commit, one per chain (`git worktree add --detach ~/post1-wt/<chain> <sha>`, `results` linked as `fourarm/run.sh` does), the generated folder copied to `~/stages-2611c/post1/`, then `touch ~/POST1.go` and `setsid nohup bash run_post1.sh`. One timed job at a time, `ORDER.txt` is the order, removing the go file stops it at the next stage boundary.
+6. **Landing.** `land_stage.py --pin <commit> --post1-reps 3 --dry-run`, read the per-cell decisions, then `--apply`. A build-only row anywhere else is refused.
+7. **Rows and state of the live pass are never touched.** The rerun writes `runs_page_<pin>_2611_post1.jsonl` and `rep-state-2026-11-<pin>[-<build>]-post1.tsv`.
+
+## What the paper may say
+
+For a rerun cell, only the post1 figures of the fields above, labelled with the wheel (`26.11.1.post1`, sha256 in the artifact), and the 26.11.1 figures for everything else of the cell, labelled as 26.11.1. The 26.11.1 rows of a rerun cell stay in the raw logs and are not printed (DECISIONS #298). A cell that was not rerun keeps its 26.11.1 rows. The served arms never change (they do not use the wheels).
+
+## n and repetitions
+
+DECISIONS #240: a speed claim needs 3 runs per arm at 2x or more, 5 or more at 1.3x to 2x, 10 or more below 1.3x, pre-registered in the README and the brief. The landing's `--post1-reps N` is that n: a cell with fewer post1 repetitions than N is refused, never filled from 26.11.1. The paper's n=3 is `--post1-reps 3`; a pass 4 or 5 of the live chain would need the same number of post1 repetitions.
+
+## Rollback
+
+`uv run --no-project python post1_rows.py strip --label 26.11.1.post1 results/runs.jsonl` deletes every row that carries the wheel label (a full post1 row, or a cell merged from build-only rows), then `land_stage.py` without `--post1-reps` lands the live rows file again, which restores the replaced 26.11.1 rows, and the page is rebuilt by the landing.
+
+## Not verified here
+
+Nothing was run on the mini or in docker. The freeze's handling of the added columns (`row_kind`, `wheel_*`, `build_*`, `timed_*`, `build_window_*`) and of two `engine_version` strings in one freeze (the one-version gate of the four-arm freeze) is checked on the landing's dry run, not before. `verify_pair_c25.sh` may compare the wheel version with the engine jar version and has not been read for `.postN`.