diff --git a/benchmarks/experiments/CAMPAIGN.md b/benchmarks/experiments/CAMPAIGN.md index 155bab603d1..0afadc3aced 100644 --- a/benchmarks/experiments/CAMPAIGN.md +++ b/benchmarks/experiments/CAMPAIGN.md @@ -8,7 +8,7 @@ Per campaign, once: 1. Settle the instrument before any cell runs: the query set, the measurement set, the durability rule, the fairness invariants, and the answer checking. A change mid-campaign splits the rows, so this is the only free moment (DECISIONS #82d, #88, #89, #90; PROTOCOL.md). The method pages in the package documentation (`bindings/python/docs/benchmarks/`) describe the same instrument without numbers, so they change in this step too, never later. 2. Smoke every adapter, one cell per engine per lane, and keep the evidence (the smoke's rows and logs). Smoke wherever a machine is free: nothing in the fairness contract touches a smoke, and the bench host is reserved only while a published cell runs, so a smoke on the real corpora there is both faster and a better test than a capped slice on the laptop. One runner per host, always: `runner.py` sweeps every `dbbench=1` container on the machine at startup, so it takes a host-wide lock first (`$TMPDIR/dbbench-runner.lock`, whichever checkout it runs from) and a second runner is refused at startup (BUGS F58). -3. Pin ArcadeDB: build the wheel for the release the campaign runs (the upstream sync is retired, so the engine tree does not move and the jars come from the release image; `bindings/python/docs/development/build-architecture.md#local-build`), run our Java reproductions on the built engine to confirm every fix it should carry is intact, and record the engine commit inside that wheel as the campaign's pin (the pin is the commit, not the version string, DECISIONS #49). For a stable release the pin is the release's own jar commit (26.10.1: `d36b4ca3a`), the wheel is the official jars plus this tree's bridge and package, and the pair is placed by the recipe in `.notes/bench/repin-round5/REPIN-START.md`. The served arm's matched pair is built from the same image by `build_matched_pair.sh` (HANDOFF.md). One wheel, one pin, held start to finish. +3. Pin ArcadeDB: build the wheel for the release the campaign runs (the upstream sync is retired, so the engine tree does not move and the jars come from the release image; `bindings/python/docs/development/build-architecture.md#local-build`), run our Java reproductions on the built engine to confirm every fix it should carry is intact, and record the engine commit inside that wheel as the campaign's pin (the pin is the commit, not the version string, DECISIONS #49). For a stable release the pin is the release's own jar commit (for example 26.10.1: `d36b4ca3a`), the wheel is the official jars plus this tree's bridge and package, and the pair is placed by the recipe in `.notes/bench/repin-round5/REPIN-START.md`. The served arm's matched pair is built from the same image by `build_matched_pair.sh`. One wheel, one pin, held start to finish. 4. Re-pin every comparator to its latest stable release, smoking the risky jumps on their own (COMPARATORS.md, DECISIONS #87). 5. Run the campaign, landing each stage into the freeze (section 6 below). 6. Regenerate the query budgets from the campaign's own rows (`python3 derive_budgets.py > budgets.py` reads the freeze, `results/runs_paper.csv`) so the next campaign's caps are measurements at the corpora it will run rather than projections, and commit the table with the freeze (DECISIONS #106). @@ -69,9 +69,9 @@ Beyond the lane's own metrics: **One cap per tier, identical for every engine.** A cap is a ceiling rather than a reservation, so an engine that needs less is unaffected by a tier whose cap is generous. A per-backend cap would be the unfairness the caps exist to prevent. -**One heap per tier for ArcadeDB, and each vendor's own rule for Neo4j and Elasticsearch (DECISIONS #202).** ArcadeDB embedded and ArcadeDB server receive the same `{heap}` from one table (`-Xmx`). Neo4j takes the heap and page cache its own `neo4j-admin server memory-recommendation` prints for the cell's cap, and Elasticsearch a heap of at most half the cap and under 26 GiB (`vendor_memory.py`; at the half-cap tiers the Elasticsearch heap is unchanged). Non-JVM comparators have no equivalent knob, which is why heap sizing is a documented resource-fitting override rather than a per-engine advantage. Two JVM processes are exceptions and run at their own default heap: QuestDB, which `runner.JVM_BACKENDS` lists, so its rows record the tier heap although no heap is passed to its server, and mongot, the search process inside the MongoDB Search image, whose heap nothing sets (section 7, row 30). +**One heap per tier for ArcadeDB, and each vendor's own rule for Neo4j, Elasticsearch and CrateDB (DECISIONS #202).** ArcadeDB embedded and ArcadeDB server receive the same `{heap}` from one table (`-Xmx`). Neo4j takes the heap and page cache its own `neo4j-admin server memory-recommendation` prints for the cell's cap, Elasticsearch a heap of at most half the cap and under 26 GiB, and CrateDB a quarter of the cap between 1 and 30.5 GiB (`vendor_memory.py`, `CRATEDB_HEAP_FRACTION`; at the half-cap tiers the Elasticsearch heap is unchanged). Non-JVM comparators have no equivalent knob, which is why heap sizing is a documented resource-fitting override rather than a per-engine advantage. Two JVM processes are exceptions and run at their own default heap: QuestDB, which `runner.JVM_BACKENDS` lists, so its rows record the tier heap although no heap is passed to its server, and mongot, the search process inside the MongoDB Search image, whose heap nothing sets (section 7, row 30). -**heap = 0.50 x cap, with three exceptions** (`runner.HEAP_BY_SCALE` against `MEM_BY_SCALE`): deep10m at 0.67 (24g in 36g), and the lifecycle tiers lc1m at 0.67 (16g in 24g) and lc10m at 0.86 (24g in 28g). The runner prints this table at startup and marks any deviation. +**ArcadeDB heap = 0.50 x cap, with three exceptions** (`runner.HEAP_BY_SCALE` against `MEM_BY_SCALE`): deep10m at 0.67 (24g in 36g), and the lifecycle tiers lc1m at 0.67 (16g in 24g) and lc10m at 0.86 (24g in 28g). The runner prints this table at startup and marks any deviation. 0.50 is a deliberate split between heap and page cache, not a default nobody revisited. At every tier except deep10m, `peak_anon` sits BELOW the committed heap (4g heap against 2.7 GiB touched at tiny; 8g against 6.1-6.5 GiB at small), so the JVM never needed the heap it already had and raising the ratio would change nothing measurable. The other half of the cap is not idle: it holds the OS page cache for the engine's own files, which the cold/warm split prices at 9-18x for ArcadeDB against 1.0x for engines resident from load. @@ -141,7 +141,7 @@ Rules every campaign keeps: **A new ArcadeDB release replaces the old one until the final DBBench release (DECISIONS #189).** When the new version's ArcadeDB pass has landed: run ONE scripted comparison of the old and new ArcadeDB rows (a regression worth filing before the next cut is the only use left for the old rows); then remove the old version's ArcadeDB rows, page data, pin-specific code paths, pinned test fixtures, wording, jars, wheels and images; the raw old rows stay only in mini's `~/queue_archive` until the paper is submitted. Comparator rows are kept, in a set that names no ArcadeDB version. A running chain is never relabelled for this (every stage pulls main at its start). Bindings release history and engine code are out of scope. After the final release, a new version adds rows and the page keeps the history (PAGE-SPEC.md section 1). -**The re-pin checklist: everything a measurement owes the harness, not just the engine.** This is the ONE list of re-pin changes; HANDOFF points here rather than keeping a copy. Each row names the decision behind it. A row that is built and verified is condensed to a line that keeps its number, because code comments, tests, and DECISIONS cite rows by number: never renumber, reorder, or delete one. An open row, or one that carries an instruction a future run needs, stays whole. Nothing on the list changes a running chain, because a mid-run change splits the rows; every item is settled before the first cell of the next measurement (step 1 of the routine). An item that moves ArcadeDB's numbers is the user's call (the "free for them, decisive for us" rule). Once decided, an item becomes a DECISIONS entry naming what it amends or retires, changes PROTOCOL.md, FAIRNESS.md, and the method pages in the same commit, and is condensed here once built. **Scope** says whose numbers an item moves: ArcadeDB's alone, or comparators too, which then need re-measuring with it. +**The re-pin checklist: everything a measurement owes the harness, not just the engine.** This is the ONE list of re-pin changes. Each row names the decision behind it. A row that is built and verified is condensed to a line that keeps its number, because code comments, tests, and DECISIONS cite rows by number: never renumber, reorder, or delete one (exception: a row whose instrument or page was retired is deleted and its number stays unused: 45, 46, 53, 79, 80, 83, 86). An open row, or one that carries an instruction a future run needs, stays whole. Nothing on the list changes a running chain, because a mid-run change splits the rows; every item is settled before the first cell of the next measurement (step 1 of the routine). An item that moves ArcadeDB's numbers is the user's call (the "free for them, decisive for us" rule). Once decided, an item becomes a DECISIONS entry naming what it amends or retires, changes PROTOCOL.md, FAIRNESS.md, and the method pages in the same commit, and is condensed here once built. **Scope** says whose numbers an item moves: ArcadeDB's alone, or comparators too, which then need re-measuring with it. | # | change | scope | status | evidence | |---|---|---|---|---| @@ -168,7 +168,7 @@ Rules every campaign keeps: | 21 | Every override is disclosed where a reader meets it, and the manifest records engine configuration | every lane with an override | built (2026-10-03), on main: `override_registry.py`, `fairness_check` F15, `page_check`, `test_overrides.py` (fails on a NOWHERE in PROTOCOL section 7). Not done: the page's overrides table showing which way each override moves a number, and the release asset's manifest does not carry the table | PROTOCOL.md section 7; FAIRNESS.md F15 | | 22 | The lifecycle lane reads the vector index's statistics in one extra untimed cycle after the timed ones, not inside every timed session: `l5_lifecycle.cycle()` calls `_vector_stats()` between the action timer and `close()`, and at the October pin that read raises the close it precedes, so the embedded dense-vector "open and close" cell is not a session that touched nothing | ArcadeDB embedded lifecycle, vector situation only | open; the lifecycle table discloses it under the table (`export_web` `vector_stats`, no number) and the sentence retires with this row; #8630, in 26.10.1, removed most of what the read added to the close | BUGS F159: laptop, 1M vectors, clean close 6.4 -> 15.3 ms with the read before #8630 (`de966e79c5`), 3.0 -> 3.5 after (`b8ab990821`); mini's pin reads 88.5 ms and its split is not measured; `.notes/bench/repros/vector-clean-close/` | | 23 | LadybugDB's graph arm runs with its thread pool and buffer pool fitted to the cell (`max_num_threads` from `sched_getaffinity`, `buffer_pool_size` at 0.8 of the cgroup limit), both on the row | LadybugDB, the graph lane (OLTP and analytics) | **built (DECISIONS #125)**: `l2_graph.py`; the 26.10.1 measurement runs every LadybugDB graph cell fitted | BUGS F160; FAIRNESS.md F6; LadybugDB/ladybug#1070 | -| 24 | The SurrealDB lifecycle arm streams its build input in batches of `INGEST_BATCH`, as the ArcadeDB arm streams row by row | SurrealDB, the lifecycle lane | built (DECISIONS #126, #134): `l5_lifecycle_surreal.py`; every SurrealDB lifecycle cell is re-run in the 26.10.1 measurement, and the October lifecycle table withholds its peak memory and 10M vector row until then (`export_web.LIFECYCLE_STALE`) | BUGS F161 | +| 24 | The SurrealDB lifecycle arm streams its build input in batches of `INGEST_BATCH`, as the ArcadeDB arm streams row by row | SurrealDB, the lifecycle lane | built (DECISIONS #126, #134): `l5_lifecycle_surreal.py`; every SurrealDB lifecycle cell is re-run in the 26.10.1 measurement; the exporter's withholding of its peak memory and 10M vector row until then was removed with the 2026-11-only publish side (#432) | BUGS F161 | | 25 | Served ArcadeDB analytics and the timed point read go through `POST /query`, which runs an idempotent statement without a transaction (ArcadeData/arcadedb#8775, #8792); `POST /command` stays for writes and untimed verification scans | ArcadeDB served: `l1_tpc` `olap` | built (2026-10-02): `l1_tpc.py`; EXPLAIN plans the filtered aggregate `(parallel)` over `/query` and sequential over `/command`, same answer | BUGS F163 | | 26 | PostgreSQL + AGE joins both graph tables: a `pgage_graph` adapter on `dbbench:pg-age`, oltp in both durability classes plus olap | PG+AGE only; rows 2, 12, and 14 apply | built and smoked (2026-10-02): `l2_graph.py`; digests match Neo4j's and LadybugDB's except the CRUD read-backs (apache/age#2587, `equivalence_check.KNOWN_DISAGREEMENTS`; those cells publish with a note) | DECISIONS #128, #131, #134 | | 27 | MongoDB's graph arm: `$graphLookup` against the chained `$lookup` form | MongoDB, the graph lane | settled the other way by row 56 (DECISIONS #131 item 2, then #151): every hop and analytics query is a chained `$lookup`, because a single-direction `$graphLookup` cannot state the undirected question exactly | `l2_graph.MongoGraph` docstring; PROTOCOL.md section 7 | @@ -204,7 +204,7 @@ Rules every campaign keeps: | 60 | LSQB's q5, q6, q8, and q9 run LSQB's own text on every engine; every graph analytics row records `lsqb_text` | every engine on l2olap | built (the user, DECISIONS #154 items 1 and 2), except the `sf1full` exclusion: `graph_common.py`, `l2_graph.py`, `test_lsqb_text.py`. **The exclusion of q6 and q9 at `sf1full` (`graph_common.TIER_EXCLUDED`, DECISIONS #109) is NOT lifted: it needs a one-cell first-touch probe on mini and a maintainer decision (DECISIONS #161 D2: lift it after the probe, at the end of the campaign).** | BUGS F174 | | 61 | ArcadeDB's Graph Analytical View covers every vertex and edge type of the loaded graph, as upstream's LSQB runner builds it; the no-view arm (`BENCH_GAV=0`) stays as the like-for-like row | ArcadeDB only (l2olap with-view arms) | built (the user, DECISIONS #154 item 3): `l2_graph.py` (rows record `gav_types`, a narrower cover refuses the cell), `test_gav_scope.py` | `.notes/bench/repros/perf-lsqb-q9-20261004` | | 62 | ArcadeDB's sparse arms return ids, as every comparator does (`SELECT id, score FROM (...)`); every ArcadeDB sparse row records `sparse_result` (`id`) | ArcadeDB only (l3s and its multipass warm passes) | built (the user, DECISIONS #153 item 1, adopted as parity): `l3_sparse.py`, `sparse_multipass_driver.py`, `test_sparse_rows.py` | `.notes/bench/repros/perf-sparse-20261004`: the ids and their order are identical on 50 of 50 queries | -| 63 | The documents lane loads ArcadeDB embedded through the bindings' columnar insert (own issue #150) instead of `insert_many` | ArcadeDB embedded only | built (the user's choice over my recommendation, DECISIONS #153 item 2): `Database.insert_columns` (`7a36dc650e`), `ArcadeTPC.build` in `l1_tpc.py`, `test_insert_columns.py`, `test_docs_columnar_load.py`; the row records `columnar_insert`. **The measured wheel must contain the bindings commit** (row 74) | `.notes/bench/repros/perf-docs-ingest-20261004` | +| 63 | The documents lane loads ArcadeDB embedded through the bindings' columnar insert (own issue #150) instead of `insert_many` | ArcadeDB embedded only | built (the user's choice over my recommendation, DECISIONS #153 item 2): `Database.insert_columns` (`7a36dc650e`), `ArcadeTPC.build` in `l1_tpc.py`, `bindings/python/tests/test_insert_columns.py`, `test_docs_columnar_load.py`; the row records `columnar_insert`. **The measured wheel must contain the bindings commit** (row 74) | `.notes/bench/repros/perf-docs-ingest-20261004` | | 64 | The sparse multipass driver keeps each comparator's version, as the dense driver does: `sparse_multipass_driver.py` stores the adapter's own string as `lib_version` and restores `engine_version` for the ArcadeDB arms | every comparator's sparse multipass warm passes (`l3s_mp`); no timing moves | **built (DECISIONS #150's remit, own issue #158)**: `test_sparse_mp_versions.py`; the October files stay as recorded | BUGS F173 | | 65 | The documents OLTP and durability operations get a disjoint untimed warm-up for every engine, as row 55 gives the graph reads | docs_oltp and the document rows of the durability table | built (the user, DECISIONS #157): `l1_tpc.py` (`OLTP_WARMUP`, `BENCH_DOCS_OLTP_WARMUP`; the bench host's to choose), `test_docs_oltp_warmup.py`; rows record `oltp_warmup`. Not for e2 | `.notes/bench/repros/perf-commit-20261004` | | 66 | `arango_common.calibrate_nprobe` fails fast when the index does not answer (recall at nProbe = nLists below a floor of 0.5), and ArangoDB moves to 3.12.12 with the other comparators | ArangoDB: the dense table (l3d), and the release move reaches every ArangoDB arm | built (the user, DECISIONS #156 item 3): `test_arango_calibration.py`; the digest is in `runner.py` and COMPARATORS.md | BUGS F175 (`.notes/bench/repros/arango-recall-20261004`) | @@ -214,24 +214,24 @@ Rules every campaign keeps: | 70 | The TimescaleDB time-series arm runs `ANALYZE p` after its `(host, ts DESC)` index, inside its ingest timer, as the plain PostgreSQL arm does | TimescaleDB, the time-series lane only | **owed at the re-pin, done on repin-prep-2 (2026-10-06, the user's decision), now on main**: `l4_tsbs.py`. October's TimescaleDB l4 ingest rows (417314c18) were measured WITHOUT the ANALYZE and stay as measured | `test_ingest_prose.py`; `test_l4_analyze.py` | | 71 | The served graph arm over Bolt in place of HTTP | ArcadeDB served, the graph arm only | **decided against (DECISIONS #162 item 2)**: with the lean HTTP client (row 72) one bound read costs 121 us over HTTP against 171 to 194 us over Bolt, and upstream's question on which protocol a Python client should use (ArcadeData/arcadedb#9165) is unanswered; the branch `levers-served-bolt` (`21a6ffd47e`) stays unmerged and off by default | `.notes` repros/levers-audit-20261006/LEVERS.md | | 72 | Every ArcadeDB SERVED arm that talks HTTP uses one persistent `http.client` connection per host (`lean_http.py`) in place of `requests.Session` | every ArcadeDB served arm that posts to the server; no comparator client moves | built (the maintainer's decision, DECISIONS #162 item 1); rows record `arcadedb_http_client` (the headline client is described under row 82); `test_lean_http.py`, `test_lean_http_live.py`, `test_e4_client_axis.py`. **Owed at the re-pin:** the e4 stage re-runs, and the page prose around the table (its caption and the "wire format" paragraph) is rewritten from the new columns. October's served rows (417314c18) carry no `arcadedb_http_client` | `.notes` repros/levers-audit-20261006/LEVERS.md | -| 73 | JPype is pinned to 1.7.1 in the benchmark images and recorded per row; reason and the measurement: `.notes/bench/repros/jpype-master-check-20261006/`. The wheel declares `jpype1>=1.5.0` with no upper bound, so an image rebuilt on the day of a JPype release would change the instrument in the middle of a run | every embedded ArcadeDB arm and the host-side Python-cost stage | **built (on main), owed at the re-pin**: `bench_common.JPYPE_PIN` and `Dockerfile.bench`'s `ARG JPYPE_VERSION=1.7.1`; every generated stage that runs an ArcadeDB arm reads JPype back out of `dbbench:arcadedb` and aborts if it differs; rows record `jpype_version`; `test_jpype_pin.py`. Moving the pin is a deliberate edit of `JPYPE_PIN` and the Dockerfile default between campaigns, never an image rebuild | jpype#1379 | +| 73 | JPype is pinned to 1.7.1 in the benchmark images and recorded per row; reason and the measurement: `.notes/bench/repros/jpype-master-check-20261006/`. The wheel declares `jpype1>=1.5.0` with no upper bound, so an image rebuilt on the day of a JPype release would change the instrument in the middle of a run | every embedded ArcadeDB arm and the host-side Python-cost stage | **built (on main)**: `bench_common.JPYPE_PIN` and `Dockerfile.bench`'s `ARG JPYPE_VERSION=1.7.1`; every generated stage that runs an ArcadeDB arm reads JPype back out of `dbbench:arcadedb` and aborts if it differs; rows record `jpype_version`; `test_jpype_pin.py`. Moving the pin is a deliberate edit of `JPYPE_PIN` and the Dockerfile default between campaigns, never an image rebuild | jpype#1379 | | 74 | The wheel the 26.10.1 measurement runs on carries the call-overhead change of bindings PR #214 (`fabbda7ab7`) and the columnar insert (row 63) | ArcadeDB embedded arms only | built and verified (2026-10-06): the stages bake a wheel built from this tree (commit `8989cbb0e8`) over the official 26.10.1 jars, sha256 `83a2a40369a411622696aed77361fd3b1c3d23b675d0a784b38b0b022c13c41b` (`PAIR VERIFIED d36b4ca3a`). It is NOT the file PyPI holds for the bindings 26.10.1 release; the page's note on what the Python package costs should name the change when it lands | #214's own A/B (laptop, relative only) | | 75 | The served ArcadeDB arms that bulk-load through `POST /api/v1/batch` run with `-Darcadedb.server.httpBodyContentMaxSize=68719476736` (64 GiB) in place of 26.10.1's default of 100 MiB | ArcadeDB served only: the graph arm (l2, restart lane) and the cross-model arm (e2) | built and verified on the laptop (2026-10-07), on main: `runner.py` (`HTTP_BODY_MAX_OPT`), `override_registry.py` (`arcadedb_http_body_limit`), PROTOCOL.md section 7, `test_overrides.py`. Mini picks it up at its next `git pull`, which is the user's to time | the 26.10.1 sf10 served graph cell failed rep 1; `.notes/bench/repros/served-body-limit-20261006/` | | 76 | **`pgvector==0.5.0` and `neo4j-rust-ext==6.3.1.0` go into the `client` image (DECISIONS #173, #174).** In `build_images.sh`, the `[client]=` line: `neo4j==6.3.1` becomes `neo4j==6.3.1 neo4j-rust-ext==6.3.1.0` and `pgvector==0.5.0` is appended; register both in the COMPARATORS.md pin table. The harness side is on main and waits for the image: `l3d_dense.PgVector.connect` and `e2_hybrid.PgAgeE2.__init__` import `pgvector.psycopg` lazily (a cell on an image without the package fails at connect with ImportError, by design), and `neo4j-rust-ext` is a one-`.so` overlay that needs no code. The same PR moved `PostgresTPC` (and `postgres_tuned`) to autocommit for single statements, the pgvector dense and cross-model product loads to binary COPY, and the six QuestDB reads to `binary=True` (PROTOCOL.md section 7) | the `client` image: comparators only; no ArcadeDB arm, no tier 1 stage | **code built, image rebuild pending; lands at the tier 1 to tier 2 boundary of the 26.10.1 measurement (qP10 rebuilds the client image at its start)**, between qP09's ALL-DONE and qP10's first cell, never inside a running stage: rebuild, `verify_pair`, `queue_lint`, then check inside the image that `neo4j.RUST_AVAILABLE` is True and `import pgvector.psycopg` works before the first cell; the image digest change is expected and recorded | Row 76 is the rebuild; the code half is the PR that added it. Rows stamped by #173 (`neo4j_rust_ext`, `redis_hiredis`, `es_json_serializer`) land in the same rebuild | -| 77 | ArcadeDB's query heap budget (`arcadedb.queryMaxHeapRAM`, default half the JVM heap) raised for the cells the default refuses: the l1tpc tpch10 olap `top_parts` GROUP BY in both ArcadeDB arms (DECISIONS #175, ArcadeData/arcadedb#9402) | ArcadeDB embedded and served, the documents lane, ONLY in the repair stage | **mechanism written, default OFF (humemai/arcadedb-embedded-python#260); the repair stage is generated after qP09 and run before the tier 1 to tier 2 boundary.** `BENCH_ARCADE_QUERY_MAX_HEAP_RAM_MB` adds the JVM option to the embedded arm and to the served arm's JAVA_OPTS and stamps `arcadedb_query_max_heap_ram_mb` and a source on the row; `override: arcadedb_query_ram`. Which rows are the headline (the default rows with the refusal shown, or the raised rows as an escape hatch like row 21's cap and row 75's body limit) is decided at landing. | not an instrument change: unset, nothing differs | +| 77 | ArcadeDB's query heap budget (`arcadedb.queryMaxHeapRAM`, default half the JVM heap) raised for the cells the default refuses: the l1tpc tpch10 olap `top_parts` GROUP BY in both ArcadeDB arms (DECISIONS #175, ArcadeData/arcadedb#9402) | ArcadeDB embedded and served, the documents lane, ONLY in the repair stage | **mechanism built, default OFF (humemai/arcadedb-embedded-python#260, closed 2026-10-07).** `BENCH_ARCADE_QUERY_MAX_HEAP_RAM_MB` adds the JVM option to the embedded arm and to the served arm's JAVA_OPTS and stamps `arcadedb_query_max_heap_ram_mb` and a source on the row; `override: arcadedb_query_ram`. Which rows are the headline (the default rows with the refusal shown, or the raised rows as an escape hatch like row 21's cap and row 75's body limit) is decided at landing. | not an instrument change: unset, nothing differs | | 78 | The comparator client paths are recorded per row (issue #261; DECISIONS #173, #174): `neo4j_rust_ext`, `redis_hiredis`, `es_json_serializer`, and `pg_client_path` (`autocommit`, `copy_binary`, `binary_cursor`); `qdrant_transport` (#172) was already stamped | comparators only: no ArcadeDB arm | built (DECISIONS #173, #174) | issue #261 | | 81 | The sparse lane warms up before its timed pass and the runner holds the CPU idle state for each timed cell (DECISIONS #177). Text on the page too: CPU idle states are not controlled on the host, and single-client times under a millisecond include the cores' wake-up latency (PAGE-SPEC section 0a, `setup.cpu_idle_note`) | the sparse lane (warm-up); every lane's rows (hold) | **built (humemai/arcadedb-embedded-python#278).** `bench_common.SPARSE_WARMUP_DEFAULT` untimed searches per engine, 1,000, one full round as the study behind the warm-up used, drawn from queries DISJOINT from the timed set so a query or result cache cannot flatter an engine: the dataset's extra queries when it holds them, else a fixed-seed sample of documents used as queries; the row stamps `sparse_warmup_queries` as run and `sparse_warmup_source` = `extra_queries` or `document_sample`. The hold is `/dev/cpu_dma_latency` at `BENCH_CPU_DMA_LATENCY_US` for the length of each timed cell: 10 microseconds in the 2026-11 pass, because the tree default `bench_common.CPU_DMA_LATENCY_DEFAULT_US` (0, no sleep state deeper than polling) kept every core in a busy-wait loop and made the host hot and slow (PROTOCOL.md section 7 has the A/B); it needs write access to the device on the host, and a hold that cannot be taken never fails the cell: rows stamp `cpu_dma_latency_us`, `cpu_dma_latency_held`, and `cpu_dma_latency_error` when it failed. `BENCH_SPARSE_WARMUP=N` and `BENCH_CPU_DMA_LATENCY_US=N` set the count and the target (PROTOCOL.md section 7); neither can be turned off. `override: sparse_warmup`, `override: cpu_dma_latency`; `test_idle_and_warmup.py`, `test_instrument_method.py` | part of the instrument | | 82 | The served ArcadeDB headline is the benchmark's own minimal HTTP client (`lean_http.LeanSession`, stamp `lean_http (http.client, keep-alive)`); the page and the paper name it and say that the other engines are measured through their vendors' own Python clients | every ArcadeDB SERVED arm that talks HTTP through `lean_http.Session()`; no comparator client moves | **built (humemai/arcadedb-embedded-python#282, the instrument's client row; the headline selection is DECISIONS #268 and #285).** Each served cell runs as a second job on this client (`runner.ARCADEDB_HTTP_ARMS`, run_id ending `_clientlean`, `arcadedb_http_client_role` = `sensitivity` as stamped; the runner sets `BENCH_ARCADEDB_HTTP_CLIENT=lean` for it). `make_paper_tables.load_canonical` keeps the rows of this client and sets the other client's rows aside before any dedupe, table, gate or claim, so `runs_paper.csv`, the paper tables and the page hold these rows only; `export_web.py` refuses a frozen row or an overlay file that names another client, and `page_check.py` fails a page that names the vendor's driver. An arm with no job on this client (`arcadedb_imgdefaults_server`, and the PQ server arm) has no published cell; the docs transaction table says so for the first. This is the client the 26.10.1 pass and the published October rows used, so those rows stay comparable; rows measured with two clients are never mixed in one column. The `/query` and `/command` calls, the `GET /server` probe, the `/batch` bulk load, transactions and NDJSON streams all go through this client in these jobs. The PostgreSQL wire has no adapter in this harness. Measured on the official 26.10.1 image (d36b4ca3a), 7 interleaved rounds: the bound Cypher point read costs the client 55.5 us of CPU per call with `lean` against 11.6 for a bare-socket variant that was tried and retired, identical answer digests; the 100-row read is engine-bound and moves less | part of the instrument | -| 84 | The dense lane's id index for the delete (humemai/arcadedb-embedded-python#291, DECISIONS #178 index parity): ArcadeDB, pgvector, Neo4j, Memgraph, FalkorDB, DuckDB, and LanceDB get an index on the vector id before or right after the load, the other engines already delete through their own key | the dense lane (l3d), every arm | **built.** Every dense row stamps `dense_id_index` (the statement, or `key`); `override: dense_id_index`; `test_dense_id_index.py`. The index build is inside `ingest_s`. The /next page says ArcadeDB's delete ran without an id index in rows that lack the stamp (`_dense_id_index_note`, list line `dense_id_index`; `test_next_measurement_wording.py` holds the wording) | part of the instrument | +| 84 | The dense lane's id index for the delete (humemai/arcadedb-embedded-python#291, DECISIONS #178 index parity): ArcadeDB, pgvector, Neo4j, Memgraph, FalkorDB, DuckDB, and LanceDB get an index on the vector id before or right after the load, the other engines already delete through their own key | the dense lane (l3d), every arm | **built.** Every dense row stamps `dense_id_index` (the statement, or `key`); `override: dense_id_index`; `test_dense_id_index.py`. The index build is inside `ingest_s`. The dense table's generated note names the engines whose rows stamp it (`export_web._dense_id_index_note`, held by `test_dense_id_index.py`; the /next page and its wording test were deleted with #432) | part of the instrument | | 85 | Streamed results for the served ArcadeDB time-series arms, `BENCH_ARCADEDB_TS_NDJSON=1`, default OFF: both served arms ask the query endpoint for NDJSON and read the records line by line, as the PostgreSQL and MongoDB arms read rows as their server sends them | the time-series lane (l4), `arcadedb_ts_doc_server` and `arcadedb_ts_native_server` only | **mechanism written, default OFF; MEASURED 2026-10-09 and NOT USED at 26.11.1:** at ts100 the switch helped the documents arm (0.853 [0.713, 0.928]) and hurt the native arm (1.087 [1.083, 1.088]), identical rows, so it stays off and the page promises nothing; it needs a reader that parses as data arrives and a ts1000 re-measure first. Unset, nothing differs (`test_ts_ndjson.py`); set, rows stamp `arcadedb_ts_result_format` = `ndjson`; `override: ts_ndjson` | not an instrument change: unset, nothing differs | -| 87 | Two failures of the 26.10.1 pass that were not results (humemai/arcadedb-embedded-python#344, the failure triage of 2026-10-20): (1) `chroma_dense` never ran, because the override registry `overrides.py` shadowed chromadb's dependency `overrides` on `/work` (`ImportError: cannot import name 'overrides'`); the registry is now `override_registry.py`. (2) Memgraph's storage access timeout (default 1 s) refused the deep10m vector index build in both dense arms; the dense arms now run `--storage-access-timeout-sec=172800`, longer than any cell cap | Chroma dense (every scale) and Memgraph dense (both precisions); no other arm moves (the rename changes no measurement) | **built on branch issue/344-chroma-memgraph, not on main**: `override_registry.py` and every import, `runner.MEMGRAPH_STORAGE_ACCESS_TIMEOUT_S`, the read-back `memgraph_storage_access_timeout_s`, PROTOCOL.md section 7, COMPARATORS.md. Mini picks it up at the `git pull` of the stage after the merge, which is the user's to time; a repair stage reruns Chroma dense and Memgraph dense deep10m | `.notes/bench/repros/failure-triage-20261020` | +| 87 | Two failures of the 26.10.1 pass that were not results (humemai/arcadedb-embedded-python#344, the failure triage of 2026-10-20): (1) `chroma_dense` never ran, because the override registry `overrides.py` shadowed chromadb's dependency `overrides` on `/work` (`ImportError: cannot import name 'overrides'`); the registry is now `override_registry.py`. (2) Memgraph's storage access timeout (default 1 s) refused the deep10m vector index build in both dense arms; the dense arms now run `--storage-access-timeout-sec=172800`, longer than any cell cap | Chroma dense (every scale) and Memgraph dense (both precisions); no other arm moves (the rename changes no measurement) | **built, merged to main (humemai/arcadedb-embedded-python#345, 2026-10-20)**: `override_registry.py` and every import, `runner.MEMGRAPH_STORAGE_ACCESS_TIMEOUT_S`, the read-back `memgraph_storage_access_timeout_s`, PROTOCOL.md section 7, COMPARATORS.md. Mini picks it up at the `git pull` of the stage after the merge, which is the user's to time; a repair stage reruns Chroma dense and Memgraph dense deep10m | `.notes/bench/repros/failure-triage-20261020` | | 88 | Product quantization arms for the dense lane (humemai/arcadedb-embedded-python#416), default OFF: ArcadeDB embedded and served (`quantization: 'PRODUCT'`, 32 subspaces of 256 centroids, SQL `vectorNeighbors` at the engine's default re-rank), Qdrant product quantization x16 with rescore, Milvus `HNSW_PQ`, ArangoDB FAISS `IVF,PQ32x8`; LanceDB none (its `IVF_HNSW_PQ` trains a codebook per IVF partition, a different setting) | the dense lane (l3d), a separate "pq" row family beside fp32 and int8 | **mechanism written, gated: the arms exist only under `BENCH_DENSE_PQ=1` (`bench_common.dense_pq_enabled`; `test_dense_pq.py` holds the gate, the ArcadeDB `rerank` refusal and the page grouping).** Off until the measured ArcadeDB release carries the upstream PQ fix (ArcadeData/arcadedb#10545, merged for 26.11.1, carrying #10516 to #10519); both ArcadeDB arms probe `rerank` at connect and refuse a build whose SQL rejects it, so a pre-fix release cannot measure exact search under a PQ label. Rows stamp `quantization` PRODUCT, `pq_subspaces`, `pq_clusters`, `pq_rerank`, and the engine's read-back (`qdrant_quantization`, `milvus_index_type`, `ivf_factory`); export_web prints only PQ rows stamped instrument 2026-11. The settings and the laptop smoke are in QUANTIZATION.md. | the PQ rows only; with the switch unset nothing differs | | 89 | The dense lane sweeps the search beam on the built index and headlines latency at matched recall (humemai/arcadedb-embedded-python#425): `ef` 50, 100, 200, 400 for the HNSW engines, `nProbe` 0.5 to 4 times the calibrated value for ArangoDB, one pass for the engines with no beam; headline = lowest p50 whose recall@10 reaches 0.95 (0.90 and 0.99 secondary), with the winning beam shown, `not reached` when none does; the cold pass is unchanged | the dense lane (l3d), every arm; the table gains matched-recall columns and each entry a `beam_curve` | **built:** `dense_beam.py`, `l3d_dense.Base.set_beam`, `export_web._beam_metrics` and `_beam_extras`, `test_dense_ef_sweep.py`; PROTOCOL.md section 2, FAIRNESS.md, QUANTIZATION.md. No cell is skipped | a 2026-11 dense stage; the page's rendering of the curve is a follow-up | | 90 | The ArcadeDB version a cell runs is the stage's pin, read out of what will run before the first cell, and no ArcadeDB image tag floats (humemai/arcadedb-embedded-python#470). `runner.py` takes the pin from the `ARCADEDB_WHEEL` file name and, for every ArcadeDB arm of the batch, reads the version out of the image that will run (embedded: the installed `arcadedb-embedded` version and the engine jar the wheel ships; served: the engine jar of the server image) and refuses to start on any difference, or when the batch has an ArcadeDB arm and no pin; the readback is in the manifest (`engine_pin`). The embedded image is `dbbench:arcadedb-`, written by `build_images.sh`, and the runner refuses every other ArcadeDB tag (`dbbench:arcadedb` stays only as the logical build target name in `runner.BACKENDS`); the comparator images keep `dbbench:`, their engines being pinned by `==` or digest and read back by `version_pin_check.py --runtime` | the embedded and served ArcadeDB arms of every lane | built, verified by unit tests | `engine_pin.py`, `test_engine_pin.py`; the trigger: the laptop's `dbbench:arcadedb` still held 26.10.1 beside `dbbench:arcadedb-26102` at 26.10.2 (2026-10-22, `.notes/bench/repros/ts-doc-oom-20261022/`), and every earlier check read rows after the run | | 91 | The composed embedded stack joins the cross-model lane: SQLite (documents) + LanceDB (vectors, fp32 `IVF_HNSW_FLAT`) + LadybugDB (graph) in one Python process, the default trio of the Cognee-style agent-memory builders and the real alternative to ArcadeDB for that audience (the owner, 2026-10-25) | `composed_embedded_sqlite_lance_ladybug`, the `e2` lane (hybrid in both durability classes, and the atomicity workload) | **built and dry-run on the laptop (2026-10-25), untimed**: `ladybug==0.21.2` added to the dense image's package line (an existing pin, no new version; 0.22.0 since 2026-10-25, row 92), the arm in `runner.BACKENDS` and `LANES["e2"]`, an `INDEX_DECISIONS` entry, the generator's `composed-embedded` engine (Tier 1, after `qdrant-neo4j`) and the regenerated `stage_drafts_2611.sha256`. Both ends of the lane's checks pass on its rows: the retrieval, hop and filtered-candidate digests equal LadybugDB's and DuckDB's at e2, 40 of 40 atomicity trials torn (nothing spans the stores), F10 and F14 clean | `COMPARATORS.md` "The composed embedded stack"; `repros/composed-embedded-lance/probe.py` | | 92 | LadybugDB moves to 0.22.0 on the client and dense images, and its `vector` extension is baked into the client image at build time, pinned by sha256 | every LadybugDB arm: `ladybug_graph`, `ladybug_dense`, `ladybug_e2`, `composed_embedded_sqlite_lance_ladybug`, `ladybug_lifecycle` | **built (2026-10-25)**: `build_images.sh` (`ladybug==0.22.0`, `LADYBUG_EXTENSIONS`), `Dockerfile.bench`, `bake_ladybug_extensions.py`, `test_ladybug_pin.py`. 0.21.2 can no longer `LOAD vector` (the vendor overwrote the v0.21.0 download at 04:28 UTC on 2026-10-25, LadybugDB/ladybug#1390). Untimed laptop cells on 0.22.0: dense micro with the mutation pass (0.20.4 control dies at the re-insert, 0.22.0 completes), graph micro, e2 (`ladybug_e2` and the composed arm). Open: the fdatasync counts in `bench_common` were measured at 0.21.2 and are not re-measured on 0.22.0; the DuckDB extensions are not baked yet (COMPARATORS.md, "Runtime extensions") | PR on humemai/arcadedb-embedded-python; `.notes` repros/ladybug-vector-ext-20261025 | -**What the 2026-11 instrument measures (DECISIONS #196).** The harness measures one instrument and every method choice named here is the behaviour, not a switch. (1) The sparse lane warms up every engine on queries apart from the timed set, and the runner holds the CPU idle-exit latency at 10 microseconds for each timed cell (row 81); every timed cell of the pass also starts behind a pre-cell temperature gate (PROTOCOL.md section 2). (2) The dense lane indexes the vector id for the delete on every engine that would scan (row 84). (3) The served ArcadeDB headline is the official `arcadedb-driver`, with the lean HTTP client as the twin sensitivity row (rows 72, 82). (4) The l4 queries with large answers read each engine on its documented bulk path: for `q_high` and `q_groupby` ArcadeDB embedded `to_arrow()`, DuckDB `to_arrow_table()`, SQLite `fetchall()` (no columnar path), with `to_list()` / `fetchall()` timed beside it as `q_high_plain_ms` and `q_groupby_plain_ms`; rows stamp `read_path`, `read_path_queries`, `read_path_plain`, and `read_path_sensitivity`; small results do not move; a missing pyarrow or bridge jar falls back to `to_list()` and the row says `read_path_arrow_missing`; the typed read's digest must equal the plain read's after the lane's declared coerce, and a mismatch ends the cell. (5) The ArcadeDB arms, embedded and served, leave `-Xms` unset (the bindings' documented practice and the maintainers' guidance for a fixed memory budget; DECISIONS #200 for the served arms) and keep `-Xmx` from the cell's memory budget; Elasticsearch and Neo4j keep the initial heap they have (PROTOCOL.md section 7, "JVM initial heap"). `BENCH_ARCADEDB_TS_NDJSON` stays off (row 85, measured mixed). Every row stamps `instrument` = `2026-11`; a `BENCH_INSTRUMENT` of anything else is refused, the runner forwards its own value to every cell so a container cannot disagree with the launcher, and `test_instrument_method.py` holds the list. The 2026-10 and September instruments are retired: no cell is measured with them again. +**What the 2026-11 instrument measures (DECISIONS #196).** The harness measures one instrument and every method choice named here is the behaviour, not a switch. (1) The sparse lane warms up every engine on queries apart from the timed set, and the runner holds the CPU idle-exit latency at 10 microseconds for each timed cell (row 81); every timed cell of the pass also starts behind a pre-cell temperature gate (PROTOCOL.md section 2). (2) The dense lane indexes the vector id for the delete on every engine that would scan (row 84). (3) The served ArcadeDB headline is the benchmark's own lean HTTP client (`lean_http.LeanSession`, row 82); the vendor driver's rows are set aside at load and never printed (rows 72, 82). (4) The l4 queries with large answers read each engine on its documented bulk path: for `q_high` and `q_groupby` ArcadeDB embedded `to_arrow()`, DuckDB `to_arrow_table()`, SQLite `fetchall()` (no columnar path), with `to_list()` / `fetchall()` timed beside it as `q_high_plain_ms` and `q_groupby_plain_ms`; rows stamp `read_path`, `read_path_queries`, `read_path_plain`, and `read_path_sensitivity`; small results do not move; a missing pyarrow or bridge jar falls back to `to_list()` and the row says `read_path_arrow_missing`; the typed read's digest must equal the plain read's after the lane's declared coerce, and a mismatch ends the cell. (5) The ArcadeDB arms, embedded and served, leave `-Xms` unset (the bindings' documented practice and the maintainers' guidance for a fixed memory budget; DECISIONS #200 for the served arms) and keep `-Xmx` from the cell's memory budget; Elasticsearch and Neo4j keep the initial heap they have (PROTOCOL.md section 7, "JVM initial heap"). `BENCH_ARCADEDB_TS_NDJSON` stays off (row 85, measured mixed). Every row stamps `instrument` = `2026-11`; a `BENCH_INSTRUMENT` of anything else is refused, the runner forwards its own value to every cell so a container cannot disagree with the launcher, and `test_instrument_method.py` holds the list. The 2026-10 and September instruments are retired: no cell is measured with them again. **Retired: settled with no change, listed so nobody re-opens them.** Raising ArcadeDB's statement-cache settings (a knob only ArcadeDB would get; item 2 removes the literal path instead). The sparse ingest path (already the faster one). Durable commit cost (one sync per commit, equal to SQLite's; not a defect). `expected_edge_count` as a speed lever (no measurable effect; item 1 adopts it only because it is the documented setting). **Done, and off the list:** the dense table's phases partition their total (`setup_s`, since 2026-09-22, BUGS F101), every stage estimates its whole cell against its cap before running (`cell_cost_check.py`, BUGS F74), one engine wears one version per page (`version_consistency_check`, a publish gate, BUGS F61), and `queue_lint` refuses the process-name wait that only holds for one exact launch line (BUGS F59; every stage waits on its predecessor's ALL-DONE marker). @@ -239,14 +239,14 @@ The detail behind item 8: * **The sparse corpus generator should yield NumPy** (measured, NOT changed). `bigann_sparse._stream_csr` calls `.tolist()` on every document, so ArcadeDB's sparse arm converts NumPy to a list inside the timer and hands JPype a list. Handing it the array instead measured **3.7% of the whole embedded build** (24.93 s to 24.01 s, 100,000 documents at 120 non-zeros) and moves no comparison (at 1M documents ArcadeDB already builds in 56.7 s against Milvus 99.2 s and Qdrant 92.3 s), which is not worth changing a generator shared by six adapters and three other scripts in front of a multi-day stage. At the re-pin: yield NumPy, have each comparator call `.tolist()` in its own adapter (the same work, still inside its own timer), and leave `sparse_common`'s synthetic generator alone, which builds Python lists natively. -**ONE LINE OF DEVELOPMENT.** The instrument is developed on `main`, never on a long-lived branch (BUGS F65): mini pulls `main` at every stage start, so the instrument is landed on `main` before the first cell and never during the run, and a change that must not reach a running campaign waits for the next one. Anything else lands on `main`, behind a flag if it must, or waits. A later instrument is added the same way: `BENCH_INSTRUMENT` (`bench_common.INSTRUMENT`, default `2026-11`) names it, and code for it asks `instrument_at_least(...)` (`test_instrument_selector.py`). +**ONE LINE OF DEVELOPMENT.** The instrument is developed on `main`, never on a long-lived branch (BUGS F65): the generated stages `git pull -q --ff-only` `main` when they start (`make_2611_stages.py`), so the instrument is landed on `main` before the first cell and never during the run, and a change that must not reach a running campaign waits for the next one. The live 2026-11 chain does not all follow `main`: it runs from pinned detached worktrees on the bench host (the four-arm ArcadeDB pass from `~/fourarm-wt/`, whose generator changes are in the unmerged draft PR #554), and only the comparator stages from qC49 on pull `main` at their start. Anything else lands on `main`, behind a flag if it must, or waits. A later instrument is added the same way: `BENCH_INSTRUMENT` (`bench_common.INSTRUMENT`, default `2026-11`) names it, and code for it asks `instrument_at_least(...)` (`test_instrument_selector.py`). **The next campaign's stages are generated by `make_2611_stages.py`.** One template carries every guard the campaign learned (cap table, shadowed definitions, budgets against the cap, whole-cell cost, images present and current, comparator digests frozen at generation, the JPype pin read out of the image, the pinned wheel by sha256, the cap rule, the ALL-DONE trap); the instrument, `REPS` and the banner are its parameters. The generator selects the multi-model subset, narrowed by a profile (next paragraph). The subset is `ENGINES` in the file: ArcadeDB (embedded and served), SurrealDB (embedded and served), ArangoDB, Memgraph, PostgreSQL with pgvector, AGE and TimescaleDB (and plain PostgreSQL where a lane uses it), MongoDB with MongoDB Search, DuckDB with VSS and DuckPGQ, SQLite with sqlite-vec, the composed Qdrant plus Neo4j stack, Neo4j and Qdrant on their own (graph and dense; sparse on the second list), and LadybugDB on every lane it supports. Every registered arm is either in the subset or in `OUTSIDE`, so a newly registered arm forces a decision (`--check`). The core lanes are graph (interactive and analytics), cross-model, documents and dense; under `--profile full`, `--second-list` adds lifecycle, then sparse and time series, after them. TimescaleDB runs whenever the time-series lane does (so not under the default profile); `--without-timescaledb` leaves it out. **THE PAPER PROFILE (the owner, 2026-10-25: the paper's experiments must finish by 2026-11-10; amended 2026-11-03, DECISIONS #232 and #235: the paper's numbers land about 2026-11-15, and the submission is 2026-11-24 for the SIGMOD 2027 industrial track or 2026-12-01 for the ICDE 2027 industry track, the fallback).** `--profile paper-2611` is the default and narrows the subset in the open (`PROFILES`, `DROP_WHY` in the file). (1) Core lanes only: graph, cross-model, documents and dense. The second list (lifecycle, sparse, time series) is out of scope for the paper, for the comparators and for the ArcadeDB chain, and `--second-list` is refused unless `--profile full` is given. (2) Neo4j and Memgraph stay in the graph lane (and cross-model where they are); their dense-lane cells (`neo4j_dense`, `neo4j_dense_int8`, `memgraph_dense`, `memgraph_dense_int8`) are out. The composed embedded stack (SQLite + LanceDB + LadybugDB in one process, `composed_embedded_sqlite_lance_ladybug`, the owner, 2026-10-25) is a Tier 1 cross-model arm: engine `composed-embedded`, right after the served composed stack, six lane cells per pass (hybrid in both durability classes and atomicity, at e2 and e2_500k), on the lane `e2` only; it runs in the dense image, so rebuild `dbbench:dense` (`./build_images.sh dense`) before the first pass. (3) The dense lane is fp32 only, for every engine: no `_int8` and no `_pq` arm, the comparators' and ArcadeDB's alike (`arangodb_dense_int8`, `arcadedb_dense_embedded_int8`, `arcadedb_dense_server_int8`, `mongodb_dense_int8`, `qdrant_dense_int8`, `sqlite_vec_dense_int8`, and the PQ arms when `BENCH_DENSE_PQ=1`). Quantized arms are not like-for-like across engines, and quantization is out of the paper's scope: fp32 for every engine, ArcadeDB's own INT8 and BINARY out by the owner's rule, jvector PQ, BQ and NVQ studied after the paper. ArcadeDB keeps its fp32 dense arms, embedded and served. (4) The target is N=3 (`TARGET_REPS`; `--project` prices 3 repetitions by default). The chain keeps the passes 4 and 5 after pass 3, in #503's order, so they run only if time is left; stopping after the first six groups is a complete result. Since 2026-11-03 the order on the bench host is: pass 3 (qC48), then the re-runs (qC14 at turbo off, qC15..qC23 at the 10 us idle hold, DECISIONS #234), then the four-arm ArcadeDB pass on the released 26.11.1 wheels, then passes 4 and 5 (qC49..qC80, which wait for `PASS45.go`). Amended 2026-11-05: if the 26.11.1 wheels are on PyPI and the four-arm runner is deployed when `REORDER reruns finished` is logged in the host's `STATUS.txt`, the four-arm pass (n=3 for the four ArcadeDB arms) starts first, and the window re-runs of qC14..qC17 follow it, still before `PASS45.go`; otherwise the order above stands. So the paper's comparators are N=3 for sure and N=5 where passes 4 and 5 finish before the tables freeze (DECISIONS #232, amendment of 2026-11-03). `--check` prints every dropped cell with its reason and count (lane cells per pass: 148 second-list, 8 Neo4j and Memgraph dense, 12 quantized dense) and proves that none is scheduled, in both chains whatever the arguments select (`profile_problems`); `--check --drafts DIR` lints written scripts for the same. `--profile full` is the whole subset as it was before the profile. Every generated stage exports `BENCH_INSTRUMENT=2026-11` before its first Python call, reads the selection back from `bench_common` and aborts if the tree does not select it, writes `runs_page__2611.jsonl` (and overlay directories with the same suffix) so one rows file never holds two instruments, and fixes `REPS` at generation (`--reps`, default 5, at most 5; the target is 3). The generator re-executes itself with the variable set and refuses a different value in the environment. `land_stage.py` lands exactly these files (DECISIONS #196; the 2026-10-25 change that closed the gap between the pull and the freeze): it pulls `runs_page__2611.jsonl` and the overlay directories with the same suffix, and `screen_pulled_rows` refuses the file before anything is merged unless every row is stamped `2026-11` and names the pin's engine commit, so a retired 2026-10 file or a file that mixes instruments never reaches `results/runs.jsonl`. The first landing meets no store (the October one is retired) and `merge_campaign.py` creates it. `export_web` no longer demands the `e4decomp_` artifacts of a landing scoped away from e4, because the chain has no e4 stage and `--only-lanes l2` would otherwise wait for one. A rehearsal restores what it wrote, the site payload (removed when the run created it) and the PAGE-SPEC.md block included. Failed cells (`rc != 0` or an error) merge as failures and are not frozen; censored queries keep their flag and publish no number. No stage script calls `land_stage.py` (the generated `qC` scripts only name it in a comment about the rows file); a landing is run by hand on the laptop. The comparators are the default mode and run on the release pair already installed (`ARCADEDB_WHEEL`, `ARCADEDB_SERVER_IMAGE`, `ARCADEDB_ENGINE_COMMIT`): `python3 make_2611_stages.py --after --status-file STATUS.txt --id-prefix qS --out DIR`. `--after` is required, with no default (a marker name, or `none`), and the first stage waits for `grep -qx " ALL-DONE"`, each later one for the stage before it. The marker must be the LAST item still to run, one that writes its line whether it runs or is skipped: a marker already in the machine's STATUS.txt makes the first stage start at once, beside the running jobs (2026-10-23: `qMA06 ALL-DONE`, written by a skipped study). So `--status-file` is required with a marker (a copy of that STATUS.txt, or `none` when there is no queue) and the generator refuses a marker the file already holds; `--check --drafts DIR --status-file STATUS.txt` makes the same test on drafts that exist. The subset is emitted in ENGINE TIERS in the user's exact pass order (2026-10-24; `--tiers`, on by default, `--no-tiers` for the plain rep-major chain). N defaults to 5, the target is 3 (the paper profile), and the order is `schedule_order`: n=1 T1, n=2 T1, n=3 T1, n=1 T2, n=2 T2, n=3 T2, n=4 T1, n=4 T2, n=5 T1, n=5 T2. Every prefix of it is a complete set of equal replicates for the tiers covered so far (each tier has run passes 1..k, none skipped, and a pass is whole before the next starts); `--reps 3` stops after the first six groups, which is the chain of Tier 1 passes 1-3 and then Tier 2 passes 1-3 as before. Tier 1 is ArcadeDB (the four 26.11.1 arms, when `--arcadedb-release` adds them), SurrealDB, ArangoDB, PostgreSQL with pgvector and AGE, MongoDB with Search, the composed Qdrant plus Neo4j stack, and the embedded references SQLite, LadybugDB and DuckDB; Tier 2 is Memgraph, Neo4j and Qdrant on their own, Redis 8, CrateDB and TimescaleDB (`TIER_ENGINES`, with `ARM_TIER` for TimescaleDB, an arm of the `postgresql` engine that the user lists in Tier 2). Every stage holds the arms of one tier, and stage ids stay one run of consecutive numbers (a stage that mixed both tiers is split in two, so the stage count grows, and each stage repeats its guards and image checks; past 99 stages the ids take three digits, so `qS100` cannot sort before `qS11`). `--check` prints the schedule and proves the order and the prefixes (the plan collapsed into (tier, pass) groups equals `schedule_order`, each group holds the whole pass), and `--check --drafts DIR` lints a draft folder for the same. An arm with no tier runs after Tier 2 in each block. The ArcadeDB arms are Tier 1 but not in this chain: DBBench keeps one ArcadeDB version per host, so they run only once the benchmark pin is 26.11.1 (next paragraph). Inside a tier the order is the one below: a pass is pass k of every stage of the tier (core lanes, then the second list), and each stage runs ONE repetition (`--only-reps $PASS`). With `--no-tiers` the chain is pass 1 of every stage, then pass 2, up to pass N. If time runs out, every engine of a tier has the same n; repetitions days apart show drift. A stage whose environment declares fewer reps (`REPS=k`) is left out of the later passes. Inside a lane the engines run in `ENGINE_PRIORITY` (SurrealDB, ArangoDB, PostgreSQL, MongoDB, Redis 8, composed Qdrant+Neo4j, Neo4j, Qdrant, DuckDB, SQLite, Memgraph, LadybugDB, CrateDB; ArcadeDB's arms lead it when they join; Redis and CrateDB are skipped until their adapters are in `ENGINES`). The rep-1-first rule and the cap rule hold across passes: a pass above 1 runs a cell only when rep 1 of it was clean, and only a timeout of rep 1 censors the larger sizes. WHICH REPS EXIST is `~/rep-state-2026-11-.tsv` on the host (one `label, rep, ok|fail` line per finished repetition; the rows themselves carry the rep number in the runner's results file): stop after any pass, and the next pass, or a restarted stage, picks up from it with no regeneration; N can go to 3 or stop at 2. Delete that file to start a chain over. The ArcadeDB pass is `--arcadedb-release X.Y.Z` with `--engine-commit`, `--image-digest sha256:...`, `--wheel-name`, `--wheel-sha256` and `--wheel-url`; it refuses unless all five are given, because the release may not exist when the drafts are made. It writes `00`, the pair install (official image by digest, wheel by sha256, `build_matched_pair.sh`, `verify_pair_c25.sh`), and ArcadeDB's lane stages behind it. THE ARCADEDB GATE: the ArcadeDB chain is its own chain (Tier 1 only, passes 1..N in order), generated with `--after` the last comparator stage, so the comparators run the schedule first. `00` waits after its `--after` for the marker file `$HOME/conf/go-arcadedb-2611` on the host (it never creates it; it sits before the ALL-DONE trap, so a killed waiter hands nothing on), and every ArcadeDB lane stage aborts without it. The user or the agent touches that file when `CURRENT_PIN` moves to 26.11.1, and not before; `--check --drafts` lints that the gate is in the pair install and in every ArcadeDB stage and in no comparator stage. The ArcadeDB pair's guards are split by what a stage runs. Every stage exports the pinned wheel (`ARCADEDB_WHEEL`, with `ARCADEDB_SERVER_IMAGE` and `ARCADEDB_ENGINE_COMMIT`) before `build_images.sh`, which bakes it into every generic image (client, duckdb, dense), and so every stage guards that the file exists at the pinned path and has the pinned sha256; `build_images.sh` copies it from that path and nothing in it or the Dockerfiles reads `dist/` itself. Only a stage that runs an ArcadeDB arm (the `--arcadedb-release` pass) also guards the rest: exactly one wheel in `dist/` (it protects `verify_pair_c25.sh`), the server image, the wheel and JPype read back out of `dbbench:arcadedb`, and `verify_pair_c25.sh`. A comparator stage on a host whose `dbbench:arcadedb` is the previous release's therefore runs. `build_images.sh` has no PyPI fallback: it refuses to build the `arcadedb` image (named or by default) without `ARCADEDB_WHEEL`, and refuses a wheel that is not a file, and refuses a wheel whose `arcadedb_embedded/core.py` lacks `_INSERT_MANY_CHUNK` (`wheel_guard.py`: before 26.10.2 `insert_many` parsed the whole input into one Java array, which the kernel killed on l4/ts1000/arcadedb_ts_doc on 2026-10-08; use a 26.10.2 or newer wheel); the images that take no wheel can be built alone by naming them. Every ABORT hands on: the `ALL-DONE` trap is set right after the wait, before the first guard, so a stage that aborts before its START line still writes its marker and the next stage runs (its own guards then run too). The ABORT line is in `STATUS.txt`, where `mini_watch.sh` reports it; to resume, fix the cause and run the aborted stage's script again by hand after the chain, and landing reads only the rows that exist. `make_2611_stages.py --check --drafts DIR` lints both rules. -The default order is the core block first, then (under `--profile full`) the second list. Inside the core block the large tiers (deep10m, tpch10, the full SF1 network) run after the small ones, and the cells that dominate the hours (Memgraph's deep10m dense cells, which the paper profile does not stage, SurrealDB's documents at tpch1, the full SF1 network's analytics) run last of all in stages of their own, where they are easiest to drop if time runs short; `BIGGEST` in the file names them, and `--order paper` keeps the table order. Splitting moves stage boundaries only, and `--check` proves the cells are the same. `--project TABLE` prints projected hours per drafted stage from a per-cell hours table (tab separated: lane, workload, scale, backend, class, hours at 3 repetitions, hours at 5 repetitions; a cell with no clean rep carries its one failed attempt). `--check --drafts DIR` lints written scripts: the instrument exported and never 2026-10, the 2611 rows file, only subset arms, each stage waiting for the one before it. Drafts are written to a folder outside the tree and copied to the bench host by hand on the owner's go. Which stage is running is state, not method: it is in `STATUS.txt`'s tail and in HANDOFF, never here. `test_stage_generator_2611.py` holds the written drafts byte for byte against `stage_drafts_2611.sha256`, so a stage cannot move without the list changing in the same commit. +The default order is the core block first, then (under `--profile full`) the second list. Inside the core block the large tiers (deep10m, tpch10, the full SF1 network) run after the small ones, and the cells that dominate the hours (Memgraph's deep10m dense cells, which the paper profile does not stage, SurrealDB's documents at tpch1, the full SF1 network's analytics) run last of all in stages of their own, where they are easiest to drop if time runs short; `BIGGEST` in the file names them, and `--order paper` keeps the table order. Splitting moves stage boundaries only, and `--check` proves the cells are the same. `--project TABLE` prints projected hours per drafted stage from a per-cell hours table (tab separated: lane, workload, scale, backend, class, hours at 3 repetitions, hours at 5 repetitions; a cell with no clean rep carries its one failed attempt). `--check --drafts DIR` lints written scripts: the instrument exported and never 2026-10, the 2611 rows file, only subset arms, each stage waiting for the one before it. Drafts are written to a folder outside the tree and copied to the bench host by hand on the owner's go. Which stage is running is state, not method: it is in `STATUS.txt`'s tail and in `.notes/bench/START-HERE.md`, never here. `test_stage_generator_2611.py` holds the written drafts byte for byte against `stage_drafts_2611.sha256`, so a stage cannot move without the list changing in the same commit. **What a stage costs is projected from its components, never from an uplift.** Multiplying the previous campaign's cells by an assumed per-lane uplift goes wrong in both directions: a larger query set lengthens a stage several times over, and a stage that runs fewer workloads than the cells it is priced from runs shorter. Project each cell from its measured ingest, first touches, and budget ceiling, as `cell_cost_check` does, and replace a stage's projection with its measured cost once it runs. The current projection lives in HANDOFF. diff --git a/benchmarks/experiments/PAGE-SPEC.md b/benchmarks/experiments/PAGE-SPEC.md index d00a82b7e40..6f1ae961dff 100644 --- a/benchmarks/experiments/PAGE-SPEC.md +++ b/benchmarks/experiments/PAGE-SPEC.md @@ -83,7 +83,7 @@ Comparators are pinned by image digest and carry a version name, recorded per ro **A dash means that and only that (DECISIONS #111).** An engine that RAN and did not finish keeps its row and prints the outcome in the cell: `>2h` for the tier's cap (derived from `runner.TIMEOUT_BY_SCALE`, so `>15m` through `>48h`), `OOM` for an envelope failure (the kernel's kill at the cell's memory cap, or the engine's own out-of-memory error inside that cap, the same for every engine), `lost` for a dropped connection, `err` for a failure inside budget whose cause will not classify, `wd` for a withheld wrong answer, `re-run` for a cell measured before a fix to our own harness and being measured again. A vanished row reads as an untested engine, which flatters whichever engines finished. The legend is generated from the marks present and refuses one it cannot define. A harness failure never gets an engine mark, and a cell skipped because we expected it to be slow is published as its bound or not at all, never as a blank. The cap each cell runs under, and the censoring that follows from it at larger sizes, are the cap rule of PROTOCOL.md section 2 (DECISIONS #188); `export_web` applies it at report time to every row. The queue chain is CAMPAIGN.md section 6. -**`re-run` marks our own mistake, and clears itself (DECISIONS #117).** A cell or a row measured before a fix to this harness stays on the table with `re-run` where its number would be, plus a generated note saying what was wrong. A marked cell carries text rather than a median, so nothing that ranks or averages reads it as a measurement. `export_web.STALE_UNTIL_RERUN` (and `LIFECYCLE_STALE` for the lifecycle table, which folds by situation and size) lists each case as a table, an engine, a column (or none), and a test that recognises a row measured BEFORE the fix. It is keyed on row fields, never on a date: the re-measured rows supersede the old ones under the canonical-row rule and fail the test, so the numbers return at the landing that carries them with no list to edit. Each case names the BUGS entry for its fix, and a row it withdraws is kept out of the censored-cell notes, so our kill never prints as the engine's. `_rows_behind` reads a table whose lane has one workload (mapped to `None`) as any workload. +**`re-run` marks our own mistake, and clears itself (DECISIONS #117): removed.** The exporter's list of cases and the test that recognised a row measured before a harness fix were deleted with #432 (the publish side renders 2026-11 rows only), and nothing marks a cell `re-run` any more. A row measured before a fix is superseded by its re-measured row under the canonical-row rule. **Sizes (DECISIONS #108).** Each table carries TWO sizes wherever a second one is affordable and informative, because one point cannot tell a query that moved elevenfold from one that did not move. The large size is a ROW GROUP beside the small one, not a switch, and an engine with no row at it is a declared outcome under #103g (censored, envelope failure, or failed inside budget), never a reason to withhold every other engine's rows. Graph analytics is the single one-size exception, the full SF1 network, because the full network at SF10 is not feasible. No table accumulates two generations of ONE size and no new table is made for a size. diff --git a/benchmarks/experiments/PROTOCOL.md b/benchmarks/experiments/PROTOCOL.md index 72f241fe078..07997d2c090 100644 --- a/benchmarks/experiments/PROTOCOL.md +++ b/benchmarks/experiments/PROTOCOL.md @@ -35,7 +35,7 @@ The admission filter is `make_paper_tables.load_canonical()`. Every test below r ## 2. How a cell is measured - Published latency, throughput, percentile and memory cells run serial, one at a time, on the full cpuset. Permutation protects a ratio; it cannot restore an absolute level or a tail that never happened. **GATE** `runner.py` paper tier refuses `workers != 1` -- Every row names the instrument it was measured under (`instrument`, from `bench_common.INSTRUMENT`, which is `2026-11`; any other `BENCH_INSTRUMENT` is refused) and the machine (`bench_host`, refused at paper tier when unset). Rows from two instruments never share a table: a five-query analytics row beside a two-query one, or a relaxed-durability write beside an fsync one, is not a comparison. **GATE** `load_canonical`, `export_web._instrument_of`, `fairness_check.check_durability` +- Every row names the instrument it was measured under (`instrument`, from `bench_common.INSTRUMENT`, which is `2026-11`; any other `BENCH_INSTRUMENT` is refused) and the machine (`bench_host`, refused at paper tier when unset). Rows from two instruments never share a table: a five-query analytics row beside a two-query one, or a relaxed-durability write beside an fsync one, is not a comparison. **GATE** `load_canonical`, `publish_instrument.instrument_of`, `fairness_check.check_durability` - **Every timed cell of the 2026-11 pass starts behind a pre-cell temperature gate.** With one busy core the package throttles continuously at 89 C, so a cell's speed depended on where the previous cell left the box. After the skip checks and before the runner starts, `run_cell` and `run_overlay` of a generated stage wait until the coretemp package (found by name) is at or below 70 C, poll every 10 s, wait at most 15 min, and then start the cell anyway. A wait from a hot start is long: the first cell of the restarted pass waited 341 s from 83.0 C. The outcome goes to the stage's STATUS log ("TEMP GATE") and into the row: `precell_gate_temp_c` (the package temperature when the cell was let go), `precell_gate_wait_s`, `precell_gate_ok` (false: the cap passed with the package still above the limit, or no sensor could be read) and `precell_gate_max_c` (the limit). A cell run without the gate carries none of the four. The pass was restarted twice for the two amendments of 2026-10-26 (hold 10 us instead of 0, then this gate), and the rows measured before each (22 under hold 0, 3 without the gate) were moved aside on the bench host and are not published. **GATE** `temp_gate` in the stages `make_2611_stages.py` writes (it delays a cell and cannot refuse one, so a cell past the cap is flagged on its row and not dropped), `runner.precell_gate_stamp` for the fields (DECISIONS #177, amendment 2026-10-26) - **Host condition: compare engines within one pass, not across passes.** The bench host is a mobile-class part in a small chassis. It is thermally limited and throttles under sustained load, and its speed drifts between passes: a November A/B arm of one LadybugDB cell took x1.56 (hold 10) and x1.61 (no hold) the CPU time that cell recorded in the October pass (qP10; DECISIONS #177, amendment 2026-10-26). The gate starts every cell with the package at or below 70 C and the hold keeps the cores out of the deep idle states, so the cells of one pass start from the same condition; neither undoes the drift between passes. So compare engines only against rows of the same pass. A ratio of a November number over an October one carries the host's drift, so it is neither an engine effect nor a version effect. **REMEMBERED**; the thermal fields, `cpu_dma_latency_*` and `precell_gate_*` on every row show the condition a cell ran in (READING-RESULTS.md) - **Rows measured under a condition this method forbids are excluded at landing, never edited.** A committed window list (`excluded_windows.json`, applied by `land_stage.py` step 2) drops every row whose `ts_utc` falls in a window and keeps the raw rows. The windows of the 2026-11 pass: (1) qC14, 2026-10-31T16:33Z to 2026-11-01T05:25Z, when turbo came back on after the stage's start gate (DECISIONS #228; qC14 is re-run with turbo off). (2) 2026-11-01T05:26:21Z to the start of qC24, when the sequential relaunch ran the stages at idle-state hold 0 us instead of 10 us: the relaunch scripts ran `bash qCnn.sh` directly and bypassed `launch.sh`, which exports `BENCH_CPU_DMA_LATENCY_US=10`; the hold is restored from qC24 (the stage scripts export it themselves) and qC15..qC23 are re-run at 10 us (DECISIONS #234); the window is entered in the list when qC24 starts, so that its end is exact. It also covers (3), the 65 s of 2026-11-03T09:44:20Z to 09:45:24Z in which turbo flipped on again inside qC22. A stage's gate reads `no_turbo` at the stage's start only, so a flip inside a stage is caught by the host watcher and answered with a window. The dense large-tier (deep10m) rep 1, qC06, ran slow for thermal reasons that no throttle counter shows (no clock sample exists for that stage; `repros/dense-rep1-20261103`), so it is not a speed reference for the dense large-tier build, index or query columns. **GATE** `land_stage.py` step 2 with `excluded_windows.json` (`test_land_stage_windows.py`); the rest is **REMEMBERED** @@ -125,7 +125,7 @@ Every row carries these fields. The code is `memory_probe.py`, and `bench_common - Never recompute a claim your own way: `import make_paper_tables` and ask `load_canonical()`. **HAND-RUN** `claims_check` - Pin prose to a generated table cell (prose to table to data), address cells by meaning (system, tier, metric) and never by column index, and parse the k/M suffixes when reading a cell back. **HAND-RUN** `claims_check.sparse_cell`, `cell()` - Verify separately that the committed tables regenerate byte-identically from the data (generate into a temp dir and diff). Otherwise every cell-pinned claim passes against a stale file. **HAND-RUN** `claims_check --regen` -- Tolerance is the number's own printed rounding, half a unit in the last digit, so a wider window cannot pass a number the table no longer prints. **GATE** `page_check._check_prose`, which derives it as `0.5 * 10 ** -decimals`; **REMEMBERED** for `claims_check`, where tolerance is a hand-typed third tuple element and several are an order of magnitude past half a unit +- Tolerance is the number's own printed rounding, half a unit in the last digit, so a wider window cannot pass a number the table no longer prints. **GATE** `page_check._pin_check`, which derives it as `0.5 * 10 ** -decimals`; **REMEMBERED** for `claims_check`, where tolerance is a hand-typed third tuple element and several are an order of magnitude past half a unit - A selector that returns no rows, or raises, is a FAILURE, never a skip. **HAND-RUN** `claims_check` NODATA/ERROR paths - Pin ranks as well as values, and fail loudly when a name stops resolving; never let a lookup fall back to a sentinel. **HAND-RUN** `_dense_rank` raises on unresolved `DENSE_ROWS` - Pin a ratio by recomputing it from its operands, and pin the endpoints of any range stated as one summary number. Both operands can be right and the quotient stale. **HAND-RUN** `claims_check` ratio selectors @@ -163,7 +163,7 @@ Every row carries these fields. The code is `memory_probe.py`, and `bench_common ## 5. How the checkers themselves must behave -- A check that could not run is carried to the summary as NOT CHECKED, not as a pass. **GATE** `_CAPTION_CHECK_RAN` and its warning, and `page_check._check_prose` returning `(0, 1)`; **REMEMBERED** for the rest, which is most of them +- A check that could not run is carried to the summary as NOT CHECKED, not as a pass. **GATE** `_CAPTION_CHECK_RAN` and its warning, and `page_check._check_page_atomicity` returning `(0, 1)` when its results log is missing; **REMEMBERED** for the rest, which is most of them - Handle both artifact shapes (object and array of per-pass records), and count an unreadable file as a finding rather than skipping it. **GATE** `_versions_in`, `run_conditions` - Search for a version at every nesting level, since a key nested under `run_conditions` is invisible to a top-level scan, and reject populated but uninformative values ("unknown...", "?", "n/a", anything with no digit). **GATE** `_nested_version_keys` and `_is_real_version`, neither of which the overlay audit calls - Exclude sidecars (`_buildstats.json`, `_gc.json`, `_manifest.json`) from version and condition audits. A checker that cries wolf is a checker nobody runs. **GATE** `SIDECAR_SUFFIXES`