benchmarks: the PostgreSQL-wire twin of the served SQL cells, with an answers-identical gate

benchmarks/experiments/CAMPAIGN.md

@@ -150,7 +150,7 @@ Rules every campaign keeps:
 | 3 | Dense ingest through `graph_batch(use_wal=True)` with Java float arrays | ArcadeDB only | **built (DECISIONS #116; 1.4x on the laptop)**: `l3d_dense.py` | DECISIONS #116 item 3 |
 | 4 | The served document and vector arms load through `INSERT ... CONTENT :rows` with the batch bound as one parameter | ArcadeDB only | built (DECISIONS #116, ArcadeData/arcadedb#8337): `l1_tpc.py` (`ArcadeServerTPC.load_batch`, `BENCH_SERVED_LOAD_BATCH`), `l3d_dense.py`, `l3_sparse.py`; parity, since every comparator loads through its vendor's bulk path. **No batch-size sweep is recorded** | `.notes` repros/served-doc-ingest; BUGS F151, F152 |
 | 5 | The strict durability setting is ArcadeDB `txWalFlush=1`, matched by effect to SQLite `synchronous=FULL`, not `txWalFlush=2` | ArcadeDB only | built (the user, DECISIONS #147; own issue #138): `bench_common.ARCADE_STRICT_TX_WAL_FLUSH`, `test_strict_durability.py`; the October string is kept for rows measured at 2; PROTOCOL section 7 has the row | `concepts/transactions.adoc` in upstream's docs; strace of the three settings on the laptop, 2026-09-24 |
-| 6 | Served SQL arms over the Postgres wire (psycopg) instead of HTTP (`requests`) | ArcadeDB only | unblocked: #8288 fixed in ArcadeData/arcadedb#8308, and pgjdbc's default prepare in #8315 (#8244), both merged for 26.10.1 and verified; **decided 2026-09-26 (DECISIONS #116): measure on the bench host at the re-pin and switch if it is the faster read path** (the laptop cannot time below a millisecond; the maintainer already says it is, on #8332). **Still open:** the served arms read through the lean HTTP client (row 72) now, and the comparison on mini is still to run (DECISIONS #162 item 3) | `.notes` repros/protocol-latency |
+| 6 | Served SQL arms over the Postgres wire (psycopg) instead of HTTP (`requests`) | ArcadeDB only | unblocked: #8288 fixed in ArcadeData/arcadedb#8308, and pgjdbc's default prepare in #8315 (#8244), both merged for 26.10.1 and verified; **decided 2026-09-26 (DECISIONS #116): measure on the bench host at the re-pin and switch if it is the faster read path** (the laptop cannot time below a millisecond; the maintainer already says it is, on #8332). **Built for the 26.11.1 four-arm pass (DECISIONS #166, #171, #237 (3)):** the served SQL cells run a third time as a twin job over the wire, ArcadeDB tier 1 only (DECISIONS #166) and at the small size tier only: no twin at `runner.LARGE_SCALES` (deep10m, tpch10, sf1full), because the twin re-runs the whole cell including load and index build (`runner.PGWIRE_SQL_ARMS`, run_id ending `_clientpgwire`, `lean_http.PgSession` over psycopg, never prepared, autocommit), a labelled sensitivity row and never a headline (it flatters ArcadeDB most, #166 addendum 2). Cells: l1 and l1tpc `arcadedb_server` (oltp, olap), l3s `arcadedb_sparse_server` and `_fp32`, l3d `arcadedb_dense_server` and `_int8`, l4 `arcadedb_ts_doc_server` and `arcadedb_ts_native_server`; not l2 (Cypher stays on HTTP, #162), e2, lifecycle, restart, e4. Only `/query` SQL goes over the wire: every write, DDL, transaction, bulk load and probe stays on HTTP (#171 (c)), so a twin row's write-phase numbers are lean HTTP and are not quoted. The twin's server starts the postgres plugin with explicit settings (`runner.PGWIRE_SERVER_SETTINGS`), read back from the engine and refused if unlisted or different. Answers-identical gate (`pgwire_gate.py`): each `res_<query>_digest` of the twin must equal the headline's for the same cell, under the lane's own declared coercions (l1tpc, l4), else the row is kept with `error` set and the reason recorded; l3s and l3d record no digest and are compared by `recall_at_10` within 0.01 of the headline median; l1 records neither, so its wire row is `unchecked` until the lane records digests; the smoke is in the PR body. Still open: the mini run and the numbers | `.notes` repros/protocol-latency |
 | 7 | What the memory column means: the peak, the working set after a full GC, or the peak with the heap printed beside it | every JVM arm | **decided (the user, DECISIONS #116 item 7): the peak, what an operator has to provision, with the heap setting printed beside it for JVM engines; built** (a `JVM heap` column from each cell's rows in `export_web`; the JVM memory note on every table that shows a JVM arm, BUGS F135). **No generated sentence may phrase the column as efficiency ("uses less memory"): a JVM's peak is the heap it was given; the like-for-like answer is the memory sweep.** **Still open, the memory sweep (the user, 2026-09-27), planned after the chain and the re-pin; it is an instrument change:** the smallest memory each arm completes a cell with. Design, settled before it runs: JVM arms (ArcadeDB both deployments, Neo4j) sweep the heap at a fixed cap, non-JVM arms the container cap; "completes" means the same answer digest within a stated slowdown of the full-memory run; one or two tables first (documents SF10, graph SF10); mini only, one pin; it costs days of bench time | DECISIONS #116 item 7 |
 | 8 | The sparse corpus generator yields NumPy, and each adapter converts in its own timer | comparators too (same work, moved) | measured 3.7% of ArcadeDB's build; do it with the next sparse measurement | bullet below |
 | 9 | The re-pin build: the official 26.10.1 with every fix we filed verified by its repro, comparators to latest stable (DECISIONS #129, relaxed by #163 to #165) | routine (steps 3 and 4 of the routine) | **built and verified.** The pin is the release's own jar commit `d36b4ca3a`; each fix we filed was verified by its repro on the laptop (the list is `.notes` issues-trail), and #8626, #8852, #8797, and #8725 landed after and are expected to move published cells. **Read the rows with these in hand, and say so on the page:** some cells get SLOWER (the embedded cross-model strict cells, whose `LSM_VECTOR` build had switched that thread's WAL off, #8129 fixed by #8221, and the sparse table's int8 arms, #8582); the dense and cross-model tables moved with the vector library (#8518), so compare recall at the same settings before reading a latency change; ranks can hold while times drop, because item 2 binds values on every engine. Laptop ratios are not projections of the page; only the re-measure decides. **Still owed:** re-verify #7146, #7183, #7184, and #7190 at 10M on the 26.10.1 pair; #7260 is fixed in 26.11.1 (PR #9289) and not in 26.10.1 | `.notes` issues-trail |
@@ -221,7 +221,7 @@ Rules every campaign keeps:
 | 77 | ArcadeDB's query heap budget (`arcadedb.queryMaxHeapRAM`, default half the JVM heap) raised for the cells the default refuses: the l1tpc tpch10 olap `top_parts` GROUP BY in both ArcadeDB arms (DECISIONS #175, ArcadeData/arcadedb#9402) | ArcadeDB embedded and served, the documents lane, ONLY in the repair stage | **mechanism written, default OFF (humemai/arcadedb-embedded-python#260); the repair stage is generated after qP09 and run before the tier 1 to tier 2 boundary.** `BENCH_ARCADE_QUERY_MAX_HEAP_RAM_MB` adds the JVM option to the embedded arm and to the served arm's JAVA_OPTS and stamps `arcadedb_query_max_heap_ram_mb` and a source on the row; `override: arcadedb_query_ram`. Which rows are the headline (the default rows with the refusal shown, or the raised rows as an escape hatch like row 21's cap and row 75's body limit) is decided at landing. | not an instrument change: unset, nothing differs |
 | 78 | The comparator client paths are recorded per row (issue #261; DECISIONS #173, #174): `neo4j_rust_ext`, `redis_hiredis`, `es_json_serializer`, and `pg_client_path` (`autocommit`, `copy_binary`, `binary_cursor`); `qdrant_transport` (#172) was already stamped | comparators only: no ArcadeDB arm | built (DECISIONS #173, #174) | issue #261 |
 | 81 | The sparse lane warms up before its timed pass and the runner holds the CPU idle state for each timed cell (DECISIONS #177). Text on the page too: CPU idle states are not controlled on the host, and single-client times under a millisecond include the cores' wake-up latency (PAGE-SPEC section 0a, `setup.cpu_idle_note`) | the sparse lane (warm-up); every lane's rows (hold) | **built (humemai/arcadedb-embedded-python#278).** `bench_common.SPARSE_WARMUP_DEFAULT` untimed searches per engine, 1,000, one full round as the study behind the warm-up used, drawn from queries DISJOINT from the timed set so a query or result cache cannot flatter an engine: the dataset's extra queries when it holds them, else a fixed-seed sample of documents used as queries; the row stamps `sparse_warmup_queries` as run and `sparse_warmup_source` = `extra_queries` or `document_sample`. The hold is `/dev/cpu_dma_latency` at `BENCH_CPU_DMA_LATENCY_US` for the length of each timed cell: 10 microseconds in the 2026-11 pass, because the tree default `bench_common.CPU_DMA_LATENCY_DEFAULT_US` (0, no sleep state deeper than polling) kept every core in a busy-wait loop and made the host hot and slow (PROTOCOL.md section 7 has the A/B); it needs write access to the device on the host, and a hold that cannot be taken never fails the cell: rows stamp `cpu_dma_latency_us`, `cpu_dma_latency_held`, and `cpu_dma_latency_error` when it failed. `BENCH_SPARSE_WARMUP=N` and `BENCH_CPU_DMA_LATENCY_US=N` set the count and the target (PROTOCOL.md section 7); neither can be turned off. `override: sparse_warmup`, `override: cpu_dma_latency`; `test_idle_and_warmup.py`, `test_instrument_method.py` | part of the instrument |
-| 82 | The served ArcadeDB headline is the vendor's own `arcadedb-driver` 0.2.0 (sync, httpx; `lean_http.OfficialSession`, stamp `arcadedb-driver 0.2.0 (sync, httpx)` with `arcadedb_http_client_role` = `headline`), and the lean HTTP client (`lean_http.LeanSession`) is the labelled sensitivity row | every ArcadeDB SERVED arm that talks HTTP through `lean_http.Session()`; no comparator client moves | **built (humemai/arcadedb-embedded-python#282, the instrument's client row).** The sensitivity row is the same cell run again as a twin job (`runner.ARCADEDB_HTTP_ARMS`, run_id ending `_clientlean`, `arcadedb_http_client_role` = `sensitivity`; the runner sets `BENCH_ARCADEDB_HTTP_CLIENT=lean` for it). `lean` is the client the 26.10.1 pass and the published October rows used, so the sensitivity row stays comparable with them; rows measured with the two clients are never mixed in one column. The driver carries `/query` and `/command` calls; the `GET /server` probe, the `/batch` bulk load, transactions, NDJSON streams and bodies with a field the driver has no argument for stay on the lean client. The PostgreSQL wire has no adapter in this harness, so its sensitivity row is not built. The `client` image gains `arcadedb-driver==0.2.0` at its next rebuild (`build_images.sh`, COMPARATORS.md pin table). Measured on the official 26.10.1 image (d36b4ca3a), 7 interleaved rounds: the bound Cypher point read costs the client 55.5 us of CPU per call with `lean` against 11.6 for a bare-socket variant that was tried and retired, identical answer digests; the 100-row read is engine-bound and moves less | part of the instrument |
+| 82 | The served ArcadeDB headline is the vendor's own `arcadedb-driver` 0.2.0 (sync, httpx; `lean_http.OfficialSession`, stamp `arcadedb-driver 0.2.0 (sync, httpx)` with `arcadedb_http_client_role` = `headline`), and the lean HTTP client (`lean_http.LeanSession`) is the labelled sensitivity row | every ArcadeDB SERVED arm that talks HTTP through `lean_http.Session()`; no comparator client moves | **built (humemai/arcadedb-embedded-python#282, the instrument's client row).** The sensitivity row is the same cell run again as a twin job (`runner.ARCADEDB_HTTP_ARMS`, run_id ending `_clientlean`, `arcadedb_http_client_role` = `sensitivity`; the runner sets `BENCH_ARCADEDB_HTTP_CLIENT=lean` for it). `lean` is the client the 26.10.1 pass and the published October rows used, so the sensitivity row stays comparable with them; rows measured with the two clients are never mixed in one column. The driver carries `/query` and `/command` calls; the `GET /server` probe, the `/batch` bulk load, transactions, NDJSON streams and bodies with a field the driver has no argument for stay on the lean client. The PostgreSQL wire is the second sensitivity row (row 6): the served SQL cells also run as a `_clientpgwire` twin (`arcadedb_client_wire` = `pgwire`, role `sensitivity`), the read statements only. The `client` image gains `arcadedb-driver==0.2.0` at its next rebuild (`build_images.sh`, COMPARATORS.md pin table). Measured on the official 26.10.1 image (d36b4ca3a), 7 interleaved rounds: the bound Cypher point read costs the client 55.5 us of CPU per call with `lean` against 11.6 for a bare-socket variant that was tried and retired, identical answer digests; the 100-row read is engine-bound and moves less | part of the instrument |
 | 84 | The dense lane's id index for the delete (humemai/arcadedb-embedded-python#291, DECISIONS #178 index parity): ArcadeDB, pgvector, Neo4j, Memgraph, FalkorDB, DuckDB, and LanceDB get an index on the vector id before or right after the load, the other engines already delete through their own key | the dense lane (l3d), every arm | **built.** Every dense row stamps `dense_id_index` (the statement, or `key`); `override: dense_id_index`; `test_dense_id_index.py`. The index build is inside `ingest_s`. The /next page says ArcadeDB's delete ran without an id index in rows that lack the stamp (`_dense_id_index_note`, list line `dense_id_index`; `test_next_measurement_wording.py` holds the wording) | part of the instrument |
 | 85 | Streamed results for the served ArcadeDB time-series arms, `BENCH_ARCADEDB_TS_NDJSON=1`, default OFF: both served arms ask the query endpoint for NDJSON and read the records line by line, as the PostgreSQL and MongoDB arms read rows as their server sends them | the time-series lane (l4), `arcadedb_ts_doc_server` and `arcadedb_ts_native_server` only | **mechanism written, default OFF; MEASURED 2026-10-09 and NOT USED at 26.11.1:** at ts100 the switch helped the documents arm (0.853 [0.713, 0.928]) and hurt the native arm (1.087 [1.083, 1.088]), identical rows, so it stays off and the page promises nothing; it needs a reader that parses as data arrives and a ts1000 re-measure first. Unset, nothing differs (`test_ts_ndjson.py`); set, rows stamp `arcadedb_ts_result_format` = `ndjson`; `override: ts_ndjson` | not an instrument change: unset, nothing differs |
 | 87 | Two failures of the 26.10.1 pass that were not results (humemai/arcadedb-embedded-python#344, the failure triage of 2026-10-20): (1) `chroma_dense` never ran, because the override registry `overrides.py` shadowed chromadb's dependency `overrides` on `/work` (`ImportError: cannot import name 'overrides'`); the registry is now `override_registry.py`. (2) Memgraph's storage access timeout (default 1 s) refused the deep10m vector index build in both dense arms; the dense arms now run `--storage-access-timeout-sec=172800`, longer than any cell cap | Chroma dense (every scale) and Memgraph dense (both precisions); no other arm moves (the rename changes no measurement) | **built on branch issue/344-chroma-memgraph, not on main**: `override_registry.py` and every import, `runner.MEMGRAPH_STORAGE_ACCESS_TIMEOUT_S`, the read-back `memgraph_storage_access_timeout_s`, PROTOCOL.md section 7, COMPARATORS.md. Mini picks it up at the `git pull` of the stage after the merge, which is the user's to time; a repair stage reruns Chroma dense and Memgraph dense deep10m | `.notes/bench/repros/failure-triage-20261020` |