diff --git a/benchmarks/experiments/CAMPAIGN.md b/benchmarks/experiments/CAMPAIGN.md index 38419a7af13..155bab603d1 100644 --- a/benchmarks/experiments/CAMPAIGN.md +++ b/benchmarks/experiments/CAMPAIGN.md @@ -213,7 +213,7 @@ Rules every campaign keeps: | 69 | One served lane gets a sensitivity arm for ArcadeDB at the image's own JVM defaults (75% heap, generational ZGC, no `-Xms`; ArcadeData/arcadedb#9167), printed beside the main arm: documents OLTP, served, the 2M-part cell, 3 repetitions | ArcadeDB served only | built (the user, DECISIONS #159): `runner.BACKENDS["arcadedb_imgdefaults_server"]` declared once in `runner.ARM_RUNS`; every served cell records `server_jvm_*` and the arm refuses to run unless the JVM reports those defaults; `fairness_check` F3b and F10 (`test_f10_restricted_arm.py`); `test_jvm_defaults_arm.py` | ArcadeData/arcadedb#9167 | | 70 | The TimescaleDB time-series arm runs `ANALYZE p` after its `(host, ts DESC)` index, inside its ingest timer, as the plain PostgreSQL arm does | TimescaleDB, the time-series lane only | **owed at the re-pin, done on repin-prep-2 (2026-10-06, the user's decision), now on main**: `l4_tsbs.py`. October's TimescaleDB l4 ingest rows (417314c18) were measured WITHOUT the ANALYZE and stay as measured | `test_ingest_prose.py`; `test_l4_analyze.py` | | 71 | The served graph arm over Bolt in place of HTTP | ArcadeDB served, the graph arm only | **decided against (DECISIONS #162 item 2)**: with the lean HTTP client (row 72) one bound read costs 121 us over HTTP against 171 to 194 us over Bolt, and upstream's question on which protocol a Python client should use (ArcadeData/arcadedb#9165) is unanswered; the branch `levers-served-bolt` (`21a6ffd47e`) stays unmerged and off by default | `.notes` repros/levers-audit-20261006/LEVERS.md | -| 72 | Every ArcadeDB SERVED arm that talks HTTP uses one persistent `http.client` connection per host (`lean_http.py`) in place of `requests.Session` | every ArcadeDB served arm that posts to the server; no comparator client moves | built (the maintainer's decision, DECISIONS #162 item 1); rows record `arcadedb_http_client` (the headline client and the lean sensitivity twin are described under row 82); `test_lean_http.py`, `test_lean_http_live.py`, `test_e4_client_axis.py`. **Owed at the re-pin:** the e4 stage re-runs, and the page prose around the table (its caption and the "wire format" paragraph) is rewritten from the new columns. October's served rows (417314c18) carry no `arcadedb_http_client` | `.notes` repros/levers-audit-20261006/LEVERS.md | +| 72 | Every ArcadeDB SERVED arm that talks HTTP uses one persistent `http.client` connection per host (`lean_http.py`) in place of `requests.Session` | every ArcadeDB served arm that posts to the server; no comparator client moves | built (the maintainer's decision, DECISIONS #162 item 1); rows record `arcadedb_http_client` (the headline client is described under row 82); `test_lean_http.py`, `test_lean_http_live.py`, `test_e4_client_axis.py`. **Owed at the re-pin:** the e4 stage re-runs, and the page prose around the table (its caption and the "wire format" paragraph) is rewritten from the new columns. October's served rows (417314c18) carry no `arcadedb_http_client` | `.notes` repros/levers-audit-20261006/LEVERS.md | | 73 | JPype is pinned to 1.7.1 in the benchmark images and recorded per row; reason and the measurement: `.notes/bench/repros/jpype-master-check-20261006/`. The wheel declares `jpype1>=1.5.0` with no upper bound, so an image rebuilt on the day of a JPype release would change the instrument in the middle of a run | every embedded ArcadeDB arm and the host-side Python-cost stage | **built (on main), owed at the re-pin**: `bench_common.JPYPE_PIN` and `Dockerfile.bench`'s `ARG JPYPE_VERSION=1.7.1`; every generated stage that runs an ArcadeDB arm reads JPype back out of `dbbench:arcadedb` and aborts if it differs; rows record `jpype_version`; `test_jpype_pin.py`. Moving the pin is a deliberate edit of `JPYPE_PIN` and the Dockerfile default between campaigns, never an image rebuild | jpype#1379 | | 74 | The wheel the 26.10.1 measurement runs on carries the call-overhead change of bindings PR #214 (`fabbda7ab7`) and the columnar insert (row 63) | ArcadeDB embedded arms only | built and verified (2026-10-06): the stages bake a wheel built from this tree (commit `8989cbb0e8`) over the official 26.10.1 jars, sha256 `83a2a40369a411622696aed77361fd3b1c3d23b675d0a784b38b0b022c13c41b` (`PAIR VERIFIED d36b4ca3a`). It is NOT the file PyPI holds for the bindings 26.10.1 release; the page's note on what the Python package costs should name the change when it lands | #214's own A/B (laptop, relative only) | | 75 | The served ArcadeDB arms that bulk-load through `POST /api/v1/batch` run with `-Darcadedb.server.httpBodyContentMaxSize=68719476736` (64 GiB) in place of 26.10.1's default of 100 MiB | ArcadeDB served only: the graph arm (l2, restart lane) and the cross-model arm (e2) | built and verified on the laptop (2026-10-07), on main: `runner.py` (`HTTP_BODY_MAX_OPT`), `override_registry.py` (`arcadedb_http_body_limit`), PROTOCOL.md section 7, `test_overrides.py`. Mini picks it up at its next `git pull`, which is the user's to time | the 26.10.1 sf10 served graph cell failed rep 1; `.notes/bench/repros/served-body-limit-20261006/` | @@ -221,7 +221,7 @@ Rules every campaign keeps: | 77 | ArcadeDB's query heap budget (`arcadedb.queryMaxHeapRAM`, default half the JVM heap) raised for the cells the default refuses: the l1tpc tpch10 olap `top_parts` GROUP BY in both ArcadeDB arms (DECISIONS #175, ArcadeData/arcadedb#9402) | ArcadeDB embedded and served, the documents lane, ONLY in the repair stage | **mechanism written, default OFF (humemai/arcadedb-embedded-python#260); the repair stage is generated after qP09 and run before the tier 1 to tier 2 boundary.** `BENCH_ARCADE_QUERY_MAX_HEAP_RAM_MB` adds the JVM option to the embedded arm and to the served arm's JAVA_OPTS and stamps `arcadedb_query_max_heap_ram_mb` and a source on the row; `override: arcadedb_query_ram`. Which rows are the headline (the default rows with the refusal shown, or the raised rows as an escape hatch like row 21's cap and row 75's body limit) is decided at landing. | not an instrument change: unset, nothing differs | | 78 | The comparator client paths are recorded per row (issue #261; DECISIONS #173, #174): `neo4j_rust_ext`, `redis_hiredis`, `es_json_serializer`, and `pg_client_path` (`autocommit`, `copy_binary`, `binary_cursor`); `qdrant_transport` (#172) was already stamped | comparators only: no ArcadeDB arm | built (DECISIONS #173, #174) | issue #261 | | 81 | The sparse lane warms up before its timed pass and the runner holds the CPU idle state for each timed cell (DECISIONS #177). Text on the page too: CPU idle states are not controlled on the host, and single-client times under a millisecond include the cores' wake-up latency (PAGE-SPEC section 0a, `setup.cpu_idle_note`) | the sparse lane (warm-up); every lane's rows (hold) | **built (humemai/arcadedb-embedded-python#278).** `bench_common.SPARSE_WARMUP_DEFAULT` untimed searches per engine, 1,000, one full round as the study behind the warm-up used, drawn from queries DISJOINT from the timed set so a query or result cache cannot flatter an engine: the dataset's extra queries when it holds them, else a fixed-seed sample of documents used as queries; the row stamps `sparse_warmup_queries` as run and `sparse_warmup_source` = `extra_queries` or `document_sample`. The hold is `/dev/cpu_dma_latency` at `BENCH_CPU_DMA_LATENCY_US` for the length of each timed cell: 10 microseconds in the 2026-11 pass, because the tree default `bench_common.CPU_DMA_LATENCY_DEFAULT_US` (0, no sleep state deeper than polling) kept every core in a busy-wait loop and made the host hot and slow (PROTOCOL.md section 7 has the A/B); it needs write access to the device on the host, and a hold that cannot be taken never fails the cell: rows stamp `cpu_dma_latency_us`, `cpu_dma_latency_held`, and `cpu_dma_latency_error` when it failed. `BENCH_SPARSE_WARMUP=N` and `BENCH_CPU_DMA_LATENCY_US=N` set the count and the target (PROTOCOL.md section 7); neither can be turned off. `override: sparse_warmup`, `override: cpu_dma_latency`; `test_idle_and_warmup.py`, `test_instrument_method.py` | part of the instrument | -| 82 | The served ArcadeDB headline is the vendor's own `arcadedb-driver` 0.2.0 (sync, httpx; `lean_http.OfficialSession`, stamp `arcadedb-driver 0.2.0 (sync, httpx)` with `arcadedb_http_client_role` = `headline`), and the lean HTTP client (`lean_http.LeanSession`) is the labelled sensitivity row | every ArcadeDB SERVED arm that talks HTTP through `lean_http.Session()`; no comparator client moves | **built (humemai/arcadedb-embedded-python#282, the instrument's client row).** The sensitivity row is the same cell run again as a twin job (`runner.ARCADEDB_HTTP_ARMS`, run_id ending `_clientlean`, `arcadedb_http_client_role` = `sensitivity`; the runner sets `BENCH_ARCADEDB_HTTP_CLIENT=lean` for it). `lean` is the client the 26.10.1 pass and the published October rows used, so the sensitivity row stays comparable with them; rows measured with the two clients are never mixed in one column. The driver carries `/query` and `/command` calls; the `GET /server` probe, the `/batch` bulk load, transactions, NDJSON streams and bodies with a field the driver has no argument for stay on the lean client. The PostgreSQL wire has no adapter in this harness, so its sensitivity row is not built. The `client` image gains `arcadedb-driver==0.2.0` at its next rebuild (`build_images.sh`, COMPARATORS.md pin table). Measured on the official 26.10.1 image (d36b4ca3a), 7 interleaved rounds: the bound Cypher point read costs the client 55.5 us of CPU per call with `lean` against 11.6 for a bare-socket variant that was tried and retired, identical answer digests; the 100-row read is engine-bound and moves less | part of the instrument | +| 82 | The served ArcadeDB headline is the benchmark's own minimal HTTP client (`lean_http.LeanSession`, stamp `lean_http (http.client, keep-alive)`); the page and the paper name it and say that the other engines are measured through their vendors' own Python clients | every ArcadeDB SERVED arm that talks HTTP through `lean_http.Session()`; no comparator client moves | **built (humemai/arcadedb-embedded-python#282, the instrument's client row; the headline selection is DECISIONS #268 and #285).** Each served cell runs as a second job on this client (`runner.ARCADEDB_HTTP_ARMS`, run_id ending `_clientlean`, `arcadedb_http_client_role` = `sensitivity` as stamped; the runner sets `BENCH_ARCADEDB_HTTP_CLIENT=lean` for it). `make_paper_tables.load_canonical` keeps the rows of this client and sets the other client's rows aside before any dedupe, table, gate or claim, so `runs_paper.csv`, the paper tables and the page hold these rows only; `export_web.py` refuses a frozen row or an overlay file that names another client, and `page_check.py` fails a page that names the vendor's driver. An arm with no job on this client (`arcadedb_imgdefaults_server`, and the PQ server arm) has no published cell; the docs transaction table says so for the first. This is the client the 26.10.1 pass and the published October rows used, so those rows stay comparable; rows measured with two clients are never mixed in one column. The `/query` and `/command` calls, the `GET /server` probe, the `/batch` bulk load, transactions and NDJSON streams all go through this client in these jobs. The PostgreSQL wire has no adapter in this harness. Measured on the official 26.10.1 image (d36b4ca3a), 7 interleaved rounds: the bound Cypher point read costs the client 55.5 us of CPU per call with `lean` against 11.6 for a bare-socket variant that was tried and retired, identical answer digests; the 100-row read is engine-bound and moves less | part of the instrument | | 84 | The dense lane's id index for the delete (humemai/arcadedb-embedded-python#291, DECISIONS #178 index parity): ArcadeDB, pgvector, Neo4j, Memgraph, FalkorDB, DuckDB, and LanceDB get an index on the vector id before or right after the load, the other engines already delete through their own key | the dense lane (l3d), every arm | **built.** Every dense row stamps `dense_id_index` (the statement, or `key`); `override: dense_id_index`; `test_dense_id_index.py`. The index build is inside `ingest_s`. The /next page says ArcadeDB's delete ran without an id index in rows that lack the stamp (`_dense_id_index_note`, list line `dense_id_index`; `test_next_measurement_wording.py` holds the wording) | part of the instrument | | 85 | Streamed results for the served ArcadeDB time-series arms, `BENCH_ARCADEDB_TS_NDJSON=1`, default OFF: both served arms ask the query endpoint for NDJSON and read the records line by line, as the PostgreSQL and MongoDB arms read rows as their server sends them | the time-series lane (l4), `arcadedb_ts_doc_server` and `arcadedb_ts_native_server` only | **mechanism written, default OFF; MEASURED 2026-10-09 and NOT USED at 26.11.1:** at ts100 the switch helped the documents arm (0.853 [0.713, 0.928]) and hurt the native arm (1.087 [1.083, 1.088]), identical rows, so it stays off and the page promises nothing; it needs a reader that parses as data arrives and a ts1000 re-measure first. Unset, nothing differs (`test_ts_ndjson.py`); set, rows stamp `arcadedb_ts_result_format` = `ndjson`; `override: ts_ndjson` | not an instrument change: unset, nothing differs | | 87 | Two failures of the 26.10.1 pass that were not results (humemai/arcadedb-embedded-python#344, the failure triage of 2026-10-20): (1) `chroma_dense` never ran, because the override registry `overrides.py` shadowed chromadb's dependency `overrides` on `/work` (`ImportError: cannot import name 'overrides'`); the registry is now `override_registry.py`. (2) Memgraph's storage access timeout (default 1 s) refused the deep10m vector index build in both dense arms; the dense arms now run `--storage-access-timeout-sec=172800`, longer than any cell cap | Chroma dense (every scale) and Memgraph dense (both precisions); no other arm moves (the rename changes no measurement) | **built on branch issue/344-chroma-memgraph, not on main**: `override_registry.py` and every import, `runner.MEMGRAPH_STORAGE_ACCESS_TIMEOUT_S`, the read-back `memgraph_storage_access_timeout_s`, PROTOCOL.md section 7, COMPARATORS.md. Mini picks it up at the `git pull` of the stage after the merge, which is the user's to time; a repair stage reruns Chroma dense and Memgraph dense deep10m | `.notes/bench/repros/failure-triage-20261020` | diff --git a/benchmarks/experiments/COMPARATORS.md b/benchmarks/experiments/COMPARATORS.md index fd8a2a9ed7e..4fc35fb73b1 100644 --- a/benchmarks/experiments/COMPARATORS.md +++ b/benchmarks/experiments/COMPARATORS.md @@ -72,7 +72,7 @@ Pins follow DECISIONS #87 (the latest self-hosted stable release at campaign tim | Neo4j driver (Neo4j AND Memgraph arms) | `neo4j==6.4.1` | client package | | Neo4j driver Rust extension (the vendor's documented accelerator, DECISIONS #173) | `neo4j-rust-ext==6.4.1.0` | client package | | pgvector Python adapter (binary COPY, DECISIONS #174) | `pgvector==0.5.1` | client package | -| ArcadeDB official Python driver (sync, httpx; the served ArcadeDB arms' headline client, CAMPAIGN section 7 row 72) | `arcadedb-driver==0.2.0` | client package | +| ArcadeDB Python client package installed in the client image (measured by the harness, not a published client; the served ArcadeDB headline is the benchmark's own `lean_http.py`, CAMPAIGN section 7 row 82) | `arcadedb-driver==0.2.0` | client package | **Held below the newest, and why.** diff --git a/benchmarks/experiments/PROTOCOL.md b/benchmarks/experiments/PROTOCOL.md index 3f99dbe7b77..72f241fe078 100644 --- a/benchmarks/experiments/PROTOCOL.md +++ b/benchmarks/experiments/PROTOCOL.md @@ -264,7 +264,7 @@ An override nobody can find is a defect, whichever way it moves the number. Ever | all dense arms | `EF_CONSTRUCTION=100`, `EF_SEARCH=100` | operating-point | Shared beam. At the 26.10.1 tag an explicit `efSearch` is a fixed beam of `max(k, efSearch)` while the adaptive default is `max(2k, 20)` on graphs of 10,000 or more vectors, so it is NOT a no-op for ArcadeDB; some comparators move down from their own defaults, unaudited. | lane docstring only | | ArcadeDB server (all arms) | `ARCADEDB_OPTS_GC=` is cleared explicitly, and `JAVA_OPTS` carries `-XX:+UseCompactObjectHeaders` | operating-point | Makes E4 a transport comparison. `ARCADEDB_OPTS_GC` survives a `JAVA_OPTS` override, so clearing it puts both arms on one collector. | DECISIONS #54, enforced by `verify_pair_c25.sh` | | ArcadeDB server vs embedded | both arms run ONE set of upstream jars on ONE JVM (Corretto 25) | fairness | The axis isolates transport, not JVM major, libc, collector, or object-header layout; `-XX:+UseCompactObjectHeaders` is rejected by JDK 21. | DECISIONS #54, enforced by `verify_pair_c25.sh` (jars AND JVM major on both sides) | -| ArcadeDB SERVED arms | the headline HTTP client is the official `arcadedb-driver` 0.2.0 (sync); the same cell runs again on the lean HTTP client as a twin job | the vendor's own client, the same class as the comparators' official clients | Protocol audit round 3: the driver costs about 3x the lean client on a one-row read, so a served latency claim names the client. Bulk loads, the server probe, and transactions stay on the lean HTTP client. | row fields `arcadedb_http_client` and `arcadedb_http_client_role` (`headline` or `sensitivity`) | +| ArcadeDB SERVED arms | the headline HTTP client is the benchmark's own minimal HTTP client (`lean_http.py`: one persistent `http.client` connection per host); the page and the paper name it | the other engines are measured through their vendors' own Python clients, so client weight differs between the served engines | A served latency claim names the client. Bulk loads, the server probe, and transactions use the same client. The freeze keeps the rows measured through this client only (DECISIONS #285), so an arm with no job on this client has no published cell. | row fields `arcadedb_http_client` and `arcadedb_http_client_role` (stamped as the job ran; the rows of this client carry `sensitivity`) | | l4 `q_high` and `q_groupby` | ArcadeDB embedded `to_arrow()`, DuckDB `to_arrow_table()`, SQLite `fetchall()` (no columnar path); `to_list()` / `fetchall()` timed beside it as `_plain_ms` | each engine on its documented best path for a result of thousands of rows | The bindings' docs recommend a bulk read for large results; small results (1 to 100 rows) are unchanged. | row fields `read_path`, `read_path_queries`, `read_path_plain`, `read_path_sensitivity`, `_plain_ms`, `_plain_digest`, `_plain_digest_match` (the cell ends if the typed and plain digests differ; DuckDB's Arrow rows are read by position, like its `fetchall()`) | | JVM initial heap (`-Xms`) per engine | **ArcadeDB, embedded and served: unset** (only `-Xmx` from the cell's memory budget; the served arms through `ARCADEDB_OPTS_MEMORY=-Xmx{heap}`, DECISIONS #200). Elasticsearch: `ES_JAVA_OPTS=-Xms{heap} -Xmx{heap}`, unchanged. Neo4j: `server.memory.heap.initial_size` = `max_size`, unchanged. QuestDB: the harness sets no heap (image defaults), unchanged | ArcadeDB as its maintainers advise for a container with a fixed memory budget (no `-Xms`; ArcadeData/arcadedb#9167) and as the bindings document, so both deployments run one JVM configuration; each other server engine as its own documentation recommends | ArcadeDB's server script (`package/src/main/scripts/server.sh`) leaves the heap to the JVM unless `ARCADEDB_OPTS_MEMORY` is set and shows `-Xms2G -Xmx2G` as its example; Elasticsearch: set `Xms` and `Xmx` to the same value (https://www.elastic.co/guide/en/elasticsearch/reference/current/advanced-configuration.html#set-jvm-heap-size); Neo4j: set initial and maximum heap to the same value (https://neo4j.com/docs/operations-manual/current/performance/memory-configuration/); QuestDB: no setting by the harness (https://questdb.com/docs/configuration/). | `server_heap` / the cell's heap fields, as before | | ArcadeDB server (all arms) | `ARCADEDB_OPTS_MEMORY=-Xmx{heap}` (image ships 2G) | resource fitting | Heap parity per tier. The 2 GB default cost the server 15% p50 and 35% build time at 10M. | T5 caption, FAIRNESS.md F3, `server_heap` read back | diff --git a/benchmarks/experiments/export_web.py b/benchmarks/experiments/export_web.py index 77a50dd92fd..70a07aa88eb 100644 --- a/benchmarks/experiments/export_web.py +++ b/benchmarks/experiments/export_web.py @@ -2894,7 +2894,7 @@ def _lc_short_reason(text): LIFECYCLE_WITHHELD = {"graph_gav": "its query grew from a bounded set of seeds to an unbounded 2-hop, so it stays off this table until the query is bounded again."} # PAGE-SPEC rule 7 -def _lifecycle_table(all_rows, twin_rows=()): +def _lifecycle_table(all_rows): """Built from the FROZEN CSV, like every other table in this file. The first version called load_canonical(), which is make_paper_tables' name @@ -2906,12 +2906,6 @@ def _lifecycle_table(all_rows, twin_rows=()): rows = [r for r in all_rows if r.get("lane") == "lifecycle"] if not rows: return None - twin_by = {} - for r in twin_rows: - if r.get("lane") == "lifecycle" and not r.get("lifecycle_situation_unexpressible"): - twin_by.setdefault((r.get("workload"), r.get("scale"), str(r.get("backend", "")).endswith("_server"), - _lifecycle_engine(r.get("backend", ""))), []).append(r) - # Rows fold by situation, size, deployment, and ENGINE. An ArcadeDB # label names only the situation; the comparator arms (DECISIONS #95a) share the table, so a # comparator row names its engine inside the parentheses and carries an @@ -2987,17 +2981,6 @@ def _engine_of(r): got = _agg(rs, field) if got is not None: entry["metrics"][label] = got - _trs = twin_by.get((situation, scale, _srv, engine)) - if _trs and entry["metrics"]: - twin_metrics = {} - for field, label in (("jvm_start_ms", "JVM start ms"), ("first_open_ms", "first open ms"), - ("cold_process_ms", "cold process ms"), - *[(f"{k}_session_ms", LIFECYCLE_SCENARIO_LABELS[k]) for k in LIFECYCLE_PAGE_SCENARIOS], - (MEM_PEAK_FIELD, "peak memory GiB")): - got = _agg(_trs, field) - if got is not None: - twin_metrics[label] = got - _attach_twin(entry, twin_metrics, _trs) if entry["metrics"]: entries.append(entry) @@ -3598,7 +3581,7 @@ def _multimodel_table(finished): } -def _l4_table(all_rows, twin_rows=()): +def _l4_table(all_rows): grouped = _l4_canonical(all_rows) if not grouped: # A LANE SCOPED OUT IS NOT A MISSING ARTIFACT, the third place today @@ -3631,7 +3614,6 @@ def _l4_table(all_rows, twin_rows=()): # Five queries, one p99, one cold column, in the order the section reads # (OCT_TABLE_METRICS), resolved once so the column list and the cells agree. _l4_cols = OCT_TABLE_METRICS["l4"] + [MEM_PHASE_COLUMN] - twin_grouped = _l4_canonical(twin_rows) or {} entries = [] # Tier first, then the engine order the section reads, so the table shows # each corpus as its own row group the way every other multi-tier lane @@ -3672,16 +3654,6 @@ def _l4_table(all_rows, twin_rows=()): if got is not None: entry["metrics"][lab] = got break - _trs = twin_grouped.get(key) - if _trs and entry["metrics"]: - twin_metrics = {} - for field, lab in _l4_cols: - for candidate in ((field,) if isinstance(field, str) else field): - got = _agg(_trs, candidate) - if got is not None: - twin_metrics[lab] = got - break - _attach_twin(entry, twin_metrics, _trs) if entry["metrics"]: entries.append(entry) @@ -4843,37 +4815,94 @@ def _arango_ivf_note(table, rows): return _gen(" ".join(parts), *dict.fromkeys(values)) -def _attach_twin(entry, twin_metrics, twin_rows): - """Put the twin beside the headline entry, over the labels the headline cell has (the same metric labels, no more).""" - kept = {k: v for k, v in twin_metrics.items() if k in entry["metrics"]} - if kept: - entry["client_twin"] = {"client": _twin_client(twin_rows), "metrics": kept} +# THE ONE CLIENT THE SERVED ARCADEDB HEADLINE RUNS THROUGH (DECISIONS #268, #285). The row's `arcadedb_http_client` is +# read from the session that ran; a row that names any other client does not belong in a printed table. +# REMOVE WHEN: upstream ships the transport fix (ArcadeData/arcadedb-drivers #188 merged and released) and the decision +# to publish that client is made; until then the page names the benchmark's own client only. +_SERVED_CLIENT_SOURCE = "benchmarks/experiments/lean_http.py" +_OTHER_CLIENT = re.compile(r"arcadedb[-_ ]?driver", re.I) + + +def _lean_client_name(): + import lean_http + return lean_http.LEAN_NAME + +def _is_served_arcadedb(backend): + b = str(backend or "") + return b.startswith("arcadedb") and b.endswith("_server") -def _twin_client(rs): - """The client a twin cell's rows name (one name in a clean freeze), or None.""" - named = collections.Counter(str(r.get("arcadedb_http_client") or "") for r in rs) - return named.most_common(1)[0][0] or None if named else None + +_OTHER_CLIENT_STAMP = re.compile(r'"arcadedb_http_client"\s*:\s*"[^"]*arcadedb[-_ ]?driver', re.I) + + +def _refuse_other_client_overlays(): + """The multipass overlays (dense_mp5, sparse_mp) are read by the page without the freeze, so the rule is held here: an + overlay file whose rows name another client refuses the page. A served arm's twin job writes the same file name as the + other client's job, so the second job to finish wins; this catches the case where it was not the benchmark's own.""" + pin = os.environ.get("BENCH_ENGINE_COMMIT", "").strip() + bad = [] + for d in sorted(RESULTS.glob(f"*_{pin}")) if pin else []: + for fp in sorted(d.glob("*.json")) if d.is_dir() else []: + if _OTHER_CLIENT_STAMP.search(fp.read_text(encoding="utf-8", errors="replace")): + bad.append(f"{d.name}/{fp.name}") + if bad: + raise SystemExit(f"REFUSING to print: {len(bad)} overlay file(s) were not measured through the benchmark's own HTTP " + f"client ({', '.join(bad[:4])}{' ...' if len(bad) > 4 else ''}); re-run the arm's lean job") + + +def _refuse_other_client_rows(rows): + """Refuse a frozen row of a served ArcadeDB arm that names a client other than the benchmark's own.""" + _refuse_other_client_overlays() + bad = collections.Counter( + (str(r.get("backend")), str(r.get("arcadedb_http_client"))) + for r in rows if _OTHER_CLIENT.search(str(r.get("arcadedb_http_client") or ""))) + if bad: + for (be, client), n in sorted(bad.items()): + print(f" {n} row(s) of {be} name the client {client!r}") + raise SystemExit( + "REFUSING to print: the frozen rows hold a served ArcadeDB row that was not measured through the benchmark's " + "own HTTP client. make_paper_tables.load_canonical sets those rows aside; re-freeze before the page is built.") + + +def _served_client_table(table): + """True when the table prints a served ArcadeDB entry.""" + return any(_is_served_arcadedb(e.get("backend_key")) for e in table.get("entries") or []) def _arcadedb_client_note(table): - """What the served ArcadeDB rows ran through, from the rows: the official driver is the headline and the lean - HTTP client's twin is printed beside it as a labelled sensitivity (`client_twin` on the entry), never pooled - into the headline median. None for a table with no served ArcadeDB row.""" - heads = sorted({str(r.get("arcadedb_http_client")) for r in _FROZEN_ROWS - if str(r.get("backend") or "").startswith("arcadedb") and str(r.get("backend")).endswith("_server") - and str(r.get("arcadedb_http_client") or "").strip() not in ("", "None") - and any(e.get("backend_key") == r.get("backend") for e in table.get("entries") or [])}) - twins = sorted({str((e.get("client_twin") or {}).get("client")) for e in table.get("entries") or [] - if (e.get("client_twin") or {}).get("client")}) + """What the served ArcadeDB rows ran through: the benchmark's own minimal HTTP client, named from the rows. None + for a table with no served ArcadeDB row. A row naming another client, or two clients under one table, refuses.""" + heads = {str(r.get("arcadedb_http_client")) for r in _FROZEN_ROWS + if _is_served_arcadedb(r.get("backend")) + and str(r.get("arcadedb_http_client") or "").strip() not in ("", "None") + and any(e.get("backend_key") == r.get("backend") for e in table.get("entries") or [])} if not heads: return None - text = (f"ArcadeDB's served rows time every call through {_join_and(heads)}, the vendor's official Python client, " - f"the same kind of client the other engines are measured through.") - if twins: - text += (f" The same cells also ran through the benchmark's own lean HTTP client ({_join_and(twins)}); " - f"each is printed on its row as a labelled sensitivity and is not part of the headline median.") - return _gen(text, *heads, *twins) + lean = _lean_client_name() + if heads != {lean}: + raise SystemExit(f"REFUSING to print table {table.get('id')!r}: its served ArcadeDB rows name {sorted(heads)}, " + f"and the one client the page may name is {lean!r}") + text = (f"ArcadeDB's served rows time every call through the benchmark's own minimal HTTP client, {lean} " + f"(source: {_SERVED_CLIENT_SOURCE}), which keeps one persistent http.client connection per host. " + f"The other engines are measured through their vendors' own Python clients, so client weight differs " + f"between the served engines.") + return _gen(text, lean, _SERVED_CLIENT_SOURCE) + + +def _imgdefaults_not_measured_note(table): + """The image-defaults ArcadeDB arm has rows only from a client other than the benchmark's own, so its cell is not + printed and the table says why (the arm is a declared absence). None on every other table, and where it is printed. + REMOVE WHEN: a lean-client job exists for this arm (runner.ARCADEDB_HTTP_ARMS) and its rows are frozen.""" + if table.get("id") not in ARM_TABLES.get(JVM_DEFAULTS_ARM, ()) or not _served_client_table(table): + return None + if any(e.get("backend_key") == JVM_DEFAULTS_ARM for e in table.get("entries") or []): + return None + label = display_name(JVM_DEFAULTS_ARM) + text = (f"{label} is not printed: it was not measured through the benchmark's own HTTP client, which every " + f"served ArcadeDB row on this table uses.") + _declare_absence(table["id"], label, None, "withheld", text) + return _gen(text, label) def _oct_conditions(table): @@ -4953,6 +4982,9 @@ def _oct_conditions(table): _client = _arcadedb_client_note(table) if _client: tail.append(_client) + _imgnote = _imgdefaults_not_measured_note(table) + if _imgnote: + tail.append(_imgnote) jvm = _jvm_memory_note(table) if jvm: tail.append(jvm) @@ -6993,11 +7025,10 @@ def _restructure_tables(tables, rows): mp = {(e["backend"], e["scale"]): e for e in by["l3smp"]["entries"]} for e in by["l3s"]["entries"]: m = e["metrics"] - for mm in (m, (e.get("client_twin") or {}).get("metrics") or {}): - if "p50 ms" in mm: - mm["cold p50 ms"] = mm.pop("p50 ms") - if "p99 ms" in mm: - mm["cold p99 ms"] = mm.pop("p99 ms") + if "p50 ms" in m: + m["cold p50 ms"] = m.pop("p50 ms") + if "p99 ms" in m: + m["cold p99 ms"] = m.pop("p99 ms") src = mp.get((e["backend"], e["scale"])) if src: for k in ("warm p50 ms", "warm p99 ms", "gain"): @@ -7021,11 +7052,11 @@ def _restructure_tables(tables, rows): # pass to show. t = by["l3s"] for e in t["entries"]: - for mm in (e["metrics"], (e.get("client_twin") or {}).get("metrics") or {}): - if "p50 ms" in mm: - mm["cold p50 ms"] = mm.pop("p50 ms") - if "p99 ms" in mm: - mm["cold p99 ms"] = mm.pop("p99 ms") + mm = e["metrics"] + if "p50 ms" in mm: + mm["cold p50 ms"] = mm.pop("p50 ms") + if "p99 ms" in mm: + mm["cold p99 ms"] = mm.pop("p99 ms") t["columns"] = [{"p50 ms": "cold p50 ms", "p99 ms": "cold p99 ms"}.get(c, c) for c in t["columns"]] if "l1tpc" in by: # THE PAGE SHOWS THE STANDARD DOCUMENT BENCHMARKS ONLY: TPC-C is the @@ -7035,9 +7066,6 @@ def _restructure_tables(tables, rows): def clone(e, keep): n = dict(e) n["metrics"] = {k: v for k, v in e["metrics"].items() if k in keep} - if e.get("client_twin"): - n["client_twin"] = dict(e["client_twin"], metrics={k: v for k, v in e["client_twin"]["metrics"].items() - if k in keep}) return n # Six transactional operations (#82a) and five analytical queries (#82), # each a warm median, with a ninety-ninth percentile on the headline @@ -7096,14 +7124,9 @@ def main() -> int: rows = list(csv.DictReader(FROZEN.open())) _PI.refuse_other_instruments(rows, FROZEN_NAME) _refuse_gav_without_supertype(rows) - # THE SERVED ARCADEDB CLIENT TWIN IS NOT THE HEADLINE. Every served ArcadeDB cell runs twice: through the official - # driver (role `headline`) and through the benchmark's lean HTTP client (role `sensitivity`). The twin's rows are kept - # apart from every table's rows, so they are never pooled into a headline median, and are printed beside the - # headline cell as `client_twin`. - _twin_rows = [r for r in rows if str(r.get("arcadedb_http_client_role") or "") == "sensitivity"] - if _twin_rows: - _twin_ids = {id(r) for r in _twin_rows} - rows = [r for r in rows if id(r) not in _twin_ids] + # EVERY SERVED ARCADEDB HEADLINE CELL COMES FROM THE BENCHMARK'S OWN HTTP CLIENT (DECISIONS #268, #285). The freeze + # (make_paper_tables.load_canonical) sets the other client's rows aside, so none is here; one that is refuses the page. + _refuse_other_client_rows(rows) # THE CAP RULE, AT REPORT TIME (DECISIONS #188): an arm-cell whose pass-1 median is over its rule cap is shown censored # (_censored_cells) and its numbers are not shown, so its rows leave the tables here. Strict-class rows are the # durability table's and stay. @@ -7153,9 +7176,6 @@ def main() -> int: if _dropped: print(f" one setting per table: {len(_dropped)} strict rows set aside for the durability table only") rows = [r for r in rows if r not in _dropped] - # The twin rows get the same one-setting rule: a strict row is the durability table's, never a twin cell's. - _twin_rows = [r for r in _twin_rows if not (str(r.get("durability_class")) == "strict" and ( - r.get("lane"), r.get("workload"), r.get("backend"), str(r.get("scale"))) in _has_relaxed)] names = _image_version_names() # The tabular lanes run OLTP and OLAP as separate workloads over ONE corpus @@ -7192,10 +7212,6 @@ def main() -> int: # have sprung it. grouped[(r["lane"], r["scale"], r["backend"], wl, r.get("gav") or "")].append(r) - twin_grouped = defaultdict(list) - for r in _twin_rows: - wl = "" if r["lane"] in MERGED_WORKLOAD_LANES else r.get("workload", "") - twin_grouped[(r["lane"], r["scale"], r["backend"], wl, r.get("gav") or "")].append(r) tables = [] for lane, spec in LANES.items(): @@ -7212,12 +7228,10 @@ def main() -> int: # Row-level, not group-level: merged lanes (l1, l1tpc) group both # workloads under one key with a blank label, so a group-level # test never matched and the per-query OLAP table was empty (BUGS F30). - _trs = twin_grouped.get((ln, scale, backend, workload, gav), []) if spec.get("only_workload"): rs = [r for r in rs if r.get("workload") == spec["only_workload"]] if not rs: continue - _trs = [r for r in _trs if r.get("workload") == spec["only_workload"]] # The no-settle ablation is MEASURED but not PUBLISHED, and the # page was the only place it appeared. Neither paper reports it: # T4 is a clean head-to-head, one ArcadeDB row against three @@ -7340,24 +7354,17 @@ def main() -> int: "host": rs[0].get("host") or None, "metrics": {}, } - twin_metrics = {} for field, label in _metrics_for(lane, spec, src_lane): - src, tsrc = rs, _trs + src = rs if field in MEM_FIELDS and src_lane in MERGED_WORKLOAD_LANES: # See MEM_FIELDS above: this lane's rows were merged across # workloads, which is right for a metric that only one # workload produces and wrong for one both produce. Report # the transactional run rather than a median straddling two. src = [r for r in rs if r.get("workload") == "oltp"] or rs - tsrc = [r for r in _trs if r.get("workload") == "oltp"] or _trs got = _agg(src, field) if got is not None: entry["metrics"][label] = got - if tsrc: - tgot = _agg(tsrc, field) - if tgot is not None: - twin_metrics[label] = tgot - _attach_twin(entry, twin_metrics, _trs) if lane == "l3d": _beam_extras(entry, rs) if entry["metrics"]: @@ -7498,7 +7505,7 @@ def main() -> int: # Python-cost table are built from artifacts of the campaign, not from the # frozen rows; the durability table sits beside the session-cost table # (DECISIONS #90) and is built from the same rows the write tables are. - _extras = [_sparse_multipass_table(), _l4_table(rows, _twin_rows), _lifecycle_table(rows, _twin_rows), _durability_table(_all_rows), + _extras = [_sparse_multipass_table(), _l4_table(rows), _lifecycle_table(rows), _durability_table(_all_rows), _e4_table(), _python_cost_table()] for extra in _extras: if extra and extra["entries"]: @@ -7715,6 +7722,8 @@ def main() -> int: for table in tables: rel = SOURCES.get(table["id"]) paths = [rel] if isinstance(rel, str) else list(rel or []) + if _served_client_table(table) and _SERVED_CLIENT_SOURCE not in paths: + paths.append(_SERVED_CLIENT_SOURCE) table["source_paths"] = paths table["source_urls"] = [f"{REPO}/{p}" for p in paths] # Kept so an older page build still renders something true: the first diff --git a/benchmarks/experiments/fixtures/publish_2611/build_fixture.py b/benchmarks/experiments/fixtures/publish_2611/build_fixture.py index cda1da3c1c9..3587e90976d 100644 --- a/benchmarks/experiments/fixtures/publish_2611/build_fixture.py +++ b/benchmarks/experiments/fixtures/publish_2611/build_fixture.py @@ -3,7 +3,7 @@ The publish tests (test_publish_2611_e2e.py) run make_paper_tables and export_web on `fixtures/publish_2611/results`: a few real rows per lane, restamped `instrument: 2026-11`, plus the row fields the 2026-11 instrument adds (per-phase memory, -the dense search-beam sweep and matched-recall headline, product-quantization arms, the served-client headline and lean twin, +the dense search-beam sweep and matched-recall headline, product-quantization arms, the served ArcadeDB rows of both clients, l4 plain reads, the switch stamps). The numbers are real 2026-10 measurements; the stamps and added fields are synthetic, so the fixture proves structure and arithmetic, never a published value. Only a developer with a populated results/ needs this script; the committed output is the fixture. @@ -27,6 +27,7 @@ HERE = Path(__file__).resolve().parent sys.path.insert(0, str(HERE.parents[1])) import dense_beam # noqa: E402 +import lean_http # noqa: E402 PIN_OLD, PIN_NEW = "417314c18", "261100fix" KEEP = {"e2": {"e2"}, "e4": {"e2"}, "l1tpc": {"tpch1"}, "l2": {"sf1"}, "l3d": {"small"}, "l3s": {"tiny", "small", "medium"}, @@ -100,18 +101,9 @@ def _add_2611_fields(rows): r["arcadedb_http_client"] = "arcadedb-driver 0.2.0 (sync, httpx)" r["arcadedb_http_client_role"] = "headline" out.append(r) - # the labelled lean twin of every served ArcadeDB cell: faster by a fixed factor, so a test can tell it from the headline - twins = [] - for r in out: - if r.get("arcadedb_http_client_role") == "headline": - t = dict(r) - t["arcadedb_http_client"] = "lean (requests)" - t["arcadedb_http_client_role"] = "sensitivity" - t["run_id"] = f"{r.get('run_id', 'x')}_clientlean" - for k, v in list(t.items()): - if k.endswith("_ms") and isinstance(v, (int, float)) and not isinstance(v, bool): - t[k] = round(v * 0.7, 4) - twins.append(t) + # the lean-client row of every served ArcadeDB cell, the headline since DECISIONS #285: faster by a fixed factor, so a + # test can tell it from the vendor-driver row, which the freeze sets aside + twins = [_lean_twin(r) for r in out if r.get("arcadedb_http_client_role") == "headline"] # the product-quantization arms (#419) beside the fp32 arm each is derived from: synthetic, there are no real PQ rows. The # stamps are the ones l3d_dense writes (quantization PRODUCT for ArcadeDB, pq_subspaces, pq_clusters, pq_rerank, the # read-back fields); the numbers are the fp32 arm's, slower queries and a lower recall, as PQ trades them. @@ -267,11 +259,48 @@ def _patch_overlays(): f.write_text(new) +def _lean_twin(r): + """The benchmark's own client's row of a served ArcadeDB cell whose vendor-driver row is `r` (30 percent faster).""" + t = dict(r) + t["arcadedb_http_client"] = lean_http.LEAN_NAME + t["arcadedb_http_client_role"] = "sensitivity" + t["run_id"] = f"{r.get('run_id', 'x')}_clientlean" + for k, v in list(t.items()): + if k.endswith("_ms") and isinstance(v, (int, float)) and not isinstance(v, bool): + t[k] = round(v * 0.7, 4) + return t + + +# The served arms the harness runs on both clients (runner.ARCADEDB_HTTP_ARMS) have a lean row per cell; the image-defaults arm +# does not (vendor-driver rows only, so its cell is not published), and the PQ server arm does not either, so the fixture gives +# that one a lean row to keep the PQ table's coverage. +NO_LEAN_TWIN = ("arcadedb_imgdefaults_server",) +LEAN_TWIN_ADDED = ("arcadedb_dense_server_pq",) + + +def _lean_client(rows): + """Idempotent: the lean rows carry the client's real name, the arm without a lean twin has none, the PQ server arm has one.""" + out = [] + for r in rows: + if r.get("arcadedb_http_client_role") == "sensitivity": + if r["backend"] in NO_LEAN_TWIN: + continue + r = dict(r, arcadedb_http_client=lean_http.LEAN_NAME) + out.append(r) + have = {(r["lane"], r["backend"], r["scale"], r.get("workload"), r["rep"], r.get("durability_class")) + for r in out if r.get("arcadedb_http_client_role") == "sensitivity"} + for r in list(out): + if (r["backend"] in LEAN_TWIN_ADDED and r.get("arcadedb_http_client_role") == "headline" + and (r["lane"], r["backend"], r["scale"], r.get("workload"), r["rep"], r.get("durability_class")) not in have): + out.append(_lean_twin(r)) + return out + + def realistic_2611(rows, deep10m=True): import bench_common import fairness_check as FC import override_registry as OR - rows = _agreeing_answers(_roster_fill([_pin_and_versions(r) for r in rows])) + rows = _lean_client(_agreeing_answers(_roster_fill([_pin_and_versions(r) for r in rows]))) relaxed = next(r["durability"] for r in rows if r["backend"] == "arcadedb_embedded" and r.get("durability_class") == "relaxed") cap, body = OR.runner_cap(), OR.runner_body_limit() hash_lanes = {lane: (wl, pairs) for lane, wl, pairs in FC.ARCADEDB_HASH_ID_INDEXES} @@ -327,7 +356,7 @@ def _deep10m_envelope_rows(rows): out = [] for r in rows: if (r["lane"] != "l3d" or r["scale"] != "small" or r["backend"] not in overlay or r["backend"] in have - or int(r.get("rep") or 0) != 1): + or int(r.get("rep") or 0) != 1 or r.get("arcadedb_http_client_role") == "headline"): continue have.add(r["backend"]) d = dict(r) diff --git a/benchmarks/experiments/fixtures/publish_2611/results/runs.jsonl.gz b/benchmarks/experiments/fixtures/publish_2611/results/runs.jsonl.gz index 1b4844da118..1fe86f5e383 100644 Binary files a/benchmarks/experiments/fixtures/publish_2611/results/runs.jsonl.gz and b/benchmarks/experiments/fixtures/publish_2611/results/runs.jsonl.gz differ diff --git a/benchmarks/experiments/lean_http.py b/benchmarks/experiments/lean_http.py index c672ed647e8..5b6d4531dc0 100644 --- a/benchmarks/experiments/lean_http.py +++ b/benchmarks/experiments/lean_http.py @@ -5,9 +5,9 @@ read costs 468 to 628 us through `requests.Session`, 119 to 150 us through a persistent `http.client` connection (3.9x), the server being the same. The comparators are driven by their vendors' own clients, which differ in weight: the Postgres and Mongo wire clients are light, but python-arango is `requests`, qdrant-client over REST is httpx plus pydantic, and elasticsearch-py is urllib3, so `requests` was heavier than most -and the served ArcadeDB column measured the client library as much as the engine. This client is lighter than the official `arcadedb-driver` -(httpx, published 2026-10-06), which measured 3.2x slower on a one-row read (protocol audit round 3); the 26.11.1 re-measure headlines the -official driver and keeps this client as a disclosed sensitivity row. This module is the +and the served ArcadeDB column measured the client library as much as the engine. This client is the headline client of the served ArcadeDB +rows: the page and the paper name it as the benchmark's own minimal HTTP client, and say that the other engines are measured through their +vendors' own Python clients, so client weight differs between the served engines (DECISIONS #268, #285). This module is the one place the lanes get their ArcadeDB HTTP session from; a lane changes by one import. The interface is the part of `requests.Session` the lanes use: `.auth` (a (user, password) tuple, sent as Basic), `.get(url, auth=, headers=, timeout=)`, @@ -33,9 +33,9 @@ A session is for one thread. `row_fields(session)` is what a row records. -THE CLIENT (CAMPAIGN section 7 rows 72 and 82): the headline is `official`, the vendor's own `arcadedb-driver` (sync, httpx) for every -`/query` and `/command` call, and this client is the labelled sensitivity row (the runner schedules it as a twin cell and sets -`BENCH_ARCADEDB_HTTP_CLIENT=lean` for it). Bulk loads, the server probe, transactions and NDJSON streams stay on this client. +THE CLIENT (CAMPAIGN section 7 rows 72 and 82): the published cell of a served ArcadeDB arm is the one measured through this client +(the runner schedules the cell as a second job and sets `BENCH_ARCADEDB_HTTP_CLIENT=lean` for it; the freeze keeps those rows and sets +the other client's rows aside). Bulk loads, the server probe, transactions and NDJSON streams use this client in every job. """ import base64 import gzip @@ -53,13 +53,14 @@ CLIENT_ENV = "BENCH_ARCADEDB_HTTP_CLIENT" LEAN = "lean" OFFICIAL = "official" -OFFICIAL_PIN = "0.2.0" # the arcadedb-driver release the 2026-11 headline measures; recorded per row from the installed package +OFFICIAL_PIN = "0.2.0" # the vendor package release the default session measures; recorded per row from the installed package LEAN_NAME = "lean_http (http.client, keep-alive)" def client_choice(): - """'official' (the headline, the default) or 'lean' (the sensitivity twin), from BENCH_ARCADEDB_HTTP_CLIENT, which the runner - sets for the twin cell; anything else is refused rather than guessed.""" + """'official' (the default while the pass runs: a mid-run change would split rows) or 'lean' (the client whose rows are + published), from BENCH_ARCADEDB_HTTP_CLIENT, which the runner sets for the second job of each cell; anything else is refused + rather than guessed.""" v = (os.environ.get(CLIENT_ENV) or "").strip().lower() if v in ("", OFFICIAL): return OFFICIAL @@ -448,7 +449,7 @@ def _driver_version(): # ------------------------------------------------------------------------------------------------------ the factory def Session(): # noqa: N802 - requests' own spelling, so a lane changes by one import - """The ArcadeDB served arms' session: the official driver (the headline), or `LeanSession` when BENCH_ARCADEDB_HTTP_CLIENT=lean.""" + """The ArcadeDB served arms' session: the default session, or `LeanSession` (the published client) when BENCH_ARCADEDB_HTTP_CLIENT=lean.""" return OfficialSession() if client_choice() == OFFICIAL else LeanSession() diff --git a/benchmarks/experiments/make_paper_tables.py b/benchmarks/experiments/make_paper_tables.py index c8301f71405..9144990ea4c 100644 --- a/benchmarks/experiments/make_paper_tables.py +++ b/benchmarks/experiments/make_paper_tables.py @@ -208,6 +208,7 @@ def _commit_matches(row_commit): # Rows dropped for an unsettled disk reading. UNSETTLED_DISK = [] _OTHERS_SAID = [] +_OFFICIAL_SAID = [] def _write_withheld_recall(): @@ -269,8 +270,24 @@ def _assert_lanes_registered(): f"export_web.SCALE_LABELS, then fairness_check.LANE_SCRIPT.") -def load_canonical(apply_corpus=True, include_twins=False): +# THE SERVED ARCADEDB HEADLINE IS THE BENCHMARK'S OWN HTTP CLIENT (DECISIONS #268, #285). The harness still measures a served +# cell through the vendor's driver as well (role `headline`, client name `arcadedb-driver ...`) and through lean_http (role +# `sensitivity`, the name the freeze keeps for it); the rows of the first are measured and not published, so the freeze +# sets them aside here, before any dedupe, table, gate or claim sees them. An arm with no lean row therefore has no cell. +# REMOVE WHEN: upstream ships the transport fix (ArcadeData/arcadedb-drivers #188 merged and released) and the owner decides +# to publish that client; flip this test, not the rows. +OFFICIAL_CLIENT_SET_ASIDE = collections.Counter() + + +def is_official_client_row(r): + """True for a row measured through the vendor's own Python driver, by its role stamp or its recorded client name.""" + return (r.get("arcadedb_http_client_role") == "headline" + or "arcadedb-driver" in str(r.get("arcadedb_http_client") or "")) + + +def load_canonical(apply_corpus=True): _assert_lanes_registered() + OFFICIAL_CLIENT_SET_ASIDE.clear() del WITHHELD_RECALL[:] # per call, or the sidecar counts every earlier call's rows again del UNSETTLED_DISK[:] CORPUS_EXCLUDED.clear() @@ -306,6 +323,9 @@ def load_canonical(apply_corpus=True, include_twins=False): continue if r["scale"] not in PAPER_SCALES.get(r["lane"], []): continue + if is_official_client_row(r): + OFFICIAL_CLIENT_SET_ASIDE[r.get("backend")] += 1 + continue # A PARTIAL LANDING FREEZES ONLY ITS OWN LANES. `land_stage --only-lanes` # filtered what was PULLED and nothing downstream, so the freeze still # swept the whole canonical store -- and a landing of l2 and e2 was @@ -546,22 +566,20 @@ def load_canonical(apply_corpus=True, include_twins=False): # sweep row and a GAV-ablation row could once have shadowed a published # cell. Rows before #90 carry no class and default to relaxed, which is # what they were. - # THE CLIENT ROLE IS PART OF THE KEY. A served ArcadeDB cell runs twice: on the headline client and, as a twin job, on - # the lean HTTP client (`arcadedb_http_client_role` = sensitivity). Same lane, scale, workload, backend and rep, so - # without the role the later of the two would replace the other on ts_utc. k = (r["lane"], r["scale"], r.get("n_docs"), r.get("workload"), r["backend"], r.get("gav") is not False, r["rep"], - r.get("durability_class") or "relaxed", r.get("arcadedb_http_client_role") == "sensitivity") + r.get("durability_class") or "relaxed") if k not in best or r["ts_utc"] > best[k]["ts_utc"]: best[k] = r _write_withheld_recall() if UNSETTLED_DISK: print(f" blanked the disk reading of {len(UNSETTLED_DISK)} row(s): it had not settled; " f"their other measurements stand") - # The paper tables, the gates and the claims read the headline rows. The twin rows go to the freeze (include_twins) so - # the exporter can show them beside the headline, labelled, and are never pooled into a headline median. - out = [r for r in best.values() if include_twins or r.get("arcadedb_http_client_role") != "sensitivity"] - return out + if OFFICIAL_CLIENT_SET_ASIDE and not _OFFICIAL_SAID: + _OFFICIAL_SAID.append(1) + print(f" set aside {sum(OFFICIAL_CLIENT_SET_ASIDE.values())} row(s) not measured through the benchmark's own HTTP client: " + + ", ".join(f"{b} {n}" for b, n in sorted(OFFICIAL_CLIENT_SET_ASIDE.items(), key=str))) + return list(best.values()) def cells(rows, key): @@ -1025,6 +1043,10 @@ def _dense_multipass(): continue p0 = passes[0] be = p0.get("backend") + if is_official_client_row(p0): + # the same rule as load_canonical: a served ArcadeDB overlay measured through the other client is not published + print(f" set aside {os.path.basename(fp)}: not measured through the benchmark's own HTTP client") + continue label = NAMES.get(be, be) if be == "arcadedb_dense_embedded": q = str(p0.get("quantization", "")).lower() @@ -1381,7 +1403,7 @@ def main(freeze=True): rows = load_canonical() print(f"{len(rows)} canonical rows") if freeze: - freeze_paper_rows(load_canonical(include_twins=True)) + freeze_paper_rows(rows) tabular_table(rows) graph_table(rows) sparse_table(rows) diff --git a/benchmarks/experiments/page_check.py b/benchmarks/experiments/page_check.py index 03797df61a6..1bb8381b8b4 100644 --- a/benchmarks/experiments/page_check.py +++ b/benchmarks/experiments/page_check.py @@ -629,8 +629,12 @@ def main() -> int: "(CAMPAIGN section 7 row 69)") j_bad = _check_jvm_defaults_disclosure(payload) print(f" {j_bad} table(s) missing the sentence") + print("\nno page text names the vendor's Python driver for ArcadeDB (DECISIONS #285)") + b_bad = _check_banned_terms(payload) + print(f" {b_bad} banned term(s) in page text") return 1 if (i_bad or d_bad or a_bad or l_bad or c_bad - or r_bad or m_bad or not u_ok or k_bad or o_bad or n_bad or iv_bad or ov_bad or j_bad or l_notice_bad) else 0 + or r_bad or m_bad or not u_ok or k_bad or o_bad or n_bad or iv_bad or ov_bad or j_bad or l_notice_bad + or b_bad) else 0 # -------------------------------------------------------------------------- @@ -1611,6 +1615,37 @@ def _check_jvm_defaults_disclosure(payload): return bad +# THE VENDOR'S DRIVER IS NOT NAMED ON THE PAGE (DECISIONS #268, #285). The served ArcadeDB headline is the benchmark's own +# HTTP client and the page says so; the vendor's driver is measured by the harness and is neither printed nor named as a +# comparison. Any string in the payload (a condition, a label, a cell note, a source) that names it fails the gate. +# REMOVE WHEN: upstream ships the transport fix (ArcadeData/arcadedb-drivers #188 merged and released) and the owner decides +# to publish that client; then replace this pattern with the disclosure that client needs. +BANNED_TERMS = re.compile(r"arcadedb[-_ ]?driver|official\s+(?:\w+\s+){0,3}?(?:driver|client)", re.I) + + +def _check_banned_terms(payload): + """Walk every string of the payload for BANNED_TERMS. Returns bad count.""" + bad = [] + + def walk(node, path=""): + if isinstance(node, str): + m = BANNED_TERMS.search(node) + if m: + bad.append((path, m.group(0), node[:100])) + elif isinstance(node, dict): + for k, v in node.items(): + walk(k, f"{path}.") + walk(v, f"{path}.{k}" if path else str(k)) + elif isinstance(node, list): + for i, v in enumerate(node): + walk(v, f"{path}[{i}]") + + walk(payload) + for path, term, ctx in bad[:12]: + print(f" BANNED {path}: {term!r} in {ctx!r}") + return len(bad) + + def _check_one_list_not_comparable(payload): """A dense cell the freeze withheld and that ArangoDB's one-list defect decided (BUGS F175, DECISIONS #156: recall below the floor, 10,000 or more diff --git a/benchmarks/experiments/test_export_client_twin.py b/benchmarks/experiments/test_export_client_twin.py index bf8890ee8a5..20f267e954a 100644 --- a/benchmarks/experiments/test_export_client_twin.py +++ b/benchmarks/experiments/test_export_client_twin.py @@ -1,6 +1,7 @@ -"""What the exporter adds for 2026-11 rows, on the publish fixture: the served ArcadeDB client twin kept out of every headline -median and attached to the headline entry, the l4 plain-read columns, and the sentences that say what the rows stamp (the idle -hold, the sparse warm-up, the official driver). +"""What the exporter adds for 2026-11 rows, on the publish fixture: every served ArcadeDB headline cell comes from the +benchmark's own HTTP client (DECISIONS #268, #285), the vendor's driver is neither printed nor named, the cell of the arm with +no lean row is dropped and said so, the l4 plain-read columns, and the sentences that say what the rows stamp (the idle hold, +the sparse warm-up). Run with `python -m pytest test_export_client_twin.py -q` from this directory (needs numpy and pandas, as the exporter does). """ @@ -18,6 +19,7 @@ HERE = Path(__file__).resolve().parent FIXTURE = HERE / "fixtures" / "publish_2611" / "results" PIN = "261100fix" +LEAN = "lean_http (http.client, keep-alive)" # lean_http.LEAN_NAME def _run(script, results, **extra): @@ -35,61 +37,104 @@ def _stage(dest): @pytest.fixture(scope="module") -def payloads(tmp_path_factory): - """(payload exported from the freeze, payload exported from the same freeze plus the twin's rows).""" - results = _stage(tmp_path_factory.mktemp("twin") / "results") - assert _run("make_paper_tables.py", results).returncode == 0 +def built(tmp_path_factory): + """(results dir, the paper-table run's stdout, the payload exported from the freeze).""" + results = _stage(tmp_path_factory.mktemp("lean") / "results") + proc = _run("make_paper_tables.py", results) + assert proc.returncode == 0, proc.stderr[-2000:] assert _run("export_web.py", results).returncode == 0 - plain = json.loads((results / "web_benchmarks.json").read_text()) - rows = list(csv.DictReader((results / "runs_paper.csv").open())) - fields = list(rows[0]) - twins = [json.loads(l) for l in (results / "runs.jsonl").open() if l.strip()] - twins = [r for r in twins if r.get("arcadedb_http_client_role") == "sensitivity" and not r.get("error")] - assert twins, "the fixture carries the lean twin's rows" - with (results / "runs_paper.csv").open("w", newline="") as fh: - w = csv.DictWriter(fh, fieldnames=fields) - w.writeheader() - w.writerows(rows) - for r in twins: - w.writerow({k: ("" if r.get(k) is None else r.get(k)) for k in fields}) - assert _run("export_web.py", results).returncode == 0 - return plain, json.loads((results / "web_benchmarks.json").read_text()) + return results, proc.stdout, json.loads((results / "web_benchmarks.json").read_text()) + + +@pytest.fixture(scope="module") +def payloads(built): + """(payload, payload): the old two-payload fixture's shape, for the tests below that read the plain one.""" + return built[2], built[2] def _table(p, tid): return next(t for t in p["tables"] if t["id"] == tid) -def test_the_twin_never_moves_a_headline_cell(payloads): - plain, twinned = payloads - for t in twinned["tables"]: - want = {(e["backend"], e.get("scale")): e["metrics"] for e in _table(plain, t["id"])["entries"]} - for e in t["entries"]: - assert e["metrics"] == want[(e["backend"], e.get("scale"))], (t["id"], e["backend"]) - - -def test_the_twin_rides_on_the_headline_entry_over_the_same_labels(payloads): - _plain, p = payloads - seen = set() - for t in p["tables"]: - for e in t["entries"]: - twin = e.get("client_twin") - if not twin: - continue - seen.add(t["id"]) - assert twin["client"] == "lean (requests)" and e["deployment"] in ("server", "all three") or t["id"] == "lifecycle" - assert twin["metrics"] and set(twin["metrics"]) <= set(e["metrics"]), (t["id"], set(twin["metrics"]) - set(e["metrics"])) - for stat in twin["metrics"].values(): - assert set(stat) == {"median", "min", "max", "n"} - assert {"docs_oltp", "docs_olap", "l2", "l3s", "l4", "lifecycle"} <= seen - assert not [e for t in p["tables"] for e in t["entries"] if "client_twin" in e and not e["is_arcadedb"]] - - -def test_the_page_names_the_official_driver_and_the_labelled_sensitivity(payloads): - _plain, p = payloads - text = " ".join(_table(p, "docs_oltp")["conditions"]) - assert "arcadedb-driver 0.2.0 (sync, httpx)" in text and "labelled sensitivity" in text and "lean (requests)" in text - assert not any("client" in c and "labelled sensitivity" in c for c in _table(p, "durability")["conditions"]) +def _raw_rows(results): + return [json.loads(l) for l in (results / "runs.jsonl").open() if l.strip()] + + +def test_the_fixture_holds_both_clients_and_the_freeze_keeps_only_the_lean_rows(built): + results, out, _p = built + raw = [r for r in _raw_rows(results) if r.get("arcadedb_http_client")] + assert {r["arcadedb_http_client_role"] for r in raw} == {"headline", "sensitivity"} + assert any("arcadedb-driver" in r["arcadedb_http_client"] for r in raw) + frozen = [r for r in csv.DictReader((results / "runs_paper.csv").open()) if r.get("arcadedb_http_client")] + assert frozen and {r["arcadedb_http_client"] for r in frozen} == {LEAN} + # the arm with no lean row has no frozen row at all, and the freeze says what it set aside + assert not [r for r in csv.DictReader((results / "runs_paper.csv").open()) if r["backend"] == "arcadedb_imgdefaults_server"] + assert "not measured through the benchmark's own HTTP client" in out and "arcadedb_imgdefaults_server" in out + + +def test_no_twin_and_no_vendor_driver_anywhere_on_the_page(built): + _results, _out, p = built + text = json.dumps(p) + assert "client_twin" not in text + for banned in ("arcadedb-driver", "official driver", "official Python client", "sensitivity", "httpx"): + assert banned not in text, banned + import page_check + assert page_check._check_banned_terms(p) == 0 + + +def test_the_page_names_the_benchmarks_own_client_and_says_the_others_differ(built): + _results, _out, p = built + for tid in ("docs_oltp", "docs_olap", "l2", "l3s", "l4", "lifecycle"): + text = " ".join(_table(p, tid)["conditions"]) + assert LEAN in text and "benchmarks/experiments/lean_http.py" in text, tid + assert "vendors' own Python clients" in text and "client weight differs between the served engines" in text, tid + assert "lean_http.py" in " ".join(_table(p, tid)["source_urls"]), tid + + +def test_the_arm_without_a_lean_row_is_not_printed_and_the_table_says_so(built): + _results, _out, p = built + t = _table(p, "docs_oltp") + assert not [e for e in t["entries"] if e["backend_key"] == "arcadedb_imgdefaults_server"] + label = "ArcadeDB (server, image JVM defaults)" + text = " ".join(t["conditions"]) + assert f"{label} is not printed: it was not measured through the benchmark's own HTTP client" in text + assert any(a["backend"] == label and a["kind"] == "withheld" for a in t["declared_absences"]) + assert "image's own JVM settings" not in text + + +def test_the_exporter_refuses_a_frozen_row_of_the_other_client(built, tmp_path): + results, _out, _p = built + bad = tmp_path / "results" + shutil.copytree(results, bad) + rows = list(csv.DictReader((bad / "runs_paper.csv").open())) + served = next(r for r in rows if r["backend"] == "arcadedb_server" and r.get("arcadedb_http_client")) + served["arcadedb_http_client"] = "arcadedb-driver 0.2.0 (sync, httpx)" + with (bad / "runs_paper.csv").open("w", newline="") as fh: + w = csv.DictWriter(fh, fieldnames=list(rows[0])) + w.writeheader() + w.writerows(rows) + proc = _run("export_web.py", bad) + assert proc.returncode != 0 and "not measured through the benchmark's own HTTP client" in proc.stderr + + +def test_the_exporter_refuses_an_overlay_of_the_other_client(built, tmp_path): + results, _out, _p = built + bad = tmp_path / "results" + shutil.copytree(results, bad) + fp = bad / f"dense_mp5_{PIN}" / "mp_arcsrv_b1.json" + passes = json.loads(fp.read_text()) + for r in passes: + r["arcadedb_http_client"] = "arcadedb-driver 0.2.0 (sync, httpx)" + fp.write_text(json.dumps(passes)) + proc = _run("export_web.py", bad) + assert proc.returncode != 0 and "overlay file(s) were not measured through the benchmark's own HTTP client" in proc.stderr + + +def test_the_banned_term_gate_flags_the_driver_and_nothing_else(): + import page_check + assert page_check._check_banned_terms({"conditions": ["through the official ArcadeDB Python driver"]}) == 1 + assert page_check._check_banned_terms({"tables": [{"conditions": ["arcadedb-driver 0.2.0"]}]}) == 1 + assert page_check._check_banned_terms({"conditions": ["through the benchmark's own minimal HTTP client"]}) == 0 def test_l4_prints_the_plain_read_beside_the_bulk_read(payloads): diff --git a/benchmarks/experiments/test_publish_2611_e2e.py b/benchmarks/experiments/test_publish_2611_e2e.py index 34a72df574d..1f7e1ff6c79 100644 --- a/benchmarks/experiments/test_publish_2611_e2e.py +++ b/benchmarks/experiments/test_publish_2611_e2e.py @@ -137,26 +137,31 @@ def test_the_pq_arms_are_their_own_precision_row_beside_fp32(published): assert emb["backend"] == fp32["backend"] == "ArcadeDB (embedded)" or emb["backend"].startswith("ArcadeDB (embedded") -def test_the_served_arcadedb_twin_row_is_not_in_the_headline(published): - # every served ArcadeDB cell has a lean twin (role "sensitivity", 30 percent faster); the headline cell is the median of - # the official-client rows alone +def test_the_served_arcadedb_headline_is_the_lean_client_alone(published): + # every served ArcadeDB cell has a vendor-driver row and a lean row (30 percent faster); the headline cell is the median + # of the lean rows alone, and the freeze holds no vendor-driver row t = _table(published["payload"], "docs_oltp") rows = [r for r in _fixture_rows() if r["lane"] == "l1tpc" and r["backend"].startswith("arcadedb") and "server" in r["backend"] and r.get("workload") == "oltp" and r.get("durability_class") == "relaxed"] assert rows and {r["arcadedb_http_client_role"] for r in rows} == {"headline", "sensitivity"} - head = [r for r in rows if r["arcadedb_http_client_role"] == "headline"] - key = next(k for k in head[0] if k.endswith("_p50_ms") and isinstance(head[0][k], (int, float))) - entry = next(e for e in t["entries"] if e["backend_key"] == head[0]["backend"]) + lean = [r for r in rows if r["arcadedb_http_client_role"] == "sensitivity" and r["backend"] == "arcadedb_server"] + other = [r for r in rows if r["arcadedb_http_client_role"] == "headline" and r["backend"] == "arcadedb_server"] + assert lean and other + key = next(k for k in lean[0] if k.endswith("_p50_ms") and isinstance(lean[0][k], (int, float))) + entry = next(e for e in t["entries"] if e["backend_key"] == "arcadedb_server") got = [m["median"] for m in entry["metrics"].values() if isinstance(m, dict) and "median" in m] - want = statistics.median(r[key] for r in head if r.get("scale") == entry["scale"]) + want = statistics.median(r[key] for r in lean if r.get("scale") == entry["scale"]) assert any(g == pytest.approx(want, rel=1e-3) for g in got), (key, want, got[:8]) - # the freeze keeps the twin rows and the exporter attaches them, labelled, to the headline entry + other_median = statistics.median(r[key] for r in other if r.get("scale") == entry["scale"]) + assert not any(g == pytest.approx(other_median, rel=1e-3) for g in got), "the vendor-driver median must not be a cell" frozen = list(csv.DictReader((published["results"] / "runs_paper.csv").open())) - assert any(r.get("arcadedb_http_client_role") == "sensitivity" for r in frozen) - twin = entry["client_twin"] - assert "lean" in twin["client"] and twin["metrics"] - label = next(k for k, m in entry["metrics"].items() if isinstance(m, dict) and m.get("median") == pytest.approx(want, rel=1e-3)) - assert twin["metrics"][label]["median"] == pytest.approx(want * 0.7, rel=1e-2) + assert not any("arcadedb-driver" in (r.get("arcadedb_http_client") or "") for r in frozen) + assert not any(r.get("arcadedb_http_client_role") == "headline" for r in frozen) + assert "client_twin" not in entry + # the arm with no lean row (the image-defaults arm) has vendor-driver rows only, so it has no frozen row and no cell + assert any(r["backend"] == "arcadedb_imgdefaults_server" for r in _fixture_rows()) + assert not any(r["backend"] == "arcadedb_imgdefaults_server" for r in frozen) + assert not any(e["backend_key"] == "arcadedb_imgdefaults_server" for e in t["entries"]) def test_the_paper_tables_are_written_beside_the_freeze(published):