diff --git a/benchmarks/experiments/CAMPAIGN.md b/benchmarks/experiments/CAMPAIGN.md index 155bab603d1..7319458f6c9 100644 --- a/benchmarks/experiments/CAMPAIGN.md +++ b/benchmarks/experiments/CAMPAIGN.md @@ -150,7 +150,7 @@ Rules every campaign keeps: | 3 | Dense ingest through `graph_batch(use_wal=True)` with Java float arrays | ArcadeDB only | **built (DECISIONS #116; 1.4x on the laptop)**: `l3d_dense.py` | DECISIONS #116 item 3 | | 4 | The served document and vector arms load through `INSERT ... CONTENT :rows` with the batch bound as one parameter | ArcadeDB only | built (DECISIONS #116, ArcadeData/arcadedb#8337): `l1_tpc.py` (`ArcadeServerTPC.load_batch`, `BENCH_SERVED_LOAD_BATCH`), `l3d_dense.py`, `l3_sparse.py`; parity, since every comparator loads through its vendor's bulk path. **No batch-size sweep is recorded** | `.notes` repros/served-doc-ingest; BUGS F151, F152 | | 5 | The strict durability setting is ArcadeDB `txWalFlush=1`, matched by effect to SQLite `synchronous=FULL`, not `txWalFlush=2` | ArcadeDB only | built (the user, DECISIONS #147; own issue #138): `bench_common.ARCADE_STRICT_TX_WAL_FLUSH`, `test_strict_durability.py`; the October string is kept for rows measured at 2; PROTOCOL section 7 has the row | `concepts/transactions.adoc` in upstream's docs; strace of the three settings on the laptop, 2026-09-24 | -| 6 | Served SQL arms over the Postgres wire (psycopg) instead of HTTP (`requests`) | ArcadeDB only | unblocked: #8288 fixed in ArcadeData/arcadedb#8308, and pgjdbc's default prepare in #8315 (#8244), both merged for 26.10.1 and verified; **decided 2026-09-26 (DECISIONS #116): measure on the bench host at the re-pin and switch if it is the faster read path** (the laptop cannot time below a millisecond; the maintainer already says it is, on #8332). **Still open:** the served arms read through the lean HTTP client (row 72) now, and the comparison on mini is still to run (DECISIONS #162 item 3) | `.notes` repros/protocol-latency | +| 6 | Served SQL arms over the Postgres wire (psycopg) instead of HTTP (`requests`) | ArcadeDB only | unblocked: #8288 fixed in ArcadeData/arcadedb#8308, and pgjdbc's default prepare in #8315 (#8244), both merged for 26.10.1 and verified; **decided 2026-09-26 (DECISIONS #116): measure on the bench host at the re-pin and switch if it is the faster read path** (the laptop cannot time below a millisecond; the maintainer already says it is, on #8332). **Built for the 26.11.1 four-arm pass (DECISIONS #166, #171, #237 (3)):** the served SQL cells run a third time as a twin job over the wire, ArcadeDB tier 1 only (DECISIONS #166) and at the small size tier only: no twin at `runner.LARGE_SCALES` (deep10m, tpch10, sf1full), because the twin re-runs the whole cell including load and index build (`runner.PGWIRE_SQL_ARMS`, run_id ending `_clientpgwire`, `lean_http.PgSession` over psycopg, never prepared, autocommit), a labelled sensitivity row and never a headline (it flatters ArcadeDB most, #166 addendum 2). Cells: l1 and l1tpc `arcadedb_server` (oltp, olap), l3s `arcadedb_sparse_server` and `_fp32`, l3d `arcadedb_dense_server` and `_int8`, l4 `arcadedb_ts_doc_server` and `arcadedb_ts_native_server`; not l2 (Cypher stays on HTTP, #162), e2, lifecycle, restart, e4. Only `/query` SQL goes over the wire: every write, DDL, transaction, bulk load and probe stays on HTTP (#171 (c)), so a twin row's write-phase numbers are lean HTTP and are not quoted. The twin's server starts the postgres plugin with explicit settings (`runner.PGWIRE_SERVER_SETTINGS`), read back from the engine and refused if unlisted or different. Answers-identical gate (`pgwire_gate.py`): each `res__digest` of the twin must equal the headline's for the same cell, under the lane's own declared coercions (l1tpc, l4), else the row is kept with `error` set and the reason recorded; l3s and l3d record no digest and are compared by `recall_at_10` within 0.01 of the headline median; l1 records neither, so its wire row is `unchecked` until the lane records digests; the smoke is in the PR body. Still open: the mini run and the numbers | `.notes` repros/protocol-latency | | 7 | What the memory column means: the peak, the working set after a full GC, or the peak with the heap printed beside it | every JVM arm | **decided (the user, DECISIONS #116 item 7): the peak, what an operator has to provision, with the heap setting printed beside it for JVM engines; built** (a `JVM heap` column from each cell's rows in `export_web`; the JVM memory note on every table that shows a JVM arm, BUGS F135). **No generated sentence may phrase the column as efficiency ("uses less memory"): a JVM's peak is the heap it was given; the like-for-like answer is the memory sweep.** **Still open, the memory sweep (the user, 2026-09-27), planned after the chain and the re-pin; it is an instrument change:** the smallest memory each arm completes a cell with. Design, settled before it runs: JVM arms (ArcadeDB both deployments, Neo4j) sweep the heap at a fixed cap, non-JVM arms the container cap; "completes" means the same answer digest within a stated slowdown of the full-memory run; one or two tables first (documents SF10, graph SF10); mini only, one pin; it costs days of bench time | DECISIONS #116 item 7 | | 8 | The sparse corpus generator yields NumPy, and each adapter converts in its own timer | comparators too (same work, moved) | measured 3.7% of ArcadeDB's build; do it with the next sparse measurement | bullet below | | 9 | The re-pin build: the official 26.10.1 with every fix we filed verified by its repro, comparators to latest stable (DECISIONS #129, relaxed by #163 to #165) | routine (steps 3 and 4 of the routine) | **built and verified.** The pin is the release's own jar commit `d36b4ca3a`; each fix we filed was verified by its repro on the laptop (the list is `.notes` issues-trail), and #8626, #8852, #8797, and #8725 landed after and are expected to move published cells. **Read the rows with these in hand, and say so on the page:** some cells get SLOWER (the embedded cross-model strict cells, whose `LSM_VECTOR` build had switched that thread's WAL off, #8129 fixed by #8221, and the sparse table's int8 arms, #8582); the dense and cross-model tables moved with the vector library (#8518), so compare recall at the same settings before reading a latency change; ranks can hold while times drop, because item 2 binds values on every engine. Laptop ratios are not projections of the page; only the re-measure decides. **Still owed:** re-verify #7146, #7183, #7184, and #7190 at 10M on the 26.10.1 pair; #7260 is fixed in 26.11.1 (PR #9289) and not in 26.10.1 | `.notes` issues-trail | @@ -202,7 +202,7 @@ Rules every campaign keeps: | 58 | Every write the documents and graph tables time is inside an answer check (new-order stock, payment count, KNOWS edges after the insert and after the delete) | docs_oltp and graph OLTP (l2) | built (DECISIONS #151, F172, within #150's remit): `l1_tpc.py`, `l2_graph.py` (`edge_scan`), `test_docs_post_state.py`, `test_graph_post_state.py`; `equivalence_check` names AGE's missing first edge as the open defect apache/age#2587 | BUGS F172 (`.notes/bench/repros/validity-20261004`) | | 59 | The answer profile (`res__profile`) and the degenerate-answer check (`degenerate_check.py`: ZERO, CONSTANT, SCALE, and EMPTY answers) | every lane (one field added to every row) | **built (DECISIONS #151, within #150's remit)**: `bench_common.py`, `degenerate_check.py`, `test_degenerate_check.py`, `test_result_digest.py` | `.notes/bench/repros/validity-20261004/SUMMARY.md`: agreeing digests hid both F146 and F169 | | 60 | LSQB's q5, q6, q8, and q9 run LSQB's own text on every engine; every graph analytics row records `lsqb_text` | every engine on l2olap | built (the user, DECISIONS #154 items 1 and 2), except the `sf1full` exclusion: `graph_common.py`, `l2_graph.py`, `test_lsqb_text.py`. **The exclusion of q6 and q9 at `sf1full` (`graph_common.TIER_EXCLUDED`, DECISIONS #109) is NOT lifted: it needs a one-cell first-touch probe on mini and a maintainer decision (DECISIONS #161 D2: lift it after the probe, at the end of the campaign).** | BUGS F174 | -| 61 | ArcadeDB's Graph Analytical View covers every vertex and edge type of the loaded graph, as upstream's LSQB runner builds it; the no-view arm (`BENCH_GAV=0`) stays as the like-for-like row | ArcadeDB only (l2olap with-view arms) | built (the user, DECISIONS #154 item 3): `l2_graph.py` (rows record `gav_types`, a narrower cover refuses the cell), `test_gav_scope.py` | `.notes/bench/repros/perf-lsqb-q9-20261004` | +| 61 | ArcadeDB's Graph Analytical View covers every vertex and edge type of the loaded graph, as upstream's LSQB runner builds it; the no-view arm (`BENCH_GAV=0`) stays as the like-for-like row | ArcadeDB only (l2olap with-view arms) | built (the user, DECISIONS #154 item 3): `l2_graph.py` (rows record `gav_types`, a narrower cover refuses the cell; from 26.11.1 the list also carries the abstract `Message` supertype, upstream PR 9320, chosen from the recorded engine version, `bench_common.GAV_SUPERTYPE_LISTED_FROM`), `test_gav_scope.py` | `.notes/bench/repros/perf-lsqb-q9-20261004` | | 62 | ArcadeDB's sparse arms return ids, as every comparator does (`SELECT id, score FROM (...)`); every ArcadeDB sparse row records `sparse_result` (`id`) | ArcadeDB only (l3s and its multipass warm passes) | built (the user, DECISIONS #153 item 1, adopted as parity): `l3_sparse.py`, `sparse_multipass_driver.py`, `test_sparse_rows.py` | `.notes/bench/repros/perf-sparse-20261004`: the ids and their order are identical on 50 of 50 queries | | 63 | The documents lane loads ArcadeDB embedded through the bindings' columnar insert (own issue #150) instead of `insert_many` | ArcadeDB embedded only | built (the user's choice over my recommendation, DECISIONS #153 item 2): `Database.insert_columns` (`7a36dc650e`), `ArcadeTPC.build` in `l1_tpc.py`, `test_insert_columns.py`, `test_docs_columnar_load.py`; the row records `columnar_insert`. **The measured wheel must contain the bindings commit** (row 74) | `.notes/bench/repros/perf-docs-ingest-20261004` | | 64 | The sparse multipass driver keeps each comparator's version, as the dense driver does: `sparse_multipass_driver.py` stores the adapter's own string as `lib_version` and restores `engine_version` for the ArcadeDB arms | every comparator's sparse multipass warm passes (`l3s_mp`); no timing moves | **built (DECISIONS #150's remit, own issue #158)**: `test_sparse_mp_versions.py`; the October files stay as recorded | BUGS F173 | @@ -221,7 +221,7 @@ Rules every campaign keeps: | 77 | ArcadeDB's query heap budget (`arcadedb.queryMaxHeapRAM`, default half the JVM heap) raised for the cells the default refuses: the l1tpc tpch10 olap `top_parts` GROUP BY in both ArcadeDB arms (DECISIONS #175, ArcadeData/arcadedb#9402) | ArcadeDB embedded and served, the documents lane, ONLY in the repair stage | **mechanism written, default OFF (humemai/arcadedb-embedded-python#260); the repair stage is generated after qP09 and run before the tier 1 to tier 2 boundary.** `BENCH_ARCADE_QUERY_MAX_HEAP_RAM_MB` adds the JVM option to the embedded arm and to the served arm's JAVA_OPTS and stamps `arcadedb_query_max_heap_ram_mb` and a source on the row; `override: arcadedb_query_ram`. Which rows are the headline (the default rows with the refusal shown, or the raised rows as an escape hatch like row 21's cap and row 75's body limit) is decided at landing. | not an instrument change: unset, nothing differs | | 78 | The comparator client paths are recorded per row (issue #261; DECISIONS #173, #174): `neo4j_rust_ext`, `redis_hiredis`, `es_json_serializer`, and `pg_client_path` (`autocommit`, `copy_binary`, `binary_cursor`); `qdrant_transport` (#172) was already stamped | comparators only: no ArcadeDB arm | built (DECISIONS #173, #174) | issue #261 | | 81 | The sparse lane warms up before its timed pass and the runner holds the CPU idle state for each timed cell (DECISIONS #177). Text on the page too: CPU idle states are not controlled on the host, and single-client times under a millisecond include the cores' wake-up latency (PAGE-SPEC section 0a, `setup.cpu_idle_note`) | the sparse lane (warm-up); every lane's rows (hold) | **built (humemai/arcadedb-embedded-python#278).** `bench_common.SPARSE_WARMUP_DEFAULT` untimed searches per engine, 1,000, one full round as the study behind the warm-up used, drawn from queries DISJOINT from the timed set so a query or result cache cannot flatter an engine: the dataset's extra queries when it holds them, else a fixed-seed sample of documents used as queries; the row stamps `sparse_warmup_queries` as run and `sparse_warmup_source` = `extra_queries` or `document_sample`. The hold is `/dev/cpu_dma_latency` at `BENCH_CPU_DMA_LATENCY_US` for the length of each timed cell: 10 microseconds in the 2026-11 pass, because the tree default `bench_common.CPU_DMA_LATENCY_DEFAULT_US` (0, no sleep state deeper than polling) kept every core in a busy-wait loop and made the host hot and slow (PROTOCOL.md section 7 has the A/B); it needs write access to the device on the host, and a hold that cannot be taken never fails the cell: rows stamp `cpu_dma_latency_us`, `cpu_dma_latency_held`, and `cpu_dma_latency_error` when it failed. `BENCH_SPARSE_WARMUP=N` and `BENCH_CPU_DMA_LATENCY_US=N` set the count and the target (PROTOCOL.md section 7); neither can be turned off. `override: sparse_warmup`, `override: cpu_dma_latency`; `test_idle_and_warmup.py`, `test_instrument_method.py` | part of the instrument | -| 82 | The served ArcadeDB headline is the benchmark's own minimal HTTP client (`lean_http.LeanSession`, stamp `lean_http (http.client, keep-alive)`); the page and the paper name it and say that the other engines are measured through their vendors' own Python clients | every ArcadeDB SERVED arm that talks HTTP through `lean_http.Session()`; no comparator client moves | **built (humemai/arcadedb-embedded-python#282, the instrument's client row; the headline selection is DECISIONS #268 and #285).** Each served cell runs as a second job on this client (`runner.ARCADEDB_HTTP_ARMS`, run_id ending `_clientlean`, `arcadedb_http_client_role` = `sensitivity` as stamped; the runner sets `BENCH_ARCADEDB_HTTP_CLIENT=lean` for it). `make_paper_tables.load_canonical` keeps the rows of this client and sets the other client's rows aside before any dedupe, table, gate or claim, so `runs_paper.csv`, the paper tables and the page hold these rows only; `export_web.py` refuses a frozen row or an overlay file that names another client, and `page_check.py` fails a page that names the vendor's driver. An arm with no job on this client (`arcadedb_imgdefaults_server`, and the PQ server arm) has no published cell; the docs transaction table says so for the first. This is the client the 26.10.1 pass and the published October rows used, so those rows stay comparable; rows measured with two clients are never mixed in one column. The `/query` and `/command` calls, the `GET /server` probe, the `/batch` bulk load, transactions and NDJSON streams all go through this client in these jobs. The PostgreSQL wire has no adapter in this harness. Measured on the official 26.10.1 image (d36b4ca3a), 7 interleaved rounds: the bound Cypher point read costs the client 55.5 us of CPU per call with `lean` against 11.6 for a bare-socket variant that was tried and retired, identical answer digests; the 100-row read is engine-bound and moves less | part of the instrument | +| 82 | The served ArcadeDB headline is the benchmark's own minimal HTTP client (`lean_http.LeanSession`, stamp `lean_http (http.client, keep-alive)`); the page and the paper name it and say that the other engines are measured through their vendors' own Python clients | every ArcadeDB SERVED arm that talks HTTP through `lean_http.Session()`; no comparator client moves | **built (humemai/arcadedb-embedded-python#282, the instrument's client row; the headline selection is DECISIONS #268 and #285).** Each served cell runs as a second job on this client (`runner.ARCADEDB_HTTP_ARMS`, run_id ending `_clientlean`, `arcadedb_http_client_role` = `sensitivity` as stamped; the runner sets `BENCH_ARCADEDB_HTTP_CLIENT=lean` for it). `make_paper_tables.load_canonical` keeps the rows of this client and sets the other client's rows aside before any dedupe, table, gate or claim, so `runs_paper.csv`, the paper tables and the page hold these rows only; `export_web.py` refuses a frozen row or an overlay file that names another client, and `page_check.py` fails a page that names the vendor's driver. An arm with no job on this client (`arcadedb_imgdefaults_server`, and the PQ server arm) has no published cell; the docs transaction table says so for the first. This is the client the 26.10.1 pass and the published October rows used, so those rows stay comparable; rows measured with two clients are never mixed in one column. The `/query` and `/command` calls, the `GET /server` probe, the `/batch` bulk load, transactions and NDJSON streams all go through this client in these jobs. The vendor's driver and the PostgreSQL wire are measured as further jobs of the same cell and are not published: the driver (0.4.0, in the `dbbench:client-arcadedb` image layer; `lean_http.OfficialSession` refuses any other version, `lean_http.OFFICIAL_PIN`) keeps the role `headline` as stamped, and the served SQL cells run a `_clientpgwire` twin (`arcadedb_client_wire` = `pgwire`, read statements only, answers gated identical by `pgwire_gate.py`); `make_paper_tables.is_official_client_row` sets both kinds of rows aside with the driver's. Measured on the official 26.10.1 image (d36b4ca3a), 7 interleaved rounds: the bound Cypher point read costs the client 55.5 us of CPU per call with `lean` against 11.6 for a bare-socket variant that was tried and retired, identical answer digests; the 100-row read is engine-bound and moves less | part of the instrument | | 84 | The dense lane's id index for the delete (humemai/arcadedb-embedded-python#291, DECISIONS #178 index parity): ArcadeDB, pgvector, Neo4j, Memgraph, FalkorDB, DuckDB, and LanceDB get an index on the vector id before or right after the load, the other engines already delete through their own key | the dense lane (l3d), every arm | **built.** Every dense row stamps `dense_id_index` (the statement, or `key`); `override: dense_id_index`; `test_dense_id_index.py`. The index build is inside `ingest_s`. The /next page says ArcadeDB's delete ran without an id index in rows that lack the stamp (`_dense_id_index_note`, list line `dense_id_index`; `test_next_measurement_wording.py` holds the wording) | part of the instrument | | 85 | Streamed results for the served ArcadeDB time-series arms, `BENCH_ARCADEDB_TS_NDJSON=1`, default OFF: both served arms ask the query endpoint for NDJSON and read the records line by line, as the PostgreSQL and MongoDB arms read rows as their server sends them | the time-series lane (l4), `arcadedb_ts_doc_server` and `arcadedb_ts_native_server` only | **mechanism written, default OFF; MEASURED 2026-10-09 and NOT USED at 26.11.1:** at ts100 the switch helped the documents arm (0.853 [0.713, 0.928]) and hurt the native arm (1.087 [1.083, 1.088]), identical rows, so it stays off and the page promises nothing; it needs a reader that parses as data arrives and a ts1000 re-measure first. Unset, nothing differs (`test_ts_ndjson.py`); set, rows stamp `arcadedb_ts_result_format` = `ndjson`; `override: ts_ndjson` | not an instrument change: unset, nothing differs | | 87 | Two failures of the 26.10.1 pass that were not results (humemai/arcadedb-embedded-python#344, the failure triage of 2026-10-20): (1) `chroma_dense` never ran, because the override registry `overrides.py` shadowed chromadb's dependency `overrides` on `/work` (`ImportError: cannot import name 'overrides'`); the registry is now `override_registry.py`. (2) Memgraph's storage access timeout (default 1 s) refused the deep10m vector index build in both dense arms; the dense arms now run `--storage-access-timeout-sec=172800`, longer than any cell cap | Chroma dense (every scale) and Memgraph dense (both precisions); no other arm moves (the rename changes no measurement) | **built on branch issue/344-chroma-memgraph, not on main**: `override_registry.py` and every import, `runner.MEMGRAPH_STORAGE_ACCESS_TIMEOUT_S`, the read-back `memgraph_storage_access_timeout_s`, PROTOCOL.md section 7, COMPARATORS.md. Mini picks it up at the `git pull` of the stage after the merge, which is the user's to time; a repair stage reruns Chroma dense and Memgraph dense deep10m | `.notes/bench/repros/failure-triage-20261020` | @@ -230,6 +230,7 @@ Rules every campaign keeps: | 90 | The ArcadeDB version a cell runs is the stage's pin, read out of what will run before the first cell, and no ArcadeDB image tag floats (humemai/arcadedb-embedded-python#470). `runner.py` takes the pin from the `ARCADEDB_WHEEL` file name and, for every ArcadeDB arm of the batch, reads the version out of the image that will run (embedded: the installed `arcadedb-embedded` version and the engine jar the wheel ships; served: the engine jar of the server image) and refuses to start on any difference, or when the batch has an ArcadeDB arm and no pin; the readback is in the manifest (`engine_pin`). The embedded image is `dbbench:arcadedb-`, written by `build_images.sh`, and the runner refuses every other ArcadeDB tag (`dbbench:arcadedb` stays only as the logical build target name in `runner.BACKENDS`); the comparator images keep `dbbench:`, their engines being pinned by `==` or digest and read back by `version_pin_check.py --runtime` | the embedded and served ArcadeDB arms of every lane | built, verified by unit tests | `engine_pin.py`, `test_engine_pin.py`; the trigger: the laptop's `dbbench:arcadedb` still held 26.10.1 beside `dbbench:arcadedb-26102` at 26.10.2 (2026-10-22, `.notes/bench/repros/ts-doc-oom-20261022/`), and every earlier check read rows after the run | | 91 | The composed embedded stack joins the cross-model lane: SQLite (documents) + LanceDB (vectors, fp32 `IVF_HNSW_FLAT`) + LadybugDB (graph) in one Python process, the default trio of the Cognee-style agent-memory builders and the real alternative to ArcadeDB for that audience (the owner, 2026-10-25) | `composed_embedded_sqlite_lance_ladybug`, the `e2` lane (hybrid in both durability classes, and the atomicity workload) | **built and dry-run on the laptop (2026-10-25), untimed**: `ladybug==0.21.2` added to the dense image's package line (an existing pin, no new version; 0.22.0 since 2026-10-25, row 92), the arm in `runner.BACKENDS` and `LANES["e2"]`, an `INDEX_DECISIONS` entry, the generator's `composed-embedded` engine (Tier 1, after `qdrant-neo4j`) and the regenerated `stage_drafts_2611.sha256`. Both ends of the lane's checks pass on its rows: the retrieval, hop and filtered-candidate digests equal LadybugDB's and DuckDB's at e2, 40 of 40 atomicity trials torn (nothing spans the stores), F10 and F14 clean | `COMPARATORS.md` "The composed embedded stack"; `repros/composed-embedded-lance/probe.py` | | 92 | LadybugDB moves to 0.22.0 on the client and dense images, and its `vector` extension is baked into the client image at build time, pinned by sha256 | every LadybugDB arm: `ladybug_graph`, `ladybug_dense`, `ladybug_e2`, `composed_embedded_sqlite_lance_ladybug`, `ladybug_lifecycle` | **built (2026-10-25)**: `build_images.sh` (`ladybug==0.22.0`, `LADYBUG_EXTENSIONS`), `Dockerfile.bench`, `bake_ladybug_extensions.py`, `test_ladybug_pin.py`. 0.21.2 can no longer `LOAD vector` (the vendor overwrote the v0.21.0 download at 04:28 UTC on 2026-10-25, LadybugDB/ladybug#1390). Untimed laptop cells on 0.22.0: dense micro with the mutation pass (0.20.4 control dies at the re-insert, 0.22.0 completes), graph micro, e2 (`ladybug_e2` and the composed arm). Open: the fdatasync counts in `bench_common` were measured at 0.21.2 and are not re-measured on 0.22.0; the DuckDB extensions are not baked yet (COMPARATORS.md, "Runtime extensions") | PR on humemai/arcadedb-embedded-python; `.notes` repros/ladybug-vector-ext-20261025 | +| 93 | The strict class passes the native server command line through unchanged except for the flush item: `durability_server_patch` replaces only the argument that is `-Darcadedb.txWalFlush=0` by the strict value, because every item of a native `server_cmd` is one argument and the helper that appends the token to a string had glued it onto `-Xmx` and every other `-D` item, so the native image refused to start (humemai/arcadedb-embedded-python#573) | the strict class of every served ArcadeDB native arm (the JVM served arms and the relaxed class are not affected) | **built, not in the running pass:** the four-arm pass holds its pin and does not sync; the fix is part of the commit of the planned catch-up, which measures the strict native server cells of the passes already taken (6 cells per pass) as new rows at that pin | `runner.py`, `test_native_strict_argv.py` (fails on the old code), #573 | **What the 2026-11 instrument measures (DECISIONS #196).** The harness measures one instrument and every method choice named here is the behaviour, not a switch. (1) The sparse lane warms up every engine on queries apart from the timed set, and the runner holds the CPU idle-exit latency at 10 microseconds for each timed cell (row 81); every timed cell of the pass also starts behind a pre-cell temperature gate (PROTOCOL.md section 2). (2) The dense lane indexes the vector id for the delete on every engine that would scan (row 84). (3) The served ArcadeDB headline is the official `arcadedb-driver`, with the lean HTTP client as the twin sensitivity row (rows 72, 82). (4) The l4 queries with large answers read each engine on its documented bulk path: for `q_high` and `q_groupby` ArcadeDB embedded `to_arrow()`, DuckDB `to_arrow_table()`, SQLite `fetchall()` (no columnar path), with `to_list()` / `fetchall()` timed beside it as `q_high_plain_ms` and `q_groupby_plain_ms`; rows stamp `read_path`, `read_path_queries`, `read_path_plain`, and `read_path_sensitivity`; small results do not move; a missing pyarrow or bridge jar falls back to `to_list()` and the row says `read_path_arrow_missing`; the typed read's digest must equal the plain read's after the lane's declared coerce, and a mismatch ends the cell. (5) The ArcadeDB arms, embedded and served, leave `-Xms` unset (the bindings' documented practice and the maintainers' guidance for a fixed memory budget; DECISIONS #200 for the served arms) and keep `-Xmx` from the cell's memory budget; Elasticsearch and Neo4j keep the initial heap they have (PROTOCOL.md section 7, "JVM initial heap"). `BENCH_ARCADEDB_TS_NDJSON` stays off (row 85, measured mixed). Every row stamps `instrument` = `2026-11`; a `BENCH_INSTRUMENT` of anything else is refused, the runner forwards its own value to every cell so a container cannot disagree with the launcher, and `test_instrument_method.py` holds the list. The 2026-10 and September instruments are retired: no cell is measured with them again. @@ -244,7 +245,7 @@ The detail behind item 8: **The next campaign's stages are generated by `make_2611_stages.py`.** One template carries every guard the campaign learned (cap table, shadowed definitions, budgets against the cap, whole-cell cost, images present and current, comparator digests frozen at generation, the JPype pin read out of the image, the pinned wheel by sha256, the cap rule, the ALL-DONE trap); the instrument, `REPS` and the banner are its parameters. The generator selects the multi-model subset, narrowed by a profile (next paragraph). The subset is `ENGINES` in the file: ArcadeDB (embedded and served), SurrealDB (embedded and served), ArangoDB, Memgraph, PostgreSQL with pgvector, AGE and TimescaleDB (and plain PostgreSQL where a lane uses it), MongoDB with MongoDB Search, DuckDB with VSS and DuckPGQ, SQLite with sqlite-vec, the composed Qdrant plus Neo4j stack, Neo4j and Qdrant on their own (graph and dense; sparse on the second list), and LadybugDB on every lane it supports. Every registered arm is either in the subset or in `OUTSIDE`, so a newly registered arm forces a decision (`--check`). The core lanes are graph (interactive and analytics), cross-model, documents and dense; under `--profile full`, `--second-list` adds lifecycle, then sparse and time series, after them. TimescaleDB runs whenever the time-series lane does (so not under the default profile); `--without-timescaledb` leaves it out. **THE PAPER PROFILE (the owner, 2026-10-25: the paper's experiments must finish by 2026-11-10; amended 2026-11-03, DECISIONS #232 and #235: the paper's numbers land about 2026-11-15, and the submission is 2026-11-24 for the SIGMOD 2027 industrial track or 2026-12-01 for the ICDE 2027 industry track, the fallback).** `--profile paper-2611` is the default and narrows the subset in the open (`PROFILES`, `DROP_WHY` in the file). (1) Core lanes only: graph, cross-model, documents and dense. The second list (lifecycle, sparse, time series) is out of scope for the paper, for the comparators and for the ArcadeDB chain, and `--second-list` is refused unless `--profile full` is given. (2) Neo4j and Memgraph stay in the graph lane (and cross-model where they are); their dense-lane cells (`neo4j_dense`, `neo4j_dense_int8`, `memgraph_dense`, `memgraph_dense_int8`) are out. The composed embedded stack (SQLite + LanceDB + LadybugDB in one process, `composed_embedded_sqlite_lance_ladybug`, the owner, 2026-10-25) is a Tier 1 cross-model arm: engine `composed-embedded`, right after the served composed stack, six lane cells per pass (hybrid in both durability classes and atomicity, at e2 and e2_500k), on the lane `e2` only; it runs in the dense image, so rebuild `dbbench:dense` (`./build_images.sh dense`) before the first pass. (3) The dense lane is fp32 only, for every engine: no `_int8` and no `_pq` arm, the comparators' and ArcadeDB's alike (`arangodb_dense_int8`, `arcadedb_dense_embedded_int8`, `arcadedb_dense_server_int8`, `mongodb_dense_int8`, `qdrant_dense_int8`, `sqlite_vec_dense_int8`, and the PQ arms when `BENCH_DENSE_PQ=1`). Quantized arms are not like-for-like across engines, and quantization is out of the paper's scope: fp32 for every engine, ArcadeDB's own INT8 and BINARY out by the owner's rule, jvector PQ, BQ and NVQ studied after the paper. ArcadeDB keeps its fp32 dense arms, embedded and served. (4) The target is N=3 (`TARGET_REPS`; `--project` prices 3 repetitions by default). The chain keeps the passes 4 and 5 after pass 3, in #503's order, so they run only if time is left; stopping after the first six groups is a complete result. Since 2026-11-03 the order on the bench host is: pass 3 (qC48), then the re-runs (qC14 at turbo off, qC15..qC23 at the 10 us idle hold, DECISIONS #234), then the four-arm ArcadeDB pass on the released 26.11.1 wheels, then passes 4 and 5 (qC49..qC80, which wait for `PASS45.go`). Amended 2026-11-05: if the 26.11.1 wheels are on PyPI and the four-arm runner is deployed when `REORDER reruns finished` is logged in the host's `STATUS.txt`, the four-arm pass (n=3 for the four ArcadeDB arms) starts first, and the window re-runs of qC14..qC17 follow it, still before `PASS45.go`; otherwise the order above stands. So the paper's comparators are N=3 for sure and N=5 where passes 4 and 5 finish before the tables freeze (DECISIONS #232, amendment of 2026-11-03). `--check` prints every dropped cell with its reason and count (lane cells per pass: 148 second-list, 8 Neo4j and Memgraph dense, 12 quantized dense) and proves that none is scheduled, in both chains whatever the arguments select (`profile_problems`); `--check --drafts DIR` lints written scripts for the same. `--profile full` is the whole subset as it was before the profile. Every generated stage exports `BENCH_INSTRUMENT=2026-11` before its first Python call, reads the selection back from `bench_common` and aborts if the tree does not select it, writes `runs_page__2611.jsonl` (and overlay directories with the same suffix) so one rows file never holds two instruments, and fixes `REPS` at generation (`--reps`, default 5, at most 5; the target is 3). The generator re-executes itself with the variable set and refuses a different value in the environment. `land_stage.py` lands exactly these files (DECISIONS #196; the 2026-10-25 change that closed the gap between the pull and the freeze): it pulls `runs_page__2611.jsonl` and the overlay directories with the same suffix, and `screen_pulled_rows` refuses the file before anything is merged unless every row is stamped `2026-11` and names the pin's engine commit, so a retired 2026-10 file or a file that mixes instruments never reaches `results/runs.jsonl`. The first landing meets no store (the October one is retired) and `merge_campaign.py` creates it. `export_web` no longer demands the `e4decomp_` artifacts of a landing scoped away from e4, because the chain has no e4 stage and `--only-lanes l2` would otherwise wait for one. A rehearsal restores what it wrote, the site payload (removed when the run created it) and the PAGE-SPEC.md block included. Failed cells (`rc != 0` or an error) merge as failures and are not frozen; censored queries keep their flag and publish no number. No stage script calls `land_stage.py` (the generated `qC` scripts only name it in a comment about the rows file); a landing is run by hand on the laptop. -The comparators are the default mode and run on the release pair already installed (`ARCADEDB_WHEEL`, `ARCADEDB_SERVER_IMAGE`, `ARCADEDB_ENGINE_COMMIT`): `python3 make_2611_stages.py --after --status-file STATUS.txt --id-prefix qS --out DIR`. `--after` is required, with no default (a marker name, or `none`), and the first stage waits for `grep -qx " ALL-DONE"`, each later one for the stage before it. The marker must be the LAST item still to run, one that writes its line whether it runs or is skipped: a marker already in the machine's STATUS.txt makes the first stage start at once, beside the running jobs (2026-10-23: `qMA06 ALL-DONE`, written by a skipped study). So `--status-file` is required with a marker (a copy of that STATUS.txt, or `none` when there is no queue) and the generator refuses a marker the file already holds; `--check --drafts DIR --status-file STATUS.txt` makes the same test on drafts that exist. The subset is emitted in ENGINE TIERS in the user's exact pass order (2026-10-24; `--tiers`, on by default, `--no-tiers` for the plain rep-major chain). N defaults to 5, the target is 3 (the paper profile), and the order is `schedule_order`: n=1 T1, n=2 T1, n=3 T1, n=1 T2, n=2 T2, n=3 T2, n=4 T1, n=4 T2, n=5 T1, n=5 T2. Every prefix of it is a complete set of equal replicates for the tiers covered so far (each tier has run passes 1..k, none skipped, and a pass is whole before the next starts); `--reps 3` stops after the first six groups, which is the chain of Tier 1 passes 1-3 and then Tier 2 passes 1-3 as before. Tier 1 is ArcadeDB (the four 26.11.1 arms, when `--arcadedb-release` adds them), SurrealDB, ArangoDB, PostgreSQL with pgvector and AGE, MongoDB with Search, the composed Qdrant plus Neo4j stack, and the embedded references SQLite, LadybugDB and DuckDB; Tier 2 is Memgraph, Neo4j and Qdrant on their own, Redis 8, CrateDB and TimescaleDB (`TIER_ENGINES`, with `ARM_TIER` for TimescaleDB, an arm of the `postgresql` engine that the user lists in Tier 2). Every stage holds the arms of one tier, and stage ids stay one run of consecutive numbers (a stage that mixed both tiers is split in two, so the stage count grows, and each stage repeats its guards and image checks; past 99 stages the ids take three digits, so `qS100` cannot sort before `qS11`). `--check` prints the schedule and proves the order and the prefixes (the plan collapsed into (tier, pass) groups equals `schedule_order`, each group holds the whole pass), and `--check --drafts DIR` lints a draft folder for the same. An arm with no tier runs after Tier 2 in each block. The ArcadeDB arms are Tier 1 but not in this chain: DBBench keeps one ArcadeDB version per host, so they run only once the benchmark pin is 26.11.1 (next paragraph). Inside a tier the order is the one below: a pass is pass k of every stage of the tier (core lanes, then the second list), and each stage runs ONE repetition (`--only-reps $PASS`). With `--no-tiers` the chain is pass 1 of every stage, then pass 2, up to pass N. If time runs out, every engine of a tier has the same n; repetitions days apart show drift. A stage whose environment declares fewer reps (`REPS=k`) is left out of the later passes. Inside a lane the engines run in `ENGINE_PRIORITY` (SurrealDB, ArangoDB, PostgreSQL, MongoDB, Redis 8, composed Qdrant+Neo4j, Neo4j, Qdrant, DuckDB, SQLite, Memgraph, LadybugDB, CrateDB; ArcadeDB's arms lead it when they join; Redis and CrateDB are skipped until their adapters are in `ENGINES`). The rep-1-first rule and the cap rule hold across passes: a pass above 1 runs a cell only when rep 1 of it was clean, and only a timeout of rep 1 censors the larger sizes. WHICH REPS EXIST is `~/rep-state-2026-11-.tsv` on the host (one `label, rep, ok|fail` line per finished repetition; the rows themselves carry the rep number in the runner's results file): stop after any pass, and the next pass, or a restarted stage, picks up from it with no regeneration; N can go to 3 or stop at 2. Delete that file to start a chain over. The ArcadeDB pass is `--arcadedb-release X.Y.Z` with `--engine-commit`, `--image-digest sha256:...`, `--wheel-name`, `--wheel-sha256` and `--wheel-url`; it refuses unless all five are given, because the release may not exist when the drafts are made. It writes `00`, the pair install (official image by digest, wheel by sha256, `build_matched_pair.sh`, `verify_pair_c25.sh`), and ArcadeDB's lane stages behind it. THE ARCADEDB GATE: the ArcadeDB chain is its own chain (Tier 1 only, passes 1..N in order), generated with `--after` the last comparator stage, so the comparators run the schedule first. `00` waits after its `--after` for the marker file `$HOME/conf/go-arcadedb-2611` on the host (it never creates it; it sits before the ALL-DONE trap, so a killed waiter hands nothing on), and every ArcadeDB lane stage aborts without it. The user or the agent touches that file when `CURRENT_PIN` moves to 26.11.1, and not before; `--check --drafts` lints that the gate is in the pair install and in every ArcadeDB stage and in no comparator stage. +The comparators are the default mode and run on the release pair already installed (`ARCADEDB_WHEEL`, `ARCADEDB_SERVER_IMAGE`, `ARCADEDB_ENGINE_COMMIT`): `python3 make_2611_stages.py --after --status-file STATUS.txt --id-prefix qS --out DIR`. `--after` is required, with no default (a marker name, or `none`), and the first stage waits for `grep -qx " ALL-DONE"`, each later one for the stage before it. The marker must be the LAST item still to run, one that writes its line whether it runs or is skipped: a marker already in the machine's STATUS.txt makes the first stage start at once, beside the running jobs (2026-10-23: `qMA06 ALL-DONE`, written by a skipped study). So `--status-file` is required with a marker (a copy of that STATUS.txt, or `none` when there is no queue) and the generator refuses a marker the file already holds; `--check --drafts DIR --status-file STATUS.txt` makes the same test on drafts that exist. The subset is emitted in ENGINE TIERS in the user's exact pass order (2026-10-24; `--tiers`, on by default, `--no-tiers` for the plain rep-major chain). N defaults to 5, the target is 3 (the paper profile), and the order is `schedule_order`: n=1 T1, n=2 T1, n=3 T1, n=1 T2, n=2 T2, n=3 T2, n=4 T1, n=4 T2, n=5 T1, n=5 T2. Every prefix of it is a complete set of equal replicates for the tiers covered so far (each tier has run passes 1..k, none skipped, and a pass is whole before the next starts); `--reps 3` stops after the first six groups, which is the chain of Tier 1 passes 1-3 and then Tier 2 passes 1-3 as before. Tier 1 is ArcadeDB (the four 26.11.1 arms, when `--arcadedb-release` adds them), SurrealDB, ArangoDB, PostgreSQL with pgvector and AGE, MongoDB with Search, the composed Qdrant plus Neo4j stack, and the embedded references SQLite, LadybugDB and DuckDB; Tier 2 is Memgraph, Neo4j and Qdrant on their own, Redis 8, CrateDB and TimescaleDB (`TIER_ENGINES`, with `ARM_TIER` for TimescaleDB, an arm of the `postgresql` engine that the user lists in Tier 2). Every stage holds the arms of one tier, and stage ids stay one run of consecutive numbers (a stage that mixed both tiers is split in two, so the stage count grows, and each stage repeats its guards and image checks; past 99 stages the ids take three digits, so `qS100` cannot sort before `qS11`). `--check` prints the schedule and proves the order and the prefixes (the plan collapsed into (tier, pass) groups equals `schedule_order`, each group holds the whole pass), and `--check --drafts DIR` lints a draft folder for the same. An arm with no tier runs after Tier 2 in each block. The ArcadeDB arms are Tier 1 but not in this chain: DBBench keeps one ArcadeDB version per host, so they run only once the benchmark pin is 26.11.1 (next paragraph). Inside a tier the order is the one below: a pass is pass k of every stage of the tier (core lanes, then the second list), and each stage runs ONE repetition (`--only-reps $PASS`). With `--no-tiers` the chain is pass 1 of every stage, then pass 2, up to pass N. If time runs out, every engine of a tier has the same n; repetitions days apart show drift. A stage whose environment declares fewer reps (`REPS=k`) is left out of the later passes. Inside a lane the engines run in `ENGINE_PRIORITY` (SurrealDB, ArangoDB, PostgreSQL, MongoDB, Redis 8, composed Qdrant+Neo4j, Neo4j, Qdrant, DuckDB, SQLite, Memgraph, LadybugDB, CrateDB; ArcadeDB's arms lead it when they join; Redis and CrateDB are skipped until their adapters are in `ENGINES`). The rep-1-first rule and the cap rule hold across passes: a pass above 1 runs a cell only when rep 1 of it was clean, and only a timeout of rep 1 censors the larger sizes. WHICH REPS EXIST is `~/rep-state-2026-11-.tsv` on the host (`-jvm` or `-native` is added to the name in an ArcadeDB release chain, because both chains label their cells identically; one `label, rep, ok|fail` line per finished repetition; the rows themselves carry the rep number in the runner's results file): stop after any pass, and the next pass, or a restarted stage, picks up from it with no regeneration; N can go to 3 or stop at 2. Delete that file to start a chain over. The ArcadeDB pass is `--arcadedb-release X.Y.Z` with `--engine-commit`, `--image-digest sha256:...`, `--wheel-name`, `--wheel-sha256` and `--wheel-url`; it refuses unless all five are given, because the release may not exist when the drafts are made. It writes `00`, the pair install (official image by digest, wheel by sha256, `build_matched_pair.sh`, `verify_pair_c25.sh`), and ArcadeDB's lane stages behind it. THE ARCADEDB GATE: the ArcadeDB chain is its own chain (Tier 1 only, passes 1..N in order), generated with `--after` the last comparator stage, so the comparators run the schedule first. `00` waits after its `--after` for the marker file `$HOME/conf/go-arcadedb-2611` on the host (it never creates it; it sits before the ALL-DONE trap, so a killed waiter hands nothing on), and every ArcadeDB lane stage aborts without it. The user or the agent touches that file when `CURRENT_PIN` moves to 26.11.1, and not before; `--check --drafts` lints that the gate is in the pair install and in every ArcadeDB stage and in no comparator stage. The ArcadeDB pair's guards are split by what a stage runs. Every stage exports the pinned wheel (`ARCADEDB_WHEEL`, with `ARCADEDB_SERVER_IMAGE` and `ARCADEDB_ENGINE_COMMIT`) before `build_images.sh`, which bakes it into every generic image (client, duckdb, dense), and so every stage guards that the file exists at the pinned path and has the pinned sha256; `build_images.sh` copies it from that path and nothing in it or the Dockerfiles reads `dist/` itself. Only a stage that runs an ArcadeDB arm (the `--arcadedb-release` pass) also guards the rest: exactly one wheel in `dist/` (it protects `verify_pair_c25.sh`), the server image, the wheel and JPype read back out of `dbbench:arcadedb`, and `verify_pair_c25.sh`. A comparator stage on a host whose `dbbench:arcadedb` is the previous release's therefore runs. `build_images.sh` has no PyPI fallback: it refuses to build the `arcadedb` image (named or by default) without `ARCADEDB_WHEEL`, and refuses a wheel that is not a file, and refuses a wheel whose `arcadedb_embedded/core.py` lacks `_INSERT_MANY_CHUNK` (`wheel_guard.py`: before 26.10.2 `insert_many` parsed the whole input into one Java array, which the kernel killed on l4/ts1000/arcadedb_ts_doc on 2026-10-08; use a 26.10.2 or newer wheel); the images that take no wheel can be built alone by naming them. Every ABORT hands on: the `ALL-DONE` trap is set right after the wait, before the first guard, so a stage that aborts before its START line still writes its marker and the next stage runs (its own guards then run too). The ABORT line is in `STATUS.txt`, where `mini_watch.sh` reports it; to resume, fix the cause and run the aborted stage's script again by hand after the chain, and landing reads only the rows that exist. `make_2611_stages.py --check --drafts DIR` lints both rules. The default order is the core block first, then (under `--profile full`) the second list. Inside the core block the large tiers (deep10m, tpch10, the full SF1 network) run after the small ones, and the cells that dominate the hours (Memgraph's deep10m dense cells, which the paper profile does not stage, SurrealDB's documents at tpch1, the full SF1 network's analytics) run last of all in stages of their own, where they are easiest to drop if time runs short; `BIGGEST` in the file names them, and `--order paper` keeps the table order. Splitting moves stage boundaries only, and `--check` proves the cells are the same. `--project TABLE` prints projected hours per drafted stage from a per-cell hours table (tab separated: lane, workload, scale, backend, class, hours at 3 repetitions, hours at 5 repetitions; a cell with no clean rep carries its one failed attempt). `--check --drafts DIR` lints written scripts: the instrument exported and never 2026-10, the 2611 rows file, only subset arms, each stage waiting for the one before it. Drafts are written to a folder outside the tree and copied to the bench host by hand on the owner's go. Which stage is running is state, not method: it is in `STATUS.txt`'s tail and in HANDOFF, never here. `test_stage_generator_2611.py` holds the written drafts byte for byte against `stage_drafts_2611.sha256`, so a stage cannot move without the list changing in the same commit. diff --git a/benchmarks/experiments/COMPARATORS.md b/benchmarks/experiments/COMPARATORS.md index 4fc35fb73b1..f7997049ee1 100644 --- a/benchmarks/experiments/COMPARATORS.md +++ b/benchmarks/experiments/COMPARATORS.md @@ -72,16 +72,33 @@ Pins follow DECISIONS #87 (the latest self-hosted stable release at campaign tim | Neo4j driver (Neo4j AND Memgraph arms) | `neo4j==6.4.1` | client package | | Neo4j driver Rust extension (the vendor's documented accelerator, DECISIONS #173) | `neo4j-rust-ext==6.4.1.0` | client package | | pgvector Python adapter (binary COPY, DECISIONS #174) | `pgvector==0.5.1` | client package | -| ArcadeDB Python client package installed in the client image (measured by the harness, not a published client; the served ArcadeDB headline is the benchmark's own `lean_http.py`, CAMPAIGN section 7 row 82) | `arcadedb-driver==0.2.0` | client package | +| ArcadeDB Python client package (measured by the harness, not a published client: the served ArcadeDB headline is the benchmark's own `lean_http.py`, CAMPAIGN section 7 row 82) | `arcadedb-driver==0.4.0` | client package, in the `dbbench:client-arcadedb` layer (below) | + +**The served ArcadeDB client image (DECISIONS #258, #259).** The served ArcadeDB arms run in `dbbench:client-arcadedb`: `dbbench:client` plus one pip layer (`Dockerfile.client-arcadedb`, `build_images.sh` target `client-arcadedb`, list `ARCADEDB_DRIVER_PKGS`), so `dbbench:client` stays byte-identical and every comparator row measured on it stays on one client. The vendor's driver is measured by the harness and not published (the page headline is `lean_http.py`, DECISIONS #268, #285). `arcadedb-driver==0.2.0` remains in `dbbench:client` (the `[client]` line of `build_images.sh`, unused by any cell now); `lean_http.OfficialSession` refuses to start on any driver version but `lean_http.OFFICIAL_PIN` (0.4.0). The driver's environment is pinned at the latest PyPI release of 2026-11-07 (the driver itself asks only `attrs>=23.2`, `httpx>=0.27`, `typing-extensions>=4.10`, `websockets>=14`): `arcadedb-driver==0.4.0` (2026-11-05), `httpx==0.28.1`, `httpcore==1.0.9`, `h11==0.16.0`, `anyio==4.16.0`, `idna==3.20`, `certifi==2026.10.23`, `attrs==26.1.0`, `typing-extensions==4.16.0`, `websockets==17.4`. This layer moves httpx, anyio, certifi and the other listed libraries only inside the served ArcadeDB client image, where no comparator client runs. Scope of the driver in a served cell: `/query` and `/command` (with `limit`); bulk `/batch` loads, NDJSON streams, transactions, probes and raw text bodies stay on the lean HTTP client (DECISIONS #259 (4)). **Held below the newest, and why.** - **MongoDB 8.3.x is skipped, and 9.0.4 is the pin** (the one engine where #103d overrode #87; hold lifted 2026-10-23). 8.3.13, the newest 8.3 patch (Docker Hub 2026-10-17), still refuses to start on the bench kernel with the SERVER-121912 guard ("Linux kernel versions 6.19 and newer has a known incompatibility"), both as plain `mongo` and inside the Search image, with mongot 1.70.4 and with 1.70.5. 9.0.4 (Docker Hub 2026-10-17, amd64 manifest `sha256:b609cdfba3a8…`; 9.0.0 shipped 2026-09-28 and the release notes call 9.0 the current stable release) starts on the same kernel. Why it starts was not investigated: SERVER-125742 exempts kernels from upstream 7.0.14 but reads `uname -r`, which Ubuntu keeps at 7.0.0 (open as SERVER-136848, reported against 8.3.11 and 9.0.2); the bench kernel 7.0.0-38-generic is upstream 7.0.14, so it carries the userspace fix the guard exists for, and the old argument that an older 8.3.x lacking the warning would run a configuration its vendor calls unsafe no longer holds. Why not stay on 8.2.12, which also boots: MongoDB's compatibility table for MongoDB Search lists mongot 1.70.1 and later against MongoDB Server 8.3 only and says 8.2 "is no longer a supported source deployment for MongoDB Search", so 8.2.12 + mongot 1.70.4 was a pairing the vendor no longer supports. 9.0 is not in that table either: 9.0.4 + mongot 1.70.5 is exercised, not documented. Laptop test 2026-10-23, kernel 7.0.0-38-generic, containers on cores 12-15 with `-m 3g`/`-m 4g`, `pymongo==4.18.1`, through the repo's own `l1_tpc.MongoTPC` on 30,000 seeded synthetic lineitem rows and `mongo_common` on 3,000 seeded 32-d vectors: 8.2.12 boots, 8.3.13 does not (also with mongot 1.70.4 and 1.70.5), 9.0.4 boots; on 9.0.4 against 8.2.12 the five analytics pipelines, 200 committed new-order transactions, 50 payments, the abort (nothing changed), the write-conflict error (code 112), `with_transaction` and 100 single-record CRUD operations gave identical digests; `$vectorSearch` through mongot 1.70.5 returned the same top-10 lists as 8.2.12 + 1.70.4 on 99.3% of 300 queries (mean set overlap 0.9993, recall 1.000 against exact search) for `quantization: none`, and 0.960 mean set overlap for `scalar` (recall 0.878 against 0.873, a quantized index is not deterministic across builds); both indexes read their quantization back, and a `$vectorSearch` inside a multi-document transaction is still refused. Not tested: performance (every MongoDB row has to be re-measured on 9.0.4), a full TPC-H scale, restart/recovery, the time-series and graph lanes. mini runs kernel 7.0.0-31-generic, the same 7.0 line; a one-minute `docker run --rm mongo@sha256:b609cdfb… mongod --version`-style start check there is a sanity step, not a requirement. - **DuckDB and DuckPGQ 1.5.6**, not 2.0.0 (checked 2026-10-23, laptop, duckdb 2.0.0 from PyPI): the `duckpgq` community build (`29ca668a22`) and `vss` both exist for v2.0.0 and both LOAD, VSS HNSW works, but DuckPGQ registers no syntax: `CREATE PROPERTY GRAPH` and `GRAPH_TABLE ... MATCH` answer `Parser Error: syntax error`, with or without `SET enable_legacy_parser` or `allow_parser_override_extension`. So the graph arm cannot be measured on 2.0.0 (DECISIONS #103d: the newest version at which every arm can be measured). Lift the hold when a DuckPGQ build for 2.x parses; the recipe is in `build_images.sh`. 1.5.5 had no DuckPGQ build (HTTP 404) and 1.5.6 does (`61f5d082`). -- **SurrealDB server v3.2.4 and SDK 2.0.0**: the SDK is the newest stable. The server is held one minor behind the newest stable (`v3.3.4`, `latest` on 2026-10-23) on purpose: on 3.3.4 the l2 graph OLTP delete left one person and its edge behind with no error in 2 of 3 runs (0 of 2 on 3.2.4), so the bump waits for a repeat study (humemai/arcadedb-embedded-python#483). Lift the hold when 3.3.x or later deletes cleanly. -- **Chroma 1.5.9, sqlite-vec 0.1.9, Memgraph 3.14.0, pymongo 4.18.1, python-arango 8.3.6, falkordb 1.7.1, redis 8.1.0**: the newest stable releases (sqlite-vec's `0.1.10a4` is a pre-release). **pymilvus 3.0.2** (with the `bulk_writer` extra, which brings `minio`, `azure-storage-blob` and `ml-dtypes`) is what Milvus's compatibility table names for a 3.0.3 server. +- **SurrealDB server v3.2.4 and SDK 2.0.0**: the SDK is pinned at the October re-pin; a newer release exists (3.0.0, 2026-11-04); not changed because its rows are already measured ("Client holds" below). The server is held one minor behind the newest stable (`v3.3.4`, `latest` on 2026-10-23) on purpose: on 3.3.4 the l2 graph OLTP delete left one person and its edge behind with no error in 2 of 3 runs (0 of 2 on 3.2.4), so the bump waits for a repeat study (humemai/arcadedb-embedded-python#483). Lift the hold when 3.3.x or later deletes cleanly. +- **Chroma 1.5.9, sqlite-vec 0.1.9, Memgraph 3.14.0, redis 8.1.0**: the newest stable releases (sqlite-vec's `0.1.10a4` is a pre-release). **pymilvus 3.0.2** (with the `bulk_writer` extra, which brings `minio`, `azure-storage-blob` and `ml-dtypes`) is what Milvus's compatibility table names for a 3.0.3 server. - **SQLite 3.46.1** is not a pin: it is the stdlib `sqlite3` of the `python:3.12-slim` base, read back at connect. **ArcadeDB is not a comparator**: it is pinned to our own wheel and the matching `arcadedb-c25` image (its row above). +**Client holds (checked on PyPI 2026-11-07; DECISIONS #259 (3)).** Each comparator client library is pinned at the October re-pin (2026-09-19, individual bumps to 2026-10-25) and not changed afterwards because its rows are already measured: a newer client mid-campaign would split a comparator's rows across two client versions. Each package below has a newer release today. The hold is a decision, not a claim that the pin is the newest. + +| Client | Pinned | Newer release (PyPI date) | Why held | +|---|---|---|---| +| psycopg (binary) | 3.3.6 | 3.3.8 (2026-11-04) | rows already measured | +| pymongo | 4.18.1 | 4.19.0 (2026-10-14) | rows already measured | +| python-arango | 8.3.6 | 8.3.7 (2026-10-27) | rows already measured | +| falkordb | 1.7.1 | 1.8.1 (2026-10-25) | rows already measured | +| lancedb | 0.39.0 | 0.41.0 (2026-10-29) | rows already measured | +| surrealdb (Python SDK) | 2.0.0 | 3.0.0 (2026-11-04) | rows already measured; a major version | +| ladybug | 0.22.0 | 0.22.3 (2026-11-06) | tied to the baked `vector` extension sha256 (`LADYBUG_EXTENSIONS`); moving it means re-reading that sha and re-measuring | +| duckdb | 1.5.6 | 2.0.0 (2026-10-21) | documented hold: DuckPGQ registers no syntax on 2.0.0 (above) | +| azure-storage-blob | 12.31.0 | 12.31.1 (2026-10-29) | rows already measured (a pymilvus `bulk_writer` dependency) | +| numpy, pandas, pyarrow (harness libraries) | 2.5.3, 3.0.6, 25.0.1 | 2.5.4 (2026-10-10), 3.1.1 (2026-11-04), 26.0.1 (2026-11-03) | rows already measured; "move them deliberately" (`build_images.sh`) | + **Why some moves were smoked.** The PostgreSQL set moves as one major (#103d); AGE gated it, being the only member packaged per major (`postgresql-18-age`). PostgreSQL 18 turns data checksums on by default, a write-path cost the page keeps because it is PostgreSQL's own default. Neo4j is pinned at .1 because 2026.08.0 could fail queries through its resource monitoring, and a new index provider (`vector-2026.08`) is auto-selected for new indexes, so the dense operating point is a measurement and not an inheritance (the Neo4j row says how). **AGE 1.8.0 is not a labelling change.** Holding a 5,000-edge batch fixed and varying the vertex count, AGE 1.7.0 takes 1.56 s at 160,000 vertices and 11.14 s at 500,000 (a jump of the shape of a planner flip, so the ratio is not a constant and must not be quoted as one), where 1.8.0 takes 0.18 s at both: 62x at 500,000. October's pg_age build numbers differ from September's for that upstream reason and not for an instrument change; a pg_age cell that ran the stale image was censored at the 2 h cap (BUGS F90, F76). diff --git a/benchmarks/experiments/Dockerfile.bench b/benchmarks/experiments/Dockerfile.bench index 4ed1a3fb3be..597e352a461 100644 --- a/benchmarks/experiments/Dockerfile.bench +++ b/benchmarks/experiments/Dockerfile.bench @@ -1,6 +1,6 @@ # Generic benchmark image. Pin the package set via build ARG so each backend # (arcadedb, duckdb, ladybug, chroma, faiss, ...) gets its own pinned image later. -# arcadedb-embedded bundles its own JRE, so no Java install is needed. +# arcadedb-jvm (and the older arcadedb-embedded) bundle their own JRE, so no Java install is needed; the native `arcadedb` wheel has no JVM. FROM python:3.12-slim ARG PIP_PACKAGES="" @@ -12,16 +12,21 @@ ARG PIP_PACKAGES="" # default is bench_common.JPYPE_PIN; `make_2611_stages.py --check` refuses a # default that differs, and every stage reads the version back out of the image. ARG JPYPE_VERSION=1.7.1 -# Local wheels (e.g. a not-yet-released arcadedb-embedded) placed in +# NO JPYPE IN A NATIVE IMAGE (26.11.1): the `arcadedb` wheel is the native build, no JVM and no JPype, and its rows must say so +# (bench_common.jpype_fields reads the JPype the process imported). JPype is installed with `arcadedb-jvm`, `arcadedb_jvm-*.whl`, +# `arcadedb-embedded` and `arcadedb_embedded-*.whl`; a wheel named `arcadedb-...` (the native one) gets none. +# Local wheels (e.g. a not-yet-released arcadedb-jvm) placed in # docker/wheels/ are installed first; the dir ships empty (.gitkeep). COPY docker-wheels/ /tmp/wheels/ RUN set -e; \ if ls /tmp/wheels/*.whl >/dev/null 2>&1; then \ - pip install --no-cache-dir /tmp/wheels/*.whl "jpype1==${JPYPE_VERSION}"; \ + _jp=""; \ + for _w in /tmp/wheels/*.whl; do case "$(basename "$_w")" in arcadedb-[0-9]*) ;; *) _jp="jpype1==${JPYPE_VERSION}";; esac; done; \ + pip install --no-cache-dir /tmp/wheels/*.whl ${_jp}; \ fi; \ if [ -n "${PIP_PACKAGES}" ]; then \ _jp=""; \ - case " ${PIP_PACKAGES} " in *" arcadedb-embedded"*) _jp="jpype1==${JPYPE_VERSION}";; esac; \ + case " ${PIP_PACKAGES} " in *" arcadedb-embedded"*|*" arcadedb-jvm"*) _jp="jpype1==${JPYPE_VERSION}";; esac; \ pip install --no-cache-dir ${PIP_PACKAGES} ${_jp}; \ fi; \ if python -c "import jpype" 2>/dev/null; then \ diff --git a/benchmarks/experiments/Dockerfile.client-arcadedb b/benchmarks/experiments/Dockerfile.client-arcadedb new file mode 100644 index 00000000000..9febc3605e0 --- /dev/null +++ b/benchmarks/experiments/Dockerfile.client-arcadedb @@ -0,0 +1,13 @@ +# The image the served ArcadeDB client cells run in: the shared `dbbench:client` image plus the official `arcadedb-driver` at the release the +# harness pins (lean_http.OFFICIAL_PIN, DECISIONS #258, #259) and the driver's own dependencies, each pinned. +# +# WHY A LAYER AND NOT A NEW PIN IN `client`. dbbench:client carries every comparator's client library, and rows made on it are already +# measured. Changing the arcadedb-driver pin in PKGS[client] (a build arg) would rebuild that image under a new digest and split those rows. +# A layer on top leaves dbbench:client byte-identical; only the served ArcadeDB arms (runner.py: ARCADEDB_CLIENT_IMAGE) run here. +# `pip install ==` of the whole list in one command leaves the resolver no room to move any of them (httpx is unpinned in the driver's +# own requirements, websockets>=14). +FROM dbbench:client +ARG DRIVER_PACKAGES="" +RUN set -e; [ -n "${DRIVER_PACKAGES}" ] || { echo "DRIVER_PACKAGES is empty" >&2; exit 1; }; \ + pip install --no-cache-dir ${DRIVER_PACKAGES}; \ + python -c "import importlib.metadata as m, sys; bad = [p for p in sys.argv[1:] if m.version(p.split('==')[0]) != p.split('==')[1]]; sys.exit('not at the pin: ' + ', '.join(bad) if bad else 0)" ${DRIVER_PACKAGES} diff --git a/benchmarks/experiments/PAGE-SPEC.md b/benchmarks/experiments/PAGE-SPEC.md index d00a82b7e40..22ad34e8911 100644 --- a/benchmarks/experiments/PAGE-SPEC.md +++ b/benchmarks/experiments/PAGE-SPEC.md @@ -10,7 +10,7 @@ The DBBench page at `humem.ai/research/dbbench` is the PRIMARY artifact: its lan 1. **Serial, full cpuset.** Every published latency, throughput, percentile, and memory cell runs one at a time on mini's `cpuset 0-11` (the 12 P-core THREADS on 6 physical P-cores; see 0a), `workers=1`, `tier=paper`. Parallel shards exist for sweeps and exploration only and may never reach the page. Enforced: `runner.py` refuses `workers != 1` at paper tier; `load_canonical` drops partial cpusets. 2. **N=5, median [min-max].** Five repetitions per cell, reached in whole passes: n=3 for every engine first (the four ArcadeDB arms included), then n=4, then n=5 (PROTOCOL.md section 1). Any table with a cell below n=5 states its n in a condition, and no page sentence may assert an n that a cell it covers does not meet. Per-table operation and repetition counts are generated conditions (`export_web._counts_note`), never typed. -3. **One engine commit per table.** Every ArcadeDB row in one table comes from the same upstream commit, stamped as `engine_commit`. Comparators are digest-pinned images. See section 1. Enforced at production time: `runner.py` refuses a paper-tier ArcadeDB run when `ARCADEDB_ENGINE_COMMIT` is unset, before any cell starts, and `bench_common.run_conditions()` reads it so bespoke probes carry it too. A rule the page depends on may not be a printed reminder to a human. +3. **One engine commit per table.** Every ArcadeDB row in one table comes from the same upstream commit, stamped as `engine_commit`; the native and JVM builds of one release are two arms built from that one commit (section 1). Comparators are digest-pinned images. See section 1. Enforced at production time: `runner.py` refuses a paper-tier ArcadeDB run when `ARCADEDB_ENGINE_COMMIT` is unset, before any cell starts, and `bench_common.run_conditions()` reads it so bespoke probes carry it too. A rule the page depends on may not be a printed reminder to a human. 4. **One corpus per table.** Every row in a table describes the same dataset at the same size. A table whose rows disagree on `n_docs` (or the lane's size field) is a defect, not a comparison. The fingerprint must come from the corpus the lane loaded, not from a module constant (`l2_graph.py` and `l3_sparse.py` point it at the loaded data; BUGS.md F6 is the sparse lane running the synthetic corpus under a paper label). 5. **A knob the campaign sets must reach the cell, be honoured by the lane, and resolve inside the container.** `runner.py`'s env passthrough is a CLOSED tuple: a knob absent from it silently runs the in-script default. Three checks, each of which has failed once: DELIVERED (diff every lane's `environ.get("UPPER_CASE")` against what the runner passes), HONOURED (an unknown mode is a hard error, not a silent fallback to a literal default), and RESOLVED where the process runs (the mount is printed in the preamble and a missing corpus is refused on the host). Run the delivery diff whenever a lane gains a knob. 6. **The config PROFILE is recorded on every row.** ArcadeDB ships profiles that rewrite defaults wholesale: one sets `VECTOR_INDEX_GRAPH_BUILD_CACHE_SIZE=-1` with both cache heap percentages at 50, another pins both caches to a flat 10,000, against a default of `graphBuildCacheSize=0` (auto) and `graphBuildCacheMaxHeapPercent=25`. The profile can move a build several-fold on its own, so two rows under different profiles are not comparable. The percentage applies only on the AUTO path; quoting it for a run that pinned an absolute cache size is wrong. @@ -57,6 +57,9 @@ ArcadeDB is pinned by **upstream commit SHA**, not by a release number. The buil - Every row carries `engine_commit` (short SHA of the upstream commit the wheel and server image were both built from). - The page prints it as the engine identity, linked to the commit on GitHub. Every ArcadeDB row is named from its own engine string and commit, never from an image tag; served rows carry the row's `server_image_ref` (BUGS.md F16, F32). The page header lists the identities actually present. - Embedded and server arms in one table are built from the SAME commit, paired, and run the same JVM (DECISIONS #54). +- **Two builds, two arms (26.11.1).** ArcadeDB has two builds of one release: `arcadedb` (the native image, the default install) and `arcadedb-jvm` (JPype and a bundled JDK), both built from one engine commit. A page whose frozen rows hold the native build shows every in-process ArcadeDB arm twice, side by side, with the build last in the label: `ArcadeDB (embedded, native build)` and `ArcadeDB (embedded, JVM build)`, `ArcadeDB (embedded, fp32, native build)` and so on, in every table that has the arm (the lifecycle table puts the build after the situation: `Graph (embedded, native build)`). The native build comes first. The build is `engine_backend` on the row; a row without it, or with it empty, is the JVM build (`arm_identity.build_of`). The served ArcadeDB arm stands twice as well: `ArcadeDB (server, native build)` and `ArcadeDB (server, JVM build)`, so four ArcadeDB arms in all, from one engine commit. The JVM server is the official Docker image built at the pair's commit (`ARCADEDB_SERVER_IMAGE`); the native server is upstream's official native-image server, `arcadedata/arcadedb:-native` (`ARCADEDB_NATIVE_SERVER_IMAGE`; built by upstream's `native-image.yml` from `native/pom.xml`), started and addressed exactly like the JVM one: same client, cpuset, memory cap and split. Both get the same explicit settings (server mode, bind host and port, `txWalFlush` per durability class, the HTTP body limit and query cap where the arm sets them), the native one as `-D` arguments derived from the JVM arm's own `JAVA_OPTS`; the runner reads each value back from the running engine (`server_settings`, with `server_engine_commit` from `SELECT version()`) and refuses a cell whose engine answers otherwise. The page discloses the one difference under the server tables: the CPU baseline each artifact was compiled for (`cpu_baseline` on the row: the native server image `-march=x86-64-v3` from `native.march`, the native wheel `-march=compatibility`, the JVM compiling for the host CPU); neither baseline is changed. A server row carries `engine_backend` (a row without it is the JVM server, the only one before 26.11.1); the JVM image-defaults sensitivity arm has no native counterpart and stays one arm. A freeze of rows from before 26.11.1 has no native row, so it prints no build and renders as it always did. `BENCH_ENGINE_BACKEND=native|jvm` still freezes one build on purpose; the labels then still name it. +- **The 26.11.1 native server image is one commit after the pin, and that is accepted by an explicit rule.** Upstream's `native-image.yml` builds `arcadedata/arcadedb:26.11.1-native` at the release tag (`0f3bb2238c`, "Set release version to 26.11.1"), one commit after the commit of the JVM image, the jars and both wheels (`a3a5d12457`, the pin). `git diff` between the two is 32 files, 34 insertions and 34 deletions, every changed line a version string (poms, studio package files, the docker identifiers in `server.json`), no source. `engine_commit_equiv.json` holds that pair (generated from the git diff by `make_engine_commit_equiv.py`, which refuses any other diff); `NATIVE_SERVER_EQUIV_SHAS` in the run's pins opts in. It applies to the native SERVER image of a release pair only: `verify_native_server_image.sh` prints `NATIVE_SERVER_EQUIV= accepted for pin `, the runner stamps `server_engine_commit_equiv` on the row, rows keep `engine_commit` = the pin, and `arm_identity` and `provenance_check` read the same registry. The page says that the native server image reports `0f3bb2238c` (a version-strings-only successor of the pin). That sentence is generated (`export_web._native_server_equiv_note`, one global condition): it appears only when a printed row carries `server_engine_commit_equiv`, takes the release, both commits, the file count and the changed-line count from `engine_commit_equiv.json`, and a row that claims the equivalence with no registry entry behind it refuses the export. The client sentence under a table with served ArcadeDB rows (`_arcadedb_client_note`) names the driver string the headline rows carry, says that bulk loads, the server probe, transactions and streamed results stay on the lean HTTP client in the headline rows too, and names the lean twin and the PostgreSQL wire twin only when their rows are present (labelled sensitivities, never in a headline median). +- **Both builds, one engine commit, or no page.** `export_web` refuses a freeze whose native and JVM rows name different `engine_version` or `engine_commit` values, and `page_check` fails the same page again (`_check_build_pair`). The gate also fails an in-process ArcadeDB arm that names no build on a page that names builds, two builds of one arm and size whose `version_name` differ, and a build that has a row at a size where the other has none. That last case is not left out: the exporter prints the missing build as a declared absence of kind `not measured` (a whole-row declaration with one generated sentence per table and build, naming the arms and sizes), which the roster and the pair gate read as accounted for. A table never takes a median over both builds: the exporter groups on the build and `_one_version` refuses an entry that spans two. Where the lifecycle table holds both builds its start column is `engine start ms`, the `start_jvm()` call on the JVM build and the library load with the isolate creation on the native build, and a sentence under the table says so. **Overlays belong to the build that produced them.** The dense and sparse multipass drivers stamp `engine_backend` on every pass they write (`run_conditions`), the native chain names its files `mp__native_b.json` and `sp__native_.json` beside the JVM build's, and the exporter reads the build from the file's own content, so the in-process dense and sparse arms stand once per build like any other arm. An overlay file without the field is the JVM build, as an old row is; a page with native rows and no native overlay therefore prints the JVM arm and declares the native one `not measured`. A partial native overlay (not every build of every in-process arm) refuses the freeze (`make_paper_tables._refuse_partial_native`). The paper's pinned-sparse path validates each build on its own and, with no `BENCH_ENGINE_BACKEND`, takes the JVM build. **Censored cells and the notes read per build.** A cap, timeout or out-of-memory outcome is keyed arm, build and size, so a native outcome never removes the JVM entry; the runner stamps `engine_backend` on the row of a cell it killed, because that row comes from the runner and not from the engine process, and an old outcome without the field is the JVM build. The heap and memory notes read the frozen rows behind each build's entry. **Each chain runs the server of its own build.** The native chain (native wheel) runs every served ArcadeDB arm on the native server image and the JVM chain on the Docker JVM image, each arm once; the JVM-only arms (the image-defaults sensitivity arm, e4) stay in the JVM chain (`make_2611_stages.build_less_served_arms`). The stage exports `BENCH_SERVER_BUILD` for its build and the runner refuses one that is not its wheel's; `--check --drafts DIR --also-drafts DIR2` fails when the two chains do not run the same served arms once each, and the lint fails a script that names the other build's server. The pre-flight checks the native server's engine commit (`verify_native_server_image.sh`, `SELECT version()`) against the pin, like the JVM pair. **The pair gate covers the servers.** The same gates (one commit, a row per build or a declared absence, equal server settings between the builds of one cell: `arm_identity.server_settings_problems`) apply to the server arms. - A campaign FREEZES one commit and holds it start to finish. Upstream landing a fix mid-run does not restart the campaign; it becomes a dated changelog entry and a candidate for the next re-pin. - Re-pinning is a deliberate, dated act recorded in the changelog with what moved. No table falls back to an older pin, anywhere (DECISIONS #62). - **One ArcadeDB version until the final DBBench release, then history (DECISIONS #189).** Until the final DBBench page and repo are released (after the late-November 2026 paper), a new ArcadeDB release's pass REPLACES the old one: after one scripted old-against-new comparison (a regression is filed before the next cut), the old version's ArcadeDB rows, page data, pin-specific code, pinned test fixtures and wording are removed. Comparator rows are kept across it and carry their own engine's version, never ArcadeDB's. From the final release on, a new engine version ADDS rows and the page keeps the history: @@ -154,7 +157,7 @@ Sizes: `e2` and `e2atom` two, 50k and 500k products (the larger so retrieval rec **Time the actions, not just the ends.** Every scenario reports `open + action + close = session`, and the page quotes the session: timing `open` and `close` alone misses where a query-triggered rebuild lands. The lane's page columns are the cold-start decomposition (4 columns), then per scenario: open ms, action ms, close ms, session ms. -**Cold start is four numbers, not one.** Measured in a fresh subprocess per situation: `import_ms` (interpreter and module import), `jvm_start_ms` (`start_jvm()` alone), `first_open_ms` (first `open_database()`, JVM already up) and `cold_process_ms` (what a CLI actually waits for). A page that quotes a warm open without the cold process beside it is off by orders of magnitude for anything CLI-shaped, and embedded use is mostly CLI-shaped. +**Cold start is four numbers, not one.** Measured in a fresh subprocess per situation: `import_ms` (interpreter and module import), `jvm_start_ms` (`start_jvm()` alone), `first_open_ms` (first `open_database()`, JVM already up) and `cold_process_ms` (what a CLI actually waits for). A page that quotes a warm open without the cold process beside it is off by orders of magnitude for anything CLI-shaped, and embedded use is mostly CLI-shaped. On the native `arcadedb` build there is no JVM: `jvm_start_ms` is null and `engine_load_ms` (the engine library load and isolate creation; `start_jvm()` on the JVM build) is the same step, so compare `import_ms + engine_load_ms` across the two builds. Two columns come from a second fresh process: `cold_first_query_ms` (the situation's first read after the first open) and `cold_query_process_ms` (process start through the end of that read). The definitions are in the docstring of `l5_lifecycle.py`. ### Rendering rules, every table @@ -296,7 +299,7 @@ And a closing list of what the page does NOT measure: concurrency and load (it n ## Published tables (generated by refresh_web_page.py from results/web_benchmarks.json on every publish; do not edit by hand) -Every table carries a Size column and direction arrows; the best value per column within a size is bold on the page; ingest is a pair of columns where the lane records it. Rows: ArcadeDB first, comparators alphabetical, embedded before server, int8 before fp32. +Every table carries a Size column and direction arrows; the best value per column within a size is bold on the page; ingest is a pair of columns where the lane records it. Rows: ArcadeDB first (the native build, then its JVM build), comparators alphabetical, embedded before server, int8 before fp32. | id | title | rows | sizes | columns | |---|---|---|---|---| diff --git a/benchmarks/experiments/PROTOCOL.md b/benchmarks/experiments/PROTOCOL.md index 72f241fe078..d22ff9d6948 100644 --- a/benchmarks/experiments/PROTOCOL.md +++ b/benchmarks/experiments/PROTOCOL.md @@ -264,7 +264,7 @@ An override nobody can find is a defect, whichever way it moves the number. Ever | all dense arms | `EF_CONSTRUCTION=100`, `EF_SEARCH=100` | operating-point | Shared beam. At the 26.10.1 tag an explicit `efSearch` is a fixed beam of `max(k, efSearch)` while the adaptive default is `max(2k, 20)` on graphs of 10,000 or more vectors, so it is NOT a no-op for ArcadeDB; some comparators move down from their own defaults, unaudited. | lane docstring only | | ArcadeDB server (all arms) | `ARCADEDB_OPTS_GC=` is cleared explicitly, and `JAVA_OPTS` carries `-XX:+UseCompactObjectHeaders` | operating-point | Makes E4 a transport comparison. `ARCADEDB_OPTS_GC` survives a `JAVA_OPTS` override, so clearing it puts both arms on one collector. | DECISIONS #54, enforced by `verify_pair_c25.sh` | | ArcadeDB server vs embedded | both arms run ONE set of upstream jars on ONE JVM (Corretto 25) | fairness | The axis isolates transport, not JVM major, libc, collector, or object-header layout; `-XX:+UseCompactObjectHeaders` is rejected by JDK 21. | DECISIONS #54, enforced by `verify_pair_c25.sh` (jars AND JVM major on both sides) | -| ArcadeDB SERVED arms | the headline HTTP client is the benchmark's own minimal HTTP client (`lean_http.py`: one persistent `http.client` connection per host); the page and the paper name it | the other engines are measured through their vendors' own Python clients, so client weight differs between the served engines | A served latency claim names the client. Bulk loads, the server probe, and transactions use the same client. The freeze keeps the rows measured through this client only (DECISIONS #285), so an arm with no job on this client has no published cell. | row fields `arcadedb_http_client` and `arcadedb_http_client_role` (stamped as the job ran; the rows of this client carry `sensitivity`) | +| ArcadeDB SERVED arms | the headline HTTP client is the benchmark's own minimal HTTP client (`lean_http.py`: one persistent `http.client` connection per host); the page and the paper name it | the other engines are measured through their vendors' own Python clients, so client weight differs between the served engines | A served latency claim names the client. Bulk loads, the server probe, and transactions use the same client. The freeze keeps the rows measured through this client only (DECISIONS #285), so an arm with no job on this client has no published cell. The harness also runs the vendor's driver and a PostgreSQL-wire twin of the served SQL cells (read statements only, psycopg, `_clientpgwire`, answers gated identical by `pgwire_gate.py`); both are measured and not published, and the freeze sets their rows aside. | row fields `arcadedb_http_client` and `arcadedb_http_client_role` (stamped as the job ran; the rows of this client carry `sensitivity`) | | l4 `q_high` and `q_groupby` | ArcadeDB embedded `to_arrow()`, DuckDB `to_arrow_table()`, SQLite `fetchall()` (no columnar path); `to_list()` / `fetchall()` timed beside it as `_plain_ms` | each engine on its documented best path for a result of thousands of rows | The bindings' docs recommend a bulk read for large results; small results (1 to 100 rows) are unchanged. | row fields `read_path`, `read_path_queries`, `read_path_plain`, `read_path_sensitivity`, `_plain_ms`, `_plain_digest`, `_plain_digest_match` (the cell ends if the typed and plain digests differ; DuckDB's Arrow rows are read by position, like its `fetchall()`) | | JVM initial heap (`-Xms`) per engine | **ArcadeDB, embedded and served: unset** (only `-Xmx` from the cell's memory budget; the served arms through `ARCADEDB_OPTS_MEMORY=-Xmx{heap}`, DECISIONS #200). Elasticsearch: `ES_JAVA_OPTS=-Xms{heap} -Xmx{heap}`, unchanged. Neo4j: `server.memory.heap.initial_size` = `max_size`, unchanged. QuestDB: the harness sets no heap (image defaults), unchanged | ArcadeDB as its maintainers advise for a container with a fixed memory budget (no `-Xms`; ArcadeData/arcadedb#9167) and as the bindings document, so both deployments run one JVM configuration; each other server engine as its own documentation recommends | ArcadeDB's server script (`package/src/main/scripts/server.sh`) leaves the heap to the JVM unless `ARCADEDB_OPTS_MEMORY` is set and shows `-Xms2G -Xmx2G` as its example; Elasticsearch: set `Xms` and `Xmx` to the same value (https://www.elastic.co/guide/en/elasticsearch/reference/current/advanced-configuration.html#set-jvm-heap-size); Neo4j: set initial and maximum heap to the same value (https://neo4j.com/docs/operations-manual/current/performance/memory-configuration/); QuestDB: no setting by the harness (https://questdb.com/docs/configuration/). | `server_heap` / the cell's heap fields, as before | | ArcadeDB server (all arms) | `ARCADEDB_OPTS_MEMORY=-Xmx{heap}` (image ships 2G) | resource fitting | Heap parity per tier. The 2 GB default cost the server 15% p50 and 35% build time at 10M. | T5 caption, FAIRNESS.md F3, `server_heap` read back | diff --git a/benchmarks/experiments/PUBLISHING.md b/benchmarks/experiments/PUBLISHING.md index b0f6f14b4d1..afd567fdb12 100644 --- a/benchmarks/experiments/PUBLISHING.md +++ b/benchmarks/experiments/PUBLISHING.md @@ -187,6 +187,8 @@ It pulls `runs_page__2611.jsonl` (and, with `--overlay`, an arm's dense mul **The pulled file is screened before it is merged, and a rehearsal can run on scratch copies.** `screen_pulled_rows` refuses the whole file, before `--exclude-*` and before the merge (so `--dry-run` shows the refusal too), when any row is not stamped `2026-11` (a retired 2026-10 file, a mixed file, an unstamped row) or names an engine commit other than `--pin` (prefix either way); nothing reaches `results/runs.jsonl`. The first landing creates the store. Failed cells (`rc != 0` or an error) are reported and merged as failures, and the freeze skips them; censored queries are reported and keep their flag. `BENCH_LAND_SITE` moves the humem.ai checkout (the landing passes it to `refresh_web_page.py --site`), `BENCH_LAND_PY` the interpreter every step runs under, `BENCH_LAND_HOST`/`BENCH_LAND_REMOTE` the bench host and `BENCH_LAND_SCRATCH` the pull directory, so a rehearsal never writes the real site checkout. A landing scoped away from e4 does not need `e4decomp_`. +**`BENCH_LAND_PULLED_FILE` lands rows that were pulled elsewhere.** The four-arm chains write their rows into several results directories on the bench host, so no single `scp` has them all. Set the variable to a merged rows file and step 1 copies it in place of the `scp`; the screen, the filters, the merge and the gates then run on it as on any pulled file. A path that is not a file ends the run, it never falls back to the host. Unset, the landing pulls from the host as before. + **`--pin` defaults to `BENCH_ENGINE_COMMIT`.** Step 1 pulls the stage's rows file by exact name, so a wrong pin fetches the wrong rows and fails much later, with the merge already done; pass `--pin ` or set `BENCH_ENGINE_COMMIT` on every landing. Never `git add` a raw directory under `results/` (see the first note of the results section above). diff --git a/benchmarks/experiments/READING-RESULTS.md b/benchmarks/experiments/READING-RESULTS.md index 4d82c240b85..ed80cdbc346 100644 --- a/benchmarks/experiments/READING-RESULTS.md +++ b/benchmarks/experiments/READING-RESULTS.md @@ -26,6 +26,8 @@ The single rule, from which most of the rest follows: **`engine_version` is deliberately null for non-ArcadeDB rows in L4.** Use `backend_version`. `run_conditions()` stamps `engine_version` from the installed wheel, which is right for the ArcadeDB row and wrong for DuckDB and QuestDB, so `l4_tsbs.py` asks each backend for its own version instead. A wrong version that passes a provenance audit is worse than none. +**`engine_backend` says which ArcadeDB build produced a row: `native` (the `arcadedb` wheel) or `jvm` (`arcadedb-jvm`, the older `arcadedb_embedded`).** `engine_version` is the same string for both wheels of one release, and `backend` is the arm name, so neither can tell them apart. It is read from `arcadedb.__backend__` by `bench_common.engine_backend()` and for a served ArcadeDB arm it is the build of the server container the cell talked to (the official JVM server image or upstream's official native-image server, named by `server_build`); it is null for every comparator and for the image-defaults sensitivity arm. A server row also carries the settings the engine reported at start (`server_settings`), its own commit (`server_engine_commit`) and the CPU baseline of the artifact (`cpu_baseline`). The page shows both builds, embedded and server, as four arms, `native build` and `JVM build`, one next to the other (PAGE-SPEC section 1); a row without the field is the JVM build. + **Arms are selected by flag, not by filename.** `claims_check.ts_arm()` takes `primitive=` and `numpy_cols=` because the published row is one specific arm. Assert the flags rather than globbing the directory. **A comparator row's version and digest are read from the row** (`server_image`, `engine_version`); `runner.BACKENDS` describes future rows only (BUGS.md F32). diff --git a/benchmarks/experiments/_arcadedb_import.py b/benchmarks/experiments/_arcadedb_import.py new file mode 100644 index 00000000000..4350f92db69 --- /dev/null +++ b/benchmarks/experiments/_arcadedb_import.py @@ -0,0 +1,191 @@ +"""The one place the harness decides which ArcadeDB Python package it imports. + +Since 26.11.1 the Python bindings ship as two distributions that both install the import name `arcadedb`: +`arcadedb` (the native build) and `arcadedb-jvm` (the Java engine through JPype). Before that the one +distribution was `arcadedb-embedded`, import name `arcadedb_embedded`. The lanes used to write +`import arcadedb_embedded as arcadedb` in every ArcadeDB arm; they now call `load()` here instead: + + import _arcadedb_import + arcadedb = _arcadedb_import.load() + +`load()` imports `arcadedb` first and falls back to `arcadedb_embedded`, so the same harness runs +against the new wheels and against the `arcadedb_embedded` wheels the images carry today. An +`arcadedb_embedded` install resolves to the old package unless `arcadedb` is installed too (the +26.11.1 `arcadedb-embedded` shim re-exports `arcadedb`, so there both names give the same module). + +Everything here is lazy: importing this file imports nothing from ArcadeDB, because a lane module is +also imported by cells that never touch ArcadeDB (DuckDB, SQLite, ...), on hosts without the wheel. + +What differs between the packages, and what this module hides: + + * module name: `arcadedb` vs `arcadedb_embedded` (module_name()). + * distribution name for the version stamp: `arcadedb` / `arcadedb-jvm` / `arcadedb-embedded` + (dist_version(); `importlib.metadata.version("arcadedb-embedded")` is a PackageNotFoundError on a new install). + * which engine runs: `arcadedb.__backend__` is "native" or "jvm"; the old package has no such + attribute and is always "jvm" (backend()). + * `.jvm`: importable in both new wheels, but the native wheel ships no JARs, so + `start_jvm()` raises ArcadeDBError ("No JAR files found") there (jvm_module()). The cold-start + subprocess in l5_lifecycle.py therefore starts the native library instead (see its docstring). + * JVM-only APIs (`get_java_database`, `jpype`, Java arrays): they exist only on the JVM build and + the native build has no JPype. No lane uses them any more (l3_sparse's embedded arm went through + insert_many, plain Python lists and SQL COMPACT INDEX); a new arm that needs one must check + backend() first, and this module does not paper over it. + * loaded_backend(): backend() once load() has run, else None; it is what the `engine_backend` row field reads. +""" +import importlib +import importlib.metadata +import os + +# Import names, in order. The first one that is the real package wins. +MODULES = ("arcadedb", "arcadedb_embedded") +# Distribution names for the version stamp, same order idea: new first. +DISTS = ("arcadedb", "arcadedb-jvm", "arcadedb-embedded") + +_loaded = None + +# THE NATIVE BUILD HAS NO JVM, SO NO `jvm_kwargs`: the native package ignores `heap_size` and `jvm_args` (with a warning) and reads its isolate +# options from this environment variable when the isolate is created (arcadedb._native.lib). The lanes still pass the JVM spelling, because that +# is the one code path for both packages, so on native `load()` wraps create_database/open_database and turns the same kwargs into isolate +# arguments before the first call: heap_size -> -Xmx, and the -D and -Xms/-Xmx entries of jvm_args as they are +# (the engine reads its settings, such as arcadedb.txWalFlush, from system properties; checked on the 26.10.2 native wheel: -Darcadedb.txWalFlush=1 +# in the isolate args shows value 1 in `SELECT settings FROM schema:database`). Other JVM flags (-XX:...) have no native meaning and are left out; +# they are recorded in `native_dropped`. An argument already present in the variable wins, as it does inside the package. +NATIVE_ENV = "ARCADEDB_NATIVE_ISOLATE_ARGS" +_PASS_PREFIXES = ("-D", "-Xms", "-Xmx") +native_dropped = [] +_isolate_created = False + + +def _is_engine(mod): + # `arcadedb` could also resolve as an empty namespace directory on sys.path; require the API. + return hasattr(mod, "create_database") and hasattr(mod, "DatabaseFactory") + + +def load(): + """Return the ArcadeDB Python module: `arcadedb`, else `arcadedb_embedded`. + + Raises ImportError (naming both) when neither is installed. + """ + global _loaded + if _loaded is not None: + return _loaded + errors = [] + for name in MODULES: + try: + mod = importlib.import_module(name) + except ImportError as e: + errors.append(f"{name}: {e}") + continue + if _is_engine(mod): + _loaded = mod + _wrap_native(mod) + return mod + errors.append(f"{name}: imported but is not the ArcadeDB package ({getattr(mod, '__file__', None)})") + raise ImportError("no ArcadeDB Python package: install arcadedb or arcadedb-jvm (or the older arcadedb-embedded); " + + "; ".join(errors)) + + +def module_name(): + """The import name `load()` resolved to ("arcadedb" or "arcadedb_embedded"). For code strings run in a subprocess.""" + return load().__name__ + + +def backend(): + """"native" or "jvm": the engine this process runs. The old `arcadedb_embedded` package is always "jvm".""" + mod = load() + return getattr(mod, "__backend__", None) or getattr(mod, "_DISTRIBUTION_BACKEND", None) or "jvm" + + +def loaded_backend(): + """`backend()` when this process has loaded the package, else None (a comparator arm, a served arm's driver). + + Never imports the engine. This is what the row stamp `engine_backend` reads, so a row of a process that never touched ArcadeDB + carries an explicit None, not a guess. + """ + return backend() if _loaded is not None else None + + +def jvm_module(): + """`.jvm` (start_jvm, shutdown_jvm, jar_fingerprint ...). Importing the submodule does not start a JVM.""" + return importlib.import_module(module_name() + ".jvm") + + +def dist_version(): + """The installed distribution's version, read from package metadata without importing the engine. + + Same contract as the old `importlib.metadata.version("arcadedb-embedded")`: returns a version string, + or raises PackageNotFoundError when none of the distributions is installed (callers format that into + "unknown (PackageNotFoundError)" exactly as before). + """ + last = None + for dist in DISTS: + try: + return importlib.metadata.version(dist) + except importlib.metadata.PackageNotFoundError as e: + last = e + raise last + + +def _arg_key(tok): + """The part of an isolate argument that names the setting: `-Darcadedb.txWalFlush=1` -> `-Darcadedb.txWalFlush`, `-Xmx4g` -> `-Xmx`.""" + if tok.startswith("-D"): + return tok.split("=", 1)[0] + return tok[:4] if tok[:4] in ("-Xms", "-Xmx") else tok + + +def native_isolate_args(heap_size=None, jvm_args=None): + """The isolate arguments that mean what a JVM `heap_size` / `jvm_args` pair means; ([kept], [dropped]). Pure, no environment.""" + toks = [f"-Xmx{heap_size}"] if heap_size else [] + ja = jvm_args.split() if isinstance(jvm_args, str) else list(jvm_args or []) + dropped = [t for t in ja if not t.startswith(_PASS_PREFIXES)] + toks += [t for t in ja if t.startswith(_PASS_PREFIXES)] + return toks, dropped + + +def apply_native(jvm_kwargs=None, environ=None): + """Put the isolate arguments for `jvm_kwargs` into ARCADEDB_NATIVE_ISOLATE_ARGS and return the effective string. + + Merge rule: a setting already in the variable wins (the caller or the package's own default is not overridden), a new one is appended, + two entries for one setting in `jvm_kwargs` keep the first. Meaningful only before the isolate exists; see `_wrap_native`. + """ + env = os.environ if environ is None else environ + kw = jvm_kwargs or {} + new, dropped = native_isolate_args(kw.get("heap_size"), kw.get("jvm_args")) + have = env.get(NATIVE_ENV, "").split() + keys = {_arg_key(t) for t in have} + for t in new: + if _arg_key(t) not in keys: + have.append(t) + keys.add(_arg_key(t)) + for t in dropped: + if t not in native_dropped: + native_dropped.append(t) + env[NATIVE_ENV] = " ".join(have) + return env[NATIVE_ENV] + + +def effective_native_args(): + """The isolate arguments this process runs with ("" when none), or None when it is not a native process. What a row records.""" + return os.environ.get(NATIVE_ENV, "") if _loaded is not None and backend() == "native" else None + + +def _wrap_native(mod): + """On the native build, make `mod.create_database` / `mod.open_database` apply their `jvm_kwargs` as isolate arguments (see NATIVE_ENV).""" + if (getattr(mod, "__backend__", None) or getattr(mod, "_DISTRIBUTION_BACKEND", None) or "jvm") != "native": + return + for name in ("create_database", "open_database"): + fn = getattr(mod, name, None) + if fn is None or getattr(fn, "_bench_native_wrapped", False): + continue + + def wrapped(*a, __fn=fn, **kw): + global _isolate_created + if not _isolate_created: # the isolate takes its arguments once; a later call cannot change them (the JVM cannot restart either) + apply_native(kw.get("jvm_kwargs")) + _isolate_created = True + return __fn(*a, **kw) + + wrapped._bench_native_wrapped = True + wrapped.__name__ = getattr(fn, "__name__", name) + wrapped.__doc__ = getattr(fn, "__doc__", None) + setattr(mod, name, wrapped) diff --git a/benchmarks/experiments/arm_identity.py b/benchmarks/experiments/arm_identity.py new file mode 100644 index 00000000000..bc753a942e2 --- /dev/null +++ b/benchmarks/experiments/arm_identity.py @@ -0,0 +1,263 @@ +"""What an ArcadeDB arm is, for every script that reads rows. + +Since 26.11.1 one release is measured on two builds of the same Python package API: `arcadedb` (native) and `arcadedb-jvm` (Java engine +through JPype). Both stamp the same `engine_version` and the same `backend` arm name (`arcadedb_sparse_embedded`, ...); only the row +field `engine_backend` ("native" or "jvm") tells them apart. A consumer that groups or dedupes on `engine_version` alone would pool or +overwrite the two builds, so the identity of an in-process ArcadeDB arm is (engine_version, build), and this module is the one +definition of `build`: + + * `engine_backend` is "native" or "jvm": that value. + * a row of an ArcadeDB in-process arm WITHOUT the field, or with it empty (every row before 26.11.1, and those rows read back from the + frozen CSV): "jvm", because that is what it ran. + * a served ArcadeDB arm (`arcadedb_server`, `arcadedb_graph_server`, ...) carries the build of the SERVER it talked to since + #222: "jvm" for the official Docker server image, "native" for the native wheel's in-process server (native_server_launcher.py). + A served row without the field (every row before it) is "jvm", the only server there was. The image-defaults sensitivity arm + (`arcadedb_imgdefaults_server`), the HTTP client container and every comparator carry None: they have no native counterpart. + +`select_build(rows, want)` is the one place a freeze picks a build: with BENCH_ENGINE_BACKEND=native|jvm the other build's in-process +ArcadeDB rows leave the freeze (they stay in runs.jsonl); without it BOTH builds stay, and the page shows them side by side as two arms +(`splits_page`, `build_label`: "ArcadeDB (embedded, native build)" next to "ArcadeDB (embedded, JVM build)"). A table entry still never +pools the two (`refuse_pooled_builds`). `pair_problems(rows)` is the gate over a two-build freeze: one engine commit, one engine_version. +No import of ArcadeDB, no I/O beyond reading engine_commit_equiv.json (through engine_pin): safe in every consumer, including the ones that read old rows. +""" +import os + +import engine_pin + +BUILDS = ("native", "jvm") +ENV = "BENCH_ENGINE_BACKEND" + +# Backend names that are ArcadeDB but not an in-process arm: the served engine and its client container. +_NOT_IN_PROCESS = ("server", "http_client", "imgdefaults") + + +def is_arcadedb(r): + return str(r.get("backend") or "").startswith("arcadedb") + + +def in_process(r): + b = str(r.get("backend") or "") + return b.startswith("arcadedb") and not any(t in b for t in _NOT_IN_PROCESS) + + +def served(r): + """True for a served ArcadeDB arm that stands once per build: the HTTP server arms, not the image-defaults sensitivity arm.""" + b = str(r.get("backend") or "") + return b.startswith("arcadedb") and "server" in b and not any(t in b for t in ("http_client", "imgdefaults")) + + +def built(r): + """True for an ArcadeDB arm that stands once per build: an in-process arm or a served one (#222).""" + return in_process(r) or served(r) + + +def build_of(r): + """"native", "jvm" or None for the row (see the module docstring).""" + eb = r.get("engine_backend") + if eb in BUILDS: + return eb + # No usable field: an old row, or a row read back from the frozen CSV, where a row from before 26.11.1 has the column empty. + if built(r) and not eb: + return "jvm" + return None + + +def arm_identity(r): + """(engine_version, build): what two rows must share to be one measurement of one engine.""" + return (str(r.get("engine_version") or ""), build_of(r)) + + +def identities(rows): + """The set of (engine_version, build) over the in-process ArcadeDB rows.""" + return {arm_identity(r) for r in rows if built(r)} + + +def wanted_build(env=None): + """The build a freeze was asked for ("native" or "jvm"), or None. A value that is neither is an error, not None.""" + v = (os.environ.get(ENV, "") if env is None else env).strip().lower() + if not v: + return None + if v not in BUILDS: + raise SystemExit(f"{ENV}={v!r}: expected one of {BUILDS}") + return v + + +def select_build(rows, want=None): + """The rows a freeze may publish. + + `want` None reads BENCH_ENGINE_BACKEND. With a build, rows of the other build are dropped (served arms and comparators carry no + build and stay): a deliberate one-build freeze. Without one every row stays, and the page shows the two builds as two arms + (`splits_page`); the exporter groups on the build, so no median is taken over native and JVM rows together. + """ + want = wanted_build() if want is None else want + if want: + return [r for r in rows if not (built(r) and build_of(r) not in (None, want))] + return rows + + +def stamp_build(row, wheel_build): + """Give an ArcadeDB row (in-process or served) its build when the row names none: the build of the wheel its runner was pinned to, or, + for a served row, the server's own (`server_build`, written by the runner at launch). + + A cell killed by its cap or by the kernel writes a row from the runner, not from the engine process, so the row has no + engine_backend of its own; without this a native timeout would read as the JVM build (the default for an unstamped row). + Returns the row. + """ + if built(row) and not row.get("engine_backend"): + if served(row) and row.get("server_build") in BUILDS: + row["engine_backend"] = row["server_build"] + elif in_process(row) and wheel_build in BUILDS: + row["engine_backend"] = wheel_build + if built(row) and row.get("engine_backend") in BUILDS and not row.get("cpu_baseline"): + row["cpu_baseline"] = cpu_baseline(row["engine_backend"], served(row)) + return row + + +# THE CPU BASELINE OF EACH ARTIFACT, disclosed on the page and left as built. The native wheel is compiled with `-march=compatibility` +# (runs on any x86-64 CPU); upstream's official native server image is compiled for `native.march` (native/pom.xml: x86-64-v3 on amd64, i.e. +# AVX2/BMI2/FMA, set by ArcadeData/arcadedb#10581; override with ARCADEDB_NATIVE_SERVER_MARCH if a different image is pinned); the JVM +# server and the JVM wheel compile at run time for the CPU they run on. +NATIVE_WHEEL_MARCH = "compatibility" +NATIVE_SERVER_MARCH_DEFAULT = "x86-64-v3" + + +def native_server_march(env=None): + return ((os.environ if env is None else env).get("ARCADEDB_NATIVE_SERVER_MARCH", "").strip() or NATIVE_SERVER_MARCH_DEFAULT) + + +def cpu_baseline(build, is_server, env=None): + """The CPU baseline sentence stamped on a row of `build` ("native" / "jvm").""" + if build == "native": + return (f"native server image: -march={native_server_march(env)}" if is_server + else f"native wheel: -march={NATIVE_WHEEL_MARCH}") + return "JVM: compiled at run time for the host CPU" + + +def builds_in(rows): + """The builds the in-process ArcadeDB rows of `rows` ran on, as a set of "native" / "jvm".""" + return {build_of(r) for r in rows if built(r)} - {None} + + +def splits_page(rows): + """True when the page must name the build on every in-process ArcadeDB arm: some row ran on the native build. + + Rows from before the field are all "jvm", so a freeze of old rows is never split and renders exactly as it did. A freeze that holds + only native rows is split too: an unlabelled arm would read as the JVM build. + """ + return "native" in builds_in(rows) + + +BUILD_NAMES = {"native": "native build", "jvm": "JVM build"} + + +def build_label(label, build): + """"ArcadeDB (embedded)" -> "ArcadeDB (embedded, native build)"; a label with no parenthesis gets one.""" + name = BUILD_NAMES[build] + return f"{label[:-1]}, {name})" if label.endswith(")") else f"{label} ({name})" + + +def build_in_label(label): + """The build a label names ("native" / "jvm"), or None.""" + for b, name in BUILD_NAMES.items(): + if str(label).endswith(f", {name})") or str(label).endswith(f"({name})"): + return b + return None + + +def strip_build(label): + """The label without its build: "ArcadeDB (embedded, native build)" -> "ArcadeDB (embedded)"; a label with no build is returned as is.""" + b = build_in_label(label) + if not b: + return label + name = BUILD_NAMES[b] + label = str(label) + return label[:-len(f", {name})")] + ")" if label.endswith(f", {name})") else label[:-len(f" ({name})")] + + +def pair_problems(rows): + """What stops a two-build freeze from being published, as a list of sentences (empty: fine). + + The two builds of one release are built from ONE engine commit. A freeze whose native and JVM rows name different engine_versions + or different engine commits compares two engines, not two builds of one, so the page must not print them side by side. + """ + out = [] + by = {} + for r in rows: + if built(r) and build_of(r): + by.setdefault(build_of(r), []).append(r) + if len(by) < 2: + return out + for field in ("engine_version", "engine_commit"): + # engine_version of a served row is the server's own build string ("server: (build ...)"), not the wheel's: compare the wheels' versions + # on the in-process rows; the commit is one value for every ArcadeDB row, served ones included. + seen = {b: sorted({str(r.get(field) or "") for r in rs if field != "engine_version" or in_process(r)} - {""}) for b, rs in by.items()} + flat = {v for vs in seen.values() for v in vs} + if len(flat) > 1: + out.append(f"the two ArcadeDB builds name different {field} values {seen}; one release is built from one engine commit") + out += server_settings_problems(rows) + return out + + +# THE SETTINGS A SERVER ARM IS HELD TO (#222 (3)). Both servers (the Docker JVM image, the native wheel's in-process server) are launched with +# the same explicit settings and the runner reads each back from the engine at start (`server_settings`, key -> value in force). The keys +# below must be present on every served ArcadeDB row and equal between the two builds of one cell (same arm, scale, durability class); +# other keys the runner requested (the HTTP body limit, the query cap) are compared when both builds list them. +SERVER_SETTING_KEYS = ("arcadedb.server.mode", "arcadedb.server.httpIncomingHost", "arcadedb.server.httpIncomingPort", "arcadedb.txWalFlush", + "arcadedb.maxPageRAM") # the page cache stays at the engine default on both servers; a different effective value refuses the page +# What each server's own start line reports (ArcadeDBServer.java: mode, CPUs, MAXRAM); a native row must carry all three, and the builds must agree. +SERVER_LOG_FIELDS = ("server_mode", "server_cpus", "server_max_heap_bytes") +MAX_HEAP_TOLERANCE = 0.05 + + +def server_settings_problems(rows): + """Sentences for a served-arm settings mismatch between the builds (empty: fine). A native server row with no read-back is a problem + too: a setting written is not a setting in force.""" + out, cells = [], {} + for r in rows: + if not served(r) or build_of(r) not in BUILDS: + continue + got = r.get("server_settings") + if isinstance(got, str): + try: + import json + got = json.loads(got) + except ValueError: + got = None + if build_of(r) == "native" and not isinstance(got, dict): + out.append(f"{r.get('backend')}: a native server row has no server_settings read-back") + continue + sc, pin = str(r.get("server_engine_commit") or ""), str(r.get("engine_commit") or "") + if sc and pin and not (sc.startswith(pin[:7]) or pin.startswith(sc[:7])) and not ( + build_of(r) == "native" and engine_pin.native_server_equivalent(sc, pin)): # ENGINE-COMMIT-EQUIV: release pair, native server only + out.append(f"{r.get('backend')}: the {build_of(r)} server reports engine commit {sc[:9]} and the row names {pin[:9]}") + if build_of(r) == "native" and any(r.get(f) in (None, "") for f in SERVER_LOG_FIELDS): + out.append(f"{r.get('backend')}: a native server row has no start-line read-back ({', '.join(SERVER_LOG_FIELDS)})") + if isinstance(got, dict): + key = (r.get("backend"), str(r.get("scale")), str(r.get("durability_class") or "")) + cells.setdefault(key, {}).setdefault(build_of(r), []).append(dict(got, **{f: r.get(f) for f in SERVER_LOG_FIELDS if r.get(f) not in (None, "")})) + for (arm, scale, cls), by in sorted(cells.items(), key=str): + if len(by) < 2: + continue + a, b = (by[x][0] for x in BUILDS) + keys = set(SERVER_SETTING_KEYS) | (set(a) & set(b)) + for f in SERVER_LOG_FIELDS: # only when both builds report it (an old JVM row has none) + if a.get(f) is not None and b.get(f) is not None: + keys.add(f) + for k in sorted(keys): + if k == "server_max_heap_bytes" and a.get(k) is not None and b.get(k) is not None: + if abs(float(a[k]) - float(b[k])) > MAX_HEAP_TOLERANCE * max(float(a[k]), float(b[k])): + out.append(f"{arm} at {scale} ({cls or 'any class'}): the native server reports MAXRAM {a[k]} bytes and the JVM server {b[k]}") + continue + if str(a.get(k)) != str(b.get(k)): + out.append(f"{arm} at {scale} ({cls or 'any class'}): server setting {k} is {a.get(k)!r} on the native server and " + f"{b.get(k)!r} on the JVM server") + return out + + +def refuse_pooled_builds(rs, where): + """SystemExit when one entry's rows span two builds of ArcadeDB (the exporter's check, beside _one_version's).""" + ids = identities(rs) + if len({b for _v, b in ids}) > 1: + raise SystemExit(f"REFUSING: {where} pools ArcadeDB rows of different builds {sorted(ids, key=str)}; its median would " + f"belong to neither. Freeze one build ({ENV}) or split the arms.") + return rs diff --git a/benchmarks/experiments/bench_common.py b/benchmarks/experiments/bench_common.py index 61a24b3d7ae..8a8d0ac18c5 100644 --- a/benchmarks/experiments/bench_common.py +++ b/benchmarks/experiments/bench_common.py @@ -41,6 +41,24 @@ def instrument_at_least(version): return INSTRUMENT >= version +# THE GRAPH ANALYTICAL VIEW MUST LIST THE `Message` SUPERTYPE FROM THIS RELEASE (upstream ArcadeData/arcadedb PR 9320, issue 9301). +# Since that change `GraphTraversalProviderRegistry.findProvider` hands a view to a walk only when the view covers EVERY vertex type of +# the schema (`coversVertexType(null)`, which also counts an abstract supertype such as Message), so a view that leaves Message out is +# silently not used for walks. One place holds the release so the lane that builds the view (l2_graph._gav_types) and the exporter that +# refuses a stale row (export_web._refuse_gav_without_supertype) cannot disagree. +GAV_SUPERTYPE_LISTED_FROM = (26, 11, 1) + + +def engine_release(version): + """(major, minor, patch) of an engine version string the harness records, or None when it carries none. + + Reads the string a row already stamps as `engine_version` ("26.11.1", "server:26.11.1", "26.11.1-SNAPSHOT (build ...)"); never a wheel file + name. A SNAPSHOT of release X counts as X, as the exporter does.""" + import re + m = re.search(r"(\d+)\.(\d+)\.(\d+)", str(version or "")) + return tuple(int(x) for x in m.groups()) if m else None + + # WHAT THE INSTRUMENT MEASURES (CAMPAIGN section 7 rows 72, 81, 82, 84 and the note under that table). These are the method, # not switches: an untimed sparse warm-up on queries disjoint from the timed set, the CPU idle-state hold at the lowest latency # target, an index on the dense lane's vector id, the embedded ArcadeDB JVM's initial heap left unset, large l4 results read on @@ -494,6 +512,11 @@ def arcade_query_ram_readback(): return {"arcadedb_query_max_heap_ram_mb": got, "arcadedb_query_max_heap_ram_source": "read from the engine (GlobalConfiguration.QUERY_MAX_HEAP_RAM)"} except Exception as e: # noqa: BLE001 + if engine_backend() == "native": + # arcadedb.queryMaxHeapRAM is SCOPE.JVM: schema:database does not list it, and the native build has no JPype to ask with. + return {"arcadedb_query_max_heap_ram_mb": n, + "arcadedb_query_max_heap_ram_source": "requested as the isolate argument -Darcadedb.queryMaxHeapRAM (a JVM-scope setting: " + "SELECT settings FROM schema:database does not list it, so the engine cannot be asked)"} return {"arcadedb_query_max_heap_ram_mb": n, "arcadedb_query_max_heap_ram_source": f"requested via the JVM argument (the engine could not be asked, {e.__class__.__name__})"} @@ -518,7 +541,30 @@ class separately (ArcadeData/arcadedb#8478): yes_nometadata is the executor's "durability_server_flags; the server exposes no read-back)") -def arcade_durability_readback(fallback_cls=None): +_ENGINE_SETTINGS = {} + + +def engine_database_settings(db): + """{key: value} of the database-scoped settings the ENGINE reports for an open database, asked in SQL so it is one code path on both + builds (the native package has no JPype): `SELECT settings FROM schema:database` lists every GlobalConfiguration entry of SCOPE.DATABASE + with the value in force for that database (checked at engine e353330c5c, FetchFromSchemaDatabaseStep, and on the native wheel). + JVM-scoped settings (arcadedb.queryMaxHeapRAM) are not in the list. + """ + row = db.query("sql", "SELECT settings FROM schema:database").first() + return {s["key"]: s["value"] for s in (row.get("settings") or [])} + + +def remember_engine_settings(db): + """Read the settings now and keep them, for a lane that closes its database before the row is stamped. Returns the dict; {} when it cannot be read.""" + try: + _ENGINE_SETTINGS.clear() + _ENGINE_SETTINGS.update(engine_database_settings(db)) + except Exception: # noqa: BLE001 + _ENGINE_SETTINGS.clear() + return dict(_ENGINE_SETTINGS) + + +def arcade_durability_readback(fallback_cls=None, db=None): """ASK THE ENGINE what txWalFlush it is running at, do not assert it. #81's standard for every durability string on a row is that it was read out @@ -530,12 +576,21 @@ def arcade_durability_readback(fallback_cls=None): Returns the string for the value the engine reports, or the asserted string with a note when the engine cannot be asked -- never silence. + + HOW IT ASKS (26.11.1, one path for both packages). txWalFlush is a SCOPE.DATABASE setting, so the engine answers in SQL + (`SELECT settings FROM schema:database`) on the JVM and on the native build alike: from `db` when the caller has the open database, else from + the snapshot `remember_engine_settings(db)` took, else, on the JVM build only, from GlobalConfiguration through JPype as before. The native + build has no JPype and so can only be asked through a database; a native caller that passes none gets the asserted string and its reason. """ want = at_class(DURABILITY_ARCADEDB, fallback_cls) try: - import jpype - gc = jpype.JClass("com.arcadedb.GlobalConfiguration") - value = int(gc.TX_WAL_FLUSH.getValue()) + settings = engine_database_settings(db) if db is not None else dict(_ENGINE_SETTINGS) + if "arcadedb.txWalFlush" in settings: + value = int(settings["arcadedb.txWalFlush"]) + else: + import jpype + gc = jpype.JClass("com.arcadedb.GlobalConfiguration") + value = int(gc.TX_WAL_FLUSH.getValue()) except Exception as e: # noqa: BLE001 return want + f" (asserted: the engine could not be asked, {e.__class__.__name__})" if value == 0: @@ -786,6 +841,17 @@ def _host_identity(): return name or "unknown" +def engine_backend(): + """"native" or "jvm": the package this process loaded, else the build of the ArcadeDB server it talks to (BENCH_ARCADEDB_SERVER_BUILD, set by the runner on a served ArcadeDB arm's client), else None. Lanes that set engine_version by hand stamp it too.""" + try: + import _arcadedb_import + got = _arcadedb_import.loaded_backend() + except Exception: + got = None + # A served ArcadeDB arm's client container never imports the package: the runner names the build of the SERVER it started (#222). + return got or os.environ.get("BENCH_ARCADEDB_SERVER_BUILD", "").strip() or None + + def run_conditions(**extra): """The conditions this process is actually running under, read not asserted. @@ -908,12 +974,23 @@ def _read(path): "ts_utc": time.strftime("%Y-%m-%dT%H:%M:%S+00:00", time.gmtime()), } try: - from importlib.metadata import version as _v - out["engine_version"] = _v("arcadedb-embedded") + import _arcadedb_import + out["engine_version"] = _arcadedb_import.dist_version() except Exception as e: out["engine_version"] = f"unknown ({e.__class__.__name__})" # THE JPYPE THIS PROCESS RAN (CAMPAIGN 7 row 73): empty unless the process imported it. out.update(jpype_fields()) + # WHICH ENGINE BUILD THIS PROCESS RAN, "native" or "jvm" (arcadedb.__backend__). engine_version is the same string for the + # `arcadedb` and `arcadedb-jvm` wheels of one release, so the version alone cannot tell the two arms apart. NOT named `backend`: + # that field is already the arm name (`arcadedb_sparse_embedded`, ...). + out["engine_backend"] = engine_backend() + # THE NATIVE HEAP AND SETTINGS, as the isolate received them (ARCADEDB_NATIVE_ISOLATE_ARGS, e.g. "-Xmx4g -Darcadedb.txWalFlush=0"): the + # native counterpart of the JVM's -Xmx/-D arguments, which the native package ignores. None on the JVM build and for every other arm. + try: + import _arcadedb_import + out["native_isolate_args"] = _arcadedb_import.effective_native_args() + except Exception: + out["native_isolate_args"] = None out.update(extra) return out diff --git a/benchmarks/experiments/build_images.sh b/benchmarks/experiments/build_images.sh index 361bfa3d18b..b555563efbb 100755 --- a/benchmarks/experiments/build_images.sh +++ b/benchmarks/experiments/build_images.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # Build the per-backend bench images. ARCADEDB_WHEEL= bakes a wheel into the arcadedb image, which is tagged with -# the wheel's version, dbbench:arcadedb-, and never with a floating dbbench:arcadedb (engine_pin.py, 2026-10-22). +# the wheel's version and build, dbbench:arcadedb--native or -jvm (the older arcadedb_embedded wheel: no suffix), and never with a floating dbbench:arcadedb (engine_pin.py, 2026-10-22). set -eu cd "$(dirname "$0")" @@ -34,7 +34,7 @@ fi _want_arcadedb=1 if [ $# -gt 0 ]; then _want_arcadedb=0; for _t in "$@"; do [ "$_t" = "arcadedb" ] && _want_arcadedb=1; done; fi if [ "$_want_arcadedb" = 1 ] && [ -z "${ARCADEDB_WHEEL:-}" ]; then - echo "REFUSING: the arcadedb image needs ARCADEDB_WHEEL=; there is no PyPI fallback." >&2 + echo "REFUSING: the arcadedb image needs ARCADEDB_WHEEL=; there is no PyPI fallback." >&2 echo " Name the targets (e.g. './build_images.sh duckdb client dense') to build the images that take no wheel." >&2 exit 1 fi @@ -79,6 +79,9 @@ declare -A PKGS=( # no way back to a working extension but the vendor restoring the file; 0.22.0 is the newest stable and its # extension loads. The extension is baked into the client image below, pinned by sha256 (LADYBUG_EXTENSIONS). [client]="requests==2.34.2 psycopg[binary]==3.3.6 $HARNESS_PKGS surrealdb==2.0.0 qdrant-client==1.20.0 pymilvus[bulk_writer]==3.0.2 minio==7.2.20 azure-storage-blob==12.31.0 ml-dtypes==0.6.0 elasticsearch==9.5.1 neo4j==6.4.1 ladybug==0.22.0 pymongo==4.18.1 python-arango==8.3.6 falkordb==1.7.1 redis==8.1.0 neo4j-rust-ext==6.4.1.0 pgvector==0.5.1 arcadedb-driver==0.2.0" + # arcadedb-driver==0.2.0 on the client line stays 0.2.0 ON PURPOSE (DECISIONS #259 (2)): this line is the build arg of dbbench:client, the + # image every comparator's measured rows ran on, and any edit to it rebuilds that image under a new digest. The served ArcadeDB arms run on + # the `client-arcadedb` layer below (ARCADEDB_DRIVER_PKGS, lean_http.OFFICIAL_PIN = 0.4.0), not on this driver. # ladybug==0.22.0 is the SAME pin as the client line above, here for the composed embedded cross-model arm # (composed_embedded_sqlite_lance_ladybug, 2026-10-25): SQLite + LanceDB + LadybugDB must import in ONE process, and # lancedb lives in this image. No new version: the dense lane's arms import none of it. @@ -108,7 +111,8 @@ LADYBUG_EXTENSIONS="vector=23da57e805fed208f1eacb4b011cb0c2e917ae098fa6853763e44 # test only that. Same for the wheel path, whose basename carries the version. _pin="" if [ -n "${ARCADEDB_WHEEL:-}" ]; then - # arcadedb_embedded-26.8.1.dev20-py3-none-linux_x86_64.whl -> 26.8.1.dev20 + # arcadedb-26.11.1.dev0-py3-none-manylinux_2_28_x86_64.whl, arcadedb_jvm-26.11.1.dev0-... and the older + # arcadedb_embedded-26.8.1.dev20-py3-none-linux_x86_64.whl all carry the version in the second field -> 26.8.1.dev20 _pin=$(basename "$ARCADEDB_WHEEL" | cut -d- -f2) fi case "$_pin" in @@ -139,10 +143,18 @@ echo "arcadedb pin: $_pin" # NO FLOATING ENGINE TAG (2026-10-22, engine_pin.py). dbbench:arcadedb pointed at whichever wheel was built last, and on the laptop it still # pointed at 26.10.1 after October was retired. The image that carries ArcadeDB is named for the version it carries, and runner.py refuses # any other tag for it. +# THE BUILD IS PART OF THE TAG (26.11.1, engine_pin.backend_from_wheel): one release ships a native wheel (arcadedb--...) and a JVM wheel +# (arcadedb_jvm--...), and `dbbench:arcadedb-` could hold only one. The tag is dbbench:arcadedb--native or -jvm; the older +# arcadedb_embedded wheels keep the unsuffixed tag, so the images and rows of earlier releases still resolve. +_bk="" +case "$(basename "${ARCADEDB_WHEEL:-}" | cut -d- -f1)" in + arcadedb) _bk=native ;; + arcadedb_jvm) _bk=jvm ;; +esac ARCADEDB_TAG="" if [ -n "$_pin" ]; then case "$_pin" in - [0-9]*.[0-9]*.[0-9]*) ARCADEDB_TAG="dbbench:arcadedb-$_pin" ;; + [0-9]*.[0-9]*.[0-9]*) ARCADEDB_TAG="dbbench:arcadedb-$_pin${_bk:+-$_bk}" ;; *) echo "REFUSING: the wheel name '$(basename "$ARCADEDB_WHEEL")' carries no x.y.z version, so the arcadedb image cannot be tagged" >&2; exit 1 ;; esac fi @@ -161,8 +173,23 @@ case "$JPYPE_VERSION" in esac echo "jpype pin: $JPYPE_VERSION" -targets=("$@"); [ ${#targets[@]} -eq 0 ] && targets=(arcadedb duckdb client dense pg-age mongo-search composed) +# THE OFFICIAL DRIVER'S ENVIRONMENT, PINNED (2026-11-07, DECISIONS #259; PyPI latest of that day, COMPARATORS.md "Client holds"). arcadedb-driver 0.4.0 +# requires attrs>=23.2, httpx>=0.27, typing-extensions>=4.10 and websockets>=14; httpx pulls httpcore, h11, anyio, idna and certifi. All are pinned +# here so the driver environment of every served ArcadeDB client cell is the same on every host and every build day. Installed as a layer over +# dbbench:client (Dockerfile.client-arcadedb), which stays byte-identical. The driver version is checked against lean_http.OFFICIAL_PIN by +# test_instrument_method.py; OfficialSession refuses to start on any other version. +ARCADEDB_DRIVER_PKGS="arcadedb-driver==0.4.0 httpx==0.28.1 httpcore==1.0.9 h11==0.16.0 anyio==4.16.0 idna==3.20 certifi==2026.10.23 attrs==26.1.0 typing-extensions==4.16.0 websockets==17.4" + +targets=("$@"); [ ${#targets[@]} -eq 0 ] && targets=(arcadedb duckdb client client-arcadedb dense pg-age mongo-search composed) for be in "${targets[@]}"; do + if [ "$be" = "client-arcadedb" ]; then + # The served ArcadeDB client image: dbbench:client + the official driver (Dockerfile.client-arcadedb). It is built FROM the local dbbench:client, + # so `client` is built first (the default target list and the stage scripts' sorted list both put it first), and never touches that image. + docker image inspect dbbench:client >/dev/null 2>&1 || { echo "REFUSING: dbbench:client-arcadedb is built on dbbench:client, which is not built here; build 'client' first" >&2; exit 1; } + echo "=== dbbench:client-arcadedb ($ARCADEDB_DRIVER_PKGS)" + docker build -q -t dbbench:client-arcadedb --build-arg DRIVER_PACKAGES="$ARCADEDB_DRIVER_PKGS" -f Dockerfile.client-arcadedb . >/dev/null && echo " ok" + continue + fi if [ "$be" = "composed" ]; then # Neo4j + Qdrant in one container (Dockerfile.composed), the composed # cross-model stack's server (BUGS F133, DECISIONS #117). Both halves diff --git a/benchmarks/experiments/deployment_decomp_probe.py b/benchmarks/experiments/deployment_decomp_probe.py index 65b3fe58870..5c1a4e62040 100644 --- a/benchmarks/experiments/deployment_decomp_probe.py +++ b/benchmarks/experiments/deployment_decomp_probe.py @@ -324,7 +324,8 @@ def main() -> int: args = ap.parse_args() import requests - import arcadedb_embedded as arcadedb + import _arcadedb_import + arcadedb = _arcadedb_import.load() results: dict[str, dict] = {} meta = { @@ -354,8 +355,12 @@ def main() -> int: # has to be pinned on the JVM before the server exists. Same -Xms=-Xmx # policy as every other lane: fixed heap for latency parity, with the # working set reported separately from cgroup anon. - from arcadedb_embedded.jvm import start_jvm - start_jvm(heap_size=args.heap, jvm_args=[f"-Xms{args.heap}"]) + if _arcadedb_import.backend() == "native": + # no JVM to start: the same heap policy goes to the isolate before the first engine call (_arcadedb_import.apply_native) + _arcadedb_import.apply_native({"heap_size": args.heap, "jvm_args": [f"-Xms{args.heap}"]}) + else: + start_jvm = _arcadedb_import.jvm_module().start_jvm + start_jvm(heap_size=args.heap, jvm_args=[f"-Xms{args.heap}"]) # Arms 1 and 2 share this process, so they share the JVM, heap, GC, engine # instance and page cache. That sharing IS the control. @@ -365,6 +370,9 @@ def main() -> int: config={"host": "127.0.0.1", "http_port": 2489, "mode": "development"}, ) as server: db = server.create_database(DB_NAME) + # the engine's own durability setting, asked in SQL while the database is open (e4_decomp stamps it on the row) + import bench_common + bench_common.remember_engine_settings(db) load_embedded(db, ROWS) print(f"loaded {ROWS:,} rows into the server-managed database", flush=True) diff --git a/benchmarks/experiments/e2_hybrid.py b/benchmarks/experiments/e2_hybrid.py index afcff8881ba..a7ff3eef569 100644 --- a/benchmarks/experiments/e2_hybrid.py +++ b/benchmarks/experiments/e2_hybrid.py @@ -236,7 +236,8 @@ def index_readback(self): TXN_SCOPE = "whole" def __init__(self): - import arcadedb_embedded as arcadedb + import _arcadedb_import + arcadedb = _arcadedb_import.load() self._a = arcadedb heap = os.environ.get("ARCADEDB_HEAP", "4g") self.db = arcadedb.create_database( @@ -247,9 +248,8 @@ def __init__(self): "jvm_args": bench_common.arcade_jvm_args()}) # THE WHEEL'S version, not its name (#156). A row stamped # "arcadedb-embedded" cannot be re-measured by anyone including us. - from importlib.metadata import version as _v - self.version = _v("arcadedb-embedded") - self.durability = bench_common.arcade_durability_readback() + self.version = _arcadedb_import.dist_version() + self.durability = bench_common.arcade_durability_readback(db=self.db) def build(self, vecs, edges): db, a = self.db, self._a @@ -2329,6 +2329,7 @@ def main(): with _beat.phase("connect", backend=args.backend): b = BACKENDS[args.backend]() out["engine_version"] = b.version + out["engine_backend"] = bench_common.engine_backend() out["instrument"] = bench_common.INSTRUMENT # The lane's search breadth (graph-index arms); IVF and LanceDB calibrate by effect and say so. An engine with a lower maximum # (pgvector, 1000) reports what it APPLIED in vector_ef and the lane's value in vector_ef_lane, so a clamp is never silent. diff --git a/benchmarks/experiments/e3_recovery.py b/benchmarks/experiments/e3_recovery.py index 2148aa5783c..e564d0494c0 100644 --- a/benchmarks/experiments/e3_recovery.py +++ b/benchmarks/experiments/e3_recovery.py @@ -20,7 +20,9 @@ import sys import time -import arcadedb_embedded as arcadedb +import _arcadedb_import + +arcadedb = _arcadedb_import.load() DB = "/data-local/e3_arcade" ROWS_PER_TX = 10 @@ -70,10 +72,12 @@ def verify(): # numbers are right, but nothing can re-derive them. Fixed here so the # freeze re-run leaves evidence. try: - from importlib.metadata import version as _pkgver - out["engine_version"] = _pkgver("arcadedb-embedded") + import _arcadedb_import as _ai + out["engine_version"] = _ai.dist_version() except Exception as e: out["engine_version"] = f"unknown ({e.__class__.__name__})" + import bench_common + out["engine_backend"] = bench_common.engine_backend() outp = os.environ.get("PROBE_OUT", "") if outp: with open(outp, "w") as f: diff --git a/benchmarks/experiments/e4_decomp.py b/benchmarks/experiments/e4_decomp.py index 206c0fe8914..ef4c02da86c 100644 --- a/benchmarks/experiments/e4_decomp.py +++ b/benchmarks/experiments/e4_decomp.py @@ -107,8 +107,9 @@ def main(): meta = payload.get("meta", {}) # The runner's row: what the lane measured, in fields a reader of runs.jsonl # expects; the full artifact stays in the file. - from importlib.metadata import version as _v - payload["engine_version"] = _v("arcadedb-embedded") + import _arcadedb_import + payload["engine_version"] = _arcadedb_import.dist_version() + payload["engine_backend"] = _arcadedb_import.loaded_backend() # THE DURABILITY THE ARM RAN AT, read out of the engine rather than # asserted (#81, #90). Every 2026-10 row must record it -- F10 fails a row # that does not -- and this lane recorded nothing, which nobody noticed @@ -116,6 +117,7 @@ def main(): # of every gate, so the gate that would have caught this never saw them. # The lane runs one engine at its default class; the readback says which. import bench_common + # Asked in SQL (arcade_durability_readback, one path for both builds): the probe took the in-process server's database settings while it was open. payload["durability"] = bench_common.arcade_durability_readback() payload["durability_class"] = bench_common.DURABILITY_CLASS stamp_clients(payload) diff --git a/benchmarks/experiments/engine_commit_equiv.json b/benchmarks/experiments/engine_commit_equiv.json new file mode 100644 index 00000000000..c1fdcda8712 --- /dev/null +++ b/benchmarks/experiments/engine_commit_equiv.json @@ -0,0 +1,13 @@ +{ + "26.11.1": { + "pin": "a3a5d124572608412aa187f9c4b290e071699473", + "equivalent": [ + "0f3bb2238c3cfc79c4a1c2cf2d8c5e36489652c0" + ], + "files_changed": 32, + "version_string_lines": 68, + "non_version_lines": 0, + "evidence": "ENGINE-COMMIT-EQUIV-26.11.1.md (notes: mini-ab-prep/fourarm-2611)", + "scope": "the native server image only; release pair only" + } +} diff --git a/benchmarks/experiments/engine_pin.py b/benchmarks/experiments/engine_pin.py index d258eee5fbd..694329805a9 100644 --- a/benchmarks/experiments/engine_pin.py +++ b/benchmarks/experiments/engine_pin.py @@ -8,17 +8,22 @@ THE RULE. The stage pins one ArcadeDB version by exporting `ARCADEDB_WHEEL` (the pinned wheel file, whose name carries the version). Before the first cell, for every ArcadeDB arm in the batch, `runner.py` reads the version out of what will run and refuses to start unless it is the pin: - embedded the image's installed `arcadedb-embedded` version AND the engine jar the wheel ships (`arcadedb-engine-.jar`); + embedded the image's installed ArcadeDB distribution version AND the engine the wheel ships: `arcadedb-jvm` (JVM build) or the older + `arcadedb-embedded` carry `arcadedb-engine-.jar`; `arcadedb` (the native build, no JVM, no jars) carries the engine + inside libarcadedb.so and names its version in `_build_info.ENGINE_VERSION`, read here as the same jar name; served the engine jar of the server image (`/home/arcadedb/lib/arcadedb-engine-.jar`), which is the engine the server loads. Nothing is started to ask, so the check costs two short `docker run`s per image and measures nothing. -NO FLOATING ENGINE TAGS. An image that carries ArcadeDB is tagged `dbbench:arcadedb-` (build_images.sh writes it) and the runner refuses +NO FLOATING ENGINE TAGS. An image that carries ArcadeDB is tagged `dbbench:arcadedb--native` or `-jvm` (build_images.sh writes it, the suffix +is the build of the pinned wheel: backend_from_wheel; the older unsuffixed `dbbench:arcadedb-` still reads, for the arcadedb_embedded wheels and +the rows made on them, and is refused for a native or JVM wheel) and the runner refuses any other tag for it, `dbbench:arcadedb` included: a floating tag points at whatever was built last, which is how a retired version stayed reachable. `dbbench:arcadedb` survives only as the logical build target name in runner.BACKENDS; the runner swaps it for the versioned tag from the pin, and a batch with an ArcadeDB arm and no pin is refused. The comparator images (client, duckdb, dense, pg-age, mongo-search, composed) keep their `dbbench:` names: their engines are pinned by `==` in build_images.sh or by digest in the Dockerfiles and are read back out of the images by version_pin_check.py --runtime; they do not move with the ArcadeDB release. """ +import json import os import re import subprocess @@ -26,44 +31,93 @@ LOGICAL_IMAGE = "dbbench:arcadedb" TARGET = "arcadedb" VERSION_RE = r"\d+\.\d+\.\d+(?:\.(?:dev|post)\d+)?" -FLOATING_OK = ("client", "duckdb", "dense", "pg-age", "mongo-search", "composed") +# client-arcadedb: dbbench:client plus the official arcadedb-driver at lean_http.OFFICIAL_PIN (Dockerfile.client-arcadedb); the served ArcadeDB client image. +FLOATING_OK = ("client", "client-arcadedb", "duckdb", "dense", "pg-age", "mongo-search", "composed") SERVER_LIB = "/home/arcadedb/lib" _JAR_RE = re.compile(r"^arcadedb-engine-(.+)\.jar$") -_EMBEDDED_PROBE = ( - "import importlib.metadata as m, importlib.util as u, os; " - "s = u.find_spec('arcadedb_embedded'); " - "print(m.version('arcadedb-embedded')); " - "print(' '.join(sorted(os.listdir(os.path.join(os.path.dirname(s.origin), 'jars')))))" -) +_EMBEDDED_PROBE = """ +import importlib.metadata as m, importlib.util as u, os +s = next((x for x in (u.find_spec(n) for n in ('arcadedb', 'arcadedb_embedded')) if x and x.origin), None) +dist = None +for n in ('arcadedb', 'arcadedb-jvm', 'arcadedb-embedded'): + try: + v = m.version(n) + dist = n + break + except m.PackageNotFoundError: + pass +print(v) +print(dist) +pkg = os.path.dirname(s.origin) +jars = os.path.join(pkg, 'jars') +if os.path.isdir(jars): + print(' '.join(sorted(os.listdir(jars)))) +else: + from importlib import import_module + print('arcadedb-engine-' + import_module(os.path.basename(pkg) + '._build_info').ENGINE_VERSION + '.jar') +""" +# THE ENGINE'S OWN BUILD STAMP, asked in SQL so it is one path for the native and the JVM package: `SELECT version() AS v` returns +# " (build //)" (checked on the 26.10.2 native wheel: full 40-character sha). The commit of a native wheel is +# not readable from its files (no jars), so this is how it is verified rather than inferred. +_BUILD_PROBE = """ +import importlib, importlib.util, os, tempfile +mod = importlib.import_module(next(n for n in ('arcadedb', 'arcadedb_embedded') if importlib.util.find_spec(n))) +d = tempfile.mkdtemp() +db = mod.create_database(os.path.join(d, 'p')) +print(db.query('sql', 'SELECT version() AS v').first().get('v')) +db.close() +""" +BUILD_RE = re.compile(r"\(build ([0-9a-f]{7,40})/") +# The native distribution is `arcadedb`, the JVM one `arcadedb-jvm`; the old `arcadedb-embedded` is a JVM build. +BACKEND_OF = {"arcadedb": "native", "arcadedb-jvm": "jvm", "arcadedb-embedded": "jvm"} def pin_from_wheel(path): - """The version a wheel file name carries (arcadedb_embedded-26.10.2-cp312-...whl -> 26.10.2), or "" when it carries none.""" + """The version a wheel file name carries (arcadedb_embedded-26.10.2-cp312-...whl, arcadedb_jvm-26.11.1.dev0-py3-...whl or + arcadedb-26.11.1.dev0-py3-...whl -> the version), or "" when it carries none.""" parts = os.path.basename(path or "").split("-") return parts[1] if len(parts) > 2 and re.fullmatch(VERSION_RE, parts[1]) else "" -def versioned_image(version): - return f"{LOGICAL_IMAGE}-{version}" +def backend_from_wheel(path): + """"native" for arcadedb--*.whl, "jvm" for arcadedb_jvm--*.whl, "" for the older arcadedb_embedded wheel (or any other name). + + The build is part of the image tag since 26.11.1: one release has a native and a JVM wheel, and one tag cannot hold both. + """ + head = os.path.basename(path or "").split("-")[0].replace("_", "-") + return {"arcadedb": "native", "arcadedb-jvm": "jvm"}.get(head, "") + + +def versioned_image(version, backend=""): + """dbbench:arcadedb--native / -jvm; the older unsuffixed dbbench:arcadedb- when no build is named.""" + return f"{LOGICAL_IMAGE}-{version}" + (f"-{backend}" if backend else "") def build_target(image): """The build_images.sh target an image name stands for: dbbench:arcadedb-26.10.2 -> arcadedb, dbbench:duckdb -> duckdb.""" tag = image.split(":", 1)[1] - return TARGET if re.fullmatch(rf"{TARGET}(?:-{VERSION_RE})?", tag) else tag + return TARGET if re.fullmatch(rf"{TARGET}(?:-{VERSION_RE}(?:-(?:native|jvm))?)?", tag) else tag -def check_image_tag(image, pin=None): - """None when `image` may run, else why it is refused. Only `dbbench:*` names are judged here (digest-pinned third-party images are not ours).""" +def check_image_tag(image, pin=None, backend=""): + """None when `image` may run, else why it is refused. Only `dbbench:*` names are judged here (digest-pinned third-party images are not ours). + + `backend` ("native"/"jvm", from backend_from_wheel) is the build the stage pinned: the tag must then end in it. An unsuffixed tag is the + pre-26.11.1 form and is accepted only when no build is named (`backend` empty), so an old image can never stand in for a native or JVM wheel. + """ if not image.startswith("dbbench:"): return None tag = image.split(":", 1)[1] if tag.startswith(TARGET): - if not re.fullmatch(rf"{TARGET}-{VERSION_RE}", tag): + m = re.fullmatch(rf"{TARGET}-({VERSION_RE})(?:-(native|jvm))?", tag) + if not m: return (f"{image} is an ArcadeDB image tag without a version. A floating tag points at whatever was built last; " - f"the tag must be {LOGICAL_IMAGE}-") - if pin and tag != f"{TARGET}-{pin}": - return f"{image} names ArcadeDB {tag[len(TARGET) + 1:]}, the stage pins {pin}" + f"the tag must be {LOGICAL_IMAGE}--native or -jvm") + if pin and m.group(1) != pin: + return f"{image} names ArcadeDB {m.group(1)}, the stage pins {pin}" + if backend and m.group(2) != backend: + return (f"{image} is the {m.group(2) or 'unsuffixed (pre-26.11.1)'} image, the stage pins the {backend} build " + f"({versioned_image(pin or '', backend)})") return None if tag not in FLOATING_OK: return f"{image} is not a known benchmark image ({', '.join(FLOATING_OK)}, or {LOGICAL_IMAGE}-)" @@ -91,14 +145,68 @@ def _engine_jar_versions(names): def read_embedded(image, run=None): - """({"wheel": ..., "engine_jar": [...]}, None) read out of a bench image by running python in it, or (None, why).""" + """({"wheel": ..., "dist": ..., "backend": ..., "engine_jar": [...]}, None) read out of a bench image by running python in it, or (None, why). + + Three lines from the probe (version, distribution name, engine jar names). Two lines (version, jars) are still read, for a probe + or a test double that predates the new package names: the distribution is then the old `arcadedb-embedded`. + """ out, err = _docker(["--entrypoint", "python3", image, "-c", _EMBEDDED_PROBE], run) if err: return None, err lines = out.strip().splitlines() - if len(lines) < 2: - return None, f"the probe in {image} printed {out.strip()!r}, not a wheel version and a jar list" - return {"wheel": lines[0].strip(), "engine_jar": _engine_jar_versions(lines[1].split())}, None + if len(lines) == 2: + lines = [lines[0], "arcadedb-embedded", lines[1]] + if len(lines) < 3: + return None, f"the probe in {image} printed {out.strip()!r}, not a wheel version, a distribution name and a jar list" + dist = lines[1].strip() + return {"wheel": lines[0].strip(), "dist": dist, "backend": BACKEND_OF.get(dist), + "engine_jar": _engine_jar_versions(lines[2].split())}, None + + +def build_sha(version_string): + """The commit in a `SELECT version()` answer ("26.10.2 (build //)"), or "" when it names none.""" + m = BUILD_RE.search(str(version_string or "")) + return m.group(1) if m else "" + + +def commit_matches(reported, pinned): + """Prefix comparison at the shorter length (the engine prints 40 characters, a stage may pin 9), never under 7.""" + n = min(len(reported or ""), len(pinned or ""), 40) + return n >= 7 and reported[:n].lower() == pinned[:n].lower() + + +EQUIV_FILE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "engine_commit_equiv.json") + + +def _equiv_registry(): + try: + with open(EQUIV_FILE) as f: + return json.load(f) + except (OSError, ValueError): + return {} + + +def native_server_equivalent(reported, pinned, registry=None): + """The release whose registry entry (engine_commit_equiv.json, written by make_engine_commit_equiv.py from the git diff) says the NATIVE + SERVER image's commit `reported` is the same engine as the pin, else "". 26.11.1: the native-image.yml build at the release tag is one + version-string commit after the commit of the JVM image, the jars and both wheels. Release entries only (A.B.C keys); the pin must be the + entry's pin and the reported commit one of its equivalents, both compared as prefixes of at least 7 characters. Applies to the native server + only: callers pass nothing else here, and the embedded and JVM arms still have to equal the pin.""" + for release, e in (_equiv_registry() if registry is None else registry).items(): + if not re.fullmatch(r"\d+\.\d+\.\d+", release): + continue + if commit_matches(pinned, e.get("pin", "")) and any(commit_matches(reported, x) for x in e.get("equivalent", [])): + return release + return "" + + +def read_embedded_build(image, run=None): + """(sha, None) the engine inside a bench image reports for itself (a short-lived database, `SELECT version()`), or ("", why).""" + out, err = _docker(["--entrypoint", "python3", image, "-c", _BUILD_PROBE], run) + if err: + return "", err + sha = build_sha(out.strip().splitlines()[-1] if out.strip() else "") + return (sha, None) if sha else ("", f"the engine in {image} answered version() with {out.strip()[-120:]!r}, which names no build") def read_served(image, run=None): @@ -109,20 +217,33 @@ def read_served(image, run=None): return {"engine_jar": _engine_jar_versions(out.split())}, None -def verify(pin, embedded_images=(), served_images=(), run=None): - """(readback, problems): what each image reports, and one sentence per way it is not the pin. Empty `problems` means every image runs `pin`.""" +def verify(pin, embedded_images=(), served_images=(), run=None, backend="", commit=""): + """(readback, problems): what each image reports, and one sentence per way it is not the pin. Empty `problems` means every image runs `pin`. + + With `backend` ("native"/"jvm") an embedded image must also carry that build's distribution (arcadedb / arcadedb-jvm), read out of the image. + With `commit` (the stage's ARCADEDB_ENGINE_COMMIT) the engine inside each embedded image must also report that build commit, read in SQL. + """ problems, readback = [], {} if not pin: return readback, ["no pinned version: ARCADEDB_WHEEL is unset or its file name carries no version"] for image in sorted(set(embedded_images)): - why = check_image_tag(image, pin) + why = check_image_tag(image, pin, backend) got, err = (None, why) if why else read_embedded(image, run) readback[image] = got if err: problems.append(f"{image}: {err}") continue if got["wheel"] != pin: - problems.append(f"{image} has arcadedb-embedded {got['wheel']}, the stage pins {pin}") + problems.append(f"{image} has {got['dist']} {got['wheel']}, the stage pins {pin}") + if commit: + sha, berr = read_embedded_build(image, run) + if berr: + problems.append(f"{image}: {berr}") + elif not commit_matches(sha, commit): + problems.append(f"{image} runs engine commit {sha[:9]}, the stage pins {commit[:9]}") + readback[image]["engine_commit"] = sha + if backend and got["backend"] != backend: + problems.append(f"{image} carries {got['dist']} ({got['backend']} build), the stage pins the {backend} build") if got["engine_jar"] != [pin]: problems.append(f"{image} ships engine jar {','.join(got['engine_jar']) or '(none)'}, the stage pins {pin}") for image in sorted(set(served_images)): diff --git a/benchmarks/experiments/export_web.py b/benchmarks/experiments/export_web.py index 70a07aa88eb..97129d9e0a6 100644 --- a/benchmarks/experiments/export_web.py +++ b/benchmarks/experiments/export_web.py @@ -46,6 +46,8 @@ from runner import BACKENDS, MEM_BY_SCALE, HEAP_BY_SCALE # noqa: E402 (path set above) import bench_common # noqa: E402 +import arm_identity as _AI # noqa: E402 +import engine_pin as _EP # noqa: E402 # THE PQ ARMS CAN HAVE ROWS EVEN WHEN THEY ARE NOT MEASURING. runner registers them only under BENCH_DENSE_PQ=1 (a stage # must not start a PQ cell by accident), but a freeze that holds their rows is read here without that variable, and their @@ -281,6 +283,7 @@ def _one_version(rs, where): is indistinguishable from a correct row to every other gate -- the payload carries one string either way, so version_consistency_check sees nothing. """ + _AI.refuse_pooled_builds(rs, where) # (engine_version, build) is the arm: native and jvm rows of one release are two arms seen = {str(r.get("engine_version")) for r in rs if r.get("engine_version")} if len(seen) > 1: raise SystemExit( @@ -1088,6 +1091,24 @@ def _num(value): ] +def _overlay_files_by_build(root, stem, tail, backend_key): + """[(build, files)] for one overlay arm: "" for a comparator or a served arm and for a page that names no build, else "native" and + "jvm" read from the first pass of each file (arm_identity.build_of; an unstamped ArcadeDB file is the JVM build). The native + files carry `_native` after the stem; a glob on the plain stem never matches them.""" + names = sorted(root.glob(f"{stem}{tail}")) + sorted(root.glob(f"{stem}_native{tail}")) + if not (_SPLIT and _AI.built({"backend": backend_key})): + return [("", sorted(root.glob(f"{stem}{tail}")))] if names else [] + got = {} + for fp in names: + try: + passes = json.loads(fp.read_text(encoding="utf-8")) + except ValueError: + continue + if passes: + got.setdefault(_AI.build_of(dict(passes[0], backend=backend_key)), []).append(fp) + return [(b, got[b]) for b in ("native", "jvm") if b in got] + + def _dense_overlay_entries(scale="deep10m"): """The DEEP-10M tier, every engine, both passes. @@ -1125,10 +1146,14 @@ def _dense_overlay_entries(scale="deep10m"): if not root.is_dir(): return [] out = [] + # THE OVERLAY BELONGS TO THE BUILD THAT PRODUCED IT. The driver stamps engine_backend on every pass it writes (run_conditions); a + # native run's files are mp__native_b.json beside the JVM build's mp__b.json, and the build is read from the + # file's own content. A file without the field is the JVM build, like an old row (arm_identity.build_of). + _units = [] for arm, backend_key, label, ours in arms: - hits = sorted(root.glob(f"mp_{arm}_b*.json")) - if not hits: - continue + for _ob, hits in _overlay_files_by_build(root, f"mp_{arm}", "_b*.json", backend_key): + _units.append((arm, backend_key, _with_build(label, _ob), ours, hits, _ob)) + for arm, backend_key, label, ours, hits, _ob in _units: cold, warm, recall, build, peak, ver = [], [], [], [], [], set() for h in hits: passes = json.loads(h.read_text(encoding="utf-8")) @@ -1201,10 +1226,10 @@ def _dense_overlay_entries(scale="deep10m"): # served arm's multipass file sees only the client container (3.6 GiB # against the pair's 28), and no file carries disk. Same build, same # envelope; the overlay's own peak is the fallback. - _pk = _campaign_stat(_cb, scale, MEM_PEAK_FIELD) or _agg(peak, MEM_PEAK_FIELD) + _pk = _campaign_stat(_cb, scale, MEM_PEAK_FIELD, build=_ob) or _agg(peak, MEM_PEAK_FIELD) if _pk is not None: metrics["peak memory GiB"] = _pk - _dk = _campaign_stat(_cb, scale, "disk_data_mb") + _dk = _campaign_stat(_cb, scale, "disk_data_mb", build=_ob) if _dk is not None: metrics["disk GiB"] = _dk # Only OUR arms carry a version string; the comparator containers do @@ -1218,6 +1243,7 @@ def _dense_overlay_entries(scale="deep10m"): v = _campaign_engine_string(_cb) out.append({ "backend": label, + "build": _ob or None, "backend_key": _cb, "is_arcadedb": ours, # NOT `arm == "int8"`. arm is a FILENAME token, and only ArcadeDB's @@ -1270,7 +1296,7 @@ def _peak_mem(r): return v if v is not None else _num(r.get(MEM_PEAK_FALLBACK)) -def _campaign_stat(backend, scale, field, lanes=("l3d", "l3s")): +def _campaign_stat(backend, scale, field, lanes=("l3d", "l3s"), build=""): """_agg of one field over the newest campaign row per rep for (backend, scale): the overlay files (multipass) carry no disk reading, but the campaign cell of the same arm in the same envelope does, and the on-disk @@ -1285,6 +1311,8 @@ def _campaign_stat(backend, scale, field, lanes=("l3d", "l3s")): continue if r.get("backend") != backend or r.get("scale") != scale or r.get("error"): continue + if build and _arm_build(r) != build: # one build's cell, never the other's + continue # PAGE-SPEC 4a rule 3: settled=False blocks publication of # the DISK reading. This path feeds the dense and sparse disk # columns straight from the campaign log, bypassing @@ -1410,6 +1438,24 @@ def _agg(rows, field, across_scales=False): # # The headline query is the FIRST one in each list, so the p99 belongs to the query the section leads with. _FROZEN_ROWS = [] # set by main() once the CSV is read; the column switches read it +# TWO BUILDS, TWO ARMS (26.11.1). True when some in-process ArcadeDB row ran on the native build: every such arm then names its build and +# the two builds sit side by side (arm_identity.splits_page). False for a freeze of old rows, which render as they always did. +_SPLIT = False + + +def _arm_build(r): + """"native" or "jvm" for an in-process ArcadeDB row of a split page, else "" (served arms, comparators, an unsplit page).""" + return (_AI.build_of(r) or "") if (_SPLIT and _AI.built(r)) else "" + + +def _with_build(label, build): + """The arm label with its build ("ArcadeDB (embedded, native build)"), or the label as it is when the arm has no build.""" + return _AI.build_label(label, build) if build else label + + +def _strip_build(label): + """The label without its build suffix: the key the lane's own tables (order lists, deployments) are written in.""" + return _AI.strip_build(label) OCT_TABLE_METRICS = { # Both document tables come from this one lane spec and are split by # _restructure_tables; the union of their columns lives here. @@ -1793,6 +1839,65 @@ def _declare_absence(table_id, backend, column, kind, why): _DECLARED_ABSENCES[table_id].append(rec) +def _declare_missing_builds(tables): + """A build with no row where the other build has one is printed as not measured, never left out. + + On a split page (arm_identity.splits_page) every in-process ArcadeDB arm should stand twice, once per build. Where one build has + entries at a size and the other has none, the missing arm is declared (kind "not measured", whole row) and one sentence per table + and build names the arms and sizes. A deliberate one-build freeze (BENCH_ENGINE_BACKEND) declares nothing: the other build was + left out on purpose, and the arm labels still say which build the page is. + """ + if not _SPLIT or _AI.wanted_build(): + return + for t in tables: + have = collections.defaultdict(set) # (label without build, size) -> builds present + for e in t.get("entries", []): + if e.get("build") in _AI.BUILDS: + have[(_strip_build(str(e["backend"])), str(e.get("scale_label") or e.get("scale")))].add(e["build"]) + missing = collections.defaultdict(lambda: collections.defaultdict(list)) # build -> arm -> sizes + for (label, size), builds in sorted(have.items()): + for b in _AI.BUILDS: + if b not in builds: + missing[b][label].append(size) + for b in _AI.BUILDS: + arms = missing.get(b) + if not arms: + continue + parts, vals = [], [] + for label, sizes in sorted(arms.items()): + arm = _with_build(label, b) + _declare_absence(t["id"], arm, None, "not measured", + f"{arm} was not measured at {_join_and(sizes)}.") + parts.append(f"{arm} at {_join_and(sizes)}") + vals += [arm, *sizes] + t.setdefault("conditions", []) + note = _gen(f"The {_AI.BUILD_NAMES[b]} was not measured for {_join_and(parts)}, so those rows are absent.", *vals) + if note not in t["conditions"]: + t["conditions"].append(note) + + +def _disclose_server_builds(tables): + """Under every table that holds a native-build server arm: both server arms are upstream's official images (the JVM server image and the + native-image server), run with the same server settings that each server reports back at start, and the one disclosed difference is the + CPU baseline each artifact was compiled for (read from the rows' `cpu_baseline`, never typed here).""" + if not _SPLIT or _AI.wanted_build(): + return + nat = next((r.get("cpu_baseline") for r in _FROZEN_ROWS if _AI.served(r) and _AI.build_of(r) == "native" and r.get("cpu_baseline")), None) + wheel = next((r.get("cpu_baseline") for r in _FROZEN_ROWS if _AI.in_process(r) and _AI.build_of(r) == "native" and r.get("cpu_baseline")), None) + if not nat: + return + wheel_part = f" and the native wheel (the embedded arm) as {wheel}" if wheel else "" + for t in tables: + if not any(e.get("build") == "native" and "server" in str(e.get("backend")) for e in t.get("entries", [])): + continue + note = _gen(f"Both server arms are the official ArcadeDB server images, one built for the JVM and one compiled to a native executable, " + f"started with the same server settings, which each server reports back when it starts. The CPU baselines differ: " + f"the native server image is {nat}{wheel_part}, and the JVM server compiles for the CPU it runs on.", nat, *([wheel] if wheel else [])) + t.setdefault("conditions", []) + if note not in t["conditions"]: + t["conditions"].append(note) + + def _withhold_cells(tables): """Take out the cells whose answer was wrong, and say so on the table.""" notes = collections.defaultdict(list) @@ -1821,7 +1926,7 @@ def _withhold_cells(tables): # stale list. A tier with no message half covers Person and KNOWS only and is not concerned. _GAV_TYPES_FIELD = "gav_types" _GAV_SUPERTYPE = "Message" -_GAV_SUPERTYPE_LISTED_FROM = (26, 11, 1) +_GAV_SUPERTYPE_LISTED_FROM = bench_common.GAV_SUPERTYPE_LISTED_FROM # one release for the lane that builds the view and this refusal def _gav_message_rows(rows): @@ -2339,11 +2444,46 @@ def _cpu_idle_note(rows): return text +def _native_server_equiv_note(tables, rows=None): + """The sentences that disclose the native server image's engine commit (one per release pair a row used), [] when no row used it. + + The release has two engine commits (DECISIONS #256): the JVM image, the jars and both wheels are built at the pin, and the official native + server image is built one commit later, which changes version strings only. A served native row stamps `server_engine_commit_equiv` (the + commit the server reported) only when the runner accepted that pair, so the sentence is written when a row printed in some table carries the + field, and from engine_commit_equiv.json (the registry the runner, arm_identity and provenance_check also read) for the release and the + commits. A row that claims the equivalence with no registry entry behind it is refused, not printed.""" + rows = _FROZEN_ROWS if rows is None else rows + shown = {e.get("backend_key") for t in tables for e in t.get("entries") or []} + used = {(str(r.get("engine_commit") or ""), str(r.get("server_engine_commit_equiv") or "")) + for r in rows if str(r.get("server_engine_commit_equiv") or "").strip() and r.get("backend") in shown} + out = [] + for pin, reported in sorted(used): + release = _EP.native_server_equivalent(reported, pin) + entry = _EP._equiv_registry().get(release) if release else None + if not entry: + raise SystemExit(f"REFUSING: a row stamps server_engine_commit_equiv={reported[:9]!r} against engine_commit {pin[:9]!r} and " + f"engine_commit_equiv.json has no release pair for it") + # the registry's spelling of both commits, whatever length the row or the stage pin wrote (9 or 40 characters) + short_pin = entry["pin"][:10] + short_eq = next(x for x in entry["equivalent"] if _EP.commit_matches(reported, x))[:10] + files, lines = entry.get("files_changed"), entry.get("version_string_lines") + if not (isinstance(files, int) and isinstance(lines, int) and entry.get("non_version_lines") == 0): + raise SystemExit(f"REFUSING: the engine_commit_equiv.json entry for {release} does not state a version-strings-only difference") + text = (f"The ArcadeDB {release} native server image reports engine commit {short_eq}, and every other ArcadeDB build on the page (the JVM image, " + f"the jars, and both Python wheels) is commit {short_pin}: upstream builds the native server image at the release tag, one commit after " + f"{short_pin}. The difference between the two commits is {files:,} files and {lines:,} changed lines, every one " + f"a version string and none of them source, so the rows keep {short_pin} as their engine commit.") + if text not in out: # the same pair written with a 9-character and a 40-character pin is one sentence + out.append(_gen(text, release, short_pin, short_eq, f"{files:,}", f"{lines:,}")) + return out + + def _global_conditions(tables): reps = _reps_note(tables) out = [_R("GLOBAL", "docker"), _R("GLOBAL", "memory")] if reps: out.append(reps) + out += _native_server_equiv_note(tables) out += [_R("GLOBAL", "defaults"), _R("GLOBAL", "digest"), _R("GLOBAL", "index_order"), _R("GLOBAL", "jvm_heap")] return out @@ -2687,7 +2827,8 @@ def _l4_canonical(all_rows): # KEYED ON THE TIER TOO: grouped by display label alone, a table with # two tiers (ts100, ts1000) would publish every cell as a median across # BOTH corpora, a number belonging to neither (BUGS F110). - grouped[(label, str(r.get("scale")))].append(r) + # A split page keys the two builds apart (the label carries the build), so no l4 median pools them. + grouped[(_with_build(label, _arm_build(r)), str(r.get("scale")))].append(r) return grouped or None @@ -2732,10 +2873,11 @@ def _sparse_multipass_table(): # optional so the table renders before they land, and the sparse # condition says whether 100k has a second pass by looking, not by text. for tier in ("medium", "small", "tiny"): + _sp_units = [] for arm, backend, label in SPARSE_MP_ARMS + SPARSE_MP_OPTIONAL_ARMS: - fp = root / f"sp_{arm}_{tier}.json" - if not fp.is_file(): - continue + for _ob, _fps in _overlay_files_by_build(root, f"sp_{arm}", f"_{tier}.json", backend): + _sp_units += [(arm, backend, _with_build(label, _ob), _ob, _fps[0])] + for arm, backend, label, _ob, fp in _sp_units: passes = json.loads(fp.read_text(encoding="utf-8")) cold = [r for r in passes if r.get("rep") == 0] warm = [r for r in passes if r.get("rep", 0) >= 1] @@ -2748,6 +2890,7 @@ def _sparse_multipass_table(): w99 = statistics.median(r["query_p99_ms"] for r in warm if r.get("query_p99_ms") is not None) if any(r.get("query_p99_ms") is not None for r in warm) else None entries.append({ "backend": label, + "build": _ob or None, "backend_key": backend, "is_arcadedb": "arcade" in backend, "precision": SPARSE_PRECISION.get(backend), @@ -2925,18 +3068,29 @@ def _engine_of(r): # registered as an absence the coverage gate reads. declared.append(r) continue - by.setdefault((r.get("workload"), r.get("scale"), _srv, _engine_of(r)), []).append(r) + by.setdefault((r.get("workload"), r.get("scale"), _srv, _engine_of(r), _arm_build(r)), []).append(r) entries = [] - _lc_columns = (["JVM start ms", "first open ms", "cold process ms"] + # The engine's own start step: start_jvm() on the JVM build (`jvm_start_ms`), which the native build has no counterpart of (null); + # there the same step is `engine_load_ms` (library load and isolate creation). A page holding both builds names the column for both (below). + _start_field, _start_label = (("engine_load_ms", "engine load ms") + if any(_AI.build_of(r) == "native" for r in rows) else ("jvm_start_ms", "JVM start ms")) + # BOTH BUILDS ON ONE TABLE: the step is start_jvm() on the JVM build and library load plus isolate creation on the native one, so + # the column is named for what both are, and each arm reads its own field (the sentence under the table says which is which). + _both_builds = {"native", "jvm"} <= _AI.builds_in(rows) + if _both_builds: + _start_label = "engine start ms" + _lc_columns = ([_start_label, "first open ms", "cold process ms"] + [LIFECYCLE_SCENARIO_LABELS[k] for k in LIFECYCLE_PAGE_SCENARIOS]) - for (situation, scale, _srv, engine), rs in sorted(by.items()): + for (situation, scale, _srv, engine, build), rs in sorted(by.items()): if situation in LIFECYCLE_WITHHELD: continue + _start_field = ("engine_load_ms" if build == "native" else "jvm_start_ms") if _both_builds else _start_field _ours = engine == "ArcadeDB" _mode = "server" if _srv else "embedded" entry = { - "backend": _lifecycle_row_label(rs[0].get("backend"), situation), + "backend": _with_build(_lifecycle_row_label(rs[0].get("backend"), situation), build), + "build": build or None, "backend_key": str(rs[0].get("backend")), "is_arcadedb": _ours, "engine": engine, @@ -2957,7 +3111,7 @@ def _engine_of(r): # session number: a 5 ms open inside a process that took half a second to # reach its first database call is not a 5 ms cost to anyone launching a # CLI. - for field, label in (("jvm_start_ms", "JVM start ms"), + for field, label in ((_start_field, _start_label), ("first_open_ms", "first open ms"), ("cold_process_ms", "cold process ms")): got = _agg(rs, field) @@ -3024,6 +3178,7 @@ def _engine_of(r): "title": "Session cost, open to close", "dataset": "synthetic, one structure per row", "conditions": [_R("lifecycle", k) for k in ("session", "cold_start", "clean_close", "server_rows")] + + ([_R("lifecycle", "engine_start")] if _both_builds else []) + [_gen(f"{LIFECYCLE_SITUATION_LABELS.get(k, k)} is withheld: {v}", v) for k, v in sorted(LIFECYCLE_WITHHELD.items())] + declared_notes, "columns": _lc_columns, @@ -3217,10 +3372,10 @@ def _durability_table(all_rows): # out of this one for the same reason. if str(r.get("backend")) in OFF_PAGE_ARMS: continue - by.setdefault((str(r.get("backend")), str(r.get("scale")) if per_scale else None), + by.setdefault((str(r.get("backend")), str(r.get("scale")) if per_scale else None, _arm_build(r)), []).append(r) - for (backend, sc), rs in sorted(by.items(), key=lambda kv: (_durability_scale_rank(lane, kv[0][1]), - kv[0][0])): + for (backend, sc, build), rs in sorted(by.items(), key=lambda kv: (_durability_scale_rank(lane, kv[0][1]), + kv[0][0], kv[0][2])): if sc is not None: # ONE ROW PER CORPUS SIZE for an operation whose cost grows with # the corpus, named in the Size column beside the operation. @@ -3238,7 +3393,8 @@ def _durability_table(all_rows): no_setting = any(str(r.get("durability_no_setting")).lower() in ("true", "1") for r in rs) entry = { - "backend": display_name(backend), + "backend": _with_build(display_name(backend), build), + "build": build or None, "is_arcadedb": "arcadedb" in backend, "scale": op_key_s, "scale_label": op_label_s, @@ -3408,7 +3564,7 @@ def multimodel_engine(name): """The capability table's row for an engine name (MULTIMODEL_ALIASES).""" return MULTIMODEL_ALIASES.get(name, name) MULTIMODEL_CELLS = {"measured": "measured", "declared": "declared", "none": "no arm"} -MULTIMODEL_KINDS = {"censored": "censored", "withheld": "withheld", +MULTIMODEL_KINDS = {"censored": "censored", "withheld": "withheld", "not measured": "not measured", "unexpressible": "cannot express", "envelope": "out of memory", "failed": "failed", # A row a defect in the engine's own release decided @@ -3620,8 +3776,8 @@ def _l4_table(all_rows): # does rather than interleaving two corpora under one heading. _tier_rank = {"ts100": 0, "ts1000": 1} for key in sorted(grouped, key=lambda k: (_tier_rank.get(k[1], 99), - order.index(k[0]) if k[0] in order else 99, - k[0])): + order.index(_strip_build(k[0])) if _strip_build(k[0]) in order else 99, + _strip_build(k[0]), {"native": 0, "jvm": 1}.get(_AI.build_in_label(k[0]), 2))): label, _scale = key rs = grouped[key] entry = { @@ -3636,7 +3792,8 @@ def _l4_table(all_rows): "scale_label": f"{_l4_points(_scale)} points", "workload": L4_SHAPE["workload"], "n_docs": str(rs[0].get("n_points")), - "deployment": L4_DEPLOYMENT[label], + "deployment": L4_DEPLOYMENT[_strip_build(label)], + "build": _AI.build_in_label(label), "image": None, # One engine name plus one version. The two ArcadeDB arms recorded # this differently ("26.8.1" from one adapter, "arcadedb 26.8.1" @@ -4163,6 +4320,7 @@ def _milvus_seal_proportion(): }, "lifecycle": { "session": ("The SESSION is open + action + close. Reporting open and close alone hides work triggered between them.", []), + "engine_start": ("Engine start ms is the start_jvm() call on the JVM build, and the library load with the creation of the isolate on the native build; the two steps are the engine's own start, which is why they share a column.", []), "cold_start": ("Cold start is measured in a fresh subprocess and reported beside every session number, because a millisecond open inside a process that takes half a second to reach its first database call is not a millisecond to whoever launched it.", []), "clean_close": ("A clean close should be O(what was written), not O(what is stored): write nothing and closing should cost the same at 10k documents and 10M.", []), "server_rows": ("Server rows have no JVM start, first open, or cold process: the server is already running when the probe connects, so those three columns describe the embedded process only. The session columns are measured for both.", []), @@ -4363,8 +4521,9 @@ def _same_arm(r, _eb=e.get("backend")): # ArcadeDB ones that were. return "(" not in dn and str(_eb).startswith(dn + " (") + # one build's rows behind one build's entry (an entry that names no build reads every row of its arm) _match = lambda r: (r.get("lane") == lane and (not wl or r.get("workload") == wl) - and _same_arm(r)) + and _same_arm(r) and (not e.get("build") or _arm_build(r) == e["build"])) rs = [r for r in _FROZEN_ROWS if _match(r) and str(r.get("scale")) == str(e.get("scale"))] if not rs and not exact_only: rs = [r for r in _FROZEN_ROWS if _match(r)] @@ -4397,6 +4556,7 @@ def _heap_not_pinned(e, lane, wl): rs = [r for r in _FROZEN_ROWS if r.get("lane") == lane and (not wl or r.get("workload") == wl) and display_name(str(r.get("backend"))) == e.get("backend") + and (not e.get("build") or _arm_build(r) == e["build"]) and str(r.get("scale")) == str(e.get("scale"))] if not rs: return False @@ -4560,8 +4720,14 @@ def _pg_memory_note(table): def _gav_pair_note(table): """Only when the table carries both arms of the ablation: the sentence must not describe a pair that is not there.""" names = {str(e["backend"]) for e in table.get("entries", [])} - pairs = [(a, a[:-1] + ", GAV)") for a in sorted(names) - if a.startswith("ArcadeDB (") and "GAV" not in a and (a[:-1] + ", GAV)") in names] + def _gav_twin(a): + # the view goes before the build: "ArcadeDB (embedded, native build)" -> "ArcadeDB (embedded, GAV, native build)" + b = _AI.build_in_label(a) + if not b: + return a[:-1] + ", GAV)" + return _strip_build(a)[:-1] + f", GAV, {_AI.BUILD_NAMES[b]})" + pairs = [(a, _gav_twin(a)) for a in sorted(names) + if a.startswith("ArcadeDB (") and "GAV" not in a and _gav_twin(a) in names] if not pairs: return None listed = "; ".join(f"{a} and {b}" for a, b in pairs) @@ -4870,6 +5036,12 @@ def _served_client_table(table): return any(_is_served_arcadedb(e.get("backend_key")) for e in table.get("entries") or []) +# What a headline served row still sends over the benchmark's own lean HTTP client (lean_http.OfficialSession: everything the driver has no +# call for), stated once so the sentence under a table cannot say the driver timed "every call" (DECISIONS #259 (4)). +_SERVED_LEAN_SCOPE = ("Bulk loads, the server probe, transactions, and streamed results go through the benchmark's own lean HTTP client in " + "the headline rows too.") + + def _arcadedb_client_note(table): """What the served ArcadeDB rows ran through: the benchmark's own minimal HTTP client, named from the rows. None for a table with no served ArcadeDB row. A row naming another client, or two clients under one table, refuses.""" @@ -5191,7 +5363,11 @@ def _same_pin(r): for r in rows: if not _same_pin(r): continue - key = (r.get("lane"), str(r.get("scale")), r.get("backend"), r.get("workload")) + # In-process ArcadeDB arms are censored per build (arm + build + size): a native out-of-memory or cap decision never takes + # the JVM entry off the page, and the other way round. The build rides on the backend element as "backend@build" and + # the consumers split it again; an old row with no build is the JVM build (arm_identity.build_of). + _b = _arm_build(r) + key = (r.get("lane"), str(r.get("scale")), f"{r.get('backend')}@{_b}" if _b else r.get("backend"), r.get("workload")) err = str(r.get("error") or "") if not err: clean.add(key) @@ -5398,6 +5574,7 @@ def _censored_entries(table): if not e.get("outcome")} out, marks = [], set() for (l, scale, backend, w), secs in sorted(_censored_cells().items(), key=str): + backend, _, build = str(backend).partition("@") if l != lane or (wl and w != wl): continue if scales and str(scale) not in scales: @@ -5414,7 +5591,8 @@ def _censored_entries(table): # "SurrealDB (embedded)" twice, one for documents with ten indexes and # one for the graph, and a reader could not tell which was which. _lc = lane == "lifecycle" - out.append({"backend": _lifecycle_row_label(backend, w) if _lc else display_name(backend), + out.append({"backend": _with_build(_lifecycle_row_label(backend, w) if _lc else display_name(backend), build), + "build": build or None, **({"engine": _lifecycle_engine(backend)} if _lc else {}), "backend_key": backend, "is_arcadedb": "arcadedb" in str(backend).lower(), @@ -5903,7 +6081,8 @@ def _censored_notes(table_id): notes = [] merged = {} scales = _table_scales(table_id) - for (l, scale, backend, w), secs in sorted(_censored_cells().items(), key=str): + for (l, scale, _qbackend, w), secs in sorted(_censored_cells().items(), key=str): + backend, _, _build = str(_qbackend).partition("@") if l != lane or (wl and w != wl): continue if scales and str(scale) not in scales: @@ -5955,7 +6134,7 @@ def _censored_notes(table_id): pins = [scale_label(lane, scale), scale_label(lane, secs.at)] else: budget = f"{secs / 3600:g} hour" if secs else "its" - _phase = _CENSORED_PHASE.get((lane, str(scale), backend, w)) + _phase = _CENSORED_PHASE.get((lane, str(scale), _qbackend, w)) _in = f" It was still in {_phase} when the budget ran out." if _phase else "" tail = (f" at {scale_label(lane, scale)}: {_cell} exceeded " f"its {budget} budget, the same budget every engine on this table had, on its first " @@ -5966,7 +6145,7 @@ def _censored_notes(table_id): f"inside its budget and was not retried, so there is no row. What it reported: " f"{secs}") pins = [scale_label(lane, scale), str(secs)] - merged.setdefault((kind, tail, tuple(pins)), []).append(display_name(backend)) + merged.setdefault((kind, tail, tuple(pins)), []).append(_with_build(display_name(backend), _build)) # ONE SENTENCE FOR THE ENGINES THAT HAVE THE SAME THING TO SAY. Three # engines were censored at TPC-H SF10 and the page printed three sentences @@ -6843,7 +7022,7 @@ def _finish_table(table: dict) -> dict: _per_situation = table.get("id") == "lifecycle" def _cell_of(m): - return (str(m.get("backend_key")), str(m.get("scale")), str(m.get("backend")) if _per_situation else "") + return (str(m.get("backend_key")), str(m.get("scale")), str(m.get("backend")) if _per_situation else "", m.get("build") or "") _cens = {_cell_of(m) for m in _marked} table["entries"] = [e for e in table["entries"] if e.get("outcome") or _cell_of(e) not in _cens] entries = table["entries"] + _marked @@ -6861,7 +7040,9 @@ def key(e): base = re.sub(r"\s*\(.*$", "", str(e["backend"])).lower() return (scale_rank(e), 0 if e.get("is_arcadedb") else 1, base, DEPLOYMENT_ORDER.get(e.get("deployment"), 2), - PRECISION_ORDER.get(e.get("precision"), 2)) + PRECISION_ORDER.get(e.get("precision"), 2), + # the native build (the default install) first, its JVM twin right after it + {"native": 0, "jvm": 1}.get(e.get("build"), 2)) table["entries"] = sorted(entries, key=key) # THE HEAP BESIDE THE PEAK (DECISIONS #116 item 7, decided: the # memory column stays the peak an operator has to provision, with the heap @@ -7141,8 +7322,16 @@ def main() -> int: print(f" cap rule: {len(_cut)} rows of {len(_over)} over-the-cap arm-cells withheld, shown censored instead") _gone = {id(r) for r in _cut} rows = [r for r in rows if id(r) not in _gone] - global _FROZEN_ROWS + # TWO ARCADEDB BUILDS, TWO ARMS (26.11.1): native and jvm rows of a release share engine_version and arm names, so the tables + # group on the build (_arm_build) and label it; no cell is a median over both. BENCH_ENGINE_BACKEND freezes one build on purpose. + rows = _AI.select_build(rows) + # ONE ENGINE COMMIT FOR BOTH BUILDS: a freeze whose native and JVM rows name two engines is refused, not printed side by side. + _bad = _AI.pair_problems(rows) + if _bad: + raise SystemExit("REFUSING: " + "; ".join(_bad)) + global _FROZEN_ROWS, _SPLIT _FROZEN_ROWS = rows + _SPLIT = _AI.splits_page(rows) # WITHHELD: ArcadeDB rows whose engine_version identifies no build. # # Not rendered, and said out loud rather than dropped quietly. A number we @@ -7211,7 +7400,7 @@ def main() -> int: # rather than a wrong number, and the l2olap table below is what would # have sprung it. grouped[(r["lane"], r["scale"], r["backend"], wl, - r.get("gav") or "")].append(r) + r.get("gav") or "", _arm_build(r))].append(r) tables = [] for lane, spec in LANES.items(): @@ -7220,7 +7409,7 @@ def main() -> int: # graph lane's rows, so the spec may name the lane it reads. src_lane = spec.get("lane_source", lane) only_scales = spec.get("only_scales") - for (ln, scale, backend, workload, gav), rs in sorted(grouped.items()): + for (ln, scale, backend, workload, gav, build), rs in sorted(grouped.items()): if ln != src_lane: continue if only_scales and scale not in only_scales: @@ -7294,8 +7483,11 @@ def main() -> int: if lane == "l2olap" and gav != "False" and "arcade" in backend: label = (f"{label[:-1]}, GAV)" if label.endswith(")") else f"{label} (GAV)") + # The build goes last, after the precision and the view, so the two builds of one arm differ only in that suffix. + label = _with_build(label, build) entry = { "backend": label, + "build": build or None, # The runner's own key for the arm, so a gate can hold the # table against runner.LANES without parsing the label # (page_check._check_lane_roster). @@ -7604,6 +7796,8 @@ def main() -> int: _t.setdefault("conditions", []) if _note not in _t["conditions"]: _t["conditions"].append(_note) + _declare_missing_builds(tables) + _disclose_server_builds(tables) # THE OVERRIDES A READER MEETS (CAMPAIGN section 7 row 21). PROTOCOL.md # section 7 lists every default this benchmark overrides, and the rows its # last column marked NOWHERE were named in no sentence under any table. One diff --git a/benchmarks/experiments/fixtures/publish_2611/build_fixture.py b/benchmarks/experiments/fixtures/publish_2611/build_fixture.py index 3587e90976d..7be6eca6a72 100644 --- a/benchmarks/experiments/fixtures/publish_2611/build_fixture.py +++ b/benchmarks/experiments/fixtures/publish_2611/build_fixture.py @@ -98,7 +98,7 @@ def _add_2611_fields(rows): r[f"{q}_plain_digest_match"] = True r["read_path"] = "arrow" if be != "sqlite" else "fetchall" if be.startswith("arcadedb") and "server" in be: - r["arcadedb_http_client"] = "arcadedb-driver 0.2.0 (sync, httpx)" + r["arcadedb_http_client"] = "arcadedb-driver 0.4.0 (sync, httpx)" r["arcadedb_http_client_role"] = "headline" out.append(r) # the lean-client row of every served ArcadeDB cell, the headline since DECISIONS #285: faster by a fixed factor, so a diff --git a/benchmarks/experiments/fixtures/publish_2611/results/runs.jsonl.gz b/benchmarks/experiments/fixtures/publish_2611/results/runs.jsonl.gz index 1fe86f5e383..dbd1c0076d4 100644 Binary files a/benchmarks/experiments/fixtures/publish_2611/results/runs.jsonl.gz and b/benchmarks/experiments/fixtures/publish_2611/results/runs.jsonl.gz differ diff --git a/benchmarks/experiments/jvm_payload_check.py b/benchmarks/experiments/jvm_payload_check.py index 3c7afd4ded5..a8ce2bb41d7 100644 --- a/benchmarks/experiments/jvm_payload_check.py +++ b/benchmarks/experiments/jvm_payload_check.py @@ -182,7 +182,7 @@ def check_file(path): # where a list legitimately becomes a Java array -- flagging it is circular, # and its one internal `vector.tolist()` feeds a normalisation loop rather # than the JVM directly. - if os.path.basename(path) == "vector.py" and "arcadedb_embedded" in path: + if os.path.basename(path) == "vector.py" and ("arcadedb_embedded" in path or os.sep + "arcadedb" + os.sep in path): return [] try: src = open(path, encoding="utf-8").read() diff --git a/benchmarks/experiments/l1_tabular.py b/benchmarks/experiments/l1_tabular.py index dd090a38bff..4eec6ba3bb6 100644 --- a/benchmarks/experiments/l1_tabular.py +++ b/benchmarks/experiments/l1_tabular.py @@ -504,7 +504,8 @@ def index_readback(self): lambda: self.db.query("sql", "SELECT FROM schema:indexes").to_list()) def connect(self): - import arcadedb_embedded as arcadedb + import _arcadedb_import + arcadedb = _arcadedb_import.load() import bench_common # this retired lane imports nothing else from it heap = os.environ.get("ARCADEDB_HEAP", "4g") # -Xms is left unset; heap_size sets -Xmx from the cell's memory budget (the bindings' documented practice, PROTOCOL section 7). @@ -538,7 +539,8 @@ def reopen(self): that reopened without it would be a different operating point wearing the same label. """ - import arcadedb_embedded as arcadedb + import _arcadedb_import + arcadedb = _arcadedb_import.load() import bench_common # this retired lane imports nothing else from it heap = os.environ.get("ARCADEDB_HEAP", "4g") self.db = arcadedb.open_database( @@ -704,6 +706,8 @@ def main(): b.connect() out["connect_s"] = round(time.perf_counter() - t0, 3) out["engine_version"] = getattr(b, "version", "?") + import bench_common as _bc + out["engine_backend"] = _bc.engine_backend() out.update(getattr(b, "row_extra", None) or {}) # F6 provenance: what the engine's pool was actually sized to, and what the # cpuset allowed. A cell where these differ is oversubscribed; recording it diff --git a/benchmarks/experiments/l1_tpc.py b/benchmarks/experiments/l1_tpc.py index 9159f7f64ed..4851322521f 100644 --- a/benchmarks/experiments/l1_tpc.py +++ b/benchmarks/experiments/l1_tpc.py @@ -1056,7 +1056,8 @@ def index_readback(self): durability = bench_common.DURABILITY_ARCADEDB def connect(self): - import arcadedb_embedded as arcadedb + import _arcadedb_import + arcadedb = _arcadedb_import.load() self._a = arcadedb heap = os.environ.get("ARCADEDB_HEAP", "4g") self.db = arcadedb.create_database("/tmp/tpc_arcade", @@ -1067,10 +1068,9 @@ def connect(self): # than a default someone remembered. jvm_kwargs={"heap_size": heap, "jvm_args": bench_common.arcade_jvm_args()}) - from importlib.metadata import version as _pv - self.version = _pv("arcadedb-embedded") + self.version = _arcadedb_import.dist_version() # ASKED, not asserted (#81's standard, doubled by #90's second class). - self.durability = bench_common.arcade_durability_readback() + self.durability = bench_common.arcade_durability_readback(db=self.db) qr = bench_common.arcade_query_ram_readback() # {} unless the query heap budget override is on (DECISIONS #175) if qr: self.row_extra = {**(getattr(self, "row_extra", None) or {}), **qr} @@ -1955,6 +1955,7 @@ def main(): with _beat.phase("connect", backend=args.backend): b.connect() out["engine_version"] = b.version + out["engine_backend"] = bench_common.engine_backend() t0 = time.perf_counter() with _beat.phase("build", n=len(li)): b.build(li, part) diff --git a/benchmarks/experiments/l2_graph.py b/benchmarks/experiments/l2_graph.py index e6c955d7cca..65c8c3dafaa 100644 --- a/benchmarks/experiments/l2_graph.py +++ b/benchmarks/experiments/l2_graph.py @@ -198,7 +198,8 @@ def index_readback(self): lambda: self.db.query("sql", "SELECT FROM schema:indexes").to_list()) def connect(self): - import arcadedb_embedded as arcadedb + import _arcadedb_import + arcadedb = _arcadedb_import.load() heap = os.environ.get("ARCADEDB_HEAP", "4g") # -Xms is left unset; heap_size sets -Xmx from the cell's memory budget (the bindings' documented practice, PROTOCOL section 7). self.db = arcadedb.create_database( @@ -207,7 +208,7 @@ def connect(self): jvm_kwargs={"heap_size": heap, "jvm_args": bench_common.arcade_jvm_args()}) self.version = arcadedb.__version__ - self.durability = bench_common.arcade_durability_readback() + self.durability = bench_common.arcade_durability_readback(db=self.db) for ddl in ["CREATE VERTEX TYPE Person", "CREATE PROPERTY Person.id LONG", "CREATE PROPERTY Person.name STRING", @@ -228,7 +229,8 @@ def reopen(self): JVM engine and on none of the others, so it has to be measured separately rather than folded in. """ - import arcadedb_embedded as arcadedb + import _arcadedb_import + arcadedb = _arcadedb_import.load() heap = os.environ.get("ARCADEDB_HEAP", "4g") self.db = arcadedb.open_database( "/tmp/l2_arcade", @@ -277,17 +279,29 @@ def build(self, n_persons): # Person and KNOWS, so no LSQB query could use it. At a tier with no message half the lists are what is loaded # (Person, KNOWS: the view the five projection questions always had); at the full network they are Person plus # ldbc_snb.MSG_VERTEX_LABELS and KNOWS plus MSG_EDGE_TYPES (Message is the abstract supertype of Post and Comment - # and holds no record of its own, so it is not listed, as upstream's runner does not list it). The properties are + # and holds no record of its own, so it is not listed before ArcadeDB 26.11.1, where upstream PR 9320 makes the registry hand a view out only + # when it covers every vertex type, abstract ones included: from there `_gav_types` adds it). The properties are # the five questions' own. An ArcadeDB-only setting, the user's call; the no-view arm (BENCH_GAV=0, backend_arm # nogav) stays on the table as the like-for-like row, and the build time of the larger view is priced on the row. GAV_PROPERTIES = "PROPERTIES (id, name, age, city) EDGE PROPERTIES (since)" def _gav_types(self): - """(vertex types, edge types) the view is created over.""" + """(vertex types, edge types) the view is created over. + + From ArcadeDB 26.11.1 (upstream PR 9320) the registry hands a view to a walk only when it covers every vertex type of the schema, the + abstract `Message` supertype included, so the list carries `Message` there and not on the releases before (bench_common. + GAV_SUPERTYPE_LISTED_FROM). The release is the engine version this adapter already records on its row (`self.version`, set at + connect); one the harness cannot read is refused rather than guessed, since the wrong list is a wrong measurement.""" if not getattr(self, "_load_messages", False): return ["Person"], ["KNOWS"] import ldbc_snb as _ldbc - return ["Person"] + list(_ldbc.MSG_VERTEX_LABELS), ["KNOWS"] + list(self.MSG_EDGE_TYPES) + release = bench_common.engine_release(self.version) + if release is None: + raise RuntimeError(f"cannot choose the view's vertex types: the engine version {self.version!r} carries no release number") + vertices = ["Person"] + list(_ldbc.MSG_VERTEX_LABELS) + if release >= bench_common.GAV_SUPERTYPE_LISTED_FROM: + vertices.insert(1, "Message") + return vertices, ["KNOWS"] + list(self.MSG_EDGE_TYPES) def _gav_ddl(self): v, e = self._gav_types() @@ -3502,6 +3516,7 @@ def gen_edges(n, *a, **kw): # noqa: F811 - deliberate shadow, counted ad.connect() out["connect_s"] = round(time.perf_counter() - t0, 3) out["engine_version"] = ad.version + out["engine_backend"] = bench_common.engine_backend() # What a served engine reported about itself at connect: client library # version, thread pool, memory limit, timeouts, persistence settings # (Memgraph, FalkorDB, DuckPGQ). Read from the server, not restated from diff --git a/benchmarks/experiments/l3_sparse.py b/benchmarks/experiments/l3_sparse.py index 2949136e46c..26b57a53adf 100644 --- a/benchmarks/experiments/l3_sparse.py +++ b/benchmarks/experiments/l3_sparse.py @@ -96,6 +96,8 @@ def recall_at_k(result_ids, gt_row): # the same ids in the same order as the whole-record form (50 of 50 queries, the pin and main, laptop). SPARSE_RESULT = "id" ARCADE_SPARSE_SEARCH_SQL = "SELECT id, score FROM (SELECT expand(`vector.sparseNeighbors`({a})))" +# The settle step, one statement for every ArcadeDB arm (embedded on either package, served): engine #5144. +COMPACT_SQL = "COMPACT INDEX `Doc[tokens,weights]`" class Base: @@ -175,7 +177,8 @@ def _index_metadata(self): return _json.dumps(meta) def connect(self): - import arcadedb_embedded as arcadedb + import _arcadedb_import + arcadedb = _arcadedb_import.load() heap = os.environ.get("ARCADEDB_HEAP", "4g") # -Xms is left unset; heap_size sets -Xmx from the cell's memory budget (the bindings' documented practice, PROTOCOL section 7). jvm_args = "" @@ -190,8 +193,7 @@ def connect(self): self.db = arcadedb.create_database( "/tmp/l3_arcade", jvm_kwargs={"heap_size": heap, "jvm_args": jvm_args or None}) - from importlib.metadata import version as _pv - self.version = _pv("arcadedb-embedded") + self.version = _arcadedb_import.dist_version() self.db.command("sql", "CREATE DOCUMENT TYPE Doc") self.db.command("sql", "CREATE PROPERTY Doc.id LONG") self.db.command("sql", "CREATE PROPERTY Doc.tokens ARRAY_OF_INTEGERS") @@ -212,7 +214,8 @@ def reopen(self): the point, since it is charged to every JVM engine and to none of the others. """ - import arcadedb_embedded as arcadedb + import _arcadedb_import + arcadedb = _arcadedb_import.load() heap = os.environ.get("ARCADEDB_HEAP", "4g") self.db = arcadedb.open_database( "/tmp/l3_arcade", @@ -220,44 +223,31 @@ def reopen(self): self.idx_name = "Doc[tokens,weights]" def build(self, n_docs): - # Native document API with primitive arrays — the embedded analog of - # Qdrant/Milvus binary clients (1.8x faster than SQL INSERT batches; - # the server adapter keeps SQL-over-HTTP, its native remote surface). - import jpype - JInt, JFloat = jpype.JArray(jpype.JInt), jpype.JArray(jpype.JFloat) - jdb = self.db.get_java_database() - jdb.begin() - open_n = 0 - for i, idx, vals in gen_docs(n_docs): - d = jdb.newDocument("Doc") - d.set("id", i) - d.set("tokens", JInt(idx)) - d.set("weights", JFloat(vals)) - d.save() - open_n += 1 - if open_n == INGEST_BATCH: - jdb.commit() - jdb.begin() - open_n = 0 - jdb.commit() + # ONE PATH FOR BOTH PACKAGES (the native `arcadedb` build has no JVM, no JPype and no get_java_database(), so the + # old jdb.newDocument + JArray ingest could not run there). The documents go through the Python API's bulk insert, + # `Database.insert_many`: one batch per commit (INGEST_BATCH), the token and weight lists as the generators yield + # them (Python lists of int and float). The declared property types turn them into ARRAY_OF_INTEGERS and + # ARRAY_OF_FLOATS (the smoke reads the stored types back); the weights are stored as float32 either way, so the + # ingested values equal the ground truth's, as in the Java-array path this replaced. + self.db.insert_many( + "Doc", + ({"id": i, "tokens": idx, "weights": vals} for i, idx, vals in gen_docs(n_docs)), + commit_every=INGEST_BATCH) def post_build(self): - # ArcadeDB's settle step (fairness parity with ES forcemerge / Milvus - # flush+load / Qdrant green-wait): blocking LSM segment compaction via - # the Java API — embedded-only, no SQL/HTTP trigger exists. Measured at - # 1M docs: ~2s, query p50 9.5ms -> 7.0ms (control run: no change). - jdb = self.db.get_java_database() - for idx in jdb.getSchema().getIndexes(): - if "SparseVector" in str(idx.getClass().getSimpleName()): - idx.compact() + # ArcadeDB's settle step (fairness parity with ES forcemerge / Milvus flush+load / Qdrant green-wait): blocking LSM + # segment compaction, the SQL statement the served arm already runs (engine #5144), so embedded, native and server + # settle the same way. Measured at 1M docs with the old Java call (idx.compact()): ~2s, query p50 9.5ms -> 7.0ms + # (control run: no change). + self.db.command("sql", COMPACT_SQL) def search(self, idx, vals, k): - import jpype - ji = jpype.JArray(jpype.JInt)(idx) - jv = jpype.JArray(jpype.JFloat)(vals) + # Python lists, as the generators yield them: both packages convert a list to the int[] / float[] the function + # reads. NOT numpy arrays: the JVM build's argument conversion turns every ndarray into float[], which the + # indices argument refuses (measured on 26.11.1.dev0: "Array must be int[], long[], Integer[]..."). rows = self.db.query( "sql", ARCADE_SPARSE_SEARCH_SQL.format(a="?, ?, ?, ?"), - self.idx_name, ji, jv, k).to_json_list() + self.idx_name, idx, vals, k).to_json_list() return [int(r["id"]) for r in rows] # ids, as every comparator returns them def resolve(self, ids): @@ -278,13 +268,11 @@ class ArcadeEmbeddedFP32(ArcadeEmbedded): class ArcadeEmbeddedNoCompact(ArcadeEmbedded): - """Settle-step ablation; de-confounds the deployment axis. + """Settle-step ablation: the embedded arm without COMPACT INDEX. - The server cannot trigger compaction (no SQL/HTTP path, engine #5144), so - "embedded (compacted) vs server (as-shipped)" mixes transport cost with - settle-step cost. Skipping compaction here gives the decomposition: - embedded vs embedded_nocompact -> settle-step cost - embedded_nocompact vs server -> pure transport cost + The served arm now runs the same COMPACT INDEX, so embedded vs server is the + transport cost and embedded vs embedded_nocompact is the settle-step cost + (time spent compacting, and the query latency it buys). """ name = "arcadedb_sparse_embedded_nocompact" @@ -322,10 +310,10 @@ def connect(self): def post_build(self): # Settle step, now client-reachable over HTTP/SQL via COMPACT INDEX - # (engine #5144, in 26.8.x). This gives the server the same settle the - # embedded adapter gets via idx.compact(), so the deployment axis - # isolates transport, not the settle asymmetry. Synchronous. - self._cmd("sql", "COMPACT INDEX `Doc[tokens,weights]`") + # (engine #5144, in 26.8.x), the same statement the embedded adapter runs, + # so the deployment axis isolates transport, not the settle asymmetry. + # Synchronous. + self._cmd("sql", COMPACT_SQL) def _cmd(self, language, command, params=None): payload = {"language": language, "command": command} diff --git a/benchmarks/experiments/l3d_dense.py b/benchmarks/experiments/l3d_dense.py index 9d163e26f7f..a939d0d4f47 100644 --- a/benchmarks/experiments/l3d_dense.py +++ b/benchmarks/experiments/l3d_dense.py @@ -584,7 +584,8 @@ def quantization(self): return canonical_quant_label(os.environ.get("BENCH_DENSE_QUANT", "")) def connect(self): - import arcadedb_embedded as arcadedb + import _arcadedb_import + arcadedb = _arcadedb_import.load() self._a = arcadedb heap = os.environ.get("ARCADEDB_HEAP", "4g") # ARCADEDB_EXTRA_JVM_ARGS: space-separated extras (e.g. an @@ -649,8 +650,7 @@ def connect(self): "/tmp/l3d_arcade", jvm_kwargs={"heap_size": heap, "jvm_args": extra.strip() or None}) - from importlib.metadata import version as _pv - self.version = _pv("arcadedb-embedded") + self.version = _arcadedb_import.dist_version() def build(self, vecs): db = self.db @@ -3261,6 +3261,7 @@ def main(): b.connect() out["connect_s"] = round(time.perf_counter() - t0, 3) out["engine_version"] = b.version + out["engine_backend"] = bench_common.engine_backend() t0 = time.perf_counter() with _beat.phase("build", n=len(train)): diff --git a/benchmarks/experiments/l4_tsbs.py b/benchmarks/experiments/l4_tsbs.py index de5fe19ceff..1955dc002f4 100644 --- a/benchmarks/experiments/l4_tsbs.py +++ b/benchmarks/experiments/l4_tsbs.py @@ -274,7 +274,8 @@ class ArcadeTS(BulkRead): name = "arcadedb_ts_doc" def connect(self): - import arcadedb_embedded as arcadedb + import _arcadedb_import + arcadedb = _arcadedb_import.load() self._arcadedb = arcadedb heap = os.environ.get("ARCADEDB_HEAP", "6g") self.db = arcadedb.create_database("/tmp/l4_arcade", diff --git a/benchmarks/experiments/l5_lifecycle.py b/benchmarks/experiments/l5_lifecycle.py index 3089cd3f01e..3477e2d9d89 100644 --- a/benchmarks/experiments/l5_lifecycle.py +++ b/benchmarks/experiments/l5_lifecycle.py @@ -36,6 +36,24 @@ rows by up to 617x, because a session that never touches the index leaves it in a state close then has to resolve. That is not a neutral protocol, it is one specific case (#5747), so all three are measured and reported separately. + +THE COLD ROW ON BOTH BUILDS (26.11.1: `arcadedb-jvm` runs on a JVM through JPype, `arcadedb` is the native image with no +JVM). The cold-start columns come from two fresh subprocesses (_cold_start_code, _cold_query_code), against a warm page cache +(cold_process_cache_state), and mean the same on both builds: + + import_ms interpreter start + `import arcadedb` + engine_load_ms the build's own start step: start_jvm() on the JVM build, the engine library load + isolate creation + (arcadedb._native.lib._load(), what the first database call triggers) on native + jvm_start_ms = engine_load_ms on the JVM build, null on native (there is no JVM; kept for the old rows) + first_open_ms the first open_database, with the engine already started + cold_process_ms the whole process through close(): what a CLI that opens and closes waits for + cold_first_query_ms the situation's first read (the `read` mode's _read) in a fresh process after the first open; + null for `empty`, which has no read + cold_query_process_ms process start through the end of that read: open, engine start and one question + engine_backend "native" or "jvm", on every ArcadeDB row (arcadedb.__backend__) + +Library load = import_ms + engine_load_ms on either build. cold_open_ms / cold_close_ms (a verified page-cache eviction, then a +cycle in the lane's own process) are measured identically on both builds and are the only cold-cache numbers. """ import argparse import json @@ -48,7 +66,8 @@ import bench_common import pagecache -DB = "/lcdb/lc" # a HOST directory bind-mounted here; see _assert_fs +# a HOST directory bind-mounted here; see _assert_fs. BENCH_LC_DB moves it for a laptop smoke (the default is the runner's mount, unchanged). +DB = os.environ.get("BENCH_LC_DB", "/lcdb/lc") DIM = 64 ITERS = int(os.environ.get("BENCH_LC_ITERS", "3")) # Which scenarios to run. At very large scales the write modes trigger a full @@ -116,7 +135,8 @@ def build(situation, n, heap): Returns the build session's close time in ms. """ - import arcadedb_embedded as arcadedb + import _arcadedb_import + arcadedb = _arcadedb_import.load() _rm(DB) db = arcadedb.create_database( DB, jvm_kwargs={"heap_size": heap, "jvm_args": None}) @@ -206,6 +226,10 @@ def build(situation, n, heap): # created the structure, and it is where a close-time rebuild doubles the # cost of building (#6489). A reopen cannot show it: by then the structure # is already on disk and clean. + # The engine's own settings, asked in SQL while the database is open (the same path on both builds; bench_common.arcade_durability_readback + # reads this snapshot when it stamps the row, long after this session closed). Not inside the timed close below. + import bench_common + bench_common.remember_engine_settings(db) _bt = time.perf_counter() db.close() return (time.perf_counter() - _bt) * 1000 @@ -474,7 +498,8 @@ def _drop(db, situation): def cycle(situation, mode, cold=False): """One open/close cycle. Returns (open_ms, close_ms).""" - import arcadedb_embedded as arcadedb + import _arcadedb_import + arcadedb = _arcadedb_import.load() if cold: # Verified eviction: evict() raises if pages survive, so a cold number # cannot be reported for a database that was still cached. @@ -583,6 +608,66 @@ def measure_stale(situation, mode): return st.median(o), st.median(c), st.median(w) +def _cold_start_code(native, db): + """The source of the cold-start subprocess: process start, import, engine load, first open, close. Prints four numbers (ms). + + JVM build (and the old arcadedb_embedded wheel), byte for byte what this lane has always timed: + import_ms interpreter start + `import @MOD@` (the JPype module included) + engine_load_ms start_jvm() alone, no database (also written as jvm_start_ms) + first_open_ms first open_database, JVM already up + cold_process_ms the whole process through close() + Native build (`arcadedb`, no JVM, no JPype). The same four steps, each where it means the same thing: + import_ms interpreter start + `import arcadedb` (a ctypes module; the engine library is not loaded yet) + engine_load_ms the engine library load + isolate creation (`arcadedb._native.lib._load()`, the step the first database + call triggers); the native counterpart of start_jvm(), so jvm_start_ms is null + first_open_ms first open_database, library already loaded (the same meaning as on the JVM) + cold_process_ms the whole process through close() (the same meaning) + "Library load" in the cold row is therefore import_ms + engine_load_ms on native and import_ms + jvm_start_ms on the JVM build. + """ + if native: + head = ("import time; _t0 = time.perf_counter()\n" + "import @MOD@ as a\n" + "_ti = time.perf_counter()\n" + "from @MOD@._native import lib as _lib; _lib._load()\n") + else: + head = ("import time; _t0 = time.perf_counter()\n" + "from @MOD@.jvm import start_jvm\n" + "import @MOD@ as a\n" + "_ti = time.perf_counter()\n" + "start_jvm()\n") + return head + ( + "_tj = time.perf_counter()\n" + "db = a.open_database(%r)\n" + "_to = time.perf_counter()\n" + "db.close()\n" + "print('%%.3f %%.3f %%.3f %%.3f' %% ((_ti-_t0)*1000, (_tj-_ti)*1000, (_to-_tj)*1000, " + "(time.perf_counter()-_t0)*1000))" % db) + + +def _cold_query_code(situation, db, here): + """The source of the cold-query subprocess: the first query of `situation` in a fresh process. Prints two numbers (ms). + + Same on both builds. A fresh process opens the built database (page cache warm, as for cold_process_ms) and runs the + situation's read exactly as the `read` mode does (`_read`: the vector search, the sparse search, the counts ...), once: + cold_first_query_ms that one read, after the first open (null for `empty`, which has no read) + cold_query_process_ms process start (after this lane's own stdlib helper imports) through the end of that read: + what a script that opens a database and asks one question waits for + The open and the engine start are inside cold_query_process_ms (they are not separate columns here; the cold-start process + reports them), and the close is not: a close after a query is the `read` mode's number. + """ + return ( + "import sys; sys.path.insert(0, %r)\n" + "import l5_lifecycle as _L\n" + "import time; _t0 = time.perf_counter()\n" + "import @MOD@ as a\n" + "db = a.open_database(%r)\n" + "_tq = time.perf_counter()\n" + "_L._read(db, %r)\n" + "_te = time.perf_counter()\n" + "db.close()\n" + "print('%%.3f %%.3f' %% ((_te-_tq)*1000, (_te-_t0)*1000))" % (here, db, situation)) + + def main(): ap = argparse.ArgumentParser() ap.add_argument("--backend", required=True) @@ -640,23 +725,17 @@ def main(): # # What a cold caller actually pays is the whole process: interpreter start, import, JPype's JVM # boot, then the open. That is what this subprocess times, from its own first line. + # The import name is resolved HERE, outside the timed window (@MOD@ below), so the two import lines + # are the same statements as before (`arcadedb_embedded` on the old wheels, `arcadedb` on the new + # ones) and import_ms still times nothing else. + import _arcadedb_import + _native = _arcadedb_import.backend() == "native" _boot = subprocess.run( - [sys.executable, "-c", - "import time; _t0 = time.perf_counter()\n" - "from arcadedb_embedded.jvm import start_jvm\n" - "import arcadedb_embedded as a\n" - "_ti = time.perf_counter()\n" - "start_jvm()\n" - "_tj = time.perf_counter()\n" - "db = a.open_database(%r)\n" - "_to = time.perf_counter()\n" - "db.close()\n" - "print('%%.3f %%.3f %%.3f %%.3f' %% ((_ti-_t0)*1000, (_tj-_ti)*1000, (_to-_tj)*1000, " - "(time.perf_counter()-_t0)*1000))" - % DB], + [sys.executable, "-c", _cold_start_code(_native, DB) + .replace("@MOD@", _arcadedb_import.module_name())], capture_output=True, text=True, timeout=900) if _boot.returncode == 0 and _boot.stdout.strip(): - import_ms, jvm_start_ms, first_open_ms, cold_proc_ms = ( + import_ms, engine_load_ms, first_open_ms, cold_proc_ms = ( float(x) for x in _boot.stdout.strip().split()[-4:]) else: # NOT -1.0. A sentinel that is a float goes into a median, a mean and a @@ -667,7 +746,25 @@ def main(): sys.stderr.write( "cold-start subprocess failed (rc=%s); cold columns will be null\n%s\n" % (_boot.returncode, (_boot.stderr or "")[-2000:])) - import_ms = jvm_start_ms = first_open_ms = cold_proc_ms = None + import_ms = engine_load_ms = first_open_ms = cold_proc_ms = None + # jvm_start_ms keeps its meaning (start_jvm() alone) and exists only where there is a JVM. The native build has none, so the + # column is null there and engine_load_ms (below) carries the build's own start step. + jvm_start_ms = None if _native else engine_load_ms + # The first query in a FRESH process (see _cold_query_code), a second subprocess so the one above keeps measuring + # open-then-close without a query, as it always did. + _q = subprocess.run( + [sys.executable, "-c", _cold_query_code(args.workload, DB, os.path.dirname(os.path.abspath(__file__))) + .replace("@MOD@", _arcadedb_import.module_name())], + capture_output=True, text=True, timeout=900) + if _q.returncode == 0 and _q.stdout.strip(): + cold_first_query_ms, cold_query_process_ms = (float(x) for x in _q.stdout.strip().split()[-2:]) + else: + sys.stderr.write( + "cold-query subprocess failed (rc=%s); its columns will be null\n%s\n" + % (_q.returncode, (_q.stderr or "")[-2000:])) + cold_first_query_ms = cold_query_process_ms = None + if args.workload == "empty": + cold_first_query_ms = None # the situation has no read to time (_read returns at once) out = bench_common.run_conditions( lane="lifecycle", backend=args.backend, workload=args.workload, @@ -699,7 +796,10 @@ def main(): # and first_open_ms is ArcadeDB's own initialisation with the JVM already up. _r3 = lambda x: None if x is None else round(x, 3) out["import_ms"] = _r3(import_ms) # interpreter + module import - out["jvm_start_ms"] = _r3(jvm_start_ms) # start_jvm() alone, no database + out["jvm_start_ms"] = _r3(jvm_start_ms) # start_jvm() alone, no database; null on the native build + out["engine_load_ms"] = _r3(engine_load_ms) # the build's own start step: start_jvm() (jvm) or library load + isolate (native) + out["cold_first_query_ms"] = _r3(cold_first_query_ms) # first query of the situation, fresh process, after the first open + out["cold_query_process_ms"] = _r3(cold_query_process_ms) # that process: start through the end of the first query out["first_open_ms"] = _r3(first_open_ms) # first open, JVM already up out["cold_process_ms"] = _r3(cold_proc_ms) # what a CLI actually waits for # WHICH CACHE STATE THESE FOUR DESCRIBE. cold_open_ms above is measured after diff --git a/benchmarks/experiments/l6_restart.py b/benchmarks/experiments/l6_restart.py index afac2c05d22..bed2022117a 100644 --- a/benchmarks/experiments/l6_restart.py +++ b/benchmarks/experiments/l6_restart.py @@ -794,6 +794,7 @@ def main(): _rx.update(_ad.readbacks()) # the dense adapters' hook, after their build timer out.update({k: v for k, v in _rx.items() if k in _ov.STAMP_FIELDS}) out["engine_version"] = eng.version + out["engine_backend"] = bench_common.engine_backend() bench_common.stamp_durability(out, eng.durability) out["instrument"] = bench_common.INSTRUMENT out["restart_ping"] = eng.ping_desc diff --git a/benchmarks/experiments/land_stage.py b/benchmarks/experiments/land_stage.py index 5f33484a7e8..4295d2d8e0e 100644 --- a/benchmarks/experiments/land_stage.py +++ b/benchmarks/experiments/land_stage.py @@ -33,6 +33,7 @@ import argparse import json import os +import shutil import subprocess import sys from datetime import datetime @@ -467,7 +468,17 @@ def main(): env["BENCH_ONLY_LANES"] = ",".join(sorted(_lanes)) step(1, f"pull {host_name('runs_page', args.pin, '.jsonl')} from {HOST}") pulled = SCRATCH / "runs_page_host.jsonl" - sh(["scp", "-q", f"{HOST}:{REMOTE}/{host_name('runs_page', args.pin, '.jsonl')}", str(pulled)]) + # BENCH_LAND_PULLED_FILE: the rows were pulled and merged outside this script (the four-arm chains write into several + # results directories, so no single scp has them all). Set and present: copy it in place of the scp. Set and absent: + # stop, never fall back to the host. Unset: the scp below, unchanged. + _pulled_file = os.environ.get("BENCH_LAND_PULLED_FILE") + if _pulled_file: + if not Path(_pulled_file).is_file(): + sys.exit(f"BENCH_LAND_PULLED_FILE is set to {_pulled_file}, which is not a file; refusing to fall back " + f"to pulling from {HOST}. Unset it to pull from the host.") + shutil.copyfile(_pulled_file, pulled) + else: + sh(["scp", "-q", f"{HOST}:{REMOTE}/{host_name('runs_page', args.pin, '.jsonl')}", str(pulled)]) for arm in args.overlay: for d in (f"dense_mp5_{args.pin}", f"dense_mp5_small_{args.pin}"): (RESULTS / d).mkdir(exist_ok=True) diff --git a/benchmarks/experiments/lean_http.py b/benchmarks/experiments/lean_http.py index 5b6d4531dc0..408fa153af8 100644 --- a/benchmarks/experiments/lean_http.py +++ b/benchmarks/experiments/lean_http.py @@ -47,26 +47,34 @@ import select import socket import ssl +import threading +import time import urllib.parse import zlib CLIENT_ENV = "BENCH_ARCADEDB_HTTP_CLIENT" LEAN = "lean" OFFICIAL = "official" -OFFICIAL_PIN = "0.2.0" # the vendor package release the default session measures; recorded per row from the installed package +PGWIRE = "pgwire" # the PostgreSQL-wire twin of the served SQL arms (DECISIONS #166, #171, #237): SQL reads over psycopg, all else over lean HTTP +PG_PORT_ENV = "BENCH_ARCADEDB_PG_PORT" +PG_DEFAULT_PORT = 5432 +PGWIRE_SCOPE = ("read and query statements only: `POST /query` SQL goes over the PostgreSQL wire; every write, DDL, transaction, bulk load, probe and " + "Cypher statement stays on lean HTTP (DECISIONS #171 (c), #162)") +OFFICIAL_PIN = "0.4.0" # the vendor package release the default session measures (DECISIONS #258, #259); recorded per row from the installed package, + # and OfficialSession refuses to start on any other version, so a stale image cannot stamp 0.4.0 or run 0.2.0 unnoticed LEAN_NAME = "lean_http (http.client, keep-alive)" def client_choice(): - """'official' (the default while the pass runs: a mid-run change would split rows) or 'lean' (the client whose rows are - published), from BENCH_ARCADEDB_HTTP_CLIENT, which the runner sets for the second job of each cell; anything else is refused - rather than guessed.""" + """'official' (the default while the pass runs: a mid-run change would split rows), 'lean' (the client whose rows are published) + or 'pgwire' (the PostgreSQL-wire twin, measured and not published), from BENCH_ARCADEDB_HTTP_CLIENT, which the runner sets for the + twin jobs of each cell; anything else is refused rather than guessed.""" v = (os.environ.get(CLIENT_ENV) or "").strip().lower() if v in ("", OFFICIAL): return OFFICIAL - if v == LEAN: - return LEAN - raise ValueError(f"{CLIENT_ENV} must be 'official' or 'lean', got {v!r}") + if v in (LEAN, PGWIRE): + return v + raise ValueError(f"{CLIENT_ENV} must be 'official', 'lean' or 'pgwire', got {v!r}") # ------------------------------------------------------------------------------------------------------ exceptions @@ -373,11 +381,19 @@ class OfficialSession: """The official `arcadedb-driver` (sync, httpx) behind the part of `requests.Session` the lanes use, so a lane changes by no line. Every `POST /api/v1/query/` and `/command/` whose body is `{language, command, params?, limit?}` goes through - `ArcadeDBServer(...).db().query(...)` or `.command(...)`, which is what the vendor documents for a Python client. Everything + `ArcadeDBServer(...).db().query(...)` or `.command(...)`, which is what the vendor documents for a Python client (0.4.0: both take + `limit`; `db()` handles are kept per (root, auth, timeout, database) so the driver's per-database URL cache, PR #168, is used). Everything else goes through a `LeanSession`: the `GET /server` probe, the bulk `/batch` load, transactions (`begin`/`commit`/`rollback` and any call carrying a session id), NDJSON streams, raw text bodies, and a body with a field the driver has no argument for. Bulk loads stay on HTTP `batch` and `command` as the protocol-audit decision says; the driver's `batch_load` is a different request shape and is not used. `official_calls` and `fallback_calls` count the two (a test reads them). + + PER-CALL TIMEOUT. The driver has no per-call timeout; its only one is the `httpx.Timeout` given to `ArcadeDBServer(timeout=...)`, and its + default is None, which is NO timeout. So the call's `timeout=` (seconds or None, as for `LeanSession`, where it is a per-operation socket + timeout) becomes `httpx.Timeout(seconds)` (connect, read, write and pool each, the same per-operation meaning) on a server object of its own: + one server, with its own connection pool, per distinct (root, auth, timeout). The lanes use a handful of values (30, 60, 300, 600), so a lane + holds a few pools, not one per call. A timeout surfaces as this module's `Timeout` and any other transport failure as `ConnectionError`, + as for `LeanSession`. A timeout that is not None or a number goes to the lean session. """ client_name = None # set per instance from the installed package @@ -389,20 +405,37 @@ def __init__(self): self.version = getattr(arcadedb_driver, "__version__", None) or _driver_version() self.client_name = f"arcadedb-driver {self.version} (sync, httpx)" self.auth = None + import httpx + self._httpx = httpx + # the pin is enforced, not just stamped: the headline is one driver release for the whole pass (DECISIONS #259 (1)) + if self.version != OFFICIAL_PIN: + raise RuntimeError(f"arcadedb-driver {self.version} is installed, the harness pins {OFFICIAL_PIN} (lean_http.OFFICIAL_PIN): the served " + f"ArcadeDB client cells run in the image built by build_images.sh target client-arcadedb, not in dbbench:client") self._servers = {} + self._dbs = {} self._fallback = LeanSession() self.official_calls = 0 self.fallback_calls = 0 - def _server(self, root, auth): - key = (root, auth) + def _server(self, root, auth, timeout=None): + key = (root, auth, timeout) srv = self._servers.get(key) if srv is None: user, password = auth - srv = self._driver.ArcadeDBServer(base_url=root, auth=self._basic_auth(user, password)) + srv = self._driver.ArcadeDBServer(base_url=root, auth=self._basic_auth(user, password), + timeout=self._httpx.Timeout(timeout)) # Timeout(None) = no timeout, as the driver's own default self._servers[key] = srv return srv + def _db(self, root, auth, timeout, name): + """The driver's database handle, built once per (root, auth, timeout, database): `ArcadeDBServer.db()` makes a new handle with a new + per-database URL cache every time, which is what the driver's fast path (arcadedb-drivers PR #168) lives in.""" + key = (root, auth, timeout, name) + db = self._dbs.get(key) + if db is None: + db = self._dbs[key] = self._server(root, auth, timeout).db(name) + return db + def get(self, url, **kw): kw.setdefault("auth", self.auth) self.fallback_calls += 1 @@ -414,17 +447,19 @@ def post(self, url, json=None, data=None, auth=None, headers=None, timeout=None, if (m and auth and isinstance(json, dict) and data is None and not headers and not kw and set(json) <= {"language", "command", "params", "limit"} and "language" in json and "command" in json - and not (m["kind"] == "command" and "limit" in json)): - srv = self._server(m["root"], tuple(auth)) - db = srv.db(m["db"]) + and (timeout is None or (isinstance(timeout, (int, float)) and not isinstance(timeout, bool)))): + timeout = None if timeout is None else float(timeout) + db = self._db(m["root"], tuple(auth), timeout, m["db"]) self.official_calls += 1 + run = db.query if m["kind"] == "query" else db.command try: - if m["kind"] == "query": - env = db.query(language=json["language"], command=json["command"], params=json.get("params"), limit=json.get("limit")) - else: - env = db.command(language=json["language"], command=json["command"], params=json.get("params")) + env = run(language=json["language"], command=json["command"], params=json.get("params"), limit=json.get("limit")) except self._driver.ArcadeDBError as e: return _OfficialResponse(e.status, {"error": e.error, "detail": e.detail, "exception": e.exception}, url, e.error or "") + except self._httpx.TimeoutException as e: + raise Timeout(f"POST {url} timed out after {timeout} s") from e + except self._httpx.TransportError as e: + raise ConnectionError(f"POST {url}: {e}") from e return _OfficialResponse(200, {"result": env.result, "limit": env.limit, "returned": env.returned, "truncated": env.truncated}, url, "OK") self.fallback_calls += 1 return self._fallback.post(url, json=json, data=data, auth=auth or self.auth, headers=headers, timeout=timeout, **kw) @@ -436,9 +471,205 @@ def close(self): except Exception: # noqa: BLE001 - closing must not fail a cell pass self._servers.clear() + self._dbs.clear() + self._fallback.close() + + +# ------------------------------------------------------------------------------------------------------- the PostgreSQL wire +# THE PG-WIRE TWIN (DECISIONS #166, #171, #237; CAMPAIGN section 7 row 6, PROTOCOL.md "ArcadeDB SERVED arms"). The same served SQL cell run again with +# its read statements on ArcadeDB's PostgreSQL-wire plugin through psycopg, as a labelled sensitivity row beside the headline (the official HTTP driver) +# and the lean HTTP twin. It flatters ArcadeDB most (the lightest client of the three, #166 addendum 2), so it is never a headline. +# SCOPE. Only `POST /query/` with language `sql` goes to the wire. Every `/command` (writes, DDL, `sqlscript` transactions), `/batch` (bulk +# loads, #171 (c)), `/server` probe, any other language (Cypher stays on HTTP, #162), a session id, a statement whose params the wire cannot carry +# (a dict) and an NDJSON stream the lane did not ask the wire for, go to a `LeanSession`; each is counted by reason in `fallback_reasons`. +# NEVER PREPARED (`prepare_threshold=None`): psycopg's default prepares a statement at its 5th execution, and the plugin's prepared path sends rows of +# a schemaless type under one row description (upstream ArcadeData/arcadedb#9009), so every execution is an unnamed Parse/Bind/Execute. +_SQL_NAMED = re.compile(r":([A-Za-z_][A-Za-z0-9_]*)") + + +def pg_translate(sql, params): + """(psycopg SQL, params) for an ArcadeDB statement with `:name` placeholders (the HTTP form every lane writes): each becomes `%(name)s`, a bound + parameter on the wire; text inside quotes and backticks is left alone; with params, a literal `%` is doubled for psycopg. Returns None when a + placeholder has no entry in `params` (the HTTP path then reports the error in the server's own words).""" + params = params or {} + out, i, n, used = [], 0, len(sql), [] + while i < n: + c = sql[i] + if c in "'\"`": + j = i + 1 + while j < n: + if sql[j] == "\\": + j += 2 + continue + if sql[j] == c: + break + j += 1 + seg = sql[i:j + 1] + out.append(seg.replace("%", "%%") if params else seg) + i = j + 1 + continue + m = _SQL_NAMED.match(sql, i) if c == ":" and not (i and (sql[i - 1].isalnum() or sql[i - 1] in "_:")) else None + if m: + if m.group(1) not in params: + return None + out.append(f"%({m.group(1)})s") + used.append(m.group(1)) + i = m.end() + continue + out.append("%%" if c == "%" and params else c) + i += 1 + return "".join(out), {k: params[k] for k in used} + + +def pg_param_ok(v): + """Can this value travel as a bound PostgreSQL parameter: None, a bool, int, float, str, or a flat list or tuple of them.""" + if v is None or isinstance(v, (bool, int, float, str)): + return True + return isinstance(v, (list, tuple)) and all(x is None or isinstance(x, (bool, int, float, str)) for x in v) + + +class PgSession: + """The part of `requests.Session` the lanes use, with a served SQL read sent over the PostgreSQL wire (see the block above); the rest goes through + a `LeanSession`. `pg_calls`, `fallback_calls` and `fallback_reasons` count the two; `summary()` is what the cell's log carries.""" + + client_name = None + + def __init__(self): + import psycopg + from psycopg.rows import dict_row + self._psycopg, self._dict_row = psycopg, dict_row + self.version = getattr(psycopg, "__version__", "unknown") + impl = getattr(getattr(psycopg, "pq", None), "__impl__", "unknown") + self.client_name = f"psycopg {self.version} (PostgreSQL wire, {impl}, unnamed statements)" + self.auth = None + self._conns = {} + self._fallback = LeanSession() + self.pg_calls = 0 + self.fallback_calls = 0 + self.fallback_reasons = {} + + def _port(self): + return int(os.environ.get(PG_PORT_ENV) or PG_DEFAULT_PORT) + + def _conn(self, host, db, auth): + key = (host, db, tuple(auth)) + cx = self._conns.get(key) + if cx is None or cx.closed: + cx = self._psycopg.connect(host=host, port=self._port(), dbname=db, user=auth[0], password=auth[1], autocommit=True, + prepare_threshold=None, row_factory=self._dict_row, connect_timeout=30) + self._conns[key] = cx + if not getattr(self, "_beat", None): + # a lane may exit without closing (os._exit), so the counters reach the cell's log on a clock too + self._beat = threading.Thread(target=self._heartbeat, daemon=True) + self._beat.start() + return cx + + def _heartbeat(self, every=10): + last = None + while True: + time.sleep(every) + cur = self.summary() + if cur != last: + print("PGWIRE_SUMMARY " + _json.dumps(cur, sort_keys=True), flush=True) + last = cur + + def _fall(self, reason): + self.fallback_calls += 1 + if reason not in self.fallback_reasons: + print(f"PGWIRE_FALLBACK {reason}", flush=True) # the first sight of each reason, so the cell's log names it even if the lane exits hard + self.fallback_reasons[reason] = self.fallback_reasons.get(reason, 0) + 1 + + def summary(self): + return {"pg_calls": self.pg_calls, "fallback_calls": self.fallback_calls, "fallback_reasons": dict(sorted(self.fallback_reasons.items()))} + + def _route(self, url, json, data, headers, kw): + """(match, reason): the data-plane match when this call goes to the wire, else None and why not.""" + m = _DATA_URL.match(url) + if not m: + return None, "other-endpoint" + if m["kind"] != "query": + return None, "command" + if not (isinstance(json, dict) and data is None and not kw and not headers): + return None, "headers-or-raw-body" + if not (set(json) <= {"language", "command", "params", "limit"} and "language" in json and "command" in json): + return None, "body-fields" + if str(json["language"]).lower() != "sql": + return None, f"language-{str(json['language']).lower()}" + if not all(pg_param_ok(v) for v in (json.get("params") or {}).values()): + return None, "param-type" + return m, None + + def _run(self, m, json, auth): + """Run one routed statement on the wire; returns (rows, truncated). The rows are dicts, the shape the HTTP answers have.""" + t = pg_translate(json["command"], json.get("params")) + if t is None: + raise _Unroutable("param-missing") + sql, args = t + host = m["root"].split("://", 1)[1].rsplit(":", 1)[0] + cur = self._conn(host, m["db"], auth).execute(sql, args or None) + rows = cur.fetchall() if cur.description is not None else [] + limit = json.get("limit") + truncated = False + if limit and limit > 0 and len(rows) > limit: + rows, truncated = rows[:limit], True + return rows, truncated + + def get(self, url, **kw): + kw.setdefault("auth", self.auth) + self._fall("get") + return self._fallback.get(url, **kw) + + def post(self, url, json=None, data=None, auth=None, headers=None, timeout=None, **kw): + auth = auth or self.auth + m, why = self._route(url, json, data, headers, kw) if auth else (None, "no-auth") + if m is not None: + try: + rows, truncated = self._run(m, json, tuple(auth)) + except _Unroutable as e: + why = str(e) + except self._psycopg.Error as e: + self.pg_calls += 1 + return _OfficialResponse(500, {"error": str(e).splitlines()[0] if str(e) else type(e).__name__, "detail": str(e), + "exception": type(e).__name__}, url, type(e).__name__) + else: + self.pg_calls += 1 + if self.pg_calls == 1: + print("PGWIRE_SUMMARY " + _json.dumps(self.summary(), sort_keys=True), flush=True) # the first call proves the wire in the log + return _OfficialResponse(200, {"result": rows, "limit": json.get("limit", -1), "returned": len(rows), "truncated": truncated}, url, "OK") + self._fall(why or "other") + return self._fallback.post(url, json=json, data=data, auth=auth or self.auth, headers=headers, timeout=timeout, **kw) + + def ndjson_rows(self, url, json): + """The rows of a query a lane asked to stream as NDJSON, read over the wire instead (the wire has no NDJSON): a list, or None when this + statement does not go to the wire and the caller streams it over HTTP.""" + auth = self.auth + m, why = self._route(url, json, None, None, {}) if auth else (None, "no-auth") + if m is None: + return None + try: + rows, _ = self._run(m, json, tuple(auth)) + except _Unroutable: + return None + except self._psycopg.Error as e: + raise HTTPError(f"500 Server Error: {type(e).__name__} for url: {url}") from e + self.pg_calls += 1 + return rows + + def close(self): + print("PGWIRE_SUMMARY " + _json.dumps(self.summary(), sort_keys=True), flush=True) + for cx in self._conns.values(): + try: + cx.close() + except Exception: # noqa: BLE001 - closing must not fail a cell + pass + self._conns.clear() self._fallback.close() +class _Unroutable(Exception): + """A statement the wire cannot take after all (a placeholder with no value): it goes over HTTP.""" + + def _driver_version(): try: from importlib.metadata import version @@ -449,8 +680,9 @@ def _driver_version(): # ------------------------------------------------------------------------------------------------------ the factory def Session(): # noqa: N802 - requests' own spelling, so a lane changes by one import - """The ArcadeDB served arms' session: the default session, or `LeanSession` (the published client) when BENCH_ARCADEDB_HTTP_CLIENT=lean.""" - return OfficialSession() if client_choice() == OFFICIAL else LeanSession() + """The ArcadeDB served arms' session: the default session, `LeanSession` (the published client) when BENCH_ARCADEDB_HTTP_CLIENT=lean, `PgSession` when it is pgwire.""" + choice = client_choice() + return OfficialSession() if choice == OFFICIAL else PgSession() if choice == PGWIRE else LeanSession() def row_fields(session): @@ -460,6 +692,13 @@ def row_fields(session): out["arcadedb_http_client_role"] = "headline" elif os.environ.get("BENCH_ARCADEDB_CLIENT_ROLE"): out["arcadedb_http_client_role"] = os.environ["BENCH_ARCADEDB_CLIENT_ROLE"] + if isinstance(session, PgSession): + # the role stays `sensitivity` (every filter that keeps twins out of a headline keys on it); the wire is the second stamp that tells the + # two twins apart (make_paper_tables' key, export_web's split) + out["arcadedb_client_wire"] = PGWIRE + out["arcadedb_pgwire_client"] = {"psycopg": session.version, "prepare_threshold": None, "autocommit": True, "result_format": "text", + "port": session._port()} + out["arcadedb_pgwire_scope"] = PGWIRE_SCOPE return out @@ -502,7 +741,11 @@ def ndjson_records(body): def post_ndjson(session, url, json, timeout=None): - """POST a query asking for NDJSON, raise on an HTTP error, and return its rows (`ndjson_records`).""" + """POST a query asking for NDJSON, raise on an HTTP error, and return its rows (`ndjson_records`). On the PG-wire twin a SQL query is read over the wire.""" + if isinstance(session, PgSession): + rows = session.ndjson_rows(url, json) + if rows is not None: + return rows r = session.post(url, json=json, headers={"Accept": NDJSON_ACCEPT}, timeout=timeout) r.raise_for_status() return ndjson_records(r.content) diff --git a/benchmarks/experiments/make_2611_stages.py b/benchmarks/experiments/make_2611_stages.py index a2b9af20546..4931091a1d0 100644 --- a/benchmarks/experiments/make_2611_stages.py +++ b/benchmarks/experiments/make_2611_stages.py @@ -21,6 +21,10 @@ only if time is left. PROFILES lists every drop and its reason, `--check` prints what was dropped and proves that nothing of it is scheduled (profile_problems; `--check --drafts` the same on written scripts), and `--profile full` is the whole subset as it was before the profile. + ONE CHAIN PER WHEEL. A release measured on both ArcadeDB builds is two generator runs, one per wheel (arcadedb- native, arcadedb_jvm- JVM), + each with its own --id-prefix and draft folder; the stage scripts derive the image tag suffix (-native / -jvm) from the wheel they are pinned to, + and a freeze names one build (BENCH_ENGINE_BACKEND=native|jvm, arm_identity.py). + The stage list is DERIVED from STAGES and runner.LANES, filtered by engine and lane, and `--check` proves the filtered list runs exactly the cells the unfiltered one would for those engines and lanes. Every registered arm is either in the subset or named in OUTSIDE, so a newly registered arm forces a decision here. @@ -66,7 +70,7 @@ ARCADEDB_WHEEL=... ARCADEDB_SERVER_IMAGE=... ARCADEDB_ENGINE_COMMIT=<40 hex> \\ python3 make_2611_stages.py --after qR15 --status-file STATUS.txt --id-prefix qS --out DIR [--engines a,b] [--lanes graph,dense] [--profile paper-2611|full] [--second-list (full only)] python3 make_2611_stages.py --arcadedb-release 26.11.1 --engine-commit <40 hex> --image-digest sha256:<64 hex> \\ - --wheel-name arcadedb_embedded-26.11.1-.whl --wheel-sha256 <64 hex> --wheel-url \\ + --wheel-name arcadedb_jvm-26.11.1-.whl --wheel-sha256 <64 hex> --wheel-url \\ --after --status-file STATUS.txt --id-prefix qT --out DIR python3 make_2611_stages.py --check --drafts DIR [--status-file STATUS.txt] # also: the drafts' first wait is not already in STATUS.txt """ @@ -82,6 +86,7 @@ import re import sys +import arm_identity as _AI import bench_common import cap_rule import runner @@ -120,6 +125,26 @@ # The sparse overlay names its files per scale; the dense one uses the runner's default mp_{label}_b{rep}.json, which is what # the dense resolver looks for. OVERLAY_ENV = {"l3s": 'BENCH_DRIVER_OUT_FMT=sp_{label}_{scale}.json'} +# THE NATIVE CHAIN'S OVERLAY FILES carry the build in their name (and every pass carries engine_backend), so the two builds of one pin +# share one directory without one chain overwriting the other: mp_fp32_native_b1.json beside mp_fp32_b1.json (arm_identity.py, export_web). +OVERLAY_ENV_NATIVE = {"l3s": 'BENCH_DRIVER_OUT_FMT=sp_{label}_native_{scale}.json', + "l3d": 'BENCH_DRIVER_OUT_FMT=mp_{label}_native_b{rep}.json'} +CHAIN_BUILD = "" # "native" while generating the native chain (set by main from the pinned wheel), else "" + + +def overlay_env(lane): + return (OVERLAY_ENV_NATIVE if CHAIN_BUILD == "native" else OVERLAY_ENV).get(lane, "") + + +def served_arcadedb_arms(): + """The served ArcadeDB arms (a client_server topology). Both chains run them (#222): the JVM chain on the Docker JVM server image, the + native chain on upstream's native server image, each arm once per chain.""" + return sorted(b for b in runner.BACKENDS if engine_of(b) == "arcadedb" and runner.BACKENDS[b].get("topology") == "client_server") + + +def build_less_served_arms(): + """The served ArcadeDB arms with no native counterpart (the JVM image-defaults sensitivity arm): the JVM chain runs them, the native one does not.""" + return [b for b in served_arcadedb_arms() if not _AI.served({"backend": b})] # THE OVERLAY LANES. The dense and sparse TABLES do not read the lane's own search rows: they read a multipass overlay # produced by a bespoke driver run through `runner.py --driver`, which builds once and then times five passes, so cold and @@ -515,8 +540,10 @@ def profile_problems(profile, stages, label=""): out.append(f"{tag}{st[0]}: {b} is a Neo4j/Memgraph dense cell, dropped by the {profile} profile") if DROP_QUANTIZED_DENSE in drops: dense = {b for st in stages if st[2] == "l3d" for b in stage_backends(st) if engine_of(b) == "arcadedb"} - if dense and dense != set(ARCADEDB_FP32_DENSE): - out.append(f"{tag}ArcadeDB's dense arms are {sorted(dense)}, expected exactly its fp32 pair {sorted(ARCADEDB_FP32_DENSE)}") + # the native chain runs the embedded half of the pair only: the served arm is the JVM chain's (served_arcadedb_arms) + want = set(ARCADEDB_FP32_DENSE) - (set(build_less_served_arms()) if CHAIN_BUILD == "native" else set()) + if dense and dense != want: + out.append(f"{tag}ArcadeDB's dense arms are {sorted(dense)}, expected exactly its fp32 {'arm' if len(want) == 1 else 'pair'} {sorted(want)}") for why in drops: if why == DROP_SECOND_LIST: continue @@ -547,7 +574,7 @@ def profile_report(profile): # (BIGGEST: Memgraph's deep10m dense cells, SurrealDB's documents at tpch1, the full SF1 network's analytics) run last of # all, in stages of their own, where they are easiest to drop if time runs short. Splitting changes the stage boundaries # only: stage_problems proves the cells are the same. -LARGE_SCALES = ("deep10m", "tpch10", "sf1full") +LARGE_SCALES = runner.LARGE_SCALES # one definition: runner.build_jobs reads it too (the PostgreSQL-wire twin runs at the small tier only) BIGGEST = (("l3d", ("memgraph_dense",), ("deep10m",)), ("l1tpc", ("surrealdb_tpc",), ("tpch1",)), ("l2", None, ("sf1full",))) @@ -796,6 +823,51 @@ def instrument_problems(env=None): _BACKENDS = re.compile(r'^BACKENDS="([^"]*)"', re.M) +_NATIVE_WHEEL = re.compile(r"dist/arcadedb-\d") + + +def served_arcadedb_in(text): + """The served ArcadeDB arms a stage script runs.""" + m = _BACKENDS.search(text) + served = set(served_arcadedb_arms()) + return sorted(b for b in (m.group(1).split() if m else []) if b in served) + + +def _chain_of(text): + """"native" for a stage script of a native-wheel chain, else "jvm".""" + return "native" if _NATIVE_WHEEL.search(text) else "jvm" + + +def chain_served_problems(*dirs): + """Each chain runs exactly the served ArcadeDB arms of its own build, each once (#222 (5)). Given the draft folders of the chains of one + release (native and JVM): one folder per build, and every served arm runs in the same lanes in both. A folder that runs the other + build's server (a script exporting the other build) fails in lint_draft.""" + out, per = [], {} + for d in dirs: + counts = collections.Counter() + builds = set() + for f in sorted(glob.glob(os.path.join(d, "*.sh"))): + text = open(f, encoding="utf-8").read() + if _BACKENDS.search(text): + builds.add(_chain_of(text)) + lane = re.search(r"--lanes (\w+) ", text) + for b in served_arcadedb_in(text): + if b not in build_less_served_arms(): # the JVM-only arms (image defaults, e4) belong to the JVM chain alone + counts[(b, lane.group(1) if lane else "?")] += 1 + if len(builds) > 1: + out.append(f"{d} mixes native and JVM stage scripts: one chain, one build") + per[d] = (next(iter(builds), ""), counts) + if len({b for b, _c in per.values()}) < len(per): + out.append("two chains of the same build were given: one release is a native chain and a JVM chain") + base = next(iter(per.values()), ("", {}))[1] + for d, (_b, counts) in per.items(): + for key in sorted(set(counts) | set(base)): + if counts.get(key, 0) != base.get(key, 0): + out.append(f"served ArcadeDB arm {key[0]} (lane {key[1]}) runs {counts.get(key, 0)}x in {d} and {base.get(key, 0)}x in the other " + f"chain; each chain runs it once") + return out + + def lint_draft(text, name, engines=None, profile=None): """Problems in one emitted stage script: it must export the instrument, name no other, write the 2611 rows file and run only subset arms (of `engines` when given), and, under a `profile`, no lane and no dense arm it drops. The pair-install @@ -820,6 +892,19 @@ def lint_draft(text, name, engines=None, profile=None): out.append(f"{name}: no BACKENDS line") else: arcade = has_arcadedb_arm(m.group(1).split()) + # THE SERVED ARM IS THE SERVER OF THE CHAIN'S OWN BUILD (#222 (5)): a native-wheel chain runs the native server image, a JVM chain the + # Docker JVM image; the runner refuses a BENCH_SERVER_BUILD that is not the wheel's build, and this lint refuses a script that names the other. + if has_arcadedb_arm(m.group(1).split()): + native_chain = _chain_of(text) == "native" + want_build = f'BENCH_SERVER_BUILD="{"native" if native_chain else "jvm"}"' + other = "jvm" if native_chain else "native" + if f'BENCH_SERVER_BUILD="{other}"' in text or (not native_chain and "ARCADEDB_NATIVE_SERVER_IMAGE" in text): + out.append(f"{name}: a {'native' if native_chain else 'JVM'}-build chain runs the {other} build's server arm") + if native_chain and served_arcadedb_in(text): + for needle, what in ((want_build, "BENCH_SERVER_BUILD for its own build"), ("export ARCADEDB_NATIVE_SERVER_IMAGE=", "the native server image pin"), + ("ABORT: native server image", "the native server's engine-commit check")): + if needle not in text: + out.append(f"{name}: a native-build chain runs served ArcadeDB arms but has no {what}") # every stage builds images with the pinned wheel baked in, so every stage guards the wheel's sha256 if "is not the wheel this chain was generated for" not in text: out.append(f"{name}: no guard for the pinned wheel's sha256, which the image build bakes into every image") @@ -1110,6 +1195,11 @@ def seed(m): ARCADE_PRE_TAIL = '''docker image inspect "$ARCADEDB_SERVER_IMAGE" >/dev/null 2>&1 || {{ say "$ID ABORT: no server image $ARCADEDB_SERVER_IMAGE"; exit 1; }} WV=$(basename "$W" | cut -d- -f2) +# THE BUILD THE WHEEL IS, and so the image tag's suffix (engine_pin.backend_from_wheel): arcadedb--*.whl is the native build, +# arcadedb_jvm--*.whl the JVM one, and the image is dbbench:arcadedb--native / -jvm. One chain is generated per wheel, so a release +# measured on both builds is two chains (two generator runs, two --id-prefix values, two draft folders) and never one stage with two engines. +WB="" +case "$(basename "$W" | cut -d- -f1)" in arcadedb) WB=-native ;; arcadedb_jvm) WB=-jvm ;; esac ''' ARCADE_PRE_BUILD = ARCADE_WHEEL_HEAD + ARCADE_WHEEL_COUNT + ARCADE_WHEEL_SHA_EXPORT + ARCADE_PRE_TAIL @@ -1119,12 +1209,19 @@ def seed(m): # The pair check below is the REPO copy, not ~/verify_pair_c25.sh: that # one is a stale copy from 2026-08-30 and would miss its own fourth # check. A script pulled with the tree cannot drift from it. -IV=$(docker run --rm --entrypoint python3 "dbbench:arcadedb-$WV" -c 'import importlib.metadata as m; print(m.version("arcadedb-embedded"))' 2>/dev/null | tr -dc "0-9a-zA-Z.-") +IV=$(docker run --rm --entrypoint python3 "dbbench:arcadedb-$WV$WB" -c 'import importlib.metadata as m +for n in ("arcadedb", "arcadedb-jvm", "arcadedb-embedded"): + try: print(m.version(n)); break + except m.PackageNotFoundError: pass' 2>/dev/null | tr -dc "0-9a-zA-Z.-") [ "$IV" = "$WV" ] || {{ say "$ID ABORT: dbbench:arcadedb carries wheel $IV, dist has $WV"; exit 1; }} say "$ID: dbbench:arcadedb carries wheel $IV, matching dist" -IJ=$(docker run --rm --entrypoint python3 "dbbench:arcadedb-$WV" -c 'import jpype; print(jpype.__version__)' 2>/dev/null | tr -dc "0-9a-zA-Z.-") -[ "$IJ" = "{jpype_version}" ] || {{ say "$ID ABORT: dbbench:arcadedb runs JPype '$IJ', the pin is {jpype_version} (Dockerfile.bench, ARG JPYPE_VERSION); rebuild dbbench:arcadedb, which installs the pin"; exit 1; }} -say "$ID: dbbench:arcadedb runs JPype $IJ, the pin" +if [ "$WB" = "-native" ]; then + say "$ID: dbbench:arcadedb$WB is the native build: no JVM, no JPype to pin" +else + IJ=$(docker run --rm --entrypoint python3 "dbbench:arcadedb-$WV$WB" -c 'import jpype; print(jpype.__version__)' 2>/dev/null | tr -dc "0-9a-zA-Z.-") + [ "$IJ" = "{jpype_version}" ] || {{ say "$ID ABORT: dbbench:arcadedb runs JPype '$IJ', the pin is {jpype_version} (Dockerfile.bench, ARG JPYPE_VERSION); rebuild dbbench:arcadedb, which installs the pin"; exit 1; }} + say "$ID: dbbench:arcadedb$WB runs JPype $IJ, the pin" +fi PAIR_IMAGE="$ARCADEDB_SERVER_IMAGE" ./verify_pair_c25.sh "$SHA" >> "$S" 2>&1 || {{ say "$ID ABORT: pair unverified at the pin"; exit 1; }} ''' @@ -1315,7 +1412,7 @@ def has_arcadedb_arm(backends): # server_not_ready. Check here, where the message can name the cause. for _img in {images}; do # the arcadedb image is tagged with the pinned version, never floating (engine_pin.py) - _tag="dbbench:$_img"; [ "$_img" = "arcadedb" ] && _tag="dbbench:arcadedb-$(basename "$ARCADEDB_WHEEL" | cut -d- -f2)" + _tag="dbbench:$_img"; [ "$_img" = "arcadedb" ] && _tag="dbbench:arcadedb-$(basename "$ARCADEDB_WHEEL" | cut -d- -f2)$(case "$(basename "$ARCADEDB_WHEEL" | cut -d- -f1)" in arcadedb) echo -native ;; arcadedb_jvm) echo -jvm ;; esac)" docker image inspect "$_tag" >/dev/null 2>&1 || {{ say "$ID ABORT: $_tag missing after build"; exit 1; }} done say "$ID: images present: {images}" @@ -1410,7 +1507,7 @@ def has_arcadedb_arm(backends): # line per finished repetition (