diff --git a/benchmarks/experiments/CAMPAIGN.md b/benchmarks/experiments/CAMPAIGN.md index 155bab603d1..a154aa403fe 100644 --- a/benchmarks/experiments/CAMPAIGN.md +++ b/benchmarks/experiments/CAMPAIGN.md @@ -230,6 +230,7 @@ Rules every campaign keeps: | 90 | The ArcadeDB version a cell runs is the stage's pin, read out of what will run before the first cell, and no ArcadeDB image tag floats (humemai/arcadedb-embedded-python#470). `runner.py` takes the pin from the `ARCADEDB_WHEEL` file name and, for every ArcadeDB arm of the batch, reads the version out of the image that will run (embedded: the installed `arcadedb-embedded` version and the engine jar the wheel ships; served: the engine jar of the server image) and refuses to start on any difference, or when the batch has an ArcadeDB arm and no pin; the readback is in the manifest (`engine_pin`). The embedded image is `dbbench:arcadedb-`, written by `build_images.sh`, and the runner refuses every other ArcadeDB tag (`dbbench:arcadedb` stays only as the logical build target name in `runner.BACKENDS`); the comparator images keep `dbbench:`, their engines being pinned by `==` or digest and read back by `version_pin_check.py --runtime` | the embedded and served ArcadeDB arms of every lane | built, verified by unit tests | `engine_pin.py`, `test_engine_pin.py`; the trigger: the laptop's `dbbench:arcadedb` still held 26.10.1 beside `dbbench:arcadedb-26102` at 26.10.2 (2026-10-22, `.notes/bench/repros/ts-doc-oom-20261022/`), and every earlier check read rows after the run | | 91 | The composed embedded stack joins the cross-model lane: SQLite (documents) + LanceDB (vectors, fp32 `IVF_HNSW_FLAT`) + LadybugDB (graph) in one Python process, the default trio of the Cognee-style agent-memory builders and the real alternative to ArcadeDB for that audience (the owner, 2026-10-25) | `composed_embedded_sqlite_lance_ladybug`, the `e2` lane (hybrid in both durability classes, and the atomicity workload) | **built and dry-run on the laptop (2026-10-25), untimed**: `ladybug==0.21.2` added to the dense image's package line (an existing pin, no new version; 0.22.0 since 2026-10-25, row 92), the arm in `runner.BACKENDS` and `LANES["e2"]`, an `INDEX_DECISIONS` entry, the generator's `composed-embedded` engine (Tier 1, after `qdrant-neo4j`) and the regenerated `stage_drafts_2611.sha256`. Both ends of the lane's checks pass on its rows: the retrieval, hop and filtered-candidate digests equal LadybugDB's and DuckDB's at e2, 40 of 40 atomicity trials torn (nothing spans the stores), F10 and F14 clean | `COMPARATORS.md` "The composed embedded stack"; `repros/composed-embedded-lance/probe.py` | | 92 | LadybugDB moves to 0.22.0 on the client and dense images, and its `vector` extension is baked into the client image at build time, pinned by sha256 | every LadybugDB arm: `ladybug_graph`, `ladybug_dense`, `ladybug_e2`, `composed_embedded_sqlite_lance_ladybug`, `ladybug_lifecycle` | **built (2026-10-25)**: `build_images.sh` (`ladybug==0.22.0`, `LADYBUG_EXTENSIONS`), `Dockerfile.bench`, `bake_ladybug_extensions.py`, `test_ladybug_pin.py`. 0.21.2 can no longer `LOAD vector` (the vendor overwrote the v0.21.0 download at 04:28 UTC on 2026-10-25, LadybugDB/ladybug#1390). Untimed laptop cells on 0.22.0: dense micro with the mutation pass (0.20.4 control dies at the re-insert, 0.22.0 completes), graph micro, e2 (`ladybug_e2` and the composed arm). Open: the fdatasync counts in `bench_common` were measured at 0.21.2 and are not re-measured on 0.22.0; the DuckDB extensions are not baked yet (COMPARATORS.md, "Runtime extensions") | PR on humemai/arcadedb-embedded-python; `.notes` repros/ladybug-vector-ext-20261025 | +| 93 | The ArcadeDB Python bindings get a wheels-only `26.11.1.post1`: the engine commit, the engine jars and the server image are identical to 26.11.1 (checked by comparing the sha256 of every engine jar of the two wheels), and only the Python and native glue change | the embedded ArcadeDB cells whose load or read goes through a changed glue path (candidates: the l1tpc LineItem load, the Part and graph vertex loads, the l3d vector load, the native one-row and read paths); never the served arms, which do not use the wheels | **pending**: which cells are rerun is decided by the untimed profile studies and a laptop A/B, not by judgment. A rerun cell is NEW ROWS at their own pin, with all repetitions from one wheel version; an ingest-only mode (the cell stops after `build()` and records the load figures and the load-phase memory) is planned for glue changes that do not touch a query path, and the freeze then takes the ingest figures of such a cell from the post1 rows and its other figures from the released rows, with the wheel version stated per cell; the page and the paper say which wheel each cell used. Nothing is rerun or re-pinned while the live passes run. | Issue humemai/arcadedb-embedded-python#581; ArcadeData/arcadedb-python #312 to #319 and #322 to #325. | **What the 2026-11 instrument measures (DECISIONS #196).** The harness measures one instrument and every method choice named here is the behaviour, not a switch. (1) The sparse lane warms up every engine on queries apart from the timed set, and the runner holds the CPU idle-exit latency at 10 microseconds for each timed cell (row 81); every timed cell of the pass also starts behind a pre-cell temperature gate (PROTOCOL.md section 2). (2) The dense lane indexes the vector id for the delete on every engine that would scan (row 84). (3) The served ArcadeDB headline is the official `arcadedb-driver`, with the lean HTTP client as the twin sensitivity row (rows 72, 82). (4) The l4 queries with large answers read each engine on its documented bulk path: for `q_high` and `q_groupby` ArcadeDB embedded `to_arrow()`, DuckDB `to_arrow_table()`, SQLite `fetchall()` (no columnar path), with `to_list()` / `fetchall()` timed beside it as `q_high_plain_ms` and `q_groupby_plain_ms`; rows stamp `read_path`, `read_path_queries`, `read_path_plain`, and `read_path_sensitivity`; small results do not move; a missing pyarrow or bridge jar falls back to `to_list()` and the row says `read_path_arrow_missing`; the typed read's digest must equal the plain read's after the lane's declared coerce, and a mismatch ends the cell. (5) The ArcadeDB arms, embedded and served, leave `-Xms` unset (the bindings' documented practice and the maintainers' guidance for a fixed memory budget; DECISIONS #200 for the served arms) and keep `-Xmx` from the cell's memory budget; Elasticsearch and Neo4j keep the initial heap they have (PROTOCOL.md section 7, "JVM initial heap"). `BENCH_ARCADEDB_TS_NDJSON` stays off (row 85, measured mixed). Every row stamps `instrument` = `2026-11`; a `BENCH_INSTRUMENT` of anything else is refused, the runner forwards its own value to every cell so a container cannot disagree with the launcher, and `test_instrument_method.py` holds the list. The 2026-10 and September instruments are retired: no cell is measured with them again.