From 2957ef7f20fa11960ddf7ea6757e110b8ced5e32 Mon Sep 17 00:00:00 2001 From: Taewoon Kim Date: Fri, 6 Nov 2026 15:03:26 +0900 Subject: [PATCH] docs: state after the 26.11.1 release, the n=3/4/5 pass order, and the doubled io figures (#555) Sweep of the public docs for statements that went stale: - README, bindings/python/README.md, bindings/python/docs/index.md: arcadedb and arcadedb-jvm 26.11.1 are final releases on PyPI (not pre-releases), so `uv add arcadedb`, no `--pre`; the arcadedb-embedded shim's last release, 26.11.1, exists; archive timing named (after the benchmark move to dbbench). - benchmarks README, PAGE-SPEC, PROTOCOL: the repetition order is n=3 for every engine first (the four ArcadeDB arms included), then n=4, then n=5. - CAMPAIGN: the 2026-11-05 order swap (four-arm pass before the window re-runs when the wheels and runner are ready). - PROTOCOL, CAMPAIGN: mini rows before the #555 fix count the stacked NVMe and device-mapper devices twice; io_stat_dedup; the freeze halves them in runs_paper.csv only, with generated/io_stat_correction.json. Co-Authored-By: Claude Sonnet 5.5 --- README.md | 8 ++++---- benchmarks/README.md | 2 +- benchmarks/experiments/CAMPAIGN.md | 4 ++-- benchmarks/experiments/PAGE-SPEC.md | 2 +- benchmarks/experiments/PROTOCOL.md | 4 ++-- bindings/python/README.md | 4 ++-- bindings/python/docs/index.md | 2 +- 7 files changed, 13 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index db2d8a1dcfa..22fb0a9b1e2 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,8 @@ # ArcadeDB Embedded for Python -> **Moved: the Python bindings now live in [ArcadeData/arcadedb-python](https://github.com/ArcadeData/arcadedb-python).** Install `arcadedb` (the native build, the default) or `arcadedb-jvm` (the JVM build); both use `import arcadedb`. [Which package?](https://github.com/ArcadeData/arcadedb-python/blob/main/docs/modules/ROOT/pages/which-package.adoc) explains the choice. The first releases of the new packages are 26.11.1, starting as pre-releases (`uv add --pre arcadedb`). `arcadedb-embedded` and `import arcadedb_embedded` are deprecated: `arcadedb-embedded` will get a last release that installs `arcadedb`. Report bindings issues and send bindings pull requests to the new repository. This repository will be archived; the benchmark tooling stays here for now. +> **Moved: the Python bindings now live in [ArcadeData/arcadedb-python](https://github.com/ArcadeData/arcadedb-python).** Install `arcadedb` (the native build, the default) or `arcadedb-jvm` (the JVM build); both use `import arcadedb`. [Which package?](https://github.com/ArcadeData/arcadedb-python/blob/main/docs/modules/ROOT/pages/which-package.adoc) explains the choice. Both packages are released as 26.11.1 (`uv add arcadedb`). `arcadedb-embedded` and `import arcadedb_embedded` are deprecated: the last release of `arcadedb-embedded`, 26.11.1, installs `arcadedb`. Report bindings issues and send bindings pull requests to the new repository. This repository will be archived once the benchmark tooling moves to [humemai/dbbench](https://github.com/humemai/dbbench) (planned for mid-November 2026, after the last measuring pass); until then the tooling stays here. -[![PyPI](https://img.shields.io/pypi/v/arcadedb?include_prereleases)](https://pypi.org/project/arcadedb/) +[![PyPI](https://img.shields.io/pypi/v/arcadedb)](https://pypi.org/project/arcadedb/) [![Docs](https://img.shields.io/badge/docs-arcadedb.com-892122)](https://docs.arcadedb.com/arcadedb-python/) --- @@ -18,8 +18,8 @@ Install the package and read the documentation from the new repository: ```bash -uv add --pre arcadedb # native build; until 26.11.1 is a final release -uv add --pre arcadedb-jvm # JVM build, only if you need it +uv add arcadedb # native build, the default +uv add arcadedb-jvm # JVM build, only if you need it ``` - Documentation: [https://docs.arcadedb.com/arcadedb-python/](https://docs.arcadedb.com/arcadedb-python/) diff --git a/benchmarks/README.md b/benchmarks/README.md index c3bd94f2e62..39daff47f82 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -3,7 +3,7 @@ Two suites. Both reach embedded ArcadeDB through the Python package (the live harness's served arms reach it over HTTP instead, on a server built from the same upstream commit as the wheel), run every backend in Docker under the -same CPU and memory caps, report five repetitions per cell, and publish the +same CPU and memory caps, report up to five repetitions per cell (n=3 for every engine first, then n=4, then n=5), and publish the rows their numbers are computed from. They differ in scope and in where they publish, not in the kind of thing they measure. diff --git a/benchmarks/experiments/CAMPAIGN.md b/benchmarks/experiments/CAMPAIGN.md index d9c506fc7f3..38419a7af13 100644 --- a/benchmarks/experiments/CAMPAIGN.md +++ b/benchmarks/experiments/CAMPAIGN.md @@ -58,7 +58,7 @@ Beyond the lane's own metrics: - **Memory.** `peak_anon_mib_sum`, `peak_shmem_mib_sum`, `peak_owned_mib_sum` (anon + shmem), summed across every container in the cell. Anon alone misses a POSIX-shmem buffer pool entirely, which is why the page's memory column is `peak_owned_mib_sum` (issue #353). - **Disk.** `SizeRw` plus `du` over the daemon-reported volume mounts, with a settle loop requiring two readings within 1%. A volume in an image with no `du` is sized from a helper container (BUGS.md F38). -- **IO.** Cumulative `rbytes`/`wbytes` from `io.stat`. +- **IO.** Cumulative `rbytes`/`wbytes` from `io.stat`. A device built on another listed device is counted once; mini rows written before that fix (issue #555) carry figures about twice their value, and the freeze halves them (PROTOCOL.md, PUBLISHING.md). - **Phases.** `build_s`, settle, query generation, ground-truth load, search wall, recall computation, and `phases_accounted_s` so unexplained time is visible rather than absorbed. The dense and sparse lanes also split `ingest_s` from `index_s` where the engine has the boundary. Every lane prints `PHASE` markers as it goes, so a cell killed by its timeout still says which phase it was in. - **Cold and warm**, separately: the first iteration after the database is opened is the cold number and the remaining iterations are the warm one, on every timed query (DECISIONS #89). A lane where the split does not apply says so instead of leaving a blank. - **Envelope.** cpuset, memory cap, heap, observed server heap and page cache, `mem_split`, image digest, engine version, engine commit. @@ -242,7 +242,7 @@ The detail behind item 8: **ONE LINE OF DEVELOPMENT.** The instrument is developed on `main`, never on a long-lived branch (BUGS F65): mini pulls `main` at every stage start, so the instrument is landed on `main` before the first cell and never during the run, and a change that must not reach a running campaign waits for the next one. Anything else lands on `main`, behind a flag if it must, or waits. A later instrument is added the same way: `BENCH_INSTRUMENT` (`bench_common.INSTRUMENT`, default `2026-11`) names it, and code for it asks `instrument_at_least(...)` (`test_instrument_selector.py`). **The next campaign's stages are generated by `make_2611_stages.py`.** One template carries every guard the campaign learned (cap table, shadowed definitions, budgets against the cap, whole-cell cost, images present and current, comparator digests frozen at generation, the JPype pin read out of the image, the pinned wheel by sha256, the cap rule, the ALL-DONE trap); the instrument, `REPS` and the banner are its parameters. The generator selects the multi-model subset, narrowed by a profile (next paragraph). The subset is `ENGINES` in the file: ArcadeDB (embedded and served), SurrealDB (embedded and served), ArangoDB, Memgraph, PostgreSQL with pgvector, AGE and TimescaleDB (and plain PostgreSQL where a lane uses it), MongoDB with MongoDB Search, DuckDB with VSS and DuckPGQ, SQLite with sqlite-vec, the composed Qdrant plus Neo4j stack, Neo4j and Qdrant on their own (graph and dense; sparse on the second list), and LadybugDB on every lane it supports. Every registered arm is either in the subset or in `OUTSIDE`, so a newly registered arm forces a decision (`--check`). The core lanes are graph (interactive and analytics), cross-model, documents and dense; under `--profile full`, `--second-list` adds lifecycle, then sparse and time series, after them. TimescaleDB runs whenever the time-series lane does (so not under the default profile); `--without-timescaledb` leaves it out. -**THE PAPER PROFILE (the owner, 2026-10-25: the paper's experiments must finish by 2026-11-10; amended 2026-11-03, DECISIONS #232 and #235: the paper's numbers land about 2026-11-15, and the submission is 2026-11-24 for the SIGMOD 2027 industrial track or 2026-12-01 for the ICDE 2027 industry track, the fallback).** `--profile paper-2611` is the default and narrows the subset in the open (`PROFILES`, `DROP_WHY` in the file). (1) Core lanes only: graph, cross-model, documents and dense. The second list (lifecycle, sparse, time series) is out of scope for the paper, for the comparators and for the ArcadeDB chain, and `--second-list` is refused unless `--profile full` is given. (2) Neo4j and Memgraph stay in the graph lane (and cross-model where they are); their dense-lane cells (`neo4j_dense`, `neo4j_dense_int8`, `memgraph_dense`, `memgraph_dense_int8`) are out. The composed embedded stack (SQLite + LanceDB + LadybugDB in one process, `composed_embedded_sqlite_lance_ladybug`, the owner, 2026-10-25) is a Tier 1 cross-model arm: engine `composed-embedded`, right after the served composed stack, six lane cells per pass (hybrid in both durability classes and atomicity, at e2 and e2_500k), on the lane `e2` only; it runs in the dense image, so rebuild `dbbench:dense` (`./build_images.sh dense`) before the first pass. (3) The dense lane is fp32 only, for every engine: no `_int8` and no `_pq` arm, the comparators' and ArcadeDB's alike (`arangodb_dense_int8`, `arcadedb_dense_embedded_int8`, `arcadedb_dense_server_int8`, `mongodb_dense_int8`, `qdrant_dense_int8`, `sqlite_vec_dense_int8`, and the PQ arms when `BENCH_DENSE_PQ=1`). Quantized arms are not like-for-like across engines, and quantization is out of the paper's scope: fp32 for every engine, ArcadeDB's own INT8 and BINARY out by the owner's rule, jvector PQ, BQ and NVQ studied after the paper. ArcadeDB keeps its fp32 dense arms, embedded and served. (4) The target is N=3 (`TARGET_REPS`; `--project` prices 3 repetitions by default). The chain keeps the passes 4 and 5 after pass 3, in #503's order, so they run only if time is left; stopping after the first six groups is a complete result. Since 2026-11-03 the order on the bench host is: pass 3 (qC48), then the re-runs (qC14 at turbo off, qC15..qC23 at the 10 us idle hold, DECISIONS #234), then the four-arm ArcadeDB pass on the released 26.11.1 wheels, then passes 4 and 5 (qC49..qC80, which wait for `PASS45.go`). So the paper's comparators are N=3 for sure and N=5 where passes 4 and 5 finish before the tables freeze (DECISIONS #232, amendment of 2026-11-03). `--check` prints every dropped cell with its reason and count (lane cells per pass: 148 second-list, 8 Neo4j and Memgraph dense, 12 quantized dense) and proves that none is scheduled, in both chains whatever the arguments select (`profile_problems`); `--check --drafts DIR` lints written scripts for the same. `--profile full` is the whole subset as it was before the profile. +**THE PAPER PROFILE (the owner, 2026-10-25: the paper's experiments must finish by 2026-11-10; amended 2026-11-03, DECISIONS #232 and #235: the paper's numbers land about 2026-11-15, and the submission is 2026-11-24 for the SIGMOD 2027 industrial track or 2026-12-01 for the ICDE 2027 industry track, the fallback).** `--profile paper-2611` is the default and narrows the subset in the open (`PROFILES`, `DROP_WHY` in the file). (1) Core lanes only: graph, cross-model, documents and dense. The second list (lifecycle, sparse, time series) is out of scope for the paper, for the comparators and for the ArcadeDB chain, and `--second-list` is refused unless `--profile full` is given. (2) Neo4j and Memgraph stay in the graph lane (and cross-model where they are); their dense-lane cells (`neo4j_dense`, `neo4j_dense_int8`, `memgraph_dense`, `memgraph_dense_int8`) are out. The composed embedded stack (SQLite + LanceDB + LadybugDB in one process, `composed_embedded_sqlite_lance_ladybug`, the owner, 2026-10-25) is a Tier 1 cross-model arm: engine `composed-embedded`, right after the served composed stack, six lane cells per pass (hybrid in both durability classes and atomicity, at e2 and e2_500k), on the lane `e2` only; it runs in the dense image, so rebuild `dbbench:dense` (`./build_images.sh dense`) before the first pass. (3) The dense lane is fp32 only, for every engine: no `_int8` and no `_pq` arm, the comparators' and ArcadeDB's alike (`arangodb_dense_int8`, `arcadedb_dense_embedded_int8`, `arcadedb_dense_server_int8`, `mongodb_dense_int8`, `qdrant_dense_int8`, `sqlite_vec_dense_int8`, and the PQ arms when `BENCH_DENSE_PQ=1`). Quantized arms are not like-for-like across engines, and quantization is out of the paper's scope: fp32 for every engine, ArcadeDB's own INT8 and BINARY out by the owner's rule, jvector PQ, BQ and NVQ studied after the paper. ArcadeDB keeps its fp32 dense arms, embedded and served. (4) The target is N=3 (`TARGET_REPS`; `--project` prices 3 repetitions by default). The chain keeps the passes 4 and 5 after pass 3, in #503's order, so they run only if time is left; stopping after the first six groups is a complete result. Since 2026-11-03 the order on the bench host is: pass 3 (qC48), then the re-runs (qC14 at turbo off, qC15..qC23 at the 10 us idle hold, DECISIONS #234), then the four-arm ArcadeDB pass on the released 26.11.1 wheels, then passes 4 and 5 (qC49..qC80, which wait for `PASS45.go`). Amended 2026-11-05: if the 26.11.1 wheels are on PyPI and the four-arm runner is deployed when `REORDER reruns finished` is logged in the host's `STATUS.txt`, the four-arm pass (n=3 for the four ArcadeDB arms) starts first, and the window re-runs of qC14..qC17 follow it, still before `PASS45.go`; otherwise the order above stands. So the paper's comparators are N=3 for sure and N=5 where passes 4 and 5 finish before the tables freeze (DECISIONS #232, amendment of 2026-11-03). `--check` prints every dropped cell with its reason and count (lane cells per pass: 148 second-list, 8 Neo4j and Memgraph dense, 12 quantized dense) and proves that none is scheduled, in both chains whatever the arguments select (`profile_problems`); `--check --drafts DIR` lints written scripts for the same. `--profile full` is the whole subset as it was before the profile. Every generated stage exports `BENCH_INSTRUMENT=2026-11` before its first Python call, reads the selection back from `bench_common` and aborts if the tree does not select it, writes `runs_page__2611.jsonl` (and overlay directories with the same suffix) so one rows file never holds two instruments, and fixes `REPS` at generation (`--reps`, default 5, at most 5; the target is 3). The generator re-executes itself with the variable set and refuses a different value in the environment. `land_stage.py` lands exactly these files (DECISIONS #196; the 2026-10-25 change that closed the gap between the pull and the freeze): it pulls `runs_page__2611.jsonl` and the overlay directories with the same suffix, and `screen_pulled_rows` refuses the file before anything is merged unless every row is stamped `2026-11` and names the pin's engine commit, so a retired 2026-10 file or a file that mixes instruments never reaches `results/runs.jsonl`. The first landing meets no store (the October one is retired) and `merge_campaign.py` creates it. `export_web` no longer demands the `e4decomp_` artifacts of a landing scoped away from e4, because the chain has no e4 stage and `--only-lanes l2` would otherwise wait for one. A rehearsal restores what it wrote, the site payload (removed when the run created it) and the PAGE-SPEC.md block included. Failed cells (`rc != 0` or an error) merge as failures and are not frozen; censored queries keep their flag and publish no number. No stage script calls `land_stage.py` (the generated `qC` scripts only name it in a comment about the rows file); a landing is run by hand on the laptop. The comparators are the default mode and run on the release pair already installed (`ARCADEDB_WHEEL`, `ARCADEDB_SERVER_IMAGE`, `ARCADEDB_ENGINE_COMMIT`): `python3 make_2611_stages.py --after --status-file STATUS.txt --id-prefix qS --out DIR`. `--after` is required, with no default (a marker name, or `none`), and the first stage waits for `grep -qx " ALL-DONE"`, each later one for the stage before it. The marker must be the LAST item still to run, one that writes its line whether it runs or is skipped: a marker already in the machine's STATUS.txt makes the first stage start at once, beside the running jobs (2026-10-23: `qMA06 ALL-DONE`, written by a skipped study). So `--status-file` is required with a marker (a copy of that STATUS.txt, or `none` when there is no queue) and the generator refuses a marker the file already holds; `--check --drafts DIR --status-file STATUS.txt` makes the same test on drafts that exist. The subset is emitted in ENGINE TIERS in the user's exact pass order (2026-10-24; `--tiers`, on by default, `--no-tiers` for the plain rep-major chain). N defaults to 5, the target is 3 (the paper profile), and the order is `schedule_order`: n=1 T1, n=2 T1, n=3 T1, n=1 T2, n=2 T2, n=3 T2, n=4 T1, n=4 T2, n=5 T1, n=5 T2. Every prefix of it is a complete set of equal replicates for the tiers covered so far (each tier has run passes 1..k, none skipped, and a pass is whole before the next starts); `--reps 3` stops after the first six groups, which is the chain of Tier 1 passes 1-3 and then Tier 2 passes 1-3 as before. Tier 1 is ArcadeDB (the four 26.11.1 arms, when `--arcadedb-release` adds them), SurrealDB, ArangoDB, PostgreSQL with pgvector and AGE, MongoDB with Search, the composed Qdrant plus Neo4j stack, and the embedded references SQLite, LadybugDB and DuckDB; Tier 2 is Memgraph, Neo4j and Qdrant on their own, Redis 8, CrateDB and TimescaleDB (`TIER_ENGINES`, with `ARM_TIER` for TimescaleDB, an arm of the `postgresql` engine that the user lists in Tier 2). Every stage holds the arms of one tier, and stage ids stay one run of consecutive numbers (a stage that mixed both tiers is split in two, so the stage count grows, and each stage repeats its guards and image checks; past 99 stages the ids take three digits, so `qS100` cannot sort before `qS11`). `--check` prints the schedule and proves the order and the prefixes (the plan collapsed into (tier, pass) groups equals `schedule_order`, each group holds the whole pass), and `--check --drafts DIR` lints a draft folder for the same. An arm with no tier runs after Tier 2 in each block. The ArcadeDB arms are Tier 1 but not in this chain: DBBench keeps one ArcadeDB version per host, so they run only once the benchmark pin is 26.11.1 (next paragraph). Inside a tier the order is the one below: a pass is pass k of every stage of the tier (core lanes, then the second list), and each stage runs ONE repetition (`--only-reps $PASS`). With `--no-tiers` the chain is pass 1 of every stage, then pass 2, up to pass N. If time runs out, every engine of a tier has the same n; repetitions days apart show drift. A stage whose environment declares fewer reps (`REPS=k`) is left out of the later passes. Inside a lane the engines run in `ENGINE_PRIORITY` (SurrealDB, ArangoDB, PostgreSQL, MongoDB, Redis 8, composed Qdrant+Neo4j, Neo4j, Qdrant, DuckDB, SQLite, Memgraph, LadybugDB, CrateDB; ArcadeDB's arms lead it when they join; Redis and CrateDB are skipped until their adapters are in `ENGINES`). The rep-1-first rule and the cap rule hold across passes: a pass above 1 runs a cell only when rep 1 of it was clean, and only a timeout of rep 1 censors the larger sizes. WHICH REPS EXIST is `~/rep-state-2026-11-.tsv` on the host (one `label, rep, ok|fail` line per finished repetition; the rows themselves carry the rep number in the runner's results file): stop after any pass, and the next pass, or a restarted stage, picks up from it with no regeneration; N can go to 3 or stop at 2. Delete that file to start a chain over. The ArcadeDB pass is `--arcadedb-release X.Y.Z` with `--engine-commit`, `--image-digest sha256:...`, `--wheel-name`, `--wheel-sha256` and `--wheel-url`; it refuses unless all five are given, because the release may not exist when the drafts are made. It writes `00`, the pair install (official image by digest, wheel by sha256, `build_matched_pair.sh`, `verify_pair_c25.sh`), and ArcadeDB's lane stages behind it. THE ARCADEDB GATE: the ArcadeDB chain is its own chain (Tier 1 only, passes 1..N in order), generated with `--after` the last comparator stage, so the comparators run the schedule first. `00` waits after its `--after` for the marker file `$HOME/conf/go-arcadedb-2611` on the host (it never creates it; it sits before the ALL-DONE trap, so a killed waiter hands nothing on), and every ArcadeDB lane stage aborts without it. The user or the agent touches that file when `CURRENT_PIN` moves to 26.11.1, and not before; `--check --drafts` lints that the gate is in the pair install and in every ArcadeDB stage and in no comparator stage. The ArcadeDB pair's guards are split by what a stage runs. Every stage exports the pinned wheel (`ARCADEDB_WHEEL`, with `ARCADEDB_SERVER_IMAGE` and `ARCADEDB_ENGINE_COMMIT`) before `build_images.sh`, which bakes it into every generic image (client, duckdb, dense), and so every stage guards that the file exists at the pinned path and has the pinned sha256; `build_images.sh` copies it from that path and nothing in it or the Dockerfiles reads `dist/` itself. Only a stage that runs an ArcadeDB arm (the `--arcadedb-release` pass) also guards the rest: exactly one wheel in `dist/` (it protects `verify_pair_c25.sh`), the server image, the wheel and JPype read back out of `dbbench:arcadedb`, and `verify_pair_c25.sh`. A comparator stage on a host whose `dbbench:arcadedb` is the previous release's therefore runs. `build_images.sh` has no PyPI fallback: it refuses to build the `arcadedb` image (named or by default) without `ARCADEDB_WHEEL`, and refuses a wheel that is not a file, and refuses a wheel whose `arcadedb_embedded/core.py` lacks `_INSERT_MANY_CHUNK` (`wheel_guard.py`: before 26.10.2 `insert_many` parsed the whole input into one Java array, which the kernel killed on l4/ts1000/arcadedb_ts_doc on 2026-10-08; use a 26.10.2 or newer wheel); the images that take no wheel can be built alone by naming them. Every ABORT hands on: the `ALL-DONE` trap is set right after the wait, before the first guard, so a stage that aborts before its START line still writes its marker and the next stage runs (its own guards then run too). The ABORT line is in `STATUS.txt`, where `mini_watch.sh` reports it; to resume, fix the cause and run the aborted stage's script again by hand after the chain, and landing reads only the rows that exist. `make_2611_stages.py --check --drafts DIR` lints both rules. diff --git a/benchmarks/experiments/PAGE-SPEC.md b/benchmarks/experiments/PAGE-SPEC.md index 371c2e882d8..d00a82b7e40 100644 --- a/benchmarks/experiments/PAGE-SPEC.md +++ b/benchmarks/experiments/PAGE-SPEC.md @@ -9,7 +9,7 @@ The DBBench page at `humem.ai/research/dbbench` is the PRIMARY artifact: its lan ## 0. The rules every published cell obeys 1. **Serial, full cpuset.** Every published latency, throughput, percentile, and memory cell runs one at a time on mini's `cpuset 0-11` (the 12 P-core THREADS on 6 physical P-cores; see 0a), `workers=1`, `tier=paper`. Parallel shards exist for sweeps and exploration only and may never reach the page. Enforced: `runner.py` refuses `workers != 1` at paper tier; `load_canonical` drops partial cpusets. -2. **N=5, median [min-max].** Five repetitions per cell, reached in two passes (PROTOCOL.md section 1). Any table with a cell below n=5 states its n in a condition, and no page sentence may assert an n that a cell it covers does not meet. Per-table operation and repetition counts are generated conditions (`export_web._counts_note`), never typed. +2. **N=5, median [min-max].** Five repetitions per cell, reached in whole passes: n=3 for every engine first (the four ArcadeDB arms included), then n=4, then n=5 (PROTOCOL.md section 1). Any table with a cell below n=5 states its n in a condition, and no page sentence may assert an n that a cell it covers does not meet. Per-table operation and repetition counts are generated conditions (`export_web._counts_note`), never typed. 3. **One engine commit per table.** Every ArcadeDB row in one table comes from the same upstream commit, stamped as `engine_commit`. Comparators are digest-pinned images. See section 1. Enforced at production time: `runner.py` refuses a paper-tier ArcadeDB run when `ARCADEDB_ENGINE_COMMIT` is unset, before any cell starts, and `bench_common.run_conditions()` reads it so bespoke probes carry it too. A rule the page depends on may not be a printed reminder to a human. 4. **One corpus per table.** Every row in a table describes the same dataset at the same size. A table whose rows disagree on `n_docs` (or the lane's size field) is a defect, not a comparison. The fingerprint must come from the corpus the lane loaded, not from a module constant (`l2_graph.py` and `l3_sparse.py` point it at the loaded data; BUGS.md F6 is the sparse lane running the synthetic corpus under a paper label). 5. **A knob the campaign sets must reach the cell, be honoured by the lane, and resolve inside the container.** `runner.py`'s env passthrough is a CLOSED tuple: a knob absent from it silently runs the in-script default. Three checks, each of which has failed once: DELIVERED (diff every lane's `environ.get("UPPER_CASE")` against what the runner passes), HONOURED (an unknown mode is a hard error, not a silent fallback to a literal default), and RESOLVED where the process runs (the mount is printed in the preamble and a missing corpus is refused on the host). Run the delivery diff whenever a lane gains a knob. diff --git a/benchmarks/experiments/PROTOCOL.md b/benchmarks/experiments/PROTOCOL.md index 545e2958d5b..3f99dbe7b77 100644 --- a/benchmarks/experiments/PROTOCOL.md +++ b/benchmarks/experiments/PROTOCOL.md @@ -26,7 +26,7 @@ The admission filter is `make_paper_tables.load_canonical()`. Every test below r - Dedupe key is `(lane, scale, n_docs, workload, backend, gav is not False, rep, durability_class)`, latest `ts_utc`, never `run_id`. Without the `gav` term the no-view ablation overwrites T3, and without the class term the strict arm of a write cell overwrites its relaxed twin (DECISIONS #90; a row with no class counts as relaxed, which is what it was). **GATE** `load_canonical` - One released engine line per table. Delete preference cascades, never reorder them. **GATE** T4 refusal in `make_paper_tables` and five of f8's six bars in `make_paper_figures`; **REMEMBERED** for f8's dense bar, whose `_dense_multipass` source reads no version field, and for every other table (F5) - Lane scripts publish; bespoke drivers investigate. Both known protocol violations were bespoke-driver rows promoted to cells, and T5's dense half is still an overlay from `dense_multipass_driver.py`, run through `runner.py --driver`. **GATE** `fairness_check.report_producers` (F6b) for `runs.jsonl` lanes, which sees no overlay record; **REMEMBERED** for every overlay-fed cell -- Every cell is N=5, or its shortfall is disclosed in `EXPECTED` and in the caption, and the disclosure is deleted when the shortfall is fixed. **N=5 is reached in two passes on the 26.10.1 measurement (DECISIONS #164): reps 1 to 3 for every arm first, reps 4 and 5 in a later pass, one whole lane at a time so every engine of a lane is topped up within days. A table landed between the passes states n=3 in its caption and in `EXPECTED`, and the later landing deletes both. Medians of three carry no rank test (the smallest two-sided Mann-Whitney p at 3 versus 3 is 0.1), so a claim under about 1.3x waits for the top-up, and the two batches are compared before they are pooled.** **GATE** `provenance_check.caption_n` over the generated `.tex` tables, skipped on a lane-scoped landing (`BENCH_ONLY_LANES`); **REMEMBERED** for T5's dense half and for the deletion half, which `if ns == {5}: continue` short-circuits +- Every cell is N=5, or its shortfall is disclosed in `EXPECTED` and in the caption, and the disclosure is deleted when the shortfall is fixed. **N=5 is reached in two passes on the 26.10.1 measurement (DECISIONS #164): reps 1 to 3 for every arm first, reps 4 and 5 in a later pass, one whole lane at a time so every engine of a lane is topped up within days. A table landed between the passes states n=3 in its caption and in `EXPECTED`, and the later landing deletes both. Medians of three carry no rank test (the smallest two-sided Mann-Whitney p at 3 versus 3 is 0.1), so a claim under about 1.3x waits for the top-up, and the two batches are compared before they are pooled.** On the 2026-11 pass the order runs across the whole roster by repetition: n=3 for every engine first, the four ArcadeDB arms (embedded and served, JVM and native) included, then n=4, then n=5, each a whole pass (CAMPAIGN.md, paper profile). The paper's comparators are n=3 for sure and n=5 where passes 4 and 5 finish before the tables freeze. **GATE** `provenance_check.caption_n` over the generated `.tex` tables, skipped on a lane-scoped landing (`BENCH_ONLY_LANES`); **REMEMBERED** for T5's dense half and for the deletion half, which `if ns == {5}: continue` short-circuits - Never publish a scale where comparators have rows and we do not; print what was withheld and why at export time. **GATE** withheld-scale guard in `export_web` - Every artifact the page cites lives in this repository, and the superseded run is not left where it can be compared against. **GATE (report only)** `provenance_check.FEEDS`: an empty feed prints "(no result files)" and counts nothing. **GATE**: every pinned reader refuses when `_` is missing - Every overlay directory and top-level result file feeding a table records run conditions and a real version; "feeds a table" means "is opened by `make_paper_tables`". **GATE** `run_conditions`, `lane_files` and `_is_real_version` for the two lane files; **REMEMBERED** for the overlay directories T4 and T5 print, whose audit accepts any non-empty string @@ -96,7 +96,7 @@ Every row carries these fields. The code is `memory_probe.py`, and `bench_common - **What compares across engines and what does not.** `mem__owned_mib` and its parts, and `mem__peak_mib`, compare across every engine: they come from the same kernel counters of the same kind of container, and the page prints one headline column from them. `jvm_*` compares across JVM engines only and says nothing about a native engine; `proc_*` compares across embedded engines only. A JVM engine's `owned` and its `jvm_used_after_gc` differ by the JVM's slack (committed but free heap, metaspace, code cache, thread stacks, native memory): the first is what the container held, the second is what the engine needed, and neither replaces the other. - **A pinned heap makes the kernel's numbers show the cap.** With `-Xms` equal to `-Xmx` the JVM commits the whole heap at start and touches it as it grows, so anon and RSS read near the cap whatever the engine needs, and a larger cap reads larger with the same workload. The ArcadeDB arms, embedded and served, do not pin the heap (`-Xms` is left unset), so their memory column reads demand up to `-Xmx`; the JVM arms that have a pinned tier heap (Elasticsearch, Neo4j) read the pin, and the other served engines keep following their vendors' documentation for the heap they start with. An unpinned heap is no cure for the headline: a JVM that has grown stays grown unless the collector returns the pages, which is why `jvm_used_after_gc_mib` sits beside it, and why a row records `jvm__committed_mib` next to the heap the engine needed. - **Disk per phase, the same way for every engine.** At the same boundaries the runner reads, for the container that holds the engine's data (the server container for a served engine, the client's for an embedded one), the bytes it is responsible for: the writable layer plus every declared volume, the reading `container_disk` already takes, here as one sample after a flush rather than the settled pair. `disk__mib` is that reading, `disk__data_mib` is the headline: it less the same container's empty-start reading (`server_disk_baseline_mb` or `client_disk_baseline_mb`), so it is what the workload left, comparable with `disk_data_mb`. `disk__wal_mib` is the write-ahead log or journal alone, found inside the container by name (`*.wal`, `pg_wal`, `journal`, `translog`, `neostore.transaction.db.*`); an engine with another layout reads 0.0, which means none was found, and it is context, not a comparison. `disk_close_mib` and `disk_close_data_mib` are the runner's existing settled reading taken after the lane has closed the database; a phase reading of the same bytes can be lower or higher than it by what background compaction and checkpointing do in between, which is the point of reading both. -- **IO per phase: `io__read_mib`, `io__write_mib`.** The bytes the cell's containers read from and wrote to disk since the previous boundary (the cgroup `io.stat` counters are cumulative, so a phase is a difference; the first phase starts when the client container does), summed over the containers. The server's startup is not in `load`. The disk and IO readings are taken before the peak restarts, so the helper processes they start inside the container never land in the next phase's peak. +- **IO per phase: `io__read_mib`, `io__write_mib`.** The bytes the cell's containers read from and wrote to disk since the previous boundary (the cgroup `io.stat` counters are cumulative, so a phase is a difference; the first phase starts when the client container does), summed over the containers. The server's startup is not in `load`. The disk and IO readings are taken before the peak restarts, so the helper processes they start inside the container never land in the next phase's peak. **Mini rows written before the fix for issue #555 count a stacked block device twice:** the runner summed every device line of `io.stat`, and on the mini the NVMe (`259:0`) and the device-mapper device on top of it (`252:0`) list identical bytes, so every `io_*_mib` figure of those rows (the per-phase fields, the `_sum` pair, and the per-side and per-container ones) is about twice its value. The fix counts a device built on another listed device once and stamps `io_stat_dedup: 1` into every row the fixed runner writes. The freeze halves the io figures of mini rows that lack the stamp, in `runs_paper.csv` only: `runs.jsonl` keeps the raw values, and `generated/io_stat_correction.json` lists each corrected row with its raw values (PUBLISHING.md). No other field of a row is affected. - **Where this stops.** A forced GC changes the heap for the phase that follows (a full collection empties young generations and caches the next phase would have found warm), which is why it runs only after the last timed phase and in `idle`, never before a timed phase, and is recorded in `jvm_forced_gc_phases`; and the lane waits for the answer, so the phases that follow a probe start a few hundred milliseconds later than they would have. A lane that prints no `PHASE` marker of the ones above (the lifecycle and restart lanes) takes no per-phase reading and keeps only the whole-cell fields. ## 3. How a comparison is kept fair diff --git a/bindings/python/README.md b/bindings/python/README.md index 5464a9c3635..522a314b118 100644 --- a/bindings/python/README.md +++ b/bindings/python/README.md @@ -1,6 +1,6 @@ # ArcadeDB Python Bindings -> **Moved: the Python bindings now live in [ArcadeData/arcadedb-python](https://github.com/ArcadeData/arcadedb-python).** Install `arcadedb` (the native build, the default) or `arcadedb-jvm` (the JVM build); both use `import arcadedb`. [Which package?](https://github.com/ArcadeData/arcadedb-python/blob/main/docs/modules/ROOT/pages/which-package.adoc) explains the choice. The first releases of the new packages are 26.11.1, starting as pre-releases (`uv add --pre arcadedb`). `arcadedb-embedded` and `import arcadedb_embedded` are deprecated: `arcadedb-embedded` will get a last release that installs `arcadedb`. Report bindings issues and send bindings pull requests to the new repository. This repository will be archived; the benchmark tooling stays here for now. +> **Moved: the Python bindings now live in [ArcadeData/arcadedb-python](https://github.com/ArcadeData/arcadedb-python).** Install `arcadedb` (the native build, the default) or `arcadedb-jvm` (the JVM build); both use `import arcadedb`. [Which package?](https://github.com/ArcadeData/arcadedb-python/blob/main/docs/modules/ROOT/pages/which-package.adoc) explains the choice. Both packages are released as 26.11.1 (`uv add arcadedb`). `arcadedb-embedded` and `import arcadedb_embedded` are deprecated: the last release of `arcadedb-embedded`, 26.11.1, installs `arcadedb`. Report bindings issues and send bindings pull requests to the new repository. This repository will be archived once the benchmark tooling moves to [humemai/dbbench](https://github.com/humemai/dbbench) (planned for mid-November 2026, after the last measuring pass); until then the tooling stays here. Native Python bindings for ArcadeDB - the multi-model database that supports Graph, Document, Key/Value, Search Engine, Time Series, and Vector models. @@ -19,7 +19,7 @@ Native Python bindings for ArcadeDB - the multi-model database that supports Gra ### Installation ```bash -uv add arcadedb-embedded # deprecated; new projects use: uv add --pre arcadedb +uv add arcadedb-embedded # deprecated; new projects use: uv add arcadedb ``` **Requirements:** diff --git a/bindings/python/docs/index.md b/bindings/python/docs/index.md index c670ce10b6e..e5c0e78f397 100644 --- a/bindings/python/docs/index.md +++ b/bindings/python/docs/index.md @@ -2,7 +2,7 @@ !!! warning "Moved to ArcadeData/arcadedb-python" - The Python bindings now live in [ArcadeData/arcadedb-python](https://github.com/ArcadeData/arcadedb-python). Install `arcadedb` (the native build, the default) or `arcadedb-jvm` (the JVM build); both use `import arcadedb`. [Which package?](https://github.com/ArcadeData/arcadedb-python/blob/main/docs/modules/ROOT/pages/which-package.adoc) explains the choice. The first releases of the new packages are 26.11.1, starting as pre-releases (`uv add --pre arcadedb`). `arcadedb-embedded` and `import arcadedb_embedded` are deprecated: `arcadedb-embedded` will get a last release that installs `arcadedb`. Report bindings issues and send bindings pull requests to the new repository. This repository will be archived; the benchmark tooling stays here for now. + The Python bindings now live in [ArcadeData/arcadedb-python](https://github.com/ArcadeData/arcadedb-python). Install `arcadedb` (the native build, the default) or `arcadedb-jvm` (the JVM build); both use `import arcadedb`. [Which package?](https://github.com/ArcadeData/arcadedb-python/blob/main/docs/modules/ROOT/pages/which-package.adoc) explains the choice. Both packages are released as 26.11.1 (`uv add arcadedb`). `arcadedb-embedded` and `import arcadedb_embedded` are deprecated: the last release of `arcadedb-embedded`, 26.11.1, installs `arcadedb`. Report bindings issues and send bindings pull requests to the new repository. This repository will be archived once the benchmark tooling moves to [humemai/dbbench](https://github.com/humemai/dbbench) (planned for mid-November 2026, after the last measuring pass); until then the tooling stays here.