benchmarks: import arcadedb (native and arcadedb-jvm), fall back to arcadedb_embedded

benchmarks/experiments/COMPARATORS.md

@@ -72,16 +72,33 @@ Pins follow DECISIONS #87 (the latest self-hosted stable release at campaign tim
 | Neo4j driver (Neo4j AND Memgraph arms) | `neo4j==6.4.1` | client package |
 | Neo4j driver Rust extension (the vendor's documented accelerator, DECISIONS #173) | `neo4j-rust-ext==6.4.1.0` | client package |
 | pgvector Python adapter (binary COPY, DECISIONS #174) | `pgvector==0.5.1` | client package |
-| ArcadeDB Python client package installed in the client image (measured by the harness, not a published client; the served ArcadeDB headline is the benchmark's own `lean_http.py`, CAMPAIGN section 7 row 82) | `arcadedb-driver==0.2.0` | client package |
+| ArcadeDB Python client package (measured by the harness, not a published client: the served ArcadeDB headline is the benchmark's own `lean_http.py`, CAMPAIGN section 7 row 82) | `arcadedb-driver==0.4.0` | client package, in the `dbbench:client-arcadedb` layer (below) |
+
+**The served ArcadeDB client image (DECISIONS #258, #259).** The served ArcadeDB arms run in `dbbench:client-arcadedb`: `dbbench:client` plus one pip layer (`Dockerfile.client-arcadedb`, `build_images.sh` target `client-arcadedb`, list `ARCADEDB_DRIVER_PKGS`), so `dbbench:client` stays byte-identical and every comparator row measured on it stays on one client. The vendor's driver is measured by the harness and not published (the page headline is `lean_http.py`, DECISIONS #268, #285). `arcadedb-driver==0.2.0` remains in `dbbench:client` (the `[client]` line of `build_images.sh`, unused by any cell now); `lean_http.OfficialSession` refuses to start on any driver version but `lean_http.OFFICIAL_PIN` (0.4.0). The driver's environment is pinned at the latest PyPI release of 2026-11-07 (the driver itself asks only `attrs>=23.2`, `httpx>=0.27`, `typing-extensions>=4.10`, `websockets>=14`): `arcadedb-driver==0.4.0` (2026-11-05), `httpx==0.28.1`, `httpcore==1.0.9`, `h11==0.16.0`, `anyio==4.16.0`, `idna==3.20`, `certifi==2026.10.23`, `attrs==26.1.0`, `typing-extensions==4.16.0`, `websockets==17.4`. This layer moves httpx, anyio, certifi and the other listed libraries only inside the served ArcadeDB client image, where no comparator client runs. Scope of the driver in a served cell: `/query` and `/command` (with `limit`); bulk `/batch` loads, NDJSON streams, transactions, probes and raw text bodies stay on the lean HTTP client (DECISIONS #259 (4)).
 
 **Held below the newest, and why.**
 
 - **MongoDB 8.3.x is skipped, and 9.0.4 is the pin** (the one engine where #103d overrode #87; hold lifted 2026-10-23). 8.3.13, the newest 8.3 patch (Docker Hub 2026-10-17), still refuses to start on the bench kernel with the SERVER-121912 guard ("Linux kernel versions 6.19 and newer has a known incompatibility"), both as plain `mongo` and inside the Search image, with mongot 1.70.4 and with 1.70.5. 9.0.4 (Docker Hub 2026-10-17, amd64 manifest `sha256:b609cdfba3a8…`; 9.0.0 shipped 2026-09-28 and the release notes call 9.0 the current stable release) starts on the same kernel. Why it starts was not investigated: SERVER-125742 exempts kernels from upstream 7.0.14 but reads `uname -r`, which Ubuntu keeps at 7.0.0 (open as SERVER-136848, reported against 8.3.11 and 9.0.2); the bench kernel 7.0.0-38-generic is upstream 7.0.14, so it carries the userspace fix the guard exists for, and the old argument that an older 8.3.x lacking the warning would run a configuration its vendor calls unsafe no longer holds. Why not stay on 8.2.12, which also boots: MongoDB's compatibility table for MongoDB Search lists mongot 1.70.1 and later against MongoDB Server 8.3 only and says 8.2 "is no longer a supported source deployment for MongoDB Search", so 8.2.12 + mongot 1.70.4 was a pairing the vendor no longer supports. 9.0 is not in that table either: 9.0.4 + mongot 1.70.5 is exercised, not documented. Laptop test 2026-10-23, kernel 7.0.0-38-generic, containers on cores 12-15 with `-m 3g`/`-m 4g`, `pymongo==4.18.1`, through the repo's own `l1_tpc.MongoTPC` on 30,000 seeded synthetic lineitem rows and `mongo_common` on 3,000 seeded 32-d vectors: 8.2.12 boots, 8.3.13 does not (also with mongot 1.70.4 and 1.70.5), 9.0.4 boots; on 9.0.4 against 8.2.12 the five analytics pipelines, 200 committed new-order transactions, 50 payments, the abort (nothing changed), the write-conflict error (code 112), `with_transaction` and 100 single-record CRUD operations gave identical digests; `$vectorSearch` through mongot 1.70.5 returned the same top-10 lists as 8.2.12 + 1.70.4 on 99.3% of 300 queries (mean set overlap 0.9993, recall 1.000 against exact search) for `quantization: none`, and 0.960 mean set overlap for `scalar` (recall 0.878 against 0.873, a quantized index is not deterministic across builds); both indexes read their quantization back, and a `$vectorSearch` inside a multi-document transaction is still refused. Not tested: performance (every MongoDB row has to be re-measured on 9.0.4), a full TPC-H scale, restart/recovery, the time-series and graph lanes. mini runs kernel 7.0.0-31-generic, the same 7.0 line; a one-minute `docker run --rm mongo@sha256:b609cdfb… mongod --version`-style start check there is a sanity step, not a requirement.
 - **DuckDB and DuckPGQ 1.5.6**, not 2.0.0 (checked 2026-10-23, laptop, duckdb 2.0.0 from PyPI): the `duckpgq` community build (`29ca668a22`) and `vss` both exist for v2.0.0 and both LOAD, VSS HNSW works, but DuckPGQ registers no syntax: `CREATE PROPERTY GRAPH` and `GRAPH_TABLE ... MATCH` answer `Parser Error: syntax error`, with or without `SET enable_legacy_parser` or `allow_parser_override_extension`. So the graph arm cannot be measured on 2.0.0 (DECISIONS #103d: the newest version at which every arm can be measured). Lift the hold when a DuckPGQ build for 2.x parses; the recipe is in `build_images.sh`. 1.5.5 had no DuckPGQ build (HTTP 404) and 1.5.6 does (`61f5d082`).
-- **SurrealDB server v3.2.4 and SDK 2.0.0**: the SDK is the newest stable. The server is held one minor behind the newest stable (`v3.3.4`, `latest` on 2026-10-23) on purpose: on 3.3.4 the l2 graph OLTP delete left one person and its edge behind with no error in 2 of 3 runs (0 of 2 on 3.2.4), so the bump waits for a repeat study (humemai/arcadedb-embedded-python#483). Lift the hold when 3.3.x or later deletes cleanly.
-- **Chroma 1.5.9, sqlite-vec 0.1.9, Memgraph 3.14.0, pymongo 4.18.1, python-arango 8.3.6, falkordb 1.7.1, redis 8.1.0**: the newest stable releases (sqlite-vec's `0.1.10a4` is a pre-release). **pymilvus 3.0.2** (with the `bulk_writer` extra, which brings `minio`, `azure-storage-blob` and `ml-dtypes`) is what Milvus's compatibility table names for a 3.0.3 server.
+- **SurrealDB server v3.2.4 and SDK 2.0.0**: the SDK is pinned at the October re-pin; a newer release exists (3.0.0, 2026-11-04); not changed because its rows are already measured ("Client holds" below). The server is held one minor behind the newest stable (`v3.3.4`, `latest` on 2026-10-23) on purpose: on 3.3.4 the l2 graph OLTP delete left one person and its edge behind with no error in 2 of 3 runs (0 of 2 on 3.2.4), so the bump waits for a repeat study (humemai/arcadedb-embedded-python#483). Lift the hold when 3.3.x or later deletes cleanly.
+- **Chroma 1.5.9, sqlite-vec 0.1.9, Memgraph 3.14.0, redis 8.1.0**: the newest stable releases (sqlite-vec's `0.1.10a4` is a pre-release). **pymilvus 3.0.2** (with the `bulk_writer` extra, which brings `minio`, `azure-storage-blob` and `ml-dtypes`) is what Milvus's compatibility table names for a 3.0.3 server.
 - **SQLite 3.46.1** is not a pin: it is the stdlib `sqlite3` of the `python:3.12-slim` base, read back at connect. **ArcadeDB is not a comparator**: it is pinned to our own wheel and the matching `arcadedb-c25` image (its row above).
 
+**Client holds (checked on PyPI 2026-11-07; DECISIONS #259 (3)).** Each comparator client library is pinned at the October re-pin (2026-09-19, individual bumps to 2026-10-25) and not changed afterwards because its rows are already measured: a newer client mid-campaign would split a comparator's rows across two client versions. Each package below has a newer release today. The hold is a decision, not a claim that the pin is the newest.
+
+| Client | Pinned | Newer release (PyPI date) | Why held |
+|---|---|---|---|
+| psycopg (binary) | 3.3.6 | 3.3.8 (2026-11-04) | rows already measured |
+| pymongo | 4.18.1 | 4.19.0 (2026-10-14) | rows already measured |
+| python-arango | 8.3.6 | 8.3.7 (2026-10-27) | rows already measured |
+| falkordb | 1.7.1 | 1.8.1 (2026-10-25) | rows already measured |
+| lancedb | 0.39.0 | 0.41.0 (2026-10-29) | rows already measured |
+| surrealdb (Python SDK) | 2.0.0 | 3.0.0 (2026-11-04) | rows already measured; a major version |
+| ladybug | 0.22.0 | 0.22.3 (2026-11-06) | tied to the baked `vector` extension sha256 (`LADYBUG_EXTENSIONS`); moving it means re-reading that sha and re-measuring |
+| duckdb | 1.5.6 | 2.0.0 (2026-10-21) | documented hold: DuckPGQ registers no syntax on 2.0.0 (above) |
+| azure-storage-blob | 12.31.0 | 12.31.1 (2026-10-29) | rows already measured (a pymilvus `bulk_writer` dependency) |
+| numpy, pandas, pyarrow (harness libraries) | 2.5.3, 3.0.6, 25.0.1 | 2.5.4 (2026-10-10), 3.1.1 (2026-11-04), 26.0.1 (2026-11-03) | rows already measured; "move them deliberately" (`build_images.sh`) |
+
 **Why some moves were smoked.** The PostgreSQL set moves as one major (#103d); AGE gated it, being the only member packaged per major (`postgresql-18-age`). PostgreSQL 18 turns data checksums on by default, a write-path cost the page keeps because it is PostgreSQL's own default. Neo4j is pinned at .1 because 2026.08.0 could fail queries through its resource monitoring, and a new index provider (`vector-2026.08`) is auto-selected for new indexes, so the dense operating point is a measurement and not an inheritance (the Neo4j row says how).
 
 **AGE 1.8.0 is not a labelling change.** Holding a 5,000-edge batch fixed and varying the vertex count, AGE 1.7.0 takes 1.56 s at 160,000 vertices and 11.14 s at 500,000 (a jump of the shape of a planner flip, so the ratio is not a constant and must not be quoted as one), where 1.8.0 takes 0.18 s at both: 62x at 500,000. October's pg_age build numbers differ from September's for that upstream reason and not for an instrument change; a pg_age cell that ran the stale image was censored at the 2 h cap (BUGS F90, F76).

benchmarks/experiments/Dockerfile.bench

@@ -1,6 +1,6 @@
 # Generic benchmark image. Pin the package set via build ARG so each backend
 # (arcadedb, duckdb, ladybug, chroma, faiss, ...) gets its own pinned image later.
-# arcadedb-embedded bundles its own JRE, so no Java install is needed.
+# arcadedb-jvm (and the older arcadedb-embedded) bundle their own JRE, so no Java install is needed; the native `arcadedb` wheel has no JVM.
 FROM python:3.12-slim
 
 ARG PIP_PACKAGES=""
@@ -12,16 +12,21 @@ ARG PIP_PACKAGES=""
 # default is bench_common.JPYPE_PIN; `make_2611_stages.py --check` refuses a
 # default that differs, and every stage reads the version back out of the image.
 ARG JPYPE_VERSION=1.7.1
-# Local wheels (e.g. a not-yet-released arcadedb-embedded) placed in
+# NO JPYPE IN A NATIVE IMAGE (26.11.1): the `arcadedb` wheel is the native build, no JVM and no JPype, and its rows must say so
+# (bench_common.jpype_fields reads the JPype the process imported). JPype is installed with `arcadedb-jvm`, `arcadedb_jvm-*.whl`,
+# `arcadedb-embedded` and `arcadedb_embedded-*.whl`; a wheel named `arcadedb-<digit>...` (the native one) gets none.
+# Local wheels (e.g. a not-yet-released arcadedb-jvm) placed in
 # docker/wheels/ are installed first; the dir ships empty (.gitkeep).
 COPY docker-wheels/ /tmp/wheels/
 RUN set -e; \
     if ls /tmp/wheels/*.whl >/dev/null 2>&1; then \
-      pip install --no-cache-dir /tmp/wheels/*.whl "jpype1==${JPYPE_VERSION}"; \
+      _jp=""; \
+      for _w in /tmp/wheels/*.whl; do case "$(basename "$_w")" in arcadedb-[0-9]*) ;; *) _jp="jpype1==${JPYPE_VERSION}";; esac; done; \
+      pip install --no-cache-dir /tmp/wheels/*.whl ${_jp}; \
     fi; \
     if [ -n "${PIP_PACKAGES}" ]; then \
       _jp=""; \
-      case " ${PIP_PACKAGES} " in *" arcadedb-embedded"*) _jp="jpype1==${JPYPE_VERSION}";; esac; \
+      case " ${PIP_PACKAGES} " in *" arcadedb-embedded"*|*" arcadedb-jvm"*) _jp="jpype1==${JPYPE_VERSION}";; esac; \
       pip install --no-cache-dir ${PIP_PACKAGES} ${_jp}; \
     fi; \
     if python -c "import jpype" 2>/dev/null; then \

benchmarks/experiments/Dockerfile.client-arcadedb

@@ -0,0 +1,13 @@
+# The image the served ArcadeDB client cells run in: the shared `dbbench:client` image plus the official `arcadedb-driver` at the release the
+# harness pins (lean_http.OFFICIAL_PIN, DECISIONS #258, #259) and the driver's own dependencies, each pinned.
+#
+# WHY A LAYER AND NOT A NEW PIN IN `client`. dbbench:client carries every comparator's client library, and rows made on it are already
+# measured. Changing the arcadedb-driver pin in PKGS[client] (a build arg) would rebuild that image under a new digest and split those rows.
+# A layer on top leaves dbbench:client byte-identical; only the served ArcadeDB arms (runner.py: ARCADEDB_CLIENT_IMAGE) run here.
+# `pip install ==` of the whole list in one command leaves the resolver no room to move any of them (httpx is unpinned in the driver's
+# own requirements, websockets>=14).
+FROM dbbench:client
+ARG DRIVER_PACKAGES=""
+RUN set -e; [ -n "${DRIVER_PACKAGES}" ] || { echo "DRIVER_PACKAGES is empty" >&2; exit 1; }; \
+    pip install --no-cache-dir ${DRIVER_PACKAGES}; \
+    python -c "import importlib.metadata as m, sys; bad = [p for p in sys.argv[1:] if m.version(p.split('==')[0]) != p.split('==')[1]]; sys.exit('not at the pin: ' + ', '.join(bad) if bad else 0)" ${DRIVER_PACKAGES}