Repository navigation
Goal: high-rate ingest and large reads in both wheels: measures, audit results and the work list #315
Description
A goal for the Python bindings: bulk ingest and large reads should be fast and use little memory in both wheels, with the standard interfaces where they exist (Arrow C Data Interface, ADBC). This issue holds the measures and links the work.
Measures, each per wheel and warm: rows per second and CPU per row for insert_many, insert_columns, insert_arrow, insert_dataframe, graph batch and time-series ingest; seconds and peak memory per phase for large reads to Arrow and dataframes; the one-row call cost. Method: ab.py ABBA with an A/A control, a warmed process, the same SQL text for every call, four cores, per-phase memory from the cgroup, and the interval rule fixed before the run (a study whose intervals are wider than the rule is rerun and not reported).
Work, by issue: #312 (native large reads have no columnar read path), #313 (JVM cold start and idle memory), #314 (the two wheels return different floats from to_json_list), and the ingest findings below. Not to be redone: the closed issues listed in each of them.
Order: the cheap bindings-only changes first (a wheels-only 26.11.1.post1 is acceptable), then the changes that need a native image rebuild. Each change is measured against the released 26.11.1 with the same method before it is claimed.
Activity
tae898 commentedon Nov 9, 2026
Ranked work from the best-practice pass (the table of our paths against the practice has 18 rows, and every item below carries a measure and uses the method of this issue). In order: native Python-side read assembly (Python only, half a day to a day); native columnar read frames (3 to 5 days plus image rebuilds, with a decoder shared with the JVM wheel); reply handling without text copies; a batched Arrow reader and
__arrow_c_stream__(1 to 2 days, Python only); closing the native ingest fallbacks (temporal, bytes and Decimal kinds, vertex types) which needs an image rebuild; then the JVM string blob and offsets, a commit-size sweep, and a PEP 249 shim.Three decisions that are the owner's, not mine. First, whether the native library exports the Arrow C Stream interface itself (hand-written structs and a release callback under native-image, an ABI commitment, no speed reason) or only a Python-level
__arrow_c_stream__; I recommend the Python-level one now. Second, whether to ship an ADBC driver, which is cheap only after the first option and carries a support commitment; the alternative is the PEP 249 shim plus__arrow_c_stream__, which covers most users. Third, whether to keep the JSON read path for the row APIs throughpost1and add the columnar frames beside it; I recommend keeping it. A smaller point only if the commit-size sweep shows a clear curve: the nativeinsert_manydefault of 1000 rows per commit is a failure-atomicity decision, not a tuning change.tae898 commentedon Nov 9, 2026
The owner answered the third decision above: no legacy burden. So the JSON read path is retired once the columnar frames exist, not kept beside them: one transport, and the row APIs build their dicts from typed buffers. That also forces one choice for the float32 rendering of #314, which I read as the exact float64 widening on both wheels. Dead and fallback paths go (for example
NativeGraphBatch._create_vertices_json_bulkhas no caller). Placement: the behaviour-neutral and parity changes can go in a wheels-only26.11.1.post1; the redesign (single columnar transport, JSON read path retired,__arrow_c_stream__) goes into the next regular release, with a changelog entry that says what changes for users.tae898 commentedon Nov 9, 2026
Design principle for the redesign, so the two wheels do not drift: the interface and the format are shared, the plumbing is per wheel. Shared: the public API and types, one value semantics (including the float rule of #314), one frame format (typed column buffers laid out like Arrow), one Python decoder that wraps a buffer (
np.frombuffer,pyarrow.foreign_buffer), and the golden parity test that runs on both wheels. Per wheel: the transport (the native wheel calls the C library throughctypeswith buffers in native memory, the JVM wheel crosses through JPype with Java arrays or direct byte buffers), the crossing cost model (cheap on native, expensive through JPype, so the JVM side batches more), ownership and freeing (a release step on native, the garbage collector on the JVM), and the memory and start-up tuning (GC and heap flags on native; the heap fraction, a CDS archive and the AOT cache on the JVM).Release placement: the small, clearly better cuts (the float semantics, dead code) can go into
26.11.1.post1, which is the version the affected paper cells are rerun on (the four time-series cells that read largeto_arrowresults); the redesign goes into a26.11.1.post2or the next regular release, so the paper's rows do not wait for it. Every cut still needs a measured win under the method of this issue and a changelog line.tae898 commentedon Nov 9, 2026
Memory is a first-class measure of this work, not an afterthought, for three reasons. The benchmark's memory columns for the embedded cells are the cell's owned memory (anonymous plus shared), and that includes the Python process and the glue, so a lighter glue lowers a published number and a heavier one raises it. The big costs the audits found are memory costs too: a million Python dicts for one
to_arrowread, rows rebuilt from columns on ingest, and a reply copied three times. And the two wheels start from different baselines (the JVM wheel's idle memory is a constant 54 to 66 MiB higher, and its default heap is half of the machine's RAM).Requirements for every glue change, next to the speed measure: report the peak and the steady resident memory, anonymous and file-backed separately, and the bytes copied per call; take the memory per phase from the cgroup inside the cell, not from the process high-water mark (the first study of this work used a last-cell high-water mark and that figure was withdrawn); bound the working set by design (batches of about 1 MB, a reader that yields batches so a read does not materialise the whole table, no intermediate row form for columnar paths); and accept a change that raises peak memory only with the reason written in the pull request. For the native wheel the young-generation sizing of the Serial GC is part of this (the eden issue I reported as oracle/graal#14758), and for the JVM wheel the heap fraction and a CDS archive are.
tae898 commentedon Nov 13, 2026
Scope update: I am widening this goal from ingest and large reads to every create, read, update and delete path of both wheels, with the same method for each (a timed A/B with an A/A control, the interval rule fixed before the run, no claim from a run that ends weak).
What I found while taking stock today:
26.11.1.post1and26.11.1.post2. The figures I posted on Native wheel: large results are slower than the JVM wheel (to_arrow 1M about 3.4x): profile and fix #312 were measured on26.11.1, before that change, and I corrected that comment.26.11.1.post2, the pair people install now. Every valid cross-wheel figure is on26.11.1. The row readers people call most (to_list,iter_dictsand the others) have never produced a valid figure on any pair.Order of work:
26.11.1.post3is cut from a pinned candidate on a release branch, never from a moving main. Every study compares the released post2 wheel (A) against the post3 candidate (B) and post2 again (Z), native against native and JVM against JVM. It ships when the touched figures show no regression and the improvement it claims.I will update this issue as each study finishes.
Tests: the nesting depth the native mark reports is the engine's, and…
Tests: the nesting depth the native mark reports is the engine's, and…
tae898 commentedon Nov 13, 2026
The first timed A/B of the post3 candidate against the released
26.11.1.post2is finished. A is the released post2 wheel, B is the candidate (post2 plus the draft PRs #382, #383, #384, #385 and #386, built locally in Docker from one integration commit, engine unchanged), and Z is post2 again as the A/A control. Native is compared with native and JVM with JVM. It ran on one laptop (cores 6 and 7), ABBA rounds, one fresh process per run, rounds 0 to 20 with round 0 dropped. The ratio is B/A of the time per operation, so below 1 means the candidate is faster. The decoded and stored values of every figure were compared between the arms: no parity failure.Figures that pass the study's rule (at least 10 clean pairs, an interval no wider than 0.10 times the median, a clean A/A control), all named improved:
iter_dicts, 1 rowiter_dicts, 100 rowsiter_dicts, 100,000 rowsinsert_arrow, timestamp and date columnsinsert_dataframe, timestamp and date columnswith db.transaction(), empty bodywith db.transaction()with one INSERTsetandsave()inside an explicit transaction, auto-transaction off (the default)No change: JVM
insert_columns(1.002 [0.982, 1.026]) and nativeinsert_columnsof a vector column in chunks, the figure for #386 (1.000 [0.991, 1.008]). Not reported because the interval is still too wide at the round cap: the JVMto_listand typed-to_listfigures at 1 and 100 rows, and the JVMto_listat 100,000 rows is unresolved. So I claim nothing forto_listfrom #382, only foriter_dicts, and the no-regression line of the study is not yet certified for those figures. The control figure of the native wheel (a point lookup that none of the PRs touch) moved by 1.007 [1.004, 1.012], which the study reports as drift and which voids nothing.What I do with it: #386 shows no measurable gain, and its change adds an integer to every native command, so I leave it out of post3. The other four go into the candidate. The release gate repeats this study on the wheels that CI builds from the tagged commit, with a higher round cap because the small-result
to_listfigures are noisy per process.Edited 2026-11-14: the one-row update figure ran inside explicit transactions with auto-transaction off, the default, and not with auto-transaction on as this text first said. It therefore does not time the path that #385 changes (#385 only changes
save()with auto-transaction on, see its own crossing table); the figure measures the post3 build on the default path, and its gain may come from #384. A timed figure for the auto-transaction-on path is still missing.tae898 commentedon Nov 14, 2026
The release gate of
26.11.1.post3is finished. It is the same study as before (A is the released26.11.1.post2wheel, B is the candidate, Z is post2 again as the A/A control, native against native and JVM against JVM, one laptop on cores 6 and 7, ABBA rounds, round 0 dropped), now on the wheels that CI built from the release-branch commit55b427a(Test JVM run 43520459781, Test native run 43520464492), not on my local build. 504 runs were kept, rounds 0 to 20. The decoded and stored values of every figure matched between the arms: no parity failure. The engine of both candidate wheels is byte-identical to the one of26.11.1(JVM: 70 of 70 engine jars; both wheels: build numbera3a5d124572608412aa187f9c4b290e071699473).Eight figures pass the rule of the study (at least 10 clean pairs, an interval no wider than 0.10 times the median, a clean A/A control) and are named improved. The ratio is B/A of the time per operation, so below 1 means the candidate is faster:
iter_dicts, 1 rowiter_dicts, 100 rowsiter_dicts, 100,000 rowsinsert_arrow, timestamp and date columnsinsert_dataframe, timestamp and date columnswith db.transaction(), empty bodywith db.transaction()with one INSERTsetandsave()inside an explicit transaction, auto-transaction off (the default)No figure is a regression or slower: the summary line reads IMPROVED 8, FASTER 1, SAME 3, UNRESOLVED 5, REGRESSION 0, SLOWER 0, PARITY-FAIL 0, WEAK 4. JVM
insert_columns(1.009 [0.983, 1.050]) and the native vectorinsert_columns(0.996 [0.990, 1.010]) show no change. The JVMto_listand typed-to_listfigures at 1 and 100 rows are still too noisy at the round cap, so I report nothing for them and claim nothing forto_list. The native control (a point lookup no PR touches) moved by 0.992 [0.989, 0.995], which the study reports as drift and which voids nothing.On memory I claim no improvement. The peak resident size is only an information figure in this study. Its B over A ratio is 0.933 [0.871, 0.968] for
iter_dictsat 100,000 rows and 0.918 [0.917, 0.919] for the empty native transaction. For JVMinsert_arrowit is 1.066 [1.014, 1.194] and forinsert_dataframeit is 1.202 [1.059, 1.299], so the faster timestamp path holds more memory at peak, which I have not looked into yet and will not call a gain.26.11.1.post3is tagged on55b427aand is on PyPI for both packages (arcadedbandarcadedb-jvm, published 2026-11-14). I ran the engine identity check on the published linux x86_64 wheels of both: the engine part is byte-identical to26.11.1(70 of 70 engine jars on the JVM wheel, build numbera3a5d124572608412aa187f9c4b290e071699473on both). #386 stays out of it.Edited 2026-11-14: the one-row update figure ran inside explicit transactions with auto-transaction off, the default, and not with auto-transaction on as this text first said. It therefore does not time the path that #385 changes (#385 only changes
save()with auto-transaction on, see its own crossing table); the figure measures the post3 build on the default path, and its gain may come from #384. A timed figure for the auto-transaction-on path is still missing.GraphBatch (JVM): create_vertices and new_edges send rows of exact sc…
Tests: the frame kinds of RID and number-list columns, and the stored…
Tests: fix the stand-in and three expectations of the native GraphBat…
tae898 commentedon Nov 14, 2026
I corrected two of my comments above (the first A/B result and the release-gate result). The one-row update figure (
setthensave()) was labelled "auto-transaction on", but the study runs it inside explicit transactions with auto-transaction off, the default. So it does not time the path that #385 changes, and its gain may come from #384; I changed the label and added a note at the end of each comment. A timed figure forsave()with auto-transaction on is still missing, and I make no speed claim for #385. The figures and intervals are unchanged.Docs for 26.11.1.post3: the four glue changes, their measured figures…