Repository navigation
JVM wheel: string and object columns cross JPype one element at a time on ingest #324
Description
Source for the practice: a literature and documentation pass of 2026-11-09 (the Arrow C Data and C Stream interface specifications, the ADBC documentation, the DuckDB, Polars and psycopg documentation, the JPype user guide, and Raasveldt and Muehleisen, VLDB 2017, on result-set transfer); the exact list of 30 sources is in my notes and I will attach it on request. Everything about our code is read at the tag 26.11.1 (commit 2f28a328be); nothing was run or timed, so there is no speed claim. Part of #315.
Read: on the write side the JVM wheel sends string and object columns across the JPype boundary element by element, which the JPype guidance names as the slow way. Numeric columns already go as JArray copies, which JPype documents as the fastest route, and the native wheel already sends strings as UTF-8 plus int32 offsets.
Proposal: send a string column as one UTF-8 blob plus an int32 offsets array and split it on the Java side in the bridge jar. This changes bridge Java, so it ships with a rebuilt wheel from the same engine commit (a post1 if wheels-only releases are allowed to rebuild the bridge jar).
Measure first: ingest a table with two string columns, rows per second and CPU per row, warmed, before and after; also check insert_many against the same table so the gain is not only a micro-benchmark.
Activity
Tests: a lone surrogate keeps the column off the blob and is refused …
JVM insert_columns: a column of str and None crosses as one UTF-8 blo…
tae898 commentedon Nov 12, 2026
Here is the figure for this issue, from the timed A/B I ran on 2026-11-10 on my laptop (Intel Core Ultra X9 388H, Linux 7.0.0, Python 3.12, two cores). A is the released
arcadedb-jvm==26.11.1wheel and B is the candidate wheel of the same engine (a CI build of therelease/26.11.1line before26.11.1.post1, sha256 starting7cc57106), which has PR #334 in it. B is not the published26.11.1.post2wheel: PR #334 is on main and in the26.11.1.post2tag, and I did not re-run this on the published file.The cell is
insert_columns(..., parallel=True)of a 10-column table with 4 string columns (4 calls of 122,880 rows after 1 warm-up call), with a control that makes the same call on the 6 numeric columns only. The rule was written before the first run: A and B alternate in paired rounds, B/A is the median of the paired ratios with a bootstrap 95 percent interval, and below 1.00 means B is faster. A WIN needs the upper end of the interval below 0.90.With the strings crossing as one UTF-8 blob plus offsets the cell drops from 1.596 s to 1.250 s over 14 clean pairs, and its peak RSS B/A is 0.893 [0.817, 0.991] (a process reading, not gated). The numeric-only control also moved, which I did not expect and have not explained, so part of the 0.779 may not come from the string path.
insert_manyon the same table (information only, no interval) went from 3.577 s to 3.385 s, a ratio of medians of 0.946; this PR does not change that path, so I do not count it as a gain. I measured wall time, not CPU per row.I am leaving the issue open for what the PR did not touch. Object columns that hold anything but
strandNone(ints, bytes, date-times, or lists mixed in one column) still convert one element at a time, and that needs a Java change. String columns ofappend_sampleson the JVM wheel still cross as aJArray(JString)built from a Python list, and Arrow and DataFrame string columns are first turned into Python lists byto_pylist()before the blob is built, so both paths still do per-element work.tae898 commentedon Nov 14, 2026
Status. #334 sends a JVM string column (
strandNone) across JPype as one UTF-8 blob plus int32 offsets. It is in26.11.1.post1and26.11.1.post2(merged 2026-11-10).Measured, on a post1 candidate wheel and not on the published file:
insert_columns(parallel=True)with 4 string and 6 numeric columns went from 1.596 s to 1.250 s, B/A 0.779 [0.756, 0.813], 14 clean pairs, A/A 1.000 [0.975, 1.026], WIN by my rule. The numeric-only control also moved, 0.930 [0.911, 0.993] (GRAY), which I have not explained, so part of the gain may not come from the string path.insert_manyon the same table (information) went from 3.577 s to 3.385 s, which this PR does not change.Open: object columns that hold anything other than
strandNone, the string columns ofappend_samples, and Arrow and DataFrame string columns, which are still turned into Python lists byto_pylist()before the blob is built. Not measured: CPU per row, and the movement of the control. The issue stays open for those.