Repository navigation
Native ingest: one datetime, date, decimal or binary column moves the whole batch off the columnar insert path #316
Description
This comes from a read-only audit of the tag 26.11.1 (commit 2f28a328be); everything below marked read was seen in the code with the file and line, nothing was run or timed, and no speed claim is made. Related closed work that this builds on and does not repeat: #229 (native point reads), #231 (native insert_many ragged lists), #235 (JVM insert_many cyclic GC), #243 (JVM GraphBatch property shapes), #174 (commit sizes).
Read: the native columnar entry only has the column kinds q d b i e f s (_native/columnar.py:8-10). One column of another kind in insert_arrow, insert_dataframe or insert_many makes every column of that batch go through per-row Python dicts, a per-row validation loop and json.dumps (dataframe.py:364-369, :517-532, _native/bulk.py:57-71). Rows of agent memory almost always have a timestamp, so I expect this to be the common case, not an edge.
Proposal: add datetime and date kinds (and, if the engine's conversion allows it, decimal and binary) to BulkOps.java and columnar.py, so those columns stay columnar. It needs a rebuilt native image from the unchanged engine, so it is a wheels-only release only if the maintainers count that as wheels-only; I did not verify a bindings-only workaround.
Measure first: ingest the same table with and without one timestamp column, rows per second and CPU per row, warmed, before and after. Done when the timestamp case is within the rule of the all-numeric case or the remaining gap is explained.
Activity
Native columnar insert: date-time, date, decimal and bytes columns st…
Native columnar insert: date-time, date, decimal and bytes columns st…
insert_arrow and insert_dataframe on the native wheel: a vector colum…
tae898 commentedon Nov 12, 2026
Here is the figure I have for this issue, from the timed A/B I ran on 2026-11-10 on my laptop (Intel Core Ultra X9 388H, Linux 7.0.0, Python 3.12, two cores). A is the released
arcadedb==26.11.1wheel and B is the candidate wheel of the same engine (a CI build of therelease/26.11.1line before26.11.1.post1, sha256 starting144091c9), which has PR #333 in it. B is not the published26.11.1.post2wheel: PR #333 is on main and in the26.11.1.post2tag, and I did not re-run this on the published file.The cell is
insert_arrowof a 10-column table (6 numeric columns, 3 string columns, andl_shipdateas an Arrowdate32), 4 calls of 122,880 rows after 1 warm-up call, 14 clean pairs. The rule was written before the first run: A and B alternate in paired rounds, B/A is the median of the paired ratios with a bootstrap 95 percent interval, and below 1.00 means B is faster. A WIN needs the upper end of the interval below 0.90.On 26.11.1 that one date column sent every column of the batch down the per-row path, and on B it stays columnar: 12.108 s to 3.825 s for the 491,520 rows, about 40.6 thousand to 128.5 thousand rows per second, B/A 0.316 [0.314, 0.318] (WIN). A against itself in the same rounds gave 1.001 [0.999, 1.004], so I do not think the harness moved it.
Two things this does not show. The issue's "done when" compares the timestamp case with the all-numeric case, and this run compares B with 26.11.1, so I do not yet know how far the date table is from the same table with no date column. B also holds more memory: peak RSS 718.3 MiB to 806.6 MiB (B/A 1.128 [1.121, 1.193]) and anonymous memory (
RssAnon) B/A 1.394 [1.362, 1.403], process readings that the rule does not gate on.Still missing: the same table through the same call with the date column as a plain numeric column and as
date32, with the memory, on the published26.11.1.post2wheel. numpydatetime64and mixed-type columns still take the row path and are not in this run. I am leaving the issue open for that run.tae898 commentedon Nov 14, 2026
Status. The native side is fixed in code: #333 adds date-time, date, decimal and bytes column kinds to the columnar insert, so such a column no longer sends the whole batch down the per-row path. It is in
26.11.1.post1and26.11.1.post2(merged 2026-11-10). The JVM counterpart, a timestamp or date column as onelong[], is #383, and it is in26.11.1.post3(on PyPI since 2026-11-14). In the timed A/B of the CI-built post3 wheel against26.11.1.post2,insert_arrowwith timestamp and date columns took 0.295 [0.283, 0.311] of the time andinsert_dataframetook 0.283 [0.278, 0.290] (20 clean pairs each, A/A control 0.998 [0.979, 1.015] and 0.993 [0.976, 1.004]).The figure above (12.108 s to 3.825 s, B/A 0.316 [0.314, 0.318], 14 clean pairs, WIN by my rule) is of a post1 candidate wheel, not of the published file. This issue's "done when" compares the timestamp case with the all-numeric case, and that is not measured yet.
What remains: the same table through
insert_arrowwith the date column asdate32and as a plain numeric column, with memory, on the published26.11.1.post2native wheel. numpydatetime64and mixed-type columns still take the row path and are covered by no figure. The issue stays open for that run.