Repository navigation
Vertex-type loads from insert_dataframe and insert_arrow go through per-row dicts and JSON in both wheels #319
Description
This comes from a read-only audit of the tag 26.11.1 (commit 2f28a328be); everything below marked read was seen in the code with the file and line, nothing was run or timed, and no speed claim is made. Related closed work that this builds on and does not repeat: #229 (native point reads), #231 (native insert_many ragged lists), #235 (JVM insert_many cyclic GC), #243 (JVM GraphBatch property shapes), #174 (commit sizes).
Read: loading a vertex type through insert_dataframe or insert_arrow builds per-row dicts and JSON in both wheels. The Java column inserters already create vertices (BulkOps.java:140, DocumentBatcher.java:115), and the returned RIDs are discarded anyway (dataframe.py:483).
Proposal: route vertex types to insert_columns in insert_table. Two points to settle first: whether the index and bucket results are the same as through graph_batch (I did not check), and the rule that column inserts refuse to run inside a caller's transaction.
Measure first: a 1M-row vertex load through insert_dataframe, rows per second and CPU per row, warmed, before and after. Done when the vertex-type load is columnar or the reason it cannot be is written here.
Activity
insert_dataframe and insert_arrow: a vertex type loads through the co…
insert_dataframe and insert_arrow: a vertex type loads through the co…
insert_arrow and insert_dataframe on the native wheel: a vector colum…
tae898 commentedon Nov 12, 2026
This is the A/B I ran on 2026-11-10 for this issue, and I read it as confounded, not as a result. It ran on my laptop (Intel Core Ultra X9 388H, Linux 7.0.0, Python 3.12, two cores). A is the released
26.11.1wheel of each family and B is the candidate wheel of each (CI builds of therelease/26.11.1line before26.11.1.post1, sha256 starting144091c9for native and7cc57106for JVM), which have PR #335 in them. B is not the published26.11.1.post2wheel: PR #335 is on main and in the26.11.1.post2tag, and I did not re-run this on the published files.Each cell loads a vertex type through
insert_arrow(e2: 2 calls of 50,000 rows with a 64-dimension vector; l3d: 10 calls of 10,000 rows with a 96-dimension vector) orinsert_dataframe(g1: 20 calls of 5,000 rows, 4 columns), 100,000 rows per cell, 14 clean pairs. Below 1.00 means B is faster.insert_arrow, e2insert_arrow, l3dinsert_dataframe, g1insert_arrow, e2insert_arrow, l3dinsert_dataframe, g1The two builds do not run the same durability, which is why no row is a verdict. A goes through
GraphBatchwith its default of the write-ahead log off, and commits per 1,000 rows on native and per 100,000 on JVM; B goes through the columnar insert with the database's own log settings (log on) and commits per 10,000 rows, as PR #335 says. The harness recorded that route for every cell, so the rule I set before the run marks all six as DURABILITY-CONFOUNDED and never as a win or a regression.On the JVM wheel the log has since been matched for the same shapes, in the A/B I posted on #348 (#348 (comment)): the log on in both arms or off in both, 12 clean pairs, the released wheel against the #348 fix that
26.11.1.post2carries. The vertex loads with a float32 vector column take 0.234 and 0.258 of the time (e2, log off and on) and 0.157 and 0.180 (l3d), andinsert_dataframeof the plain 4-column rows (g1, information only) takes 0.686 and 0.681. The memory on the vector shape stays higher and is open on #348.Not measured: the native wheel with the log matched in both arms (the harness could not read the native database's log flag, so the 0.090 and 0.073 above are not a result either), and the 1,000,000-row
insert_dataframeload this issue asks for, because every cell here holds 100,000 rows. I am leaving the issue open for those two runs.tae898 commentedon Nov 14, 2026
Status. #319's change is #335:
insert_dataframeandinsert_arrowload a vertex type through the columnar insert on both wheels. It is in26.11.1.post1and26.11.1.post2(merged 2026-11-10), and post2 also carries the JVM vector fix of #348. The two points this issue asked me to settle are settled in #335: the vertex route was read againstGraphBatch.createVertices, andtests/test_vertex_load_columnar.pycompares the loaded vertices with that route (every stored property, per-bucket record counts and index lookups). A vertex load still refuses to run inside a transaction the caller opened. One behaviour change is stated in #335: the write-ahead log is now on for a vertex load, as the database is configured.Measured: on the JVM wheel with the write-ahead log matched in both arms (the fix of #348 against
26.11.1, 12 clean pairs, B/A of time),insert_arrowe2 is 0.234 [0.213, 0.254] with the log off and 0.258 [0.230, 0.333] on, and l3d is 0.157 [0.149, 0.174] and 0.180 [0.175, 0.188]. The post1 candidate cells are DURABILITY-CONFOUNDED and carry no verdict.Not measured: the native wheel with the log matched (my harness cannot read the native database's log flag), and the 1,000,000-row
insert_dataframevertex load that this issue asks for. Memory on the JVM vector shape is still 1.3 to 1.4 times the old route and is open on #348. The issue stays open for those two runs.