[tae898]

Stacked on #330 (perf/318-single-pass-validation): this branch is cut from its head, because both PRs edit classify_vertex_rows, split_float_vectors and GraphBatch.create_vertices. The base is release/26.11.1 as for the other stacked PRs, so the diff here also shows the two commits of #330 until it merges. Merge order: #330 first.

Closes #337

I searched first: no open PR or issue does this. Issue #220 and PR #221 (JVM float[] as a candidate in type_conversion.py only, closed on 2026-11-05 as not planned after two GRAY timed runs) are the closest. This PR is the same idea for the JVM wheel plus the part #221 did not touch (a float32 array on a property that is not declared, and the whole native encoding), with no speed gate: the issue says every glue drawback gets fixed.

What I found (step 1, read from the code at 26.11.1 and at the #330 head)

GraphBatch.create_vertices on the JVM wheel, per value type, before this change:

row value	property declared ARRAY_OF_FLOATS	property not declared
Java float[] (to_java_float_array)	not a candidate (is_vector_candidate is list or ndarray), classify_vertex_rows returns None, the whole input goes through _to_java_property_matrix	same
numpy float32	candidate, split_float_vectors takes it as one float32 matrix per chunk	not a candidate (key in declared is the only test), whole input through _to_java_property_matrix
numpy float64	candidate, matrix, cast to float32 (documented; the engine stores floats for that type)	whole input through _to_java_property_matrix, stored as a list of doubles
list of floats	candidate, matrix, cast to float32	whole input through _to_java_property_matrix (one boxed double[] per row)
ragged, None, missing	the split declines the column, the chunk's text fails (an array is no JSON), that chunk goes through _to_java_property_matrix	same

A correction to the hypothesis in #337: the 26.11.1 scan in _create_vertices_json_bulk does not read every row for these shapes. It returns None at the first Java float[] it meets, which is the first row, so it costs one row. The cost is in _to_java_property_matrix.

Where the 59 to 61 percent go. _to_java_property_matrix runs once for the whole input (not per chunk) and, for each row, does a Python loop over the properties (_to_java_varargs: convert_python_to_java for every value, about ten type tests each), jpype.JArray(jpype.JObject)(row) (JPype converts the 2K-element list: a Java String per key, a boxed Long or Double per scalar), and finally assigns the N Object[] one by one into an Object[][]. In the untimed 26.11.1 profile (cells e2 and l3d, variant cv) the phase counters do not cover this function: boundary (the engine's createVertices alone) is 16.5 to 20.2 percent, the harness's own row building (caller, one to_java_float_array per row) 21.5 to 22.8 percent, and the remainder, rest, is 59.2 and 60.6 percent. So the split inside _to_java_property_matrix is read from the code, not timed: I did not time its parts separately. Peak memory is also bigger on that path, because the Python lists, the JArray wrappers and the Java Object[] of the whole input exist at once; the profile has no memory row for these two shapes, so the post1 A/B should take it (commands below).

NativeGraphBatch.create_vertices on the native wheel (the library does not look at declarations; to_java_float_array returns a float32 ndarray there, so there is no Java array):

row value	before
float32 ndarray, 1-D, non-empty	encode_gb_args: _prep(args) walks every value of every row in Python, json.dumps calls the hook _gb_default then _default for every array (about a dozen isinstance tests, one ascontiguousarray and one append per row), one {"\u0001":["B",offset,count]} tag per row is written and parsed back in Java (a HashMap, an ArrayList and a float[] per row), np.concatenate joins one small array per row, then tobytes() copies it again. Plus a [r or {} for r in chunk] per row. This is the 29.0 and 31.3 percent prep of the profile
float64 ndarray, list	number text through .tolist() / json.dumps

What changed (step 2)

• src/arcadedb/type_conversion.py: take_vector_columns(rows, declared, undeclared=False) is the one split of both wheels. A declared column qualifies as before and now also with a Java float[] or double[]. With undeclared=True any other property qualifies when its value in every row is a float32 numpy array or a Java float[] of one length (the stacked column must come out as float32, so the stored value, a float[], is what the property-matrix path stored; a float64 array, a list or a Java double[] on an undeclared property keeps the old path and its doubles). is_vector_candidate and classify_vertex_rows know the Java arrays (_java_vector_arrays, resolved once the JVM runs, () on the native wheel). _without_keys rebuilds the rows without the vector columns from columns by C iterators (itemgetter, zip, dict) when the rows share their keys in one order, and by the dict comprehension otherwise. split_float_vectors keeps its signature and return value (insert_many uses it with the declared columns only, so insert_many now also takes a Java float[] column through the matrix).
• src/arcadedb/graph_batch.py: _create_vertices_json_bulk asks for the undeclared float32 columns too; docstrings.
• src/arcadedb/_native/codec.py: encode_vertex_args writes the rows without the vector columns with one json.dumps (no hook and no _prep when the other values are plain scalars, decided by a census in C: _plain_rows), puts the matrices at the front of the side buffer and describes them by one "vecs": {"names", "dims", "offs"} entry. The buffer is a ctypes view of the matrix, not a copy (bytes when the array is read-only).
• src/arcadedb/_native/extras.py: NativeGraphBatch._create_chunk uses it; the per-row [r or {} for r in chunk] only runs when a row is None.
• native/src/main/java/org/humemai/nativelib/NativeExtras.java: withVectors turns the "vecs" entry into one float[] per row (bounds-checked against the side buffer) before GraphBatch.createVertices. The ["B", offset, count] tag stays for edge properties and for a float32 column that does not qualify; such a tag then follows the matrices in the buffer. The Java is not compiled on my laptop; the native CI job builds it.
• docs/modules/ROOT/pages/how-to/vectors.adoc: the two paragraphs on create_vertices vectors.

Stored values are the same float32 values on every path: the matrix carries the float32 bits and Matrix.row(i) copies them. A library that predates the "vecs" entry cannot meet this Python (the library and the Python ship in one wheel); lib.GB_BIN false keeps the text path.

Tests (step 3)

CI is my only run environment (a repo hook blocks pytest and java locally), so none of the tests below ran on my laptop. Locally I only ran the pure functions (take_vector_columns, _without_keys, _plain_rows, encode_vertex_args, and a ctypes c_char array as a c_char_p argument) in a plain script, and the lint gates: compile(), pyflakes (its five warnings in extras.py are on the base too), black 26.10.1, isort 9.0.2, bandit==1.9.4 (clean).

• tests/test_vertex_vector_loads.py (both wheels, to_java_float_array is the shared call): stored bits for Java float[] (JVM) or float32 ndarray (native), numpy float32, declared and not, 64 and 96 dimensions, with -0.0, the largest, the smallest normal and the smallest subnormal; the matrix against the per-vertex path; numpy float64 and lists (documented values); ragged, None, missing and a None row; a float64 row among float32 rows; other properties and different key orders; two vector properties and a date-time in one call; a load of chunk + 3 rows (the JVM chunk is 100,000, the native chunk is commit_every); insert_many; and the split itself without a database.
• tests/test_graph_batch_vector_crossings_jvm.py (JVM only): wraps VertexBatcher, FloatVectors, convert_python_to_java and _to_java_property_matrix. For 250 rows in three chunks of 100 it asserts no fallback, no per-value conversion, ["createVerticesJson"] * 3 and one Matrix per property and chunk, for a Java float[] and a float32 array, declared and not; a ragged row sends only its chunk to the fallback; a Java double[] is a vector only when declared; insert_many does no per-row insert.
• tests/test_native_vertex_vector_crossings.py (native only): spies on lib.call and on codec._gb_default, codec._default and codec._prep: three arcadedb_n_gb_bin crossings for three chunks, none of the three hooks ever called, no tag and no number in the text, the buffer is a ctypes view, the "vecs" offsets for two columns, a non-qualifying float32 array following the matrices, the text entry for float64 and lists and for GB_BIN false, and a "vecs" entry outside the buffer refused.
• tests/test_native_graph_batch_binary_vectors.py: the one assertion that expected a "B" tag per vertex row now expects "vecs".
• tests/conftest.py: the two spy files are registered (test_graph_batch_vector_crossings_jvm.py in NATIVE_NOT_APPLICABLE, test_native_vertex_vector_crossings.py in the native-only list).

What each one proves fails on the base (stated from reading the code; the run of these files on the base of #330 is linked under CI below and I have not read its result when writing this): every JVM spy case that holds a Java float[] or double[] (the matrix cases, the ragged case, the two-property case, insert_many) fails on fallback == [] or on the per-row insert count, the float32 case on a property that is not declared fails the same way, and the same case on a declared property passes on the base (it is the control); the native spy test fails on the hook counts (the base calls _gb_default once per row); the split tests fail because take_vector_columns does not exist. The value tests pin equality with the old path and are expected to pass on both.

Profile commands for the post1 A/B (step 5, nothing timed here)

Untimed phase and memory profile of the ingest shapes, cells e2 and l3d, variant cv, one fresh process per cell, 3 repetitions, once per arm (the 26.11.1.post1 wheel with this change against the one without it), both wheels:

taskset -c 8-11 uv run --no-project --offline --python 3.12 --with numpy==2.5.4 [--with JPype1==1.7.1 for the JVM wheel] \
  python -I profile.py --tree <unzipped wheel> --wheel {jvm,native} --shape {e2,l3d} --variant cv --db <fresh dir> --rep <n> --out <cell>.json
python score.py   # the phase table (caller, prep, validate, boundary, commit, rest) and the memory section (anon MiB, B/row, RSS)

The figure to read is rest on the JVM wheel (was 59.2 and 60.6 percent) and prep on the native wheel (was 31.3 and 29.0 percent), plus anon_MiB and B/row per cell.

CI (the humemai fork, because the upstream runner queue is backed up)

• native, test-native.yml with version=26.11.1.post1: https://github.com/humemai/arcadedb-python/actions/runs/42616214876
• JVM, test-jvm.yml with arcadedb-tag=26.11.1: https://github.com/humemai/arcadedb-python/actions/runs/42616219493
• examples, test-python-examples.yml with arcadedb-tag=26.11.1: https://github.com/humemai/arcadedb-python/actions/runs/42616224280
• the new test files on the unchanged insert_many and create_vertices: check the rows in the one json.dumps pass, not in a loop before it (refs #318) #330 head (branch perf/337-tests-on-base, JVM only; the red cases there are the proof that they fail on the base): https://github.com/humemai/arcadedb-python/actions/runs/42616346182

🤖 Generated with Claude Code