[tae898]

For users the values do not change: to_arrow, to_columns, to_dataframe and to_polars of the native wheel return what they returned, now read from typed column frames in the library (the layout the JVM wheel's ColumnBatcher writes) by the same Python decoder as the JVM wheel, without one Python dict per row. A few behaviours are made the same on both wheels on purpose, and I list each below. This is a draft: the CI evidence is not in yet, and I make no speed claim (the measurement is the read-path profile study and a later A/B).

Refs #312 (part of #315). The base is release/26.11.1 for 26.11.1.post1. #326 (the cherry-picks of #320 and #321) is still open, so tests/test_cross_wheel_golden.py and the #321 version of columns.py are not in this base. This branch deletes columns.py (nothing calls it any more), which will conflict with #326 in that file only: take the deletion. The golden test must pass on both wheels once #326 is in; I will rerun after it merges.

How it works:

• arcadedb_n_next_col (ArcadeNative.java, encoder in the new ColumnFrames.java, signature in _native/lib.py) returns one frame of the cursor. A frame ends at batch_size rows or when what it will hold reaches about 1 MiB, counted as the larger of the values read and rows times columns times 9 bytes, so a schemaless result with another property in every row does not allocate gigabytes for a frame. No Arrow C ABI export.
• src/arcadedb/_columnar.py holds the one decoder and the merge of frames for both wheels. results.py keeps its JPype transport and the text of its json columns; _native/colframes.py is the native glue.
• Frame types: i8, f8, b1, dt/dtu, f4v/f8v, str, dec. What a fixed layout cannot carry without changing a value is a per-column json frame column: nested lists and maps, records, BigInteger, a mixed column, a ragged vector, a DATE or time-zone-aware value in to_dataframe, and an integer beyond 2**53 next to a float. Nothing falls back to the JSON row path.
• The first reply of a query is still JSON for the row APIs, and the caller already holds those rows as dicts. A columnar read writes them as one more frame in Python (colframes.py), by the rules of the library's encoder and through the same decoder, so the cursor keeps no rows and a column has one type however a result splits between the first reply and frames. The JSON reply now names the properties that were float[] or double[] (fv, dv), because a list of numbers in a row cannot say so. This frame goes away with the JSON reply.

Behaviour that is now the same on both wheels (the old behaviour was not kept):

• A vector column is a column of declared arrays (float[], double[]) at any result size. A LIST of doubles whose values fit float32 was a float32 matrix in to_columns for 256 rows or fewer on native and a list above that, and a list on the JVM wheel; it is a list of lists everywhere now.
• The type of a column belongs to the whole result, not to where a batch ends. A batch that lacks the column or holds only nulls for it takes the type of the others (to_columns and to_arrow). float32 next to float64 vectors of one size unify to a fixed-size list of double, and a dimension that changes (inside a batch or between two) gives list<double> on both wheels (the widening is exact); never text.
• An integer beyond 2**53 next to a float stays exact: a json column, so to_columns holds the exact ints and floats and to_arrow has no common numeric type and gives exact text. The JVM ColumnBatcher used to widen it to a double and lose digits.
• A result with no rows and columns=[...] returns those columns (empty lists in to_columns, null-typed columns in to_arrow). Without columns it is {} or an empty table.
• to_columns arrays are writable (the decoder copies once; they were read-only for a single frame).
• to_dataframe: a vector cell is a numpy float32 array (a row of the to_columns matrix, no Python list per row) or None for a null vector (never a vector of NaN), and a DATETIME column is datetime64[us]. to_polars gives Array(Float32, n) for a vector column, null where the vector is null. to_columns keeps its matrix, where a null vector is a row of NaN, as before. All of this is in the docstrings of both wheels.
• batch_size below 1 raises ValueError; a batch_size of 2**31 or more is clamped to what a C int holds (it used to wrap to zero on native and loop forever). An empty frame from an open cursor is an ArcadeDBError, so a library bug cannot loop, and a library older than the wheel raises an error that says so instead of an AttributeError.

Differences from the JSON route that stay: a float[] column in to_arrow is fixed_size_list<float32> (it was list<double>, the same values) and a column of only nulls is a string column (it was the Arrow null type).

What stays: the old JSON read path (_drain_dicts, loads_tagged, arcadedb_n_next_bin) stays in this PR for to_list, to_json_list, row iteration and the first reply. Retiring it for the row APIs is a later PR, once these frames are measured. No path materialises the table twice: frames are decoded and merged once.

Not changed: to_columns still promotes an int column with nulls to float64 with NaN (the documented pandas rule). So an int column that has a null in one frame next to another frame of integers beyond 2**53 is rounded in to_columns while to_arrow keeps int64. That is the rule of the API, not a frame boundary effect, and I did not change it here.

Tests (CI only; locally I compiled every changed Python file, ran pyflakes, black, isort and bandit, and called the decoder and the first-reply frame logic directly on hand-built data):

• tests/test_column_frames.py, both wheels: the decoder on hand-built frames, then one stored table through every columnar API, plus result sizes 255 to 513, huge and invalid batch sizes, empty results with columns, writable arrays, declared and list vectors at 10, 300 and 3000 rows, double arrays after rows read past the first reply, a property that is numeric from row 30,000 on with batch_size=25_000, exact integers next to floats at 10 and 300 rows, a changing vector dimension, two cursors at once, a cursor closed early and 300 closed unread.
• tests/test_native_column_frames_reference.py, native only: the old assembly is copied into the test as a private reference and each case compares the two tables column by column with NaN equal to NaN and None equal only to None (arrays are compared as arrays, vector cells must be numpy float32); a frame budget test for a result with another property in every row; an empty frame is an error; a huge batch_size.

What I could not verify: no Java compile and no native build here (CI builds the library), so the Java is read by eye only. I did not construct a query that fails in the middle of a stream, so that path is covered by reading the code, not by a test.

🤖 Generated with Claude Code