Skip to content

Native columnar readers: infer Arrow types first, skip per-key and per-row steps, one _kind per column (refs #312) - #321

Merged
tae898 merged 2 commits into
mainfrom
perf/312-native-columns-bindings-only
Nov 9, 2026
Merged

tae898 merged 2 commits into
mainfrom
perf/312-native-columns-bindings-only

Conversation

@tae898

@tae898 tae898 commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

What does this PR do?

I read the native wheel's columnar read path in #312: to_arrow, to_columns, to_dataframe and to_polars drain the result as one Python dict per row (_drain_dicts) and then build each column with per-value Python loops in src/arcadedb/_native/columns.py. This PR changes only that Python, so it needs no native image rebuild, and it is meant to return exactly the same values as before.

  • columns.to_arrow tries pa.array(values) first. It keeps the old per-value rules (_kind and the typed pa.array calls with their text fallbacks) for a column that pyarrow cannot type, whatever the error (pa.array([2**63]) raises OverflowError, where the typed call raised ArrowInvalid), and for a column it types differently from the rules: a temporal type (the rules write timestamp[ms], inference gives timestamp[us] or date32) and an unsigned integer (an int above int64 is an error under the rules).
  • columns._names tests each row's keys against the keys already seen with one set-like comparison, instead of one Python step per key of every row. The order is still the order of first sight. I did this once per call and not once per reply batch, because the rows reach columns.py as one list; a per-batch version would have to thread the names through the cursor.
  • columns.to_columns calls _kind once per column, where it called it twice.
  • NativeResultSet._fill can hand a reply's rows straight to a list, and _drain_dicts uses that, so the drain no longer builds a tuple per row and takes it apart with a generator. Rows already in the buffer (the first reply, or rows a caller has not taken) still come out of it.

I did not do the float32-tail view in codec.loads_tagged. A vector in a row must stay a list of Python floats for the public API (to_list(), to_json_list(), == against a list, json.dumps), and a view would hand out arrays or a lazy object instead. It is a change to the columnar path only, which belongs with the native columnar read frames in #312.

The test is tests/test_native_columns_reference.py (collected on the native wheel only). It keeps the old _kind, _names, to_columns and to_arrow as a private reference and compares the new code with it on 46 single-column cases (ints, floats, bools, strings, all-null, datetimes, dates, Decimals up to 80 digits, vectors, ragged lists, dicts, bytes, ints above int64 and beyond 2^53 next to floats), on a table of all of them, on null-heavy ragged tables of 1000 rows, on chosen column lists and on empty input. It compares schema, values, column order and, where the old code raised, the exception class. A second group compares _drain_dicts with row-by-row iteration on 5000 rows (more than the first reply and two further replies), with 1 to 4999 rows taken before the drain.

Related issues

Refs #312. This is the first, bindings-only part; the native columnar read frames are not here.

Performance changes

No speed or memory figure is claimed. The measurement is the profile study (bindings-readpath-profile) and the timed A/B (ab3), both queued; the numbers will go on #312.

Docs

None needed: no output and no API changed.

Checklist

  • A regression test fails on the old code and passes on the new: the new code is checked against the old rules, not a changed behaviour; the tests that count _kind calls and the _inferred cases fail on the old code
  • The test suite that the change can affect passes (CI is the first run)
  • uvx pre-commit run --files <changed files> passes

🤖 Generated with Claude Code

tae898 and others added 2 commits November 9, 2026 16:54
…r-row steps, one _kind per column (refs #312)

Bindings-only, Python in arcadedb._native, no image rebuild. Output is meant to be identical to before:
- columns.to_arrow tries pa.array(values) first and keeps the old per-value rules for a column that inference cannot type
  (ArrowInvalid, ArrowTypeError) or types differently (temporal types, unsigned ints).
- columns._names tests each row's keys against the known ones in C instead of a Python step per key.
- columns.to_columns calls _kind once per column instead of twice.
- NativeResultSet._fill can hand a reply's rows straight to _drain_dicts, which no longer builds a tuple and a generator step
  per row.
The float32-tail view in codec.loads_tagged is not done: a vector must stay a list of floats for the row API.

tests/test_native_columns_reference.py keeps the old rules as a private reference and compares tables, columns, names and
the drain on mixed, null-heavy, ragged and vector inputs. No speed is claimed here.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…; tests compare NaN tables by value (refs #312)

pa.array([2**63]) raises OverflowError, not ArrowInvalid, so an int above int64 next to floats would have raised where
the old explicit-type call fell back to exact text. _inferred now leaves every failed inference to the rules, so the
outcome of such a column is decided by the old code. Four reference cases cover it. Table.equals is False for a NaN next to
the same NaN, so the comparison is the schema plus the repr of the values.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tae898
tae898 merged commit 6c6a5a1 into main Nov 9, 2026
42 checks passed
@tae898
tae898 deleted the perf/312-native-columns-bindings-only branch November 9, 2026 09:05
@github-actions github-actions Bot added this to the 26.12.1 milestone Nov 9, 2026
tae898 added a commit that referenced this pull request Nov 10, 2026
… modules that read the columns now

The page came with main and its first comment line named _native/columns.py (#321) as the source of to_columns, to_arrow, to_dataframe and to_polars. The release line deleted that module with the native column frames (#331), so the merge of the release line leaves the comment pointing at a file that is gone. It now names _columnar.py and _native/colframes.py. The comment does not render; no statement of the page changed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
abhirajadhikary06 pushed a commit to abhirajadhikary06/arcadedb-python that referenced this pull request Nov 10, 2026
…r-row steps, one _kind per column (refs ArcadeData#312) (ArcadeData#321)

* Native columnar readers: infer Arrow types first, skip per-key and per-row steps, one _kind per column (refs ArcadeData#312)

Bindings-only, Python in arcadedb._native, no image rebuild. Output is meant to be identical to before:
- columns.to_arrow tries pa.array(values) first and keeps the old per-value rules for a column that inference cannot type
  (ArrowInvalid, ArrowTypeError) or types differently (temporal types, unsigned ints).
- columns._names tests each row's keys against the known ones in C instead of a Python step per key.
- columns.to_columns calls _kind once per column instead of twice.
- NativeResultSet._fill can hand a reply's rows straight to _drain_dicts, which no longer builds a tuple and a generator step
  per row.
The float32-tail view in codec.loads_tagged is not done: a vector must stay a list of floats for the row API.

tests/test_native_columns_reference.py keeps the old rules as a private reference and compares tables, columns, names and
the drain on mixed, null-heavy, ragged and vector inputs. No speed is claimed here.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Native columns: a failed inference of any kind falls to the old rules; tests compare NaN tables by value (refs ArcadeData#312)

pa.array([2**63]) raises OverflowError, not ArrowInvalid, so an int above int64 next to floats would have raised where
the old explicit-type call fell back to exact text. _inferred now leaves every failed inference to the rules, so the
outcome of such a column is decided by the old code. Four reference cases cover it. Table.equals is False for a NaN next to
the same NaN, so the comparison is the schema plus the repr of the values.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
(cherry picked from commit 6c6a5a1)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
abhirajadhikary06 pushed a commit to abhirajadhikary06/arcadedb-python that referenced this pull request Nov 10, 2026
…Data#320 JVM to_json_list float parity, ArcadeData#321 native columnar read assembly)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
abhirajadhikary06 pushed a commit to abhirajadhikary06/arcadedb-python that referenced this pull request Nov 10, 2026
…ative column frames for the columnar readers)

Conflicts, three (as in the first candidate):
- src/arcadedb/_native/columns.py: deleted by ArcadeData#331, modified by ArcadeData#326 (ArcadeData#321). Took the deletion: nothing calls the module any more.
- src/arcadedb/_native/database.py NativeResultSet._fill: ArcadeData#321 returns has_rows (the drain's into= path), ArcadeData#331 calls self._note_vectors(reply) before the return. Both kept.
- tests/test_native_columns_reference.py (ArcadeData#321) imported the deleted module. Cut to its drain tests (NativeResultSet._drain_dicts against row-by-row iteration); the columns.py comparisons are superseded by ArcadeData#331's test_native_column_frames_reference.py. conftest comment updated.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
abhirajadhikary06 pushed a commit to abhirajadhikary06/arcadedb-python that referenced this pull request Nov 10, 2026
The release line (tag 26.11.1.post2 = 1454809, which contains the post1 tag
26.11.1.post1 = dbe1670) branched from the 26.11.1 tag and carries the post1
candidates (the native column frames ArcadeData#331, the other native and JVM load and read changes
of ArcadeData#327 to ArcadeData#339, and the cherry-picks of main's ArcadeData#320 and ArcadeData#321),
the post2 additions (Windows parity harness ArcadeData#295/ArcadeData#347, ArcadeData#340, the JVM Arrow
vector load change) and their test fixes. Main did not have them. The merge
uses a merge commit so that the release tag stays an ancestor of main.

Four files conflicted, each because main and the release line changed the
same lines. Resolutions, each checked against the history of both sides:

- src/arcadedb/_native/columns.py (modify/delete): deleted. Main changed it
  only with ArcadeData#321 (6c6a5a1). The release line holds the same change as the
  cherry-pick 946dfda (identical blobs of columns.py and of
  test_native_columns_reference.py at both commits) and then deleted the
  module with the native column frames (ArcadeData#331, d606cce). Nothing else imports
  it; the merged database.py reads columns through the frames code.
- src/arcadedb/_native/database.py: kept both. Main's facets lines in _fill
  and its facets property (ArcadeData#299, ArcadeData#301), then the release line's
  self._note_vectors(reply) before the return and its _note_vectors method.
- tests/conftest.py: the release side of the native-only block (the
  comment of test_native_columns_reference.py now says what the file
  compares, plus the two new native-only files), and both entries in
  WINDOWS_NOT_APPLICABLE (main's test_dev_engine_pin.py and the release
  line's test_parity_gate_awk.py).
- tests/test_native_columns_reference.py (add/add): the release side whole.
  Main's file tested the deleted module.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant