Repository navigation
Native columnar readers: infer Arrow types first, skip per-key and per-row steps, one _kind per column (refs #312) - #321
Merged
Conversation
…r-row steps, one _kind per column (refs #312) Bindings-only, Python in arcadedb._native, no image rebuild. Output is meant to be identical to before: - columns.to_arrow tries pa.array(values) first and keeps the old per-value rules for a column that inference cannot type (ArrowInvalid, ArrowTypeError) or types differently (temporal types, unsigned ints). - columns._names tests each row's keys against the known ones in C instead of a Python step per key. - columns.to_columns calls _kind once per column instead of twice. - NativeResultSet._fill can hand a reply's rows straight to _drain_dicts, which no longer builds a tuple and a generator step per row. The float32-tail view in codec.loads_tagged is not done: a vector must stay a list of floats for the row API. tests/test_native_columns_reference.py keeps the old rules as a private reference and compares tables, columns, names and the drain on mixed, null-heavy, ragged and vector inputs. No speed is claimed here. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…; tests compare NaN tables by value (refs #312) pa.array([2**63]) raises OverflowError, not ArrowInvalid, so an int above int64 next to floats would have raised where the old explicit-type call fell back to exact text. _inferred now leaves every failed inference to the rules, so the outcome of such a column is decided by the old code. Four reference cases cover it. Table.equals is False for a NaN next to the same NaN, so the comparison is the schema plus the repr of the values. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
tae898
added a commit
that referenced
this pull request
Nov 10, 2026
… modules that read the columns now The page came with main and its first comment line named _native/columns.py (#321) as the source of to_columns, to_arrow, to_dataframe and to_polars. The release line deleted that module with the native column frames (#331), so the merge of the release line leaves the comment pointing at a file that is gone. It now names _columnar.py and _native/colframes.py. The comment does not render; no statement of the page changed. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2 of 3 tasks
abhirajadhikary06
pushed a commit
to abhirajadhikary06/arcadedb-python
that referenced
this pull request
Nov 10, 2026
…r-row steps, one _kind per column (refs ArcadeData#312) (ArcadeData#321) * Native columnar readers: infer Arrow types first, skip per-key and per-row steps, one _kind per column (refs ArcadeData#312) Bindings-only, Python in arcadedb._native, no image rebuild. Output is meant to be identical to before: - columns.to_arrow tries pa.array(values) first and keeps the old per-value rules for a column that inference cannot type (ArrowInvalid, ArrowTypeError) or types differently (temporal types, unsigned ints). - columns._names tests each row's keys against the known ones in C instead of a Python step per key. - columns.to_columns calls _kind once per column instead of twice. - NativeResultSet._fill can hand a reply's rows straight to _drain_dicts, which no longer builds a tuple and a generator step per row. The float32-tail view in codec.loads_tagged is not done: a vector must stay a list of floats for the row API. tests/test_native_columns_reference.py keeps the old rules as a private reference and compares tables, columns, names and the drain on mixed, null-heavy, ragged and vector inputs. No speed is claimed here. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Native columns: a failed inference of any kind falls to the old rules; tests compare NaN tables by value (refs ArcadeData#312) pa.array([2**63]) raises OverflowError, not ArrowInvalid, so an int above int64 next to floats would have raised where the old explicit-type call fell back to exact text. _inferred now leaves every failed inference to the rules, so the outcome of such a column is decided by the old code. Four reference cases cover it. Table.equals is False for a NaN next to the same NaN, so the comparison is the schema plus the repr of the values. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> (cherry picked from commit 6c6a5a1) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
abhirajadhikary06
pushed a commit
to abhirajadhikary06/arcadedb-python
that referenced
this pull request
Nov 10, 2026
…Data#320 JVM to_json_list float parity, ArcadeData#321 native columnar read assembly) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
abhirajadhikary06
pushed a commit
to abhirajadhikary06/arcadedb-python
that referenced
this pull request
Nov 10, 2026
…ative column frames for the columnar readers) Conflicts, three (as in the first candidate): - src/arcadedb/_native/columns.py: deleted by ArcadeData#331, modified by ArcadeData#326 (ArcadeData#321). Took the deletion: nothing calls the module any more. - src/arcadedb/_native/database.py NativeResultSet._fill: ArcadeData#321 returns has_rows (the drain's into= path), ArcadeData#331 calls self._note_vectors(reply) before the return. Both kept. - tests/test_native_columns_reference.py (ArcadeData#321) imported the deleted module. Cut to its drain tests (NativeResultSet._drain_dicts against row-by-row iteration); the columns.py comparisons are superseded by ArcadeData#331's test_native_column_frames_reference.py. conftest comment updated. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
abhirajadhikary06
pushed a commit
to abhirajadhikary06/arcadedb-python
that referenced
this pull request
Nov 10, 2026
The release line (tag 26.11.1.post2 = 1454809, which contains the post1 tag 26.11.1.post1 = dbe1670) branched from the 26.11.1 tag and carries the post1 candidates (the native column frames ArcadeData#331, the other native and JVM load and read changes of ArcadeData#327 to ArcadeData#339, and the cherry-picks of main's ArcadeData#320 and ArcadeData#321), the post2 additions (Windows parity harness ArcadeData#295/ArcadeData#347, ArcadeData#340, the JVM Arrow vector load change) and their test fixes. Main did not have them. The merge uses a merge commit so that the release tag stays an ancestor of main. Four files conflicted, each because main and the release line changed the same lines. Resolutions, each checked against the history of both sides: - src/arcadedb/_native/columns.py (modify/delete): deleted. Main changed it only with ArcadeData#321 (6c6a5a1). The release line holds the same change as the cherry-pick 946dfda (identical blobs of columns.py and of test_native_columns_reference.py at both commits) and then deleted the module with the native column frames (ArcadeData#331, d606cce). Nothing else imports it; the merged database.py reads columns through the frames code. - src/arcadedb/_native/database.py: kept both. Main's facets lines in _fill and its facets property (ArcadeData#299, ArcadeData#301), then the release line's self._note_vectors(reply) before the return and its _note_vectors method. - tests/conftest.py: the release side of the native-only block (the comment of test_native_columns_reference.py now says what the file compares, plus the two new native-only files), and both entries in WINDOWS_NOT_APPLICABLE (main's test_dev_engine_pin.py and the release line's test_parity_gate_awk.py). - tests/test_native_columns_reference.py (add/add): the release side whole. Main's file tested the deleted module. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
I read the native wheel's columnar read path in #312:
to_arrow,to_columns,to_dataframeandto_polarsdrain the result as one Python dict per row (_drain_dicts) and then build each column with per-value Python loops insrc/arcadedb/_native/columns.py. This PR changes only that Python, so it needs no native image rebuild, and it is meant to return exactly the same values as before.columns.to_arrowtriespa.array(values)first. It keeps the old per-value rules (_kindand the typedpa.arraycalls with their text fallbacks) for a column that pyarrow cannot type, whatever the error (pa.array([2**63])raisesOverflowError, where the typed call raisedArrowInvalid), and for a column it types differently from the rules: a temporal type (the rules writetimestamp[ms], inference givestimestamp[us]ordate32) and an unsigned integer (an int above int64 is an error under the rules).columns._namestests each row's keys against the keys already seen with one set-like comparison, instead of one Python step per key of every row. The order is still the order of first sight. I did this once per call and not once per reply batch, because the rows reachcolumns.pyas one list; a per-batch version would have to thread the names through the cursor.columns.to_columnscalls_kindonce per column, where it called it twice.NativeResultSet._fillcan hand a reply's rows straight to a list, and_drain_dictsuses that, so the drain no longer builds a tuple per row and takes it apart with a generator. Rows already in the buffer (the first reply, or rows a caller has not taken) still come out of it.I did not do the float32-tail view in
codec.loads_tagged. A vector in a row must stay a list of Python floats for the public API (to_list(),to_json_list(),==against a list,json.dumps), and a view would hand out arrays or a lazy object instead. It is a change to the columnar path only, which belongs with the native columnar read frames in #312.The test is
tests/test_native_columns_reference.py(collected on the native wheel only). It keeps the old_kind,_names,to_columnsandto_arrowas a private reference and compares the new code with it on 46 single-column cases (ints, floats, bools, strings, all-null, datetimes, dates, Decimals up to 80 digits, vectors, ragged lists, dicts, bytes, ints above int64 and beyond 2^53 next to floats), on a table of all of them, on null-heavy ragged tables of 1000 rows, on chosen column lists and on empty input. It compares schema, values, column order and, where the old code raised, the exception class. A second group compares_drain_dictswith row-by-row iteration on 5000 rows (more than the first reply and two further replies), with 1 to 4999 rows taken before the drain.Related issues
Refs #312. This is the first, bindings-only part; the native columnar read frames are not here.
Performance changes
No speed or memory figure is claimed. The measurement is the profile study (
bindings-readpath-profile) and the timed A/B (ab3), both queued; the numbers will go on #312.Docs
None needed: no output and no API changed.
Checklist
_kindcalls and the_inferredcases fail on the old codeuvx pre-commit run --files <changed files>passes🤖 Generated with Claude Code