Skip to content

JVM to_list and iter_dicts: no object_hook for a batch with no tagged value (refs #315) - #382

Closed
tae898 wants to merge 2 commits into
ArcadeData:mainfrom
humemai:perf/315-jvm-row-reader-hook
Closed

tae898 wants to merge 2 commits into
ArcadeData:mainfrom
humemai:perf/315-jvm-row-reader-hook

Conversation

@tae898

@tae898 tae898 commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?

The JVM wheel's to_list() and iter_dicts() decode every TypedRows batch with json.loads(text, object_hook=_typed_hook(side)) and read the batch's side array first, also when the batch holds only plain values. The hook is a Python call per JSON object, and reading the side array is a JPype operation per batch (per row for iter_dicts(), which reads one row per crossing). I added _loads_typed_batch in src/arcadedb/results.py: it looks for the tag escape in the batch text first, and when it is absent it parses with plain json.loads and does not touch the side array. _loads_typed_row already does this for a single row. No Java change, and the native wheel is untouched.

How "nothing tagged" is decided, and why that is exact

The side array is not a usable signal: DECIMAL, dates, datetimes and sets are tagged without a side entry. The text is:

  • TypedRows.java writes a tag in two places only, tagged() and the r row of appendRow(), and both spell it with the six characters \u0001. quote() writes a control character inside a string as the same escape, and no path writes the raw character.
  • Every side array entry is referred to by an index inside a tag (toSide() and the r row). A text without those six characters therefore has no tag and no side index.
  • _typed_hook returns any object that lacks the tag key unchanged, so json.loads(text) gives the same value as the hooked parse.
  • The error can only go one way. A string that holds U+0001, or the text \u0001, also puts the escape into the text. The check then says "maybe tagged", the hook stays, and it changes nothing.

How it is verified

  • The tests came first, in their own commit (ff852ad): tests/test_typed_batch_guard.py. It counts the hook factory, the object_hook argument of json.loads and the batch elements that are read. A result of plain values (three batches for to_list(), a row per crossing for iter_dicts()) must use no hook and never read element 1. Every kind of value the bridge tags (Decimal, big integer, dates, datetimes, instants, sets, NaN and the infinities, float and double arrays, a map with an int key or the tag as key, tagged values nested in lists and maps) must keep the hook. One tagged row among 2,100 must hook exactly one batch. to_list() and iter_dicts() are also compared with the old always-hooked decoding of the same batches over each shape alone, a mixed result, the wide table of test_typed_rows.py and batch boundaries. The counting tests fail on the old decoder (every batch gets a hook and a read of the side array) and pass on the new one. The file is not collected on the native wheel (NATIVE_NOT_APPLICABLE).
  • On my machine I ran only light checks, no JVM: black, isort, ruff and pyflakes on the changed files, compile() of the sources, the real results.py against fake batches (the old code hooks every batch, the new one only the tagged ones), and a differential run of 30,000 random TypedRows-shaped texts, with U+0001 and the escape text inside strings, against the always-hooked decoding. Every result was equal. Those scripts are not in the PR.
  • CI: this is a draft, and test-jvm.yml skips the platform matrix for a draft, so the JVM tests have not run yet. They run when the PR is marked ready for review.

Related issues

Part of #315. It stays open: nothing here claims a speed result.

Performance changes

Not measured yet. The speed claim waits for a timed A/B of a candidate wheel against 26.11.1.post2 (interleaved runs, a control run of the baseline against itself, the median ratio with its interval).

Docs

No page or README.md describes how a batch is decoded, and no behavior changes, so none is updated.

Checklist

  • A regression test fails on the old code and passes on the new (the counting tests; confirmed against the real functions with fake batches, not yet by the JVM suite in CI)
  • The test suite that the change can affect passes (waits for CI on the ready PR)
  • uvx pre-commit run --files <changed files> passes (I ran black, isort and ruff with the versions pinned in .pre-commit-config.yaml, not pre-commit itself; the CI pre-commit job covers it)

🤖 Generated with Claude Code

tae898 and others added 2 commits November 14, 2026 02:43
…ject_hook and without reading its side array (refs ArcadeData#315)

to_list() and iter_dicts() of the JVM wheel build a json object_hook for every batch and read the
side array of every batch, whether or not the batch holds a tagged value. These tests count the hook
factory, the object_hook argument of json.loads and the batch elements that are read, for a result of
plain values (no hook, no read of the side array) and for every kind of value the bridge tags (the
hook stays). They also compare to_list() and iter_dicts() with the decoding that always builds the
hook, over single shapes, a mixed result and batch boundaries.

They fail on the current decoder (every batch gets a hook and a read of the side array) and pass once
the decoder looks for the tag escape in the batch text first.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…s text without an object_hook and without reading its side array (refs ArcadeData#315)

Every batch was decoded with json.loads(text, object_hook=_typed_hook(side)) and its side array was
read first, also when the batch held only plain values. The hook is a Python call per JSON object and
the read of the side array is a JPype operation per batch, which for iter_dicts (one row per crossing)
is one per row.

TypedRows writes every tag as {"\u0001": [kind, payload]} with the escape spelled out, writes a control
character of a string as the same escape, and never writes the raw character, and every side array
entry is referred to by a tag in the text. A text without the escape therefore holds no tag and no side
index, the hook would return each object unchanged, and plain json.loads gives the same value. The
check is a substring scan of a str that already exists (_loads_typed_row did it for one row). A string
that holds U+0001 or the text of the escape only keeps the hook, which then changes nothing.

No Java change, and the native wheel is untouched. The speed effect is not measured: it waits for a
timed A/B of a candidate wheel against 26.11.1.post2.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tae898

tae898 commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator Author

This change shipped in 26.11.1.post3 (on PyPI since 2026-11-14) through the release line, which #387 merged into main with a merge commit. The release line carries the same change as its own commits, so I close this draft. The measurements are on #315.

@tae898 tae898 closed this Nov 14, 2026
tae898 added a commit that referenced this pull request Nov 14, 2026
…, and the stale statements (refs #315) (#399)

JVM iter_dicts parses a batch with no tagged value without the object hook (#382); JVM insert_arrow and insert_dataframe carry a timestamp or date column as one long[] of epoch ticks (#383); native transaction calls return the state as one integer (#384); native save() of a saved record with auto-transaction on skips the transaction check (#385). The figures are copied by script from the scored gate study and each is checked against the public comment on #315. The n_update_one figure ran with auto-transaction off, so the page says off (the comment says on). The save() paragraph said three calls without the auto-transaction case. The quickstart was re-run on both published 26.11.1.post3 packages on Python 3.12 and printed the output shown.

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jvm The arcadedb-jvm package (JPype build) performance Speed or memory of the bindings, with a measurement python Pull requests that update python code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant