Skip to content

Native transactions: arcadedb_n_tx returns the state as one integer, not a JSON text (refs #315) - #384

Closed
tae898 wants to merge 3 commits into
ArcadeData:mainfrom
humemai:perf/315-native-tx-no-json
Closed

tae898 wants to merge 3 commits into
ArcadeData:mainfrom
humemai:perf/315-native-tx-no-json

Conversation

@tae898

@tae898 tae898 commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?

I changed the native transaction entry point arcadedb_n_tx to return the transaction state as one long, not as a malloc'd JSON text. A begin, a commit, a rollback or a mark is now one foreign call: no string built in Java, no strlen, no copy into a str, no arcadedb_free, no json.loads. It works the way arcadedb_n_cmd already returns a cursor id.

New signature: public static long tx(IsolateThread thread, long h, int op) in ArcadeNative.java (was CCharPointer), restype c_longlong in lib.py, and a new Python door lib.tx(handle, op) -> int. The integer is the state after the call: bit 0 active, bits 1 to 31 depth, bits 32 to 62 the low 31 bits of commits (callers only compare two counts for equality), and -1 for an error.

before after
entries for with db.transaction() (mark, begin, commit) 3 3
foreign calls for with db.transaction() 12 (per entry: arcadedb_n_tx, strlen, PyUnicode_FromStringAndSize, arcadedb_free) 3
foreign calls for one begin, commit, rollback or is_transaction_active 4 1
json.loads for the mark 1 0

The error path is unchanged for the caller. On -1 the Java side has stored the same errorJson(t) text per thread (as arcadedb_n_cmd does), and Python reads it with lib.last_error() and passes it to the same _error(prefix, reply). The exception type, the message (Failed to commit transaction: <java class>: <message>), .native and .java_chain are the ones from before. A closed database still raises Database is closed before any crossing.

How I verified it: only by reading and by the light checks. I did not build the wheel or run a test, because the native image build and the suite run in CI. uvx pre-commit run --files on the changed files and Bandit on the changed Python files pass. The CI run on the fork is the proof.

Tests are in tests/test_native_tx_crossings.py, in the file that already counted these crossings, and the first commit holds only the tests:

  • with db.transaction() is exactly three foreign calls, all arcadedb_n_tx, counted through a proxy of the loaded library and the two reply-reading helpers. Begin, commit, rollback, is_transaction_active and the mark are one each. A transaction never calls lib._take.
  • json.loads is not called by a transaction (the old test expected one decode for the mark).
  • The layout of the integer, -1 and the error text for an unknown handle, commit without begin, rollback with nothing to roll back, a failed statement inside a block, nested with blocks, an injected failure on each entry point, and a database closed with a transaction open.

They fail on the old code, where lib.tx does not exist and the counts are higher. tests/test_native_one_row_crossings.py now counts lib.tx under the name arcadedb_n_tx, so its existing expectations hold.

Not measured yet: the speed claim waits for a timed A/B of a candidate wheel against 26.11.1.post2. This draft only claims fewer foreign calls and no reply text, which the tests count.

Things I am not sure of: a failing commit that the engine refuses at commit time (not at the statement) is covered by the injected failure, not by a real conflict, because I did not find a deterministic way to make one on one thread. The wheel ships the library and the Python files together, so a library built before this change (ARCADEDB_NATIVE_LIB) would be read wrongly by the new Python.

Related issues

Part of #315

Performance changes

None claimed yet, see above.

Docs

No page or README text describes this entry point, so none needs an update.

Checklist

  • A regression test fails on the old code and passes on the new (CI on the fork)
  • The test suite that the change can affect passes (CI on the fork)
  • uvx pre-commit run --files <changed files> passes

🤖 Generated with Claude Code

tae898 and others added 2 commits November 14, 2026 02:58
…ly text, no free and no JSON decode (refs ArcadeData#315)

arcadedb_n_tx returns the transaction state as one integer (bit 0 active, bits 1 to 31 depth, bits 32 to 62 commits, -1 for an
error whose text lib.last_error holds), as arcadedb_n_cmd returns a cursor id. These tests pin that and that an error still
reaches Python as the same typed exception with the same message:

- with db.transaction() is three foreign calls (mark, begin, commit), all arcadedb_n_tx: no strlen, no copy into a str, no
  arcadedb_free (was twelve);
- begin, commit, rollback, is_transaction_active and the mark each make exactly one foreign call;
- no JSON decode in a transaction, not even for the mark that reads active and depth;
- the layout of the integer, -1 for an unknown handle with the error in last_error, commit without begin, rollback with nothing to
  roll back, a failed statement in a block, nested with blocks, an injected failure on each entry point, a database closed with a
  transaction open.

They fail on the old code: lib.tx does not exist there, and the foreign-call and decode counts are higher.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…not a JSON text (refs ArcadeData#315)

A begin, a commit, a rollback and a mark each crossed into the library and got back a malloc'd JSON text of about 60 bytes that
Python read with strlen, a copy into a str and arcadedb_free, and mostly never looked at. with db.transaction() is three crossings
(the mark, begin, commit), so twelve foreign calls and three strings built, copied and freed.

arcadedb_n_tx now returns a long: bit 0 active, bits 1 to 31 depth, bits 32 to 62 the low 31 bits of the commit count, -1 for an
error. It is the way arcadedb_n_cmd returns a cursor id: the error text is kept per thread and read with arcadedb_n_last_error, built
by the same errorJson as before, so the typed exception and its message do not change. Each crossing is one foreign call, and
with db.transaction() is three. The Python side reads the fields of the integer inline, so the mark, the begin and rollback states
the unwind and run_in_transaction read, and is_transaction_active no longer decode anything.

Not measured: the speed claim waits for a timed A/B of a candidate wheel against 26.11.1.post2.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tae898 tae898 added python Pull requests that update python code performance Speed or memory of the bindings, with a measurement native The arcadedb package (native library build) labels Nov 13, 2026
@tae898 tae898 self-assigned this Nov 13, 2026
… no failed assert leaves a transaction open (refs ArcadeData#315)

CI of the first push (run 43478436168) failed two tests on a wrong assumption of mine: I expected the outer begin to raise the depth
by one. LocalDatabase.getNestedTransactions() is the size of the thread's stack of transaction contexts
(DatabaseContext.DatabaseContextTL.transactions). The stack always holds the thread's base context once the thread has used the
database, so the depth is 1 while idle. The outer begin reuses that base context (depth 1, active). Only a begin while a transaction
is active pushes a context (2), the commit or rollback of that level pops it (1, still active), and the outermost end leaves the base
context on the stack (1, idle). popIfNotLastTransaction never removes the last one. The native tx entry point returns exactly that
number, as the old JSON reply did, so the change is right and the two expectations were not.

The fixture now rolls back whatever a test left open before it drops the database, so a failed assert is one failure and not also a
"Cannot drop the database in transaction" teardown error. The close-and-reopen test does the same in a finally.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tae898

tae898 commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator Author

This change shipped in 26.11.1.post3 (on PyPI since 2026-11-14) through the release line, which #387 merged into main with a merge commit. The release line carries the same change as its own commits, so I close this draft. The measurements are on #315.

@tae898 tae898 closed this Nov 14, 2026
tae898 added a commit that referenced this pull request Nov 14, 2026
…, and the stale statements (refs #315) (#399)

JVM iter_dicts parses a batch with no tagged value without the object hook (#382); JVM insert_arrow and insert_dataframe carry a timestamp or date column as one long[] of epoch ticks (#383); native transaction calls return the state as one integer (#384); native save() of a saved record with auto-transaction on skips the transaction check (#385). The figures are copied by script from the scored gate study and each is checked against the public comment on #315. The n_update_one figure ran with auto-transaction off, so the page says off (the comment says on). The save() paragraph said three calls without the auto-transaction case. The quickstart was re-run on both published 26.11.1.post3 packages on Python 3.12 and printed the output shown.

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

native The arcadedb package (native library build) performance Speed or memory of the bindings, with a measurement python Pull requests that update python code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant