Skip to content

Native: run_in_transaction takes the commit-count witness from the begin reply (4 -> 3 crossings; failed block 6 -> 4) - #522

Closed
tae898 wants to merge 3 commits into
native-previewfrom
perf/native-tx-crossings
Closed

tae898 wants to merge 3 commits into
native-previewfrom
perf/native-tx-crossings

Conversation

@tae898

@tae898 tae898 commented Oct 25, 2026 •

Copy link
Copy Markdown

Backlog row N-05 (bindings study 2026-10-25). NativeDatabase.run_in_transaction made 4 library crossings on the happy path (_transaction_stack_mark, begin, _transaction_commit_mark, commit), each one a ctypes call, a malloc'd JSON reply, string_at, a decode and a json.loads, although the begin reply already carries {"active","depth","commits"} (ArcadeNative.tx returns txState(db) for every op) and begin() discarded it.

What changed (Python only, _native/database.py; no Java change, no image rebuild): run_in_transaction takes commits_at_start from the begin reply (new internal _begin_state(), public begin() still returns None). The unwind after a failed block used to ask is_transaction_active() before every rollback and once more at the end, and the retry check asked for the commit count again; every rollback reply carries the same state, so the new _unwind_opened_since(mark) returns the state of its last crossing and the retry check reads the commit count from it. A crossing that failed leaves the state unknown and the next step reads it again, as before. _end_transactions_opened_since(mark) keeps its signature and behaviour (it is what TransactionContext calls, shared with the JVM wheel).

What was measured, counts only (lib.call invocations, warmed database, new test file test_native_tx_crossings.py): run_in_transaction happy path 4 -> 3 (mark, begin, commit); a block that raises 6 -> 4 (mark, begin, state read, rollback); a refused attempt that is retried and then commits 11 -> 7; with db.transaction() stays at 3 (mark, begin, commit; it never read the commit count) and its failing form is 3 (mark, begin, rollback). Two crossings is not reachable from Python alone: the mark has to be taken before begin (a mark taken after it counts the level begin pushed as the caller's, ArcadeData#9970/ArcadeData#9321, and the interrupt-between-push-and-return window needs the pre-begin state), so it is a crossing of its own. A Java-side record of the pre-begin state in the begin op would make it 2; that needs an image rebuild and is not part of this PR.

No speed claim is made. Whether 1 of 4 crossings is visible in a small-transaction rate would be a timed A/B through tools/ab.py, not run here.

Semantics: unchanged by design, pinned by tests that pass before and after (a nested block that fails leaves the caller's transaction open, one that succeeds commits with the caller, a refused attempt is rolled back before the retry, a duplicated key is retried once, an error without a retry kind is raised after one attempt). The tests that count crossings fail on native-preview (3 failures) and pass here. Ran with the 26.10.2.dev1 library and this checkout's Python (test_native_tx_crossings.py, test_transaction_context_interrupt.py, test_core.py, test_concurrency.py, test_run_in_transaction_retry.py: 62 passed, the JPype-only cases of the last file are skipped as before).

The P-03 guard bug found while reading is fixed in #527, which is based on native-preview and goes first. This branch is rebased onto #527's branch, so its first commit is that fix and the retry check here keeps it (the witness is read from the unwind state, the comparison and the raise sit outside the try). Merge #527 first and this PR shows only its own commit.

Repaired in this PR: tests/test_interrupt_hygiene.py patched Database.begin and Database.rollback, and run_in_transaction no longer calls either on the native Database, so the patches never applied. Two tests failed (test_a_begin_that_refuses_leaves_the_callers_transaction_alone, test_a_failing_rollback_does_not_skip_the_levels_below_it) and three interrupt tests failed with DID NOT RAISE KeyboardInterrupt, which conftest.py turns into a silent NA-JVM skip. rollback() now wraps a new _rollback_state(), as begin() wraps _begin_state(), and the tests patch the seam the active backend's run_in_transaction really calls (begin/rollback on the JPype Database, _begin_state/_rollback_state on the native one). They check what they checked before; with the unwind changed to stop at its first failing rollback, the rollback test fails. The one test with no native equivalent (an interrupt between begin and a separate commit-mark read, which the begin reply made one crossing) skips with a stated reason instead of by marker.

Local checks on this head: native (nnext library) test_interrupt_hygiene.py, test_run_in_transaction_partial_commit.py, test_native_tx_crossings.py, test_core.py, test_run_in_transaction_retry.py and test_concurrency.py: 77 passed, 10 skipped (nine JPype-only cases of the retry file, as before, and the one test above); JVM wheel wheel-main-78e97b1cff (main), the hygiene and partial-commit files: 20 passed. black, isort and the other pre-commit hooks on the touched files: clean; bandit on _native/database.py: no findings.

🤖 Generated with Claude Code

…mitted part of its work (P-03)

The guard that refuses a retry after a BATCH boundary or a commit() inside the block sat in a try whose own `except ArcadeDBError: pass` swallowed the `raise` it was written to make, so the block was replayed and its durable part applied twice. Only the read of the commit-count witness may fail quietly; the raise now sits outside that try. The JVM core already refused.

The new test uses a real conflict (a second thread commits to a record the block also wrote), so it runs on both wheels: red on the native wheel before this change (20 of 25 rows incremented twice after a BATCH boundary, all 25 after a commit() inside the block), green after, green on the JVM wheel. The contract is added to the run_in_transaction API doc.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
tae898 and others added 2 commits October 25, 2026 18:31
…e text)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gin reply, and the unwind reads no state it already has

run_in_transaction made 4 library crossings on the happy path (mark, begin, commit mark, commit) although the begin reply already carries {"active","depth","commits"}; the commit-count witness now comes from that reply: 3 crossings (the mark must stay before begin, see the docstring).

The unwind after a failed block (_end_transactions_opened_since) asked is_transaction_active() before every rollback and once more at the end, and the retry check asked for the commit count again: every rollback reply carries the same state, so the unwind returns the state of its last crossing and the retry check reads the commit count from it (a failed crossing falls back to a read). A failed block: 6 crossings -> 4; a refused attempt that is retried: 11 -> 7.

Tests first: test_native_tx_crossings.py counts lib.call invocations (sequence of arcadedb_n_tx operations) for run_in_transaction, `with db.transaction()` and the exception and retry paths, and pins the behaviour the marks exist for (nested block leaves the caller's transaction open, refused attempt rolled back before the retry, duplicated key retried once). Counts only; no speed claim.

The unwind and begin now go through two named seams, _begin_state() and _rollback_state() (begin() and rollback() are thin wrappers over them). test_interrupt_hygiene.py patched Database.begin and Database.rollback, which run_in_transaction no longer calls on the native Database: two tests failed (a refused begin, a failing rollback) and three interrupt tests failed with "DID NOT RAISE KeyboardInterrupt", which conftest.py turns into a silent "NA-JVM" skip. The tests now patch the seam the active backend's run_in_transaction really calls (begin/rollback on the JPype Database, _begin_state/_rollback_state on the native one) and check what they checked before; the one test with no native equivalent (an interrupt between begin and a separate commit-mark read, which the begin reply made one crossing) skips with a stated reason.

Based on the P-03 fix (the commit-count witness check raises outside its try), so the retry check here reads the witness from the unwind state and keeps that fix.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tae898

tae898 commented Oct 26, 2026

Copy link
Copy Markdown
Author

Closing here: the bindings moved to ArcadeData/arcadedb-python. This change has no counterpart there yet and no queued A/B. If it is wanted, it goes there as its own PR: it is now a line in the tracking issue ArcadeData/arcadedb-python#19. The branch stays.

@tae898 tae898 closed this Oct 26, 2026
tae898 added a commit to ArcadeData/arcadedb-python that referenced this pull request Nov 3, 2026
…gin reply, and the unwind reads no state it already has (#201)

Port of humemai/arcadedb-embedded-python#522 (branch perf/native-tx-crossings, never merged there) onto current main; the P-03 guard of that PR's base is already here.

run_in_transaction made 4 library crossings on the happy path (mark, begin, commit mark, commit) although the begin reply already carries {"active","depth","commits"}: the witness now comes from that reply, 3 crossings (the mark must stay before begin). The unwind after a failed block asked is_transaction_active() before every rollback and once more at the end, and the retry check asked for the commit count again; every rollback reply carries the same state, so the unwind returns the state of its last crossing and the retry check reads the commit count from it (a failed crossing falls back to a read). Failed block 6 -> 4 crossings, refused and retried attempt 11 -> 7.

begin() and rollback() stay public and return None; _begin_state() and _rollback_state() are the seams run_in_transaction calls. test_interrupt_hygiene.py patched begin and rollback, which the native run_in_transaction no longer calls, so it patches the seam the active backend calls. test_native_tx_crossings.py counts the arcadedb_n_tx operations and pins the behaviour the marks exist for. Counts only, no speed claim: a timed A/B decides it before merge.

Refs #19

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@tae898
tae898 deleted the perf/native-tx-crossings branch November 8, 2026 11:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant