[tae898]

Refs #336. I wrote the ingest-integrity tests of #336: one check that every ingest test calls, a mutation proof for every family, and the same file on both wheels. The stress script for the rare losses is a separate study on my laptop (not in this repository); its N will be quoted in #336.

Existing work searched first. gh pr list --state all --search "integrity" finds only a merged Dependabot PR, --search "data loss OR digest OR rows lost" finds nothing, and gh issue list --search "loss OR digest OR dropped rows" finds #336 itself and older bug reports (#7615 is the async loss that tests/test_async_executor.py already pins). ls tests | grep -i -E "integr|durab|loss|digest|stress" is empty. The existing tests check values and error messages path by path (test_bulk_insert.py, test_insert_columns.py, test_graph_batch.py, test_async_executor.py); none checks count, digest and reopen in one place for every path.

The check (tests/ingest_integrity.py). After a call it compares the stored type with the rows that were generated: the record count read three ways (SELECT count(*), count_type(), the length of the rows read back), an order-independent digest of every column (sum and xor of a 64-bit hash per row) read two ways (to_list() and iterating the result), and every key present exactly once. Then close() straight after the call, reopen from disk, and the same again (durability). It never trusts what the path says it did: the expected side is the generated data. It does not assert executor idleness after wait_completion() (#210), only stored data.

Tests (tests/test_ingest_integrity.py, both wheels):

• every size at every boundary for 15 families: insert_many (list, generator, parallel=True), insert_columns (sequential, parallel), insert_arrow and insert_dataframe (sequential, parallel), insert_arrow into a vertex type, create_vertices, async create_record (parallel level 1 and 4) and command, append_samples. The sizes are 0, 1, 2 and one either side of every multiple of the chunk (7 rows, patched into the module constants) and of commit_every and batch_size (5 rows), and 255, 256 and 257 (the first reply of the native read path). One more load at the real 100,000 create_vertices chunk.
• reading back 600 rows through to_columns() and to_arrow() at batch_size 1, 2, 7, 255, 256, 257, 600 and 25,000 (the shape of the Native: column frames for to_arrow, to_columns, to_dataframe and to_polars, read by the decoder the JVM wheel uses (refs #312) #331 loss).
• new_edges: edge count (SQL and Cypher) and the endpoint pairs and weights, at every boundary, plain and weighted; close() straight after the last edges.
• writer threads on one type (through run_in_transaction) and on different types with different paths, with a reader thread whose count may only grow and must end at the total.
• a bad row in the middle of a batch (duplicate key, wrong declared type, a value the encoder refuses) at five positions for insert_many, insert_columns, insert_arrow, insert_dataframe and create_vertices, with the documented outcome exact (the committed batches stored, the batch with the bad row rolled back, the error raised, the message of insert_arrow and insert_dataframe naming the committed row count); a column the declared type refuses stores nothing; a parallel duplicate is reported, nothing is stored twice and the stored keys survive a reopen.
• async: rejected records reach their own error callbacks and the executor-wide one; a per-record callback that raises loses no record without a report; close() straight after the submissions stores everything or raises.

Mutation proof (test_mutation_*, 95 cases plus the seams). Every family runs through the same function with a fault and must report it: drop the last row, drop the first, skip a middle chunk, store a row twice, change one value, swap two values, and for the synchronous families load inside a transaction nobody commits (visible in the session, gone after the reopen: the message must name the reopen and not the call). test_mutation_inside_insert_many puts the fault inside the path (the chunk function of each wheel), test_mutation_an_async_load_that_returns_before_the_wait parks the executor and checks before the wait, test_mutation_edges drops, duplicates and misroutes an edge, and two small files put a rows-dropped fault into the native columnar fast path (test_native_ingest_integrity_faults.py) and the JVM column conversion and bulk vertex path (test_jvm_ingest_integrity_faults.py, listed in NATIVE_NOT_APPLICABLE). The pure part of the check (digest, key check, every mutant) was run locally against a stub before the push.

CI. This branch is release/26.11.1 (which is the 26.11.1 tag, 2f28a328be) plus test files only, so a run of it is the run on the released behaviour; no throwaway base branch is needed. No CI result exists yet: the two fork runs I dispatched were cancelled with the rest of the fork queue. The tests will run first inside a combined candidate branch with the open PRs merged. Any test that fails on the released behaviour is a finding to file as its own issue (then a strict xfail naming it), not a test to loosen.

🤖 Generated with Claude Code