You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Ingest integrity: count and content checks after every ingest path (parallel and async included) and a stress run for rare losses #336
A goal for both wheels: no ingest path may lose, duplicate or silently alter a row, and a test must be able to catch it. Today the suite checks values and error messages path by path, but there is no one place that checks the same three things after every ingest call: the record count, an order-independent digest of every stored column, and the same two after closing and reopening the database. The new native read frames (#331) lost rows for a small batch_size and only CI found it, which is the kind of loss I want a test to find before a review does.
Scope: insert_many, insert_columns, insert_arrow, insert_dataframe, graph_batch (create_vertices, new_edges), the async executor, and time-series append, each sequential and with parallel=True or async where it exists, on both wheels. Cases: sizes at every chunk boundary (n = k * chunk and one either side of it, for the 100,000 JSON chunk, commit_every and batch_size), several writer threads on one type and on different types, a bad row in the middle of a batch (duplicate key, wrong type, refused value) with the documented outcome checked exactly (earlier chunks stored, the error raised, nothing silently dropped), a failure inside an async callback that must reach the caller, and close() straight after the last call.
Method: a CI test file for correctness on every platform (with a mutation check, so each test is shown to fail on code with a rows-dropped fault), and a stress script that repeats the cycles in fresh processes on the laptop. A run that finds no loss is reported as 0 of N, not as absence: with N cycles the rate is below 3 / N at 95 percent.
Done when: the test file is on the release branch and green on both wheels, the stress run has a published N, and every loss it finds has its own issue.
Activity