Skip to content

Ingest integrity: count and content checks after every ingest path (parallel and async included) and a stress run for rare losses #336

Open

Description

@tae898

A goal for both wheels: no ingest path may lose, duplicate or silently alter a row, and a test must be able to catch it. Today the suite checks values and error messages path by path, but there is no one place that checks the same three things after every ingest call: the record count, an order-independent digest of every stored column, and the same two after closing and reopening the database. The new native read frames (#331) lost rows for a small batch_size and only CI found it, which is the kind of loss I want a test to find before a review does.

Scope: insert_many, insert_columns, insert_arrow, insert_dataframe, graph_batch (create_vertices, new_edges), the async executor, and time-series append, each sequential and with parallel=True or async where it exists, on both wheels. Cases: sizes at every chunk boundary (n = k * chunk and one either side of it, for the 100,000 JSON chunk, commit_every and batch_size), several writer threads on one type and on different types, a bad row in the middle of a batch (duplicate key, wrong type, refused value) with the documented outcome checked exactly (earlier chunks stored, the error raised, nothing silently dropped), a failure inside an async callback that must reach the caller, and close() straight after the last call.

Method: a CI test file for correctness on every platform (with a mutation check, so each test is shown to fail on code with a rows-dropped fault), and a stress script that repeats the cycles in fresh processes on the laptop. A run that finds no loss is reported as 0 of N, not as absence: with N cycles the rate is below 3 / N at 95 percent.

Done when: the test file is on the release branch and green on both wheels, the stress run has a published N, and every loss it finds has its own issue.

Activity

  1. added
    concurrencyThreads, transactions and conflicts
    github_actionsPull requests that update GitHub Actions code
    pythonPull requests that update python code
    timeseriesTime-series ingest and queries
    on Nov 9, 2026
  2. added this to the 26.12.1 milestone on Nov 9, 2026
  3. self-assigned this
    on Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

concurrencyThreads, transactions and conflictsgithub_actionsPull requests that update GitHub Actions codepythonPull requests that update python codetimeseriesTime-series ingest and queries

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions