From 85f45f1a3b2e57433df5fd8566a3ea2a23854dc2 Mon Sep 17 00:00:00 2001 From: Lareb Zafar Date: Fri, 20 Nov 2026 18:52:12 +0530 Subject: [PATCH 1/4] Document memory write and worker progress regression tests Exercise derived effects, stored cursors and complete SQL rows so regression checks observe the boundaries they claim to protect. Signed-off-by: Lareb Zafar --- docs/development/testing.md | 68 +++++++++++++++++++++++++++++++++++-- 1 file changed, 65 insertions(+), 3 deletions(-) diff --git a/docs/development/testing.md b/docs/development/testing.md index 00177da962..81132207bd 100644 --- a/docs/development/testing.md +++ b/docs/development/testing.md @@ -6,7 +6,7 @@ Which suite a test belongs in, what each suite needs to run, how tests skip when | Suite | Path | Needs | Run locally | In CI | | ----------------------------------------------- | -------------------------------------------------------------------- | ----------------------------------------------------------------------------- | -------------------------------------------------------------- | -------------------------------------------------------------- | -| Unit | `tests/unit/` | nothing external; no ClickHouse | `make test-unit` (also runs `pytest bench -q`) | `unit` in parts, random order; `unit-coverage`, coverage ≥ 40% | +| Unit | `tests/unit/` | no external services; optional ClickHouse binary | `make test-unit` (also runs `pytest bench -q`) | `unit` in parts, random order; `unit-coverage`, coverage ≥ 40% | | Bench harnesses | `bench/` (its `test_*` files) | nothing external | `pytest bench -q` | `unit-coverage` | | Integration | `tests/integration/` | a migrated ClickHouse and a running server | `make test-integration` with `LOOM_BASE_URL` | `integration` | | Analyzer corpus | `tests/analyzer/` | a migrated ClickHouse (no server) | `pytest tests/analyzer -q` | `integration` | @@ -31,8 +31,11 @@ or the embedder. ## Unit tests -- **No ClickHouse.** A unit test stubs the client; anything that needs real SQL - semantics belongs in the integration or analyzer suite. +- **No database server.** Unit tests stub the client. The optional SQL tests use + `clickhouse-local` and skip without its binary. Use the release that + `dev/clickhouse_release.sh` resolves, as CI does; another release can have different + codecs or query behavior. Server, replica and transport behavior belongs in the + integration suite. - **Random order, in parallel.** CI runs the suite with `-n auto -p randomly` on one worker process per core and prints the seed first, so a test must pass after any other and beside any other. [`tests/README.md`](../../tests/README.md#rules-every-test-keeps) @@ -42,6 +45,10 @@ or the embedder. `now()` passes until the date ages past the threshold. Pin the clock through the code's time seam instead of hard-coding a date the check reads against the wall clock. +Keep CI's system tools beside the locked Python environment. Release-script tests, for +example, call `jq`. Record missing tools or an unsupported platform separately from a +failure introduced by the diff. + ## Integration tests The integration suite runs against a live server, the way CI's `integration` job does: @@ -119,6 +126,49 @@ LOOM_BASE_URL=http://localhost:7777 make test-integration table with an `org` column and the residue is counted; residue fails the run. `LOOM_IT_KEEP_ROWS=1` keeps the rows for debugging. +## Concurrent write regressions + +Pause a writer after its head read and after its final admission check. Complete a +competing write from a second client before releasing each pause. Read the winning row +back and compare every field, including its version. Keep an unchanged-head case beside +each contended case, and run the interleaving against a private migrated database. + +Admission tests cover the derived effects too. A definite refusal before a memory send +leaves its search documents and existing search data unchanged. Test known commits and +uncertain sends separately: privacy cleanup and lease retention depend on whether a send +may have committed. Preserve the original backend exception, including cancellation. + +Exercise the largest supported batch with nonzero control store latency and a small +connection pool. Check acquisition and release under partial failure and cancellation. +Every acquired holder must be released or retained for the documented uncertain-send +bound. Include restore batches, whose supported size can exceed a single request batch. +Use the clock seam for deadline cases. + +Cover the documented [rolling-upgrade overlap](../architecture/rolling-upgrades.md) as +well as writers that all use the same protocol. Record which interleavings a protocol +protects and the operator steps required to reach that guarantee. + +## Worker progress regressions + +Resume through the supervisor and persisted `loom.worker_status`, with fresh worker +process state between passes. Check the saved cursor after a partial pass and after a +completed scan. This exercises both the worker's cursor and the supervisor's persistence +rules. + +Include these boundaries: + +- More identities sharing one timestamp than fit in a batch, including a pass stopped + by its time budget. +- An event that becomes visible in the boundary second after the totals read, for both + short and full batches. +- A completed scan followed by late usage or a stale winning version behind its former + cursor. +- The longest supported identity through the persisted cursor's size limit. +- Totals beyond a destination counter's integer range. Reconciliation and writes must + compare the same capped target, so a saturated counter cannot starve later work. + +Absolute counter targets must remain idempotent when a boundary is folded again. + ## Framework adapters The SDK adapters whose framework is an SDK extra keep their framework out of `uv.lock`, so @@ -164,6 +214,18 @@ The [analyzer README](../../tests/analyzer/README.md) covers the case format, th migrated-target requirement and importing production shapes captured with `LOOM_CH_CAPTURE_STATEMENTS`. +### SQL write assertions + +Real-row tests check values as well as statement resolution. Compare the complete +winning row through the production identity predicate and `FINAL`. An isolated fixture +can also read a full `FINAL` snapshot and select its row in Python. + +If a filtered read disagrees with that snapshot, compare its `PREWHERE` plan on the same +ClickHouse release before changing the writer. A key-only filter can take a different +patch part read path from the production full identity predicate. Keep the complete +field comparison, and verify that the original write bug still fails under the chosen +readback. + ## CI gates Most gates are unit tests, so `make test-unit` runs them; the rest are `make lint` steps. From 651ed257ef3bb3e4f5d633feefe360ed510aa0f7 Mon Sep 17 00:00:00 2001 From: Lareb Zafar Date: Fri, 20 Nov 2026 19:03:04 +0530 Subject: [PATCH 2/4] Test warm worker passes beside cursor restarts A restart can recover a missed boundary event through cold reconciliation, so the guide must retain warm continuation regressions. Signed-off-by: Lareb Zafar --- docs/development/testing.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/development/testing.md b/docs/development/testing.md index 81132207bd..4be95a3279 100644 --- a/docs/development/testing.md +++ b/docs/development/testing.md @@ -150,10 +150,11 @@ protects and the operator steps required to reach that guarantee. ## Worker progress regressions -Resume through the supervisor and persisted `loom.worker_status`, with fresh worker -process state between passes. Check the saved cursor after a partial pass and after a -completed scan. This exercises both the worker's cursor and the supervisor's persistence -rules. +Use consecutive passes in the same process for boundary events and late usage. These +cases exercise the process watermark. Also test restarts and handovers: begin with fresh +worker process state and resume through the supervisor and persisted +`loom.worker_status`. Check the saved cursor after a partial pass and after a completed +scan to exercise the supervisor's persistence rules. Include these boundaries: From 8a0cdb6797d77461696795a3b0e638d25c1ce0e7 Mon Sep 17 00:00:00 2001 From: Lareb Zafar Date: Fri, 20 Nov 2026 19:21:36 +0530 Subject: [PATCH 3/4] Concurrency tests cover changed content at an equal version Signed-off-by: Lareb Zafar --- docs/development/testing.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/development/testing.md b/docs/development/testing.md index 4be95a3279..ee1d9703cf 100644 --- a/docs/development/testing.md +++ b/docs/development/testing.md @@ -133,6 +133,9 @@ competing write from a second client before releasing each pause. Read the winni back and compare every field, including its version. Keep an unchanged-head case beside each contended case, and run the interleaving against a private migrated database. +Include competing rows with different content at the same numeric version. A matching +version alone does not prove that the head is unchanged. + Admission tests cover the derived effects too. A definite refusal before a memory send leaves its search documents and existing search data unchanged. Test known commits and uncertain sends separately: privacy cleanup and lease retention depend on whether a send From fe354a6f0ec76579b4d9de9604c866ad3c7c76e7 Mon Sep 17 00:00:00 2001 From: Lareb Zafar Date: Fri, 20 Nov 2026 21:36:02 +0530 Subject: [PATCH 4/4] Document committed-write cleanup and private cache regression tests Cancellation during fence release must not skip effects owed by an acknowledged write. Caches of private strings need retention and erasure coverage as well as result checks. Signed-off-by: Lareb Zafar --- docs/development/testing.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/development/testing.md b/docs/development/testing.md index ee1d9703cf..d1c4729681 100644 --- a/docs/development/testing.md +++ b/docs/development/testing.md @@ -141,12 +141,20 @@ leaves its search documents and existing search data unchanged. Test known commi uncertain sends separately: privacy cleanup and lease retention depend on whether a send may have committed. Preserve the original backend exception, including cancellation. +Pause fence release after the database acknowledges a write, then cancel the caller or +fail the release. Verify that every owed audit receipt, announcement and privacy cleanup +still completes: a release failure cannot undo the committed row. + Exercise the largest supported batch with nonzero control store latency and a small connection pool. Check acquisition and release under partial failure and cancellation. Every acquired holder must be released or retained for the documented uncertain-send bound. Include restore batches, whose supported size can exceed a single request batch. Use the clock seam for deadline cases. +Prefer stateless fast paths over retaining private derived strings. If an optimization +needs a process cache, bound its size and retention. Test erasure across replicas as well +as warm-cache results, including private text retained in cache keys. + Cover the documented [rolling-upgrade overlap](../architecture/rolling-upgrades.md) as well as writers that all use the same protocol. Record which interleavings a protocol protects and the operator steps required to reach that guarantee.