Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 77 additions & 3 deletions docs/development/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Which suite a test belongs in, what each suite needs to run, how tests skip when

| Suite | Path | Needs | Run locally | In CI |
| ----------------------------------------------- | -------------------------------------------------------------------- | ----------------------------------------------------------------------------- | -------------------------------------------------------------- | -------------------------------------------------------------- |
| Unit | `tests/unit/` | nothing external; no ClickHouse | `make test-unit` (also runs `pytest bench -q`) | `unit` in parts, random order; `unit-coverage`, coverage ≥ 40% |
| Unit | `tests/unit/` | no external services; optional ClickHouse binary | `make test-unit` (also runs `pytest bench -q`) | `unit` in parts, random order; `unit-coverage`, coverage ≥ 40% |
| Bench harnesses | `bench/` (its `test_*` files) | nothing external | `pytest bench -q` | `unit-coverage` |
| Integration | `tests/integration/` | a migrated ClickHouse and a running server | `make test-integration` with `LOOM_BASE_URL` | `integration` |
| Analyzer corpus | `tests/analyzer/` | a migrated ClickHouse (no server) | `pytest tests/analyzer -q` | `integration` |
Expand All @@ -31,8 +31,11 @@ or the embedder.

## Unit tests

- **No ClickHouse.** A unit test stubs the client; anything that needs real SQL
semantics belongs in the integration or analyzer suite.
- **No database server.** Unit tests stub the client. The optional SQL tests use
`clickhouse-local` and skip without its binary. Use the release that
`dev/clickhouse_release.sh` resolves, as CI does; another release can have different
codecs or query behavior. Server, replica and transport behavior belongs in the
integration suite.
- **Random order, in parallel.** CI runs the suite with `-n auto -p randomly` on one
worker process per core and prints the seed first, so a test must pass after any other
and beside any other. [`tests/README.md`](../../tests/README.md#rules-every-test-keeps)
Expand All @@ -42,6 +45,10 @@ or the embedder.
`now()` passes until the date ages past the threshold. Pin the clock through the code's
time seam instead of hard-coding a date the check reads against the wall clock.

Keep CI's system tools beside the locked Python environment. Release-script tests, for
example, call `jq`. Record missing tools or an unsupported platform separately from a
failure introduced by the diff.

## Integration tests

The integration suite runs against a live server, the way CI's `integration` job does:
Expand Down Expand Up @@ -119,6 +126,61 @@ LOOM_BASE_URL=http://localhost:7777 make test-integration
table with an `org` column and the residue is counted; residue fails the run.
`LOOM_IT_KEEP_ROWS=1` keeps the rows for debugging.

## Concurrent write regressions

Pause a writer after its head read and after its final admission check. Complete a
competing write from a second client before releasing each pause. Read the winning row
back and compare every field, including its version. Keep an unchanged-head case beside
each contended case, and run the interleaving against a private migrated database.

Include competing rows with different content at the same numeric version. A matching
version alone does not prove that the head is unchanged.

Admission tests cover the derived effects too. A definite refusal before a memory send
leaves its search documents and existing search data unchanged. Test known commits and
uncertain sends separately: privacy cleanup and lease retention depend on whether a send
may have committed. Preserve the original backend exception, including cancellation.

Pause fence release after the database acknowledges a write, then cancel the caller or
fail the release. Verify that every owed audit receipt, announcement and privacy cleanup
still completes: a release failure cannot undo the committed row.

Exercise the largest supported batch with nonzero control store latency and a small
connection pool. Check acquisition and release under partial failure and cancellation.
Every acquired holder must be released or retained for the documented uncertain-send
bound. Include restore batches, whose supported size can exceed a single request batch.
Use the clock seam for deadline cases.

Prefer stateless fast paths over retaining private derived strings. If an optimization
needs a process cache, bound its size and retention. Test erasure across replicas as well
as warm-cache results, including private text retained in cache keys.

Cover the documented [rolling-upgrade overlap](../architecture/rolling-upgrades.md) as
well as writers that all use the same protocol. Record which interleavings a protocol
protects and the operator steps required to reach that guarantee.

## Worker progress regressions

Use consecutive passes in the same process for boundary events and late usage. These
cases exercise the process watermark. Also test restarts and handovers: begin with fresh
worker process state and resume through the supervisor and persisted
`loom.worker_status`. Check the saved cursor after a partial pass and after a completed
scan to exercise the supervisor's persistence rules.

Include these boundaries:

- More identities sharing one timestamp than fit in a batch, including a pass stopped
by its time budget.
- An event that becomes visible in the boundary second after the totals read, for both
short and full batches.
- A completed scan followed by late usage or a stale winning version behind its former
cursor.
- The longest supported identity through the persisted cursor's size limit.
- Totals beyond a destination counter's integer range. Reconciliation and writes must
compare the same capped target, so a saturated counter cannot starve later work.

Absolute counter targets must remain idempotent when a boundary is folded again.

## Framework adapters

The SDK adapters whose framework is an SDK extra keep their framework out of `uv.lock`, so
Expand Down Expand Up @@ -164,6 +226,18 @@ The [analyzer README](../../tests/analyzer/README.md) covers the case format, th
migrated-target requirement and importing production shapes captured with
`LOOM_CH_CAPTURE_STATEMENTS`.

### SQL write assertions

Real-row tests check values as well as statement resolution. Compare the complete
winning row through the production identity predicate and `FINAL`. An isolated fixture
can also read a full `FINAL` snapshot and select its row in Python.

If a filtered read disagrees with that snapshot, compare its `PREWHERE` plan on the same
ClickHouse release before changing the writer. A key-only filter can take a different
patch part read path from the production full identity predicate. Keep the complete
field comparison, and verify that the original write bug still fails under the chosen
readback.

## CI gates

Most gates are unit tests, so `make test-unit` runs them; the rest are `make lint` steps.
Expand Down