Phase 0 conformance harness, and close the query-concurrency gap - #64
Merged
Conversation
Every unchecked row in spec/13's traceability matrix was blocked on the same
thing: no way to feed the worker an assignment, serve it chunks, and ask it a
signed question. This adds the stubs (HC-1/2/3/8), the structural validators
(HC-5) and deterministic seeding (HC-12), plus the smoke path that proves they
compose. The crate grew a library target because an integration tier cannot
reach a binary-only crate, and MG-4 wants the conformance subset timed as its
own gate rather than buried in the unit-test run.
The SUT boundary is a compromise worth naming: the input side (IB-40/41/42/43/44)
really is black-box over HTTP, but the query side enters below the transport at
p2p::{validate_query, execute, build_delivery, build_log}, so the controller's
intake and IB-1/2 stay untested. That is recorded in harness::UNCOVERED and in
spec/13 rather than papered over, and a test keeps those declarations pointing
at real identifiers.
Two findings fall out. GAP-1 is confirmed: its test fails on the inert cap and
passes on the one-token fix, so it ships quarantined for Phase 1. And the CLI
could not be parsed at all in a debug build — a bare bool positional trips a
clap assertion, aborting even --help — which release builds hid by compiling
the assertion out.
Also carries the freshness-attribution spec work: a rejection whose reference
input is the worker's own clock is a worker fault, not a client one (ADR-20,
INV-26, FM-55, GAP-33).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`run_query` bound its scopeguard to `_`, so the guard dropped at the end of its own `let` statement rather than at the end of the function. The slot was released before the query ran, which left `max_parallel_queries` unenforced — the real ceiling was the transport's message-handler product — and made the running-query gauge fall back to zero as fast as it rose, so overload was invisible in metrics as well as unbounded in fact. The test lands first and was checked both ways: it fails on the defect and passes on the fix. It lives in its own binary because the OB signals are process-global `lazy_static`s, and a gauge assertion sharing a process with other query-running tests would be reading their traffic — passing for the wrong reason, and blind to a regression. Closes the register's GAP-1: RP-4, REQ-22, INV-31 and PF-1 no longer carry the exception, and P-Q-PAR is observed-enforced. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both test jobs died with SIGTERM part-way through compiling the crate — no rustc error, only "the runner has received a shutdown signal". The library target and three integration binaries took `cargo test` from two heavy link targets to six, and four concurrent codegen units over the polars/arrow monomorphisations exhaust a 16 GB runner. Clippy passing over the same --all-targets set is the tell: it emits metadata and never links. So CI drops DWARF nothing reads, caps codegen on the two jobs that link, and stops the unit job rebuilding the conformance binaries the conformance job already owns. The harness had copied the status report, the log-page budget and the overload retry hint out of the controller rather than calling them. The budget copy was already 100 KiB looser than production, so the RP-22 page bound was checked against a number the worker never uses. A harness that asserts against a reimplementation of its subject can pass while the subject is broken, so those three are now reachable from p2p and called. test_fs asserted a raw directory listing, leaving it red on any macOS checkout Finder had touched. Every production caller of `ls` already filters by name shape, so the filtering belongs in the test and not in `ls`. `conformance_metrics` read as "metrics about conformance" rather than "conformance tests that touch process-global metrics", and `conformance.rs` was a catch-all that would have silted up as the CT-2/4/9 classes land. Each file is now named for what it tests, so the next class has an obvious home instead of a default one. The split of query_concurrency stays, but for the reason that actually forces it: one test binary is one process, and the OB signals are process-global, so a gauge assertion cannot sit beside tests that run queries for other reasons. That is now stated where someone deciding where to put a new test will read it. The harness self-test moves into the harness module — it guards the harness's own honesty, so it should hold wherever the harness is used rather than in whichever binary happened to be first. Also adds the .gitattributes the repo never had. LFS was configured and the index held correct pointers, but nothing routed *.parquet through the filter, so git compared 356 MB of smudged content against 131-byte pointers and called tests/data permanently modified — one `git add -A` away from inlining all of it into the repo. The stored oids match the working tree exactly, so declaring the filter reconciles the tree with no change to the data. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mo4islona
force-pushed
the
spec-suite
branch
2 times, most recently
from
July 26, 2026 11:14
7868574 to
753d370
Compare
`fetch_add` then compare let every concurrent rejection carry the count, and the OB-6 gauge with it, above max_parallel_queries until its guard ran. The cap on executing queries still held, but the gauge lied (INV-31) and the inflated count shed queries that would have fit. `fetch_update` takes a slot only when one is free, and the gauge moves only once admission has succeeded. The harness named this worker on every chunk in the assignment document, and the reader resolves membership from the chunk's own worker_indexes rather than from the worker's chunk list. So `assigned: false` and NoChunksForWorker both built a document that assigned everything — GAP-3's advertised fault input was inert, and would have stayed inert under a test written against it. declared_gaps_cite_the_spec checked identifier shape, which NOTREAL-999 satisfies, and took a token's leading digits, which resolved a mistyped GAP-3x to GAP-3. It now requires the digits to reach the end of the token and the id to exist in spec/, parsed by the three definition forms check_spec.py recognises. Continuations like the 2 of IB-1/2 carry no prefix and are still unchecked. The cap assertion was `peak <= cap`, which a worker stuck at a single slot also satisfies; equality catches that, and running at two caps is what distinguishes honouring the configured value from honouring some ceiling. The test keeps join_all, because one task running every admission check in a burst is what makes the rejection certain — a task per query lets early ones finish first and goes flaky. What it needed was a sampler that can see a few instructions, so that thread spins rather than sleeping. The tier only ran under --all-features, so mvcc-chunks was always on and its 37 cfg sites never took the other branch, while the Dockerfile builds default features. It now runs both, matrixed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds Phase 0 of the conformance program from spec/13's build order — the harness stubs, the structural validators, deterministic seeding, and one end-to-end smoke — then uses it to close the first P0 gap in the register.
Follows #63, which landed the spec suite itself.
Phase 0 harness
tests/harness/implements HC-1/2/3/5/8/12:scheduler.rssqd-assignmentsbuilder (crypto-box headers included)origin.rsportal.rsregistry.rssqd_contract_client::Client— epoch advance, allocation change, read failurevalidators.rsseed.rs+corpus.rsTest binaries are one per subject:
e2e(assign → download → query → verify → logs pull),query_surface(RP-10/RP-20/RP-1 outcomes),query_concurrency(separate by necessity — the OB signals are process-global, so a gauge assertion can't share a process with other query-running tests).The crate gains a library target: an integration tier can't reach a binary-only crate, and MG-4 wants the conformance subset timed as its own CI job rather than buried in the unit-test run.
What it does not cover
The input side (IB-40/41/42/43/44) is genuinely black-box over HTTP, but the query side enters below the transport at
p2p::{validate_query, execute, build_delivery, build_log}. So IB-1/2 and theP2PControllerintake — queue capacity, reject fan-out, the assignment pending queue — are not under test. That's recorded inharness::UNCOVEREDand in a new "SUT boundary, as built" note in spec/13, anddeclared_gaps_cite_the_specenforces that every such declaration names a real spec identifier.GAP-1 closed
run_querybound its scopeguard to_, so the guard dropped at the end of its ownletstatement rather than at the end of the function. The slot was released before the query ran:max_parallel_querieswas unenforced (the real ceiling was the transport's message-handler product) and the running-query gauge fell back to zero as fast as it rose, so overload was invisible in metrics as well as unbounded in fact.Test landed first and was checked both ways — fails on the defect, passes on the fix, and stable over 10 consecutive runs. The register row is gone; RP-4, REQ-22, INV-31 and PF-1 no longer carry the exception, and P-Q-PAR reads observed-enforced in spec/15.
Two incidental fixes
src/cli.rs—sentry_is_enabledis declared positional (like its env-only siblings) but a barebooldefaults to a flag action, which clap rejects for positionals. Every debug-build invocation aborted,--helpincluded. Release compiled the assert out, which is why it survived..gitattributes— the repo never had one. LFS was configured and the index held correct pointers, but nothing routed*.parquetthrough the filter, so git compared 356 MB of smudged content against 131-byte pointers and calledtests/datapermanently modified — onegit add -Afrom inlining all of it. Stored oids match the working tree exactly, so declaring the filter reconciles the tree without touching the data.Also included
The freshness-attribution spec work: a rejection whose reference input is the worker's own clock is a worker fault, not a client one (ADR-20, INV-26, FM-55, GAP-33).
Verification
47 tests across 4 binaries, zero ignored.
cargo test --all-features,cargo fmt --check, clippy at the CI deny level, andspec/tools/check_spec.pyall clean.Next
GAP-2 (externally supplied content can terminate the process) and GAP-3 (no reconciliation deletion floor) remain open at P0. Their HC-1 fault knobs —
UnparseableFileUrl,NoChunksForWorker— exist, but no tests drive them yet.🤖 Generated with Claude Code