[wchwawa]

TLDR

An owner-mediated upload no longer holds the shard's exclusive write gate while it writes a 4 MiB block to the provider. The request admits the block in one short session, writes with no metadata lock, and records it in a second session, so the shard keeps answering every other request while blocks are in flight. A new probe measures that shape and fails when loaded p50 exceeds idle p50 by more than a fifth.

Problem

The owner-mediated upload path (#586, the R5 slice of #566) moved object writes from Agents to the shard owner. The executor's dispatcher runs every mutating request as one metadata write session, and that session holds the shard's exclusive command gate for the whole handler. For UploadArtifactObject the handler re-encodes the 4 MiB body for its request fingerprint, hashes it twice, and performs the provider PUT, all under the gate. Every other request of the shard, reads included, queued behind that provider round trip. No separate issue exists; this is the first R5b item recorded under #566 after the Habitat spikes.

Consequences

One Agent publishing a large artifact slowed every other Agent on the shard. With eight interactive sessions beside one unpaced 16 MiB uploader on the pre-change binary, the sessions' put p50 rose 35% and their request rate fell 40% (pooled over two cycles); a single-session measurement showed stat p90 going from 0.4 ms to 26 ms. The handler's own comment claimed the provider create ran under no metadata lock, which was false.

Example

A shard serves eight agents running a tool loop (stat a file, write a 4 KiB file) while a ninth agent publishes 16 MiB results back to back. Before this change the eight see 17.09 ms put p50 under the uploader against 12.67 ms idle, and their combined rate drops from 735.9 to 441.2 requests per second. After it they see 14.77 ms against 14.22 ms idle, at 652.6 against 709.1 requests per second, while the uploader itself moves 60.0 MiB/s.

Solution

Requests are now classified three ways instead of two: reads share the gate, ordinary mutations run as one write session, and the upload sequences itself. The upload computes the request fingerprint and the body digest before any gate. Its first session binds the request id, replays an already recorded block, proves the operation current and the token valid, and checks the staged record's length and digest, so a wrong body is refused before the owner pays for a provider write. The provider create then runs with no metadata lock held. Its second session reloads the operation and the staged record at a fresh read version and marks the object uploaded; the token is not rechecked there because its digest covers the activity deadline that the first session's own heartbeat may have moved, and the record's own fences (phase, cursor, optimistic concurrency on both rows) protect the write. A duplicate of the request that recorded first answers with the recorded outcome.

scripts/workbench/owner_upload_interference_probe.py runs long-lived workbench-session processes beside one uploader through the TLS gate's stack (read-only provider proxy for Agents, owner holds the writing credential), over idle, paced, unpaced, and idle-again conditions, repeated for a configurable number of cycles with a per-bucket timeline so a stall from elsewhere is visible as such. It is not wired into CI beyond the script compile check; its evidence is below.

Gain

Interactive latency under an unpaced uploader is back to idle: candidate stat p50 7.02 → 7.1 ms and put p50 14.22 → 14.77 ms (pooled over 2 cycles of 40 s), against 5.83 → 7.31 ms and 12.67 → 17.09 ms on the pre-change binary. Two tests pin the shape: a provider that holds its create does not block a concurrent read or write, and concurrent duplicates of one upload record the object once.

Tradeoffs

Each block now costs two acknowledged sessions instead of one, so an upload pays one more fsync per 4 MiB block; the uploader's own throughput was 63.8 MiB/s before and 60.0 MiB/s after in the probe. The paced condition of the probe is reported but not judged, because on this machine it caught stalls that belong to store maintenance rather than to the uploader; the timeline exists to tell them apart. No admission limit on concurrent uploads is added: with the gate fixed, the measured interference is within the uploader's bandwidth share, and the frame size already bounds memory per connection.

Validation

• cargo fmt --all -- --check, cargo clippy -p nokv-server --all-targets -- -D warnings (Rust 1.99.1).
• cargo test -p nokv-server --lib: 234 passed, 0 failed (232 before plus the two new tests).
• scripts/workbench/restore_composition_gate.py --build locally against the pinned RustFS image and etcd: overall_status PASS, restore_composition PASS, partial_publication_recovery PASS.
• scripts/workbench/rpc_tls_gate.py and scripts/workbench/cli_contract_gate.py against the release binary with moto: TLS gate t0–t14 PASS and CLI contract gate s1–s5, s7 PASS (overall PASS for both).
• scripts/workbench/owner_upload_interference_probe.py --sessions 8 --seconds 40 --cycles 2 with the pre-change binary as baseline on an Apple Silicon laptop with moto as the provider: verdict QUALIFIED for the candidate; the full report with per-cycle timelines is archived as evidence. Numbers above are from that run; they are one machine's and the provider is in-process moto, so absolute values are not a throughput claim.
• Not run: the Python contract gate (no SDK surface changed) and cross-host throughput.