[wchwawa]

TLDR

The object namespace recovery gate judged its restarted owner ready from a bare TCP connect on a port it had reserved long before, so a port collision let the first read after the restart fail with a transient Connection refused and turned a required check red. The gate now draws each owner's port when that owner starts and waits until the owner answers a metadata read through its published route.

Problem

On the first attempt of CI run 40152636320 for #583, the gate's restart-read-structure step failed: the CLI exhausted its three preflight attempts with Connection refused, class transient, exit 5. The retained evidence shows why. The restarted owner's log ends with server bind failed: Address already in use (os error 98): the gate had reserved that port with a throw-away socket about 17 seconds earlier, and another listener held it by the time the owner bound it. That listener also satisfied the gate's TCP readiness probe at once, so the gate read before the owner had claimed the shard, and the CLI routed to the dead first owner's endpoint. The rerun passed.

The same path carries a smaller window on every run: an owner publishes its route before it binds its listener, and a TCP probe cannot tell a listener from a serving owner.

No issue tracks this flake. This is a CI harness fix with no product change, so it uses the CI-chore exception.

Consequences

A required check fails for a reason unrelated to the change under test, and the message blames the client's transport rather than the owner's bind failure, so the maintainer reruns the job instead of fixing anything. The readiness the gate relied on was never evidence that the owner served.

Example

Before: the gate reserves both owner ports at startup, kills the first owner, starts the second on its reserved port, returns as soon as any socket accepts on that port, and reads. When the port was taken meanwhile, the owner dies on bind while the gate has already moved on, and the read fails with Connection refused.

After: the gate draws the port when it starts each owner, then runs operation status with an unknown id until the owner answers not_found through its published route. A bind failure fails the gate as owner-e2-ready: owner exited before serving (1) next to the owner log, and a slow start waits, bounded by --timeout.

Solution

start_owner draws the port immediately before spawning the owner and hands the process to wait_owner_serves, which probes operation status with a single-attempt client through the same route the Workbench reads use and grades each answer with the shared owner_serves contract check, as the live Workbench and restore composition gates already do. The loop fails fast when the owner exits and gives up at the gate's --timeout. Both owners of the gate start this way; the control-record and epoch checks after them are unchanged. wait_tcp stays in the module because the fork-restore and append-identity gates import it.

Gain

The gate no longer reports a readiness it did not observe. A port taken between reservation and bind can no longer happen, and a read between route publication and bind can no longer happen; an owner that still fails to start is reported as such. No quantified claim.

Tradeoffs

Each owner start costs a few extra CLI invocations, one every 250 ms until the owner serves, recorded in processes.jsonl under owner-e1-ready and owner-e2-ready. The evidence schema and scenario names are unchanged. The fork-restore and append-identity gates still pre-reserve owner ports and probe TCP only; they are out of scope here. A collision in the sub-second window between drawing the port and the bind remains possible and now fails loudly.

Validation

• python3 scripts/workbench/object_namespace_recovery_gate_test.py: 26 tests OK, 4 of them new.
• python3 scripts/workbench/fork_restore_recovery_gate_test.py: 19 tests OK. PYTHONPATH=scripts/workbench python3 -m unittest scripts/workbench/qualification_invocation_manifest_test.py: 5 tests OK.
• Live gate on macOS against the pinned RustFS image, Homebrew etcd and a debug binary from rustup run 1.99.1 cargo build -p nokv --bin nokv: python3 scripts/workbench/object_namespace_recovery_gate.py --binary <target>/debug/nokv --evidence-dir <dir> --timeout 60: PASS in 13 s. The recorded probes show the window the change closes: the first owner answered not_found on its 8th probe after seven logical shard is Unassigned, not Serving answers; the restarted owner answered on its 7th probe after five Connection refused answers against the stale epoch-1 route and one logical shard is Recovering, not Serving; restart-read-structure then succeeded on its first attempt.
• python3 scripts/ci/pr_description.py --pull-request-json <this description>: OK.
• git diff --check: clean.
• Not run: Rust format, clippy and tests, because no Rust code changes. The other live gates are unchanged.