Skip to content

test: drive the restore composition gate through the native CLI - #589

Merged
wchwawa merged 3 commits into
mainfrom
test/restore-composition-gate-cli
Oct 27, 2026
Merged

wchwawa merged 3 commits into
mainfrom
test/restore-composition-gate-cli

Conversation

@wchwawa

@wchwawa wchwawa commented Oct 26, 2026 •

Copy link
Copy Markdown
Member

TLDR

The restore-composition gate, the last live gate on the deprecated nokv mcp sidecar, now drives the 18-tool oracle through nokv workbench <tool> processes. Its "bounded client failure after owner loss" verdict is restated as the native CLI contract: exit 6, class outcome_unknown, next_action: resubmit_same, one stderr JSON line.

Problem

scripts/workbench/restore_composition_gate.py ran the A-to-B-to-C composition oracle through one nokv mcp process per owner epoch and retained mcp-transcript.jsonl. Its owner-loss verdict, as #586 showed, was a statement about the sidecar's error result (isError, ClientFailure, details.attempts == 3, "the MCP process survived the owner exit"), not about what a CLI caller observes. With the object-namespace gate (#587) and the live Workbench gate (#588) already moved, this gate was the only reason the sidecar's code still shipped as harness plumbing.

Merge after #588; this PR targets main and carries the #587 and #588 commits until they merge, after which its diff is its own two commits. It is the third of three gate migrations and links no issue: a test-harness chore with no product behavior change.

Consequences

Until this gate moved, nokv mcp could not be deleted, and the restore-crash evidence qualified a deprecated transport rather than the primary surface.

Example

Before, the gate sent workbench_restore over JSON-RPC to the sidecar while the feature-only crash owner exited at the dual-manifest-published boundary, and accepted an isError result saying ClientFailure after three attempts.

After, the gate runs nokv ... workbench workbench_restore '{...}' against the crash owner. The command exits 6, prints nothing to stdout, and writes one stderr line: {"status":"error","code":"ClientFailure","message":"workspace request exhausted 3 attempts: RPC transport failed: ...","retryable":true,"details":{"attempts":3,"source":"nokv-client"},"class":"outcome_unknown","next_action":"resubmit_same"}. After the successor owner reopens the same Holt directory, the same command replays with idempotent_replay: true, the same operation id, and an unchanged object inventory.

Solution

A WorkbenchCli handle replaces the Mcp session: each oracle step is its own nokv workbench <tool> process recorded in cli-transcript.jsonl and graded with the shared contract grader from #587; expected NotFound reads carry their class and next action (not_found, do_not_retry). The owner-loss call is graded as outcome_unknown with resubmit_same, and the restated validate_owner_loss_error requires ClientFailure from nokv-client after exactly three attempts plus that class and next action; the evidence records the CLI exit code in place of the sidecar's process liveness. Owner readiness for each epoch is an operation status probe of an unknown id answering not_found, shared with the live Workbench gate through cli_contract.owner_serves. The oracle plan's tool steps now name the workbench CLI verb beside the existing workspace-path mutations. AGENTS.md, the acceptance document, and the Workbench README no longer describe any gate as a sidecar caller.

Gain

No live gate launches nokv mcp: grep -rn 'nokv mcp\|"mcp"' scripts/workbench/*.py matches only the three regression tests that assert the token is absent from their gate. The restore-crash verdict is now a statement about docs/cli-contract.md, and deleting the sidecar is a product change that can be its own PR.

Tradeoffs

Two expectations are restated at the CLI level: "the MCP process survived the owner exit" becomes "the command exited with the graded failure code", and the legacy retryable flag is recorded but no longer required, as the contract directs. Each tool call is a fresh process, so the gate no longer observes a client that stays connected across the owner handoff; that is the surface the sidecar had, not the one the product supports. The sidecar's own command, module, and historical documentation remain for a follow-up removal.

Validation

  • python3 scripts/workbench/restore_composition_gate_test.py (26 tests, OK); live_workbench_test.py, cli_contract_gate_test.py, rpc_tls_gate_test.py, object_namespace_recovery_gate_test.py, fork_restore_recovery_gate_test.py, local_wal_recovery_gate_test.py, workbench_contract_test.py, loopx_stage2a_stack_test.py (OK); python3 -m py_compile over the changed scripts; git diff --check.
  • python3 scripts/workbench/restore_composition_gate.py --dry-run exits 0 with plan surfaces workbench and workspace-path.
  • Live run on this Mac with etcd 3.7.1, the pinned RustFS image, the default nokv and the feature-only nokv-restore-crash-owner built in separate targets, --timeout 60: overall_status PASS, restore_composition PASS, partial_publication_recovery PASS; 36 CLI transcript entries; the owner-loss restore exited 6 as ClientFailure/outcome_unknown/resubmit_same with details.attempts 3, the crash owner exited 86 after publishing exactly two manifest objects, the replay was idempotent with a stable inventory, and the four expected NotFound reads exited 3 as not_found/do_not_retry.
  • Not run locally: the GitHub nokv-workspace job's --build path; the Phase 1 receipt step is unchanged (the restore-composition producer declares no transcript role).

cli_contract gains published_contract(repository) and owner_serves(call,
exits, actions): an operation status probe of an unknown id proves that an
owner serves metadata reads once it answers not_found rather than a
transport failure. The live Workbench gate uses both instead of its own
copies.

Signed-off-by: wchwawa <wch19961116@gmail.com>
…e CLI

The gate ran the A-to-B-to-C oracle through one nokv mcp process per owner
epoch and judged owner loss by the sidecar's error result. Every oracle
step is now its own nokv workbench <tool> process recorded in
cli-transcript.jsonl and graded against docs/cli-contract.md. The restore
whose owner exits at the fault barrier must fail as one stderr line of
ClientFailure from nokv-client after three attempts, class
outcome_unknown, next_action resubmit_same, with the exit code of that
class; the evidence records that exit code in place of the sidecar's
process liveness, and the exact replay still converges on the successor.
Expected NotFound reads carry not_found and do_not_retry. No gate launches
the sidecar any more.

Signed-off-by: wchwawa <wch19961116@gmail.com>
@wchwawa
wchwawa requested a review from feichai0017 as a code owner October 26, 2026 05:40
@wchwawa
wchwawa changed the base branch from test/live-workbench-gate-cli to main October 26, 2026 05:40
@wchwawa wchwawa closed this Oct 26, 2026
@wchwawa wchwawa reopened this Oct 26, 2026
Signed-off-by: wchwawa <wch19961116@gmail.com>
@wchwawa
wchwawa merged commit 8520c27 into main Oct 27, 2026
10 checks passed
@wchwawa
wchwawa deleted the test/restore-composition-gate-cli branch October 27, 2026 01:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant