Skip to content

test: drive the live Workbench gate through the native CLI - #588

Merged
wchwawa merged 3 commits into
mainfrom
test/live-workbench-gate-cli
Oct 27, 2026
Merged

wchwawa merged 3 commits into
mainfrom
test/live-workbench-gate-cli

Conversation

@wchwawa

@wchwawa wchwawa commented Oct 26, 2026 •

Copy link
Copy Markdown
Member

TLDR

The live Workbench gate exercised the 18 tools through one nokv mcp process and retained mcp-transcript.jsonl. Every tool call is now its own nokv workbench <tool> process graded against the native CLI contract, the tool definitions come from nokv schema, and the pre-#423 ledger admits the resulting cli-transcript.

Problem

scripts/workbench/live_workbench.py is the required nokv-workspace live gate for the 18-tool workflow and the C06 root-authority probe, yet it reached the product through the deprecated sidecar: one long-lived MCP session for the workflow, two more for the peer root and its reconnect, and a failed MCP initialize as the wrong-AgentId probe. Its evidence role was mcp-transcript, which the digest-pinned ledger required from the live-workbench producer, so the workflow could not be qualified over the primary surface without a ledger-policy change.

Merge after #587, which moved the object-namespace gate and introduced the shared contract grader; this PR targets main and carries #587's commits until #587 merges, after which its diff is its own two commits. It is the second of three gate migrations and links no issue: it is a test-harness chore with no product behavior change.

Consequences

The sidecar stayed load-bearing in CI, Gate 0 evidence described a deprecated transport, and the acceptance document had to declare the native 18-operation transcript unsatisfiable as specified.

Example

Before, the gate wrote a JSON-RPC tools/call line to a nokv mcp child and read structuredContent back; a replayed create-only put was an isError result whose content said AlreadyExists.

After, the gate runs nokv ... workbench workbench_put_file '{...}'. The replayed create-only put exits 4 with one stderr line {"status":"error","code":"AlreadyExists",...,"class":"precondition_failed","next_action":"reread_then_retry"}, and a read under a wrong --agent-id exits 2 with AgentBindingMismatch, invalid_input, fix_input before any Workbench request is sent.

Solution

Each ToolStep runs through nokv workbench <tool> and is graded with the shared grader: a success is one JSON result without internal keys; an expected failure names its Agent code plus the contract class and next action, and must arrive as one stderr line with the exit code of that class. The plan's two AlreadyExists steps and the probe's NotFound read carry those expectations. The 18 tool definitions and contract.json come from nokv schema, checked against the frozen Rust-owned schema by the existing validator. Owner readiness is observed as operation status of an unknown id answering not_found rather than a connection failure, the probe the CLI contract gate already uses.

The C06 probe keeps its reads and writes across the two roots. The reconnect read is a fresh process that re-admits the peer root from the control store; the wrong-AgentId probe is a graded AgentBindingMismatch refusal whose message must not disclose either AgentId. cli-transcript.jsonl replaces mcp-transcript.jsonl. The typed producer path accepts one transcript role per producer, cli-transcript for the CLI and mcp-transcript for the historical LingTai gap; the ledger's live-workbench producer now requires cli-transcript, and the ledger policy digest is re-pinned.

Gain

Two of the three sidecar-bound gates now run on the primary surface. Gate 0's 18-operation transcript is a native CLI transcript, and the acceptance document no longer carries the "unsatisfiable as specified" caveat.

Tradeoffs

Three expectations are restated at the CLI level: the peer session's tools/list check was an MCP handshake artifact and becomes one nokv schema validation; "rejected before MCP initialize" becomes "refused at root admission as AgentBindingMismatch"; a reconnect is a fresh process rather than a second stdio session. The ledger policy digest changes, which is the deliberate policy change the acceptance document called out. The restore-composition gate still launches the sidecar, so nokv mcp cannot be deleted yet.

Validation

  • python3 scripts/workbench/live_workbench_test.py (25 tests, OK); typed_live_qualification_test.py (5, OK); qualification_receipt_test.py, qualification_invocation_manifest_test.py, pre423_contract_ledger_test.py, qualification_aggregate_test.py, source_bound_producer_test.py (OK); python3 scripts/workbench/pre423_contract_ledger.py PASS on the re-pinned digest; the whole workbench-contract job sequence (py_compile list, every *_test.py, bash -n start_rustfs.sh, the unittest set, qualification_invocation_check PASS) run locally.
  • Live run on this Mac with etcd 3.7.1 and the pinned RustFS image, using the rust.yml step's arguments: workbench_workflow.status == "PASS", acceptance_gates["0"].status == "NOT QUALIFIED", 50 CLI transcript entries, contract.json with 18 tools from nokv schema. The failures arrived exactly as expected: two AlreadyExists exit 4 precondition_failed/reread_then_retry, one NotFound exit 3 not_found/do_not_retry, one AgentBindingMismatch exit 2 invalid_input/fix_input.
  • Not run locally: the GitHub nokv-workspace job and the Phase 1 receipt step, whose live-workbench invocation reports NOT QUALIFIED before executing and now publishes a cli-transcript.jsonl gap stub; that path is covered by the receipt and typed unit tests.

A live producer retains its transport's exact requests and responses under
one transcript role, cli-transcript for the native CLI or mcp-transcript for
the deprecated sidecar and the historical LingTai gap, in <role>.jsonl beside
its result. The pre-#423 ledger's live-workbench producer now requires
cli-transcript, its policy digest is re-pinned, and the Phase 1 receipt step
declares that evidence.

Signed-off-by: wchwawa <wch19961116@gmail.com>
The gate drove the 18 tools through one long-lived nokv mcp process, two
more for the peer root and its reconnect, and a failed MCP initialize as
the wrong-AgentId probe. Every tool call is now its own nokv workbench
<tool> process graded against docs/cli-contract.md: a success is one JSON
result without internal keys, an expected failure names its code, class,
and next action and arrives as one stderr line with the exit code of its
class. The tool definitions come from nokv schema, owner readiness is an
operation status probe answering not_found, the reconnect read is a fresh
process that re-admits the peer root, and the wrong AgentId is refused at
root admission as AgentBindingMismatch. cli-transcript.jsonl replaces
mcp-transcript.jsonl.

Signed-off-by: wchwawa <wch19961116@gmail.com>
@wchwawa
wchwawa requested a review from feichai0017 as a code owner October 26, 2026 05:28
@wchwawa
wchwawa changed the base branch from test/object-namespace-gate-cli to main October 26, 2026 05:40
@wchwawa wchwawa closed this Oct 26, 2026
@wchwawa wchwawa reopened this Oct 26, 2026
Signed-off-by: wchwawa <wch19961116@gmail.com>
@wchwawa
wchwawa merged commit 5516f30 into main Oct 27, 2026
10 checks passed
@wchwawa
wchwawa deleted the test/live-workbench-gate-cli branch October 27, 2026 01:33
feichai0017 pushed a commit to feichai0017/NoKV that referenced this pull request Oct 29, 2026
The stdio JSON-RPC sidecar was deprecated in 2026-08 and kept only because
the live qualification producers drove the product through it. Those
producers now run the native CLI (NoKV-Lab#587, NoKV-Lab#588, NoKV-Lab#589), so the command's only
callers were its own parser and dispatch arms and historical documentation.

Delete the transport module, the `mcp` command and its two profiles, the
stderr deprecation warning, the usage lines and the CLI-only generic-profile
handler builder. `nokv mcp` now fails as an unknown command (invalid_input,
exit 2); two tests pin that the command is unknown and that the help text no
longer mentions it.

Documents that still described the sidecar as a retained transport (README,
AGENTS.md, code contract, nokv-agent, review checklist, workspace acceptance,
workbench preflight, the live Workbench and scripts READMEs, the ledger guide
and the LingTai announcement notes) now say it is removed; the crate and
Homebrew descriptions follow. The unused JSON-RPC tools/list extractor leaves
scripts/workbench/workbench_contract.py.

The pre-NoKV-Lab#423 ledger's L01 item was source-bound to the `McpProfile` enum.
It is re-stated as the generic seven-tool Agent profile kept in nokv-agent,
the api-decision producer binds it to SdkGenericAgentToolHandler and its
GenericAgentToolHandler impl, and the policy digest is re-pinned. Gates and
dispositions are unchanged. The lingtai-mcp gap producer stays: it runs
nothing, and retiring that evidence class is a ledger change of its own.

Signed-off-by: wchwawa <wch19961116@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant