Repository navigation
test: drive the live Workbench gate through the native CLI - #588
Merged
Merged
Conversation
A live producer retains its transport's exact requests and responses under one transcript role, cli-transcript for the native CLI or mcp-transcript for the deprecated sidecar and the historical LingTai gap, in <role>.jsonl beside its result. The pre-#423 ledger's live-workbench producer now requires cli-transcript, its policy digest is re-pinned, and the Phase 1 receipt step declares that evidence. Signed-off-by: wchwawa <wch19961116@gmail.com>
The gate drove the 18 tools through one long-lived nokv mcp process, two more for the peer root and its reconnect, and a failed MCP initialize as the wrong-AgentId probe. Every tool call is now its own nokv workbench <tool> process graded against docs/cli-contract.md: a success is one JSON result without internal keys, an expected failure names its code, class, and next action and arrives as one stderr line with the exit code of its class. The tool definitions come from nokv schema, owner readiness is an operation status probe answering not_found, the reconnect read is a fresh process that re-admits the peer root, and the wrong AgentId is refused at root admission as AgentBindingMismatch. cli-transcript.jsonl replaces mcp-transcript.jsonl. Signed-off-by: wchwawa <wch19961116@gmail.com>
Signed-off-by: wchwawa <wch19961116@gmail.com>
feichai0017
pushed a commit
to feichai0017/NoKV
that referenced
this pull request
Oct 29, 2026
The stdio JSON-RPC sidecar was deprecated in 2026-08 and kept only because the live qualification producers drove the product through it. Those producers now run the native CLI (NoKV-Lab#587, NoKV-Lab#588, NoKV-Lab#589), so the command's only callers were its own parser and dispatch arms and historical documentation. Delete the transport module, the `mcp` command and its two profiles, the stderr deprecation warning, the usage lines and the CLI-only generic-profile handler builder. `nokv mcp` now fails as an unknown command (invalid_input, exit 2); two tests pin that the command is unknown and that the help text no longer mentions it. Documents that still described the sidecar as a retained transport (README, AGENTS.md, code contract, nokv-agent, review checklist, workspace acceptance, workbench preflight, the live Workbench and scripts READMEs, the ledger guide and the LingTai announcement notes) now say it is removed; the crate and Homebrew descriptions follow. The unused JSON-RPC tools/list extractor leaves scripts/workbench/workbench_contract.py. The pre-NoKV-Lab#423 ledger's L01 item was source-bound to the `McpProfile` enum. It is re-stated as the generic seven-tool Agent profile kept in nokv-agent, the api-decision producer binds it to SdkGenericAgentToolHandler and its GenericAgentToolHandler impl, and the policy digest is re-pinned. Gates and dispositions are unchanged. The lingtai-mcp gap producer stays: it runs nothing, and retiring that evidence class is a ledger change of its own. Signed-off-by: wchwawa <wch19961116@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
The live Workbench gate exercised the 18 tools through one
nokv mcpprocess and retainedmcp-transcript.jsonl. Every tool call is now its ownnokv workbench <tool>process graded against the native CLI contract, the tool definitions come fromnokv schema, and the pre-#423 ledger admits the resultingcli-transcript.Problem
scripts/workbench/live_workbench.pyis the requirednokv-workspacelive gate for the 18-tool workflow and the C06 root-authority probe, yet it reached the product through the deprecated sidecar: one long-lived MCP session for the workflow, two more for the peer root and its reconnect, and a failed MCPinitializeas the wrong-AgentId probe. Its evidence role wasmcp-transcript, which the digest-pinned ledger required from thelive-workbenchproducer, so the workflow could not be qualified over the primary surface without a ledger-policy change.Merge after #587, which moved the object-namespace gate and introduced the shared contract grader; this PR targets main and carries #587's commits until #587 merges, after which its diff is its own two commits. It is the second of three gate migrations and links no issue: it is a test-harness chore with no product behavior change.
Consequences
The sidecar stayed load-bearing in CI, Gate 0 evidence described a deprecated transport, and the acceptance document had to declare the native 18-operation transcript unsatisfiable as specified.
Example
Before, the gate wrote a JSON-RPC
tools/callline to anokv mcpchild and readstructuredContentback; a replayed create-only put was anisErrorresult whose content saidAlreadyExists.After, the gate runs
nokv ... workbench workbench_put_file '{...}'. The replayed create-only put exits 4 with one stderr line{"status":"error","code":"AlreadyExists",...,"class":"precondition_failed","next_action":"reread_then_retry"}, and a read under a wrong--agent-idexits 2 withAgentBindingMismatch,invalid_input,fix_inputbefore any Workbench request is sent.Solution
Each
ToolStepruns throughnokv workbench <tool>and is graded with the shared grader: a success is one JSON result without internal keys; an expected failure names its Agent code plus the contract class and next action, and must arrive as one stderr line with the exit code of that class. The plan's twoAlreadyExistssteps and the probe'sNotFoundread carry those expectations. The 18 tool definitions andcontract.jsoncome fromnokv schema, checked against the frozen Rust-owned schema by the existing validator. Owner readiness is observed asoperation statusof an unknown id answeringnot_foundrather than a connection failure, the probe the CLI contract gate already uses.The C06 probe keeps its reads and writes across the two roots. The reconnect read is a fresh process that re-admits the peer root from the control store; the wrong-AgentId probe is a graded
AgentBindingMismatchrefusal whose message must not disclose either AgentId.cli-transcript.jsonlreplacesmcp-transcript.jsonl. The typed producer path accepts one transcript role per producer,cli-transcriptfor the CLI andmcp-transcriptfor the historical LingTai gap; the ledger'slive-workbenchproducer now requirescli-transcript, and the ledger policy digest is re-pinned.Gain
Two of the three sidecar-bound gates now run on the primary surface. Gate 0's 18-operation transcript is a native CLI transcript, and the acceptance document no longer carries the "unsatisfiable as specified" caveat.
Tradeoffs
Three expectations are restated at the CLI level: the peer session's
tools/listcheck was an MCP handshake artifact and becomes onenokv schemavalidation; "rejected before MCP initialize" becomes "refused at root admission asAgentBindingMismatch"; a reconnect is a fresh process rather than a second stdio session. The ledger policy digest changes, which is the deliberate policy change the acceptance document called out. The restore-composition gate still launches the sidecar, sonokv mcpcannot be deleted yet.Validation
python3 scripts/workbench/live_workbench_test.py(25 tests, OK);typed_live_qualification_test.py(5, OK);qualification_receipt_test.py,qualification_invocation_manifest_test.py,pre423_contract_ledger_test.py,qualification_aggregate_test.py,source_bound_producer_test.py(OK);python3 scripts/workbench/pre423_contract_ledger.pyPASS on the re-pinned digest; the wholeworkbench-contractjob sequence (py_compile list, every*_test.py,bash -n start_rustfs.sh, the unittest set,qualification_invocation_checkPASS) run locally.rust.ymlstep's arguments:workbench_workflow.status == "PASS",acceptance_gates["0"].status == "NOT QUALIFIED", 50 CLI transcript entries,contract.jsonwith 18 tools fromnokv schema. The failures arrived exactly as expected: twoAlreadyExistsexit 4precondition_failed/reread_then_retry, oneNotFoundexit 3not_found/do_not_retry, oneAgentBindingMismatchexit 2invalid_input/fix_input.nokv-workspacejob and the Phase 1 receipt step, whoselive-workbenchinvocation reports NOT QUALIFIED before executing and now publishes acli-transcript.jsonlgap stub; that path is covered by the receipt and typed unit tests.