Skip to content

docs(mcp): use recorded MCP tool names and document mocked server behavior - #339

Merged
zpzjzj merged 2 commits into
alibaba:mainfrom
karanb192:docs/mcp-tool-names
Oct 26, 2026
Merged

zpzjzj merged 2 commits into
alibaba:mainfrom
karanb192:docs/mcp-tool-names

Conversation

@karanb192

@karanb192 karanb192 commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The tool_called examples in the writing guide use server::tool names (github::create_pull_request, project-mgmt::create_publish_plan_simple). tool_called compares names exactly, and Claude Code, Codex, and Qoder CLI record MCP tools as mcp__<server>__<tool>, so a rule copied from these examples never matches. The skill-upper judge reference has the same example, so agents that write evals with skill-upper copy it too.

This PR switches the examples to the recorded form and adds a short list of how a tool_responses mock behaves.

Related issues

None. Docs-only fix, so I went straight to a PR as CONTRIBUTING allows.

Changes

  • docs/guide/writing-evals.md and docs/zh/guide/writing-evals.md:
    • tool_called examples use mcp__github__create_pull_request and mcp__project-mgmt__create_publish_plan_simple.
    • New note after the rule_based evaluation order: tool names match exactly, in the mcp__<server>__<tool> form.
    • New list under MCP configuration: stdio only; started with node, which must be on PATH (skill-up sets up Node.js only when it installs a Node-based engine CLI itself); one default response per tool; generic tool description and open input schema; an error for undeclared tools.
  • skills/skill-upper/references/judge-types.md: the same example fix plus a one-line note.

Test plan

  • make test passes
  • make verify passes (fmt + vet + lint)
  • npm run docs:build passes
  • Name forms checked against the code: codexMCPToolName in internal/agent/codex.go, the Qoder stream fixture in internal/agent/qodercli_test.go, and e2e/testdata/sandbox-mcp, which already uses mcp__agent-sandbox__.... The Claude Code adapter passes the stream name through, and Claude Code names MCP tools mcp__<server>__<tool> (Claude Code MCP docs).

Notes for reviewers

  • I left internal/judge/rule_based_test.go alone. It uses github::... names on synthetic transcripts and only tests exact matching.
  • An inline {{params...}} breaks the VitePress build, so the template bullet points to the example's name field instead.
  • I did not state the name form for Qwen Code or OpenCode because I could not confirm it from the tests.
  • The Chinese text is my translation. Please correct the wording if needed.

…avior

tool_called compares names exactly, and Claude Code, Codex, and Qoder CLI
record MCP tools as mcp__<server>__<tool>. The server::tool examples in
the writing guide and the skill-upper judge reference could never match.

Also list how a tool_responses mock behaves: stdio only, started with
node, one default response per tool, generic tool descriptions, and an
error for undeclared tools.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-25T16:44:38.327836Z fb553b5 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


Karan Bansal seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@zpzjzj zpzjzj left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zpzjzj
zpzjzj added this pull request to the merge queue Oct 26, 2026
Merged via the queue into alibaba:main with commit 738a1ee Oct 26, 2026
18 of 19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants