Add a model-in-the-loop eval harness - #26
roborourke wants to merge 1 commit into
Conversation
c244d98 to
f0d4c2e
Compare
2226dd0 to
651d7ea
Compare
Playwright test results
Details
|
f0d4c2e to
75dc6c5
Compare
651d7ea to
5de30a9
Compare
| const DEFAULT_MODEL = 'claude-haiku-4-5'; | ||
|
|
||
| /** | ||
| * Per-million-token prices, for reporting what a run cost. | ||
| * | ||
| * @see https://www.anthropic.com/pricing | ||
| */ | ||
| const PRICES = { | ||
| 'claude-haiku-4-5': { input: 1, output: 5 }, | ||
| 'claude-sonnet-5': { input: 2, output: 10 }, | ||
| 'claude-opus-5': { input: 5, output: 25 }, | ||
| }; |
There was a problem hiding this comment.
do we need something to keep an eye on these prices or availability of the models?
There was a problem hiding this comment.
hm good point. It's how well we can attribute the costs but maybe this action can request and show the cost of the run along with the PR comment. Not sure if you can otherwise label API requests. Closest is maybe using the workspace id header in requests but not sure if it allows you to do price based reporting.
5de30a9 to
da4bdfc
Compare
roborourke
left a comment
roborourke
left a comment
There was a problem hiding this comment.
We're going another route here, as MCP is a protocol it's enough just to test that. This eval is useful for local testing only but we won't use it with CI, so no longer needs to rely on an API key, we can do it through a local subscription.
The end-to-end tests prove the MCP server answers correctly. They can't prove an agent works out how to use the tools from their descriptions and schemas alone, which is what regresses quietly when someone rewords a description or changes a schema. Claude Code is the harness rather than a turn loop of our own. It connects to the MCP endpoint itself, so a run exercises a real client against the real protocol instead of a reimplementation that can drift from one. It also bills to a local subscription, so there is no API key anywhere and nothing to wire into CI. The Anthropic SDK dependency goes with it. Two CLI flags carry the weight: - `--restricted` removes Bash and the other code-running tools. Without it an agent can curl the REST API directly and pass without ever touching MCP, which would make the whole thing meaningless. - `--strict-mcp-config` ignores the developer's own MCP servers, so a run only ever sees the test site. Grading reads WordPress state over a separate REST connection, never the agent's account of its own work, so reporting success without changing anything fails. `--dry-run` runs the setup and graders with no agent and asserts every scenario fails. That replaces the scripted provider: it catches a grader that passes when nothing happened, costs nothing, and needs no model. It immediately earned its place — subscriber-blocked passed under it, because "no post was created" is equally true when the agent was refused and when it never ran. That scenario now pairs the forbidden action with a permitted fallback, so the fallback is what proves the run reached the tools and was turned away. Seven scenarios, all passing against sonnet, in three to four turns each. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
da4bdfc to
1d3d78b
Compare
|
Reworked for local-only, no API key. Rebased onto main, so it picks up the read/write/delete tool split. Claude Code is the harness now. Instead of my own turn loop against the SDK, Two flags do the heavy lifting:
The
First real run. Previously this was written but unexercised. Now: No CI wiring, per your note on #27. 🤖 Generated with Claude Code |
2 participants
Closes #17
Reworked for local-only, no API key. Rebased onto main, so it picks up the read/write/delete tool split.
Claude Code is the harness now. Instead of my own turn loop against the SDK,
run.mjsboots Playground, writes an MCP config, and runsclaude -p. That means a real client against the real protocol rather than a reimplementation of one, and it bills to your subscription.Two flags do the heavy lifting:
--restrictedremoves Bash and the other code-running tools. Without it the agent can curl the REST API directly and pass without touching MCP at all.--strict-mcp-configignores your own MCP servers, so a run only sees the test site.The
@anthropic-ai/sdkdependency is gone.package-lock.jsonis now identical to main's.--dry-runreplaces the scripted provider. It runs setup and the graders with no agent and asserts every scenario fails, which catches a grader that passes when nothing happened. It earned its keep immediately:subscriber-blockedpassed under it, because "no post was created" is equally true whether the agent was refused or never ran. That scenario now pairs the forbidden action with a permitted fallback, and the fallback is what proves the run reached the tools and was turned away.First real run. Previously this was written but unexercised. Now:
No CI wiring, per your note on #27.
🤖 Generated with Claude Code