Run the evals in CI - #27
Closed
roborourke wants to merge 1 commit into
Closed
roborourke wants to merge 1 commit into
roborourke wants to merge 1 commit into
Conversation
roborourke
force-pushed
the
feature/evals-workflow
branch
from
September 11, 2026 14:01
7f5df7d to
efce293
Compare
roborourke
force-pushed
the
feature/eval-harness
branch
from
September 11, 2026 14:01
2226dd0 to
651d7ea
Compare
Playwright test results
Details
|
roborourke
force-pushed
the
feature/eval-harness
branch
from
September 11, 2026 16:06
651d7ea to
5de30a9
Compare
roborourke
force-pushed
the
feature/evals-workflow
branch
from
September 11, 2026 16:06
efce293 to
4679556
Compare
roborourke
force-pushed
the
feature/eval-harness
branch
from
September 15, 2026 10:46
5de30a9 to
da4bdfc
Compare
Two levels, split by what they cost. The scripted run joins the existing CI workflow, so it runs on every push and pull request including forks. It replays fixed tool calls rather than asking a model, so it is free and deterministic. It checks the harness — MCP connection, tool discovery, the turn loop, every scenario's setup and grading — which is the part most likely to break and the part a model run can't isolate. The model run is its own workflow, on pushes to main and manual dispatch. No schedule: a nightly run spends money on a repository that may not have changed. It is skipped on forks, which cannot read secrets, and skips with a note rather than failing when no ANTHROPIC_API_KEY is set, so a contributor without one doesn't see a red tick they can't fix. Each scenario runs three times and passes on a majority, and only scenarios tagged `core` can fail the build. A model is not deterministic, so a single unlucky run must not block a merge. The step uses continue-on-error and the failure is re-raised afterwards, so the report is still summarised and uploaded when a scenario fails. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
roborourke
force-pushed
the
feature/evals-workflow
branch
from
September 15, 2026 10:47
4679556 to
7f77ff5
Compare
roborourke
commented
Sep 16, 2026
roborourke
left a comment
roborourke
left a comment
Collaborator
Author
There was a problem hiding this comment.
Going to close this one out, CI isn't a good place for non-deterministic tests. It's enough to have a way to confirm outputs locally via #26
This was referenced Sep 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #18
Stacked on #26. The base retargets as the PRs below it merge.
Two levels, split by what they cost.
Scripted joins the existing
CIworkflow, so it runs on every push and pull request, forks included. It replays fixed tool calls rather than asking a model, so it is free and deterministic. It checks the harness — MCP connection, tool discovery, the turn loop, each scenario's setup and grading — which is the part most likely to break and the part a model run can't isolate.Model is its own
Evalsworkflow, on pushes tomainand manual dispatch. No schedule, as you asked. Skipped on forks, which cannot read secrets. When noANTHROPIC_API_KEYis set it skips with a note rather than failing, so a contributor without one doesn't get a red tick they can't fix.Each scenario runs three times and passes on a majority, and only scenarios tagged
corecan fail the build — a single unlucky run shouldn't block a merge. The run step usescontinue-on-errorand the failure is re-raised at the end, so the report is still summarised and uploaded when something fails.Manual dispatch takes a model and a repeat count, so you can try
claude-sonnet-5or a single quick run without editing anything.🤖 Generated with Claude Code