Rename eval suites to align with agentcontrol skill renames#68
Conversation
PR #67 renamed skills from aiconfig-* to their new names but PR #65 (eval harness) had already merged with the old directory names. This reconciles the eval suite directories and all references: - aiconfig-create -> configs-create - aiconfig-update -> configs-update - aiconfig-tools -> agentcontrol-tools (evals/tools/ was taken) - aiconfig-variations -> configs-variations Updated: _manifest.js, package.json scripts, README.md, promptfooconfig.yaml descriptions/comments, and script comments.
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
Skill eval results
Only suites whose source actually changed since their last recorded score were re-run. Soft-failing while we stabilise the baseline. |
There was a problem hiding this comment.
Devin Review found 1 potential issue.
⚠️ 1 issue in files not directly in the diff
⚠️ CONTRIBUTING.md references renamed script eval:aiconfig-create that no longer exists (CONTRIBUTING.md:46)
The PR renames eval:aiconfig-create to eval:configs-create in evals/package.json:6 but does not update the matching reference in CONTRIBUTING.md:46, which still says (e.g., eval:aiconfig-create). A contributor following these instructions would get a "missing script" error. This is an incomplete mechanical transformation — the rename was applied to all files under evals/ but missed this repo-root documentation file.
View 1 additional finding in Devin Review.
Summary
aiconfig-*to their new agentcontrol names, but PR Add promptfoo eval harness for skills + consolidate MCP endpoint #65 (eval harness) had already merged with the old directory names. This reconciles the eval suite directories and all references.aiconfig-create→configs-create,aiconfig-update→configs-update,aiconfig-tools→agentcontrol-tools(evals/tools/was already taken by tool definitions),aiconfig-variations→configs-variations_manifest.jssuite names,package.jsonnpm scripts,README.md, promptfooconfig.yaml descriptions/comments, and script commentsTesting
All 70 unit tests pass (
npm testinevals/).Notes
aiconfig-toolseval suite was renamed toagentcontrol-tools(nottools) becauseevals/tools/already containsdefinitions.jsonfor the mock MCP tool definitions.Link to Devin session: https://app.devin.ai/sessions/f0a8cdfd7a884292a5580ed989cda5fb
Requested by: @ari-launchdarkly