Skip to content

Rename eval suites to align with agentcontrol skill renames#68

Merged
ari-launchdarkly merged 1 commit into
mainfrom
devin/1779452853-rename-eval-suites
May 22, 2026
Merged

Rename eval suites to align with agentcontrol skill renames#68
ari-launchdarkly merged 1 commit into
mainfrom
devin/1779452853-rename-eval-suites

Conversation

@ari-launchdarkly
Copy link
Copy Markdown
Contributor

Summary

  • PR Sattensil/agentcontrol config naming #67 renamed skill directories from aiconfig-* to their new agentcontrol names, but PR Add promptfoo eval harness for skills + consolidate MCP endpoint #65 (eval harness) had already merged with the old directory names. This reconciles the eval suite directories and all references.
  • Renames: aiconfig-createconfigs-create, aiconfig-updateconfigs-update, aiconfig-toolsagentcontrol-tools (evals/tools/ was already taken by tool definitions), aiconfig-variationsconfigs-variations
  • Updated _manifest.js suite names, package.json npm scripts, README.md, promptfooconfig.yaml descriptions/comments, and script comments

Testing

  • Manual (describe below)

All 70 unit tests pass (npm test in evals/).

Notes

  • The aiconfig-tools eval suite was renamed to agentcontrol-tools (not tools) because evals/tools/ already contains definitions.json for the mock MCP tool definitions.

Link to Devin session: https://app.devin.ai/sessions/f0a8cdfd7a884292a5580ed989cda5fb
Requested by: @ari-launchdarkly

PR #67 renamed skills from aiconfig-* to their new names but PR #65
(eval harness) had already merged with the old directory names. This
reconciles the eval suite directories and all references:

- aiconfig-create -> configs-create
- aiconfig-update -> configs-update
- aiconfig-tools -> agentcontrol-tools (evals/tools/ was taken)
- aiconfig-variations -> configs-variations

Updated: _manifest.js, package.json scripts, README.md,
promptfooconfig.yaml descriptions/comments, and script comments.
@devin-ai-integration
Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@github-actions
Copy link
Copy Markdown

Skill eval results

Skill Before After Δ
agentcontrol/configs-create 100/100 (4/4) 100/100 (4/4) no change
agentcontrol/configs-update 80/100 (4/5) 80/100 (4/5) no change
agentcontrol/configs-variations 80/100 (4/5) 80/100 (4/5) no change
agentcontrol/tools 75/100 (3/4) 75/100 (3/4) no change
feature-flags/launchdarkly-flag-create 100/100 (3/3) 100/100 (3/3) no change

Only suites whose source actually changed since their last recorded score were re-run. Soft-failing while we stabilise the baseline.

@ari-launchdarkly ari-launchdarkly marked this pull request as ready for review May 22, 2026 12:37
@ari-launchdarkly ari-launchdarkly requested a review from a team as a code owner May 22, 2026 12:37
Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

⚠️ 1 issue in files not directly in the diff

⚠️ CONTRIBUTING.md references renamed script eval:aiconfig-create that no longer exists (CONTRIBUTING.md:46)

The PR renames eval:aiconfig-create to eval:configs-create in evals/package.json:6 but does not update the matching reference in CONTRIBUTING.md:46, which still says (e.g., eval:aiconfig-create). A contributor following these instructions would get a "missing script" error. This is an incomplete mechanical transformation — the rename was applied to all files under evals/ but missed this repo-root documentation file.

View 1 additional finding in Devin Review.

Open in Devin Review

@ari-launchdarkly ari-launchdarkly enabled auto-merge (squash) May 22, 2026 12:52
@ari-launchdarkly ari-launchdarkly merged commit 0cffd4a into main May 22, 2026
13 checks passed
@ari-launchdarkly ari-launchdarkly deleted the devin/1779452853-rename-eval-suites branch May 22, 2026 12:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants