Skip to content

Migrate eval engine to skill-litmus #122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mrizzi wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Migrate eval engine to skill-litmus #122

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
9641a2e
chore(evals): add plugin field to all evals.json files
mrizzi May 5, 2026
3578e4e
ci: replace eval workflows with skill-litmus action
mrizzi May 5, 2026
5190b19
refactor: remove run-evals skill from sdlc-workflow
mrizzi May 5, 2026
5665efb
docs: update eval references from sdlc-workflow to skill-litmus
mrizzi May 5, 2026
0df540f
ci: bump skill-litmus action to v0.1.3
mrizzi May 5, 2026
e971531
ci: bump skill-litmus action to v0.1.4
mrizzi May 6, 2026
f0d16b9
ci: bump skill-litmus action to v0.1.5
mrizzi May 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
140 changes: 0 additions & 140 deletions .github/workflows/eval-baseline.yml
View file
Open in desktop

This file was deleted.

152 changes: 0 additions & 152 deletions .github/workflows/eval-pr.yml
View file
Open in desktop

This file was deleted.

35 changes: 35 additions & 0 deletions .github/workflows/eval.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: Skill Evals

on:
pull_request:
paths:
- 'plugins/sdlc-workflow/skills/**/*.md'
- 'evals/**/evals.json'
Comment on lines +4 to +7
Copy link
Copy Markdown
Contributor

@sourcery-ai sourcery-ai Bot May 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (bug_risk): Workflow will not have access to secrets.GCP_SA_KEY or pull-requests: write permissions for PRs from forks, likely breaking evals for external contributions.

For PRs from forks, this job won’t be able to use GCP_SA_KEY or pull-requests: write, so evals will fail or be skipped for external contributors.

To support forked PRs, consider either:

  • Using pull_request_target only for the secret/commenting step, with a hardened checkout (e.g., ref: ${{ github.event.pull_request.head.sha }} and restricted paths), or
  • Splitting into two jobs: one running evals on pull_request without secrets, and a pull_request_target job that just posts results using secrets.

Otherwise, external contributors will have a worse experience than maintainers’ branches.

push:
branches: [main]
paths:
- 'plugins/sdlc-workflow/skills/**/*.md'
- 'evals/**/evals.json'

jobs:
eval:
runs-on: ubuntu-latest
permissions:
contents: write
pull-requests: write
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0

- uses: google-github-actions/auth@v3
with:
credentials_json: ${{ secrets.GCP_SA_KEY }}

- uses: mrizzi/skill-litmus@v0.1.5
env:
CLAUDE_CODE_USE_VERTEX: "1"
CLOUD_ML_REGION: ${{ secrets.GCP_CLOUD_ML_REGION }}
ANTHROPIC_DEFAULT_SONNET_MODEL: "claude-sonnet-4-6"
ANTHROPIC_DEFAULT_OPUS_MODEL: "claude-opus-4-6"
ANTHROPIC_MODEL: "claude-opus-4-6"
Loading
Loading