Measure skill activation and adherence with a claude plugin eval suite

Problem

Nothing measures whether agents load the right skill, skip the wrong one, or follow the skill once it loads. CI checks skill metadata against the spec (validate-spec in .github/workflows/ci.yaml) but never checks triggering. The two evals/evals.json files (vendored gitops-knowledge and gitops-cluster-debug) grade output quality, not activation.

What spike #642 measured on the Claude host (7 days to 2026-11-04, 374 sessions plus 354 subagent transcripts):

127 native Skill invocations. 123 were the run-procedure skill, and none were domain skills like golang-pro, github-actions-hardening or gitops-knowledge.
Over the same window, at least 34 sessions edited Go, 23 edited workflow files and 11 edited Kubernetes manifests.
The runtime's skill listing showed 30 of 94 skills by name only, with no description. Ten of those were agentic-engineering skills (for example test-driven-development, secret-scanning and ways-of-working).

The zero is not proof of harm. Without a matched no-skill baseline, nobody can tell an unneeded skill from a missed one.

Proposal

Add an offline suite for claude plugin eval, the native Claude Code harness (checked on 2.1.317). It runs each case with and without the plugin and grades tool_used: Skill as a plugin-fired signal, separate from the score. Start report-only with about 8 to 12 cases from the #642 protocol:

positive cases;
near-miss negative cases (for example live-cluster debugging versus repository audit, and Actions cost versus Actions security);
an over-trigger guard (a trivial question should load no skill);
adherence cases graded by deterministic file or trace checks (test before implementation, explicit-path staging, the fixed self-review heading);
the product-context scenarios from Spike: evaluate product and customer context in engineering agent decisions #643 (scope creep, needless infrastructure, outcome tracking).

Run the suite on the agent host, on the existing monthly plugin-usefulness review in docs/plugin-effectiveness.md, with --no-publish. Keep results outside Git, as that doc already requires. Use only included subscription inference: no API keys and no paid judge.

Acceptance criteria

A suite under the plugin's eval directory with at least 8 cases covering positive, near-miss negative, over-trigger and adherence behaviour. Each case is repeated at least 3 times.
One recorded baseline run per case: activation recall, false-activation rate, adherence pass rate and the with/without score difference. The results stay in the private review record.
Any skill whose recall misses the target is fixed at the root cause (description wording, duplicate shadowing, or listing budget), then re-measured.
Docs change only by extending docs/plugin-effectiveness.md, with no new doc file (Spike: Improve first-pass agent implementation quality through reuse of established solutions #647).
Regression thresholds become blocking only after two baselines agree.

Related: #642 (protocol), #637 (domain coverage), #641 (its evaluation requirement can reuse this harness), #646 (feedback loops), #625 (scorecard).

🤖 Generated with Claude Code

🔗 Related PRs

#625 - feat(plugins): measure use and usefulness from native evidence [closed]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

🤖 Generated by the Agentic Engineer (interactive Claude Code session)

Activation cases to add to this suite: Svelte (from #652)

svelte-core-bestpractices has been bundled since 7.0.0. The suite has not landed yet, so these are the cases #652 asks for, ready to copy in:

Kind	Prompt	Expected
Should trigger	"Add a form action to this SvelteKit route that validates the RSVP input"	loads `svelte-core-bestpractices`
Near miss	"Add a page to the KSail docs site" (an Astro site)	loads no Svelte skill
Adherence	Any generated Svelte component in a project whose `package.json` declares Svelte 5	uses that major version's syntax, and `svelte-check` passes
Over-trigger guard	"What branch am I on?"	loads no skill

One change from #652's wording: the near-miss case there expected the astro skill to load. That skill left the bundle in 7.0.0, so the case can only assert that no Svelte skill loads.

🤖 Generated with Claude Code

🤖 Generated by the Agentic Engineer (interactive Claude Code session)

Activation cases to add to this suite: PostgreSQL (from #653)

Updated 2026-11-10: no PostgreSQL skill is bundled. The candidate evaluated under #653 was not adopted, because the review of its pull request found defects in its row-level-security and migration examples. The cases below now describe what a future PostgreSQL skill must pass before it is adopted.

Kind	Prompt	Expected
Should trigger	"Add a NOT NULL column with a default to the guests table in a migration"	loads the skill. Adherence: the migration is lock-safe (nullable column, backfill, then the constraint) and follows the repository's own migration rules
Should trigger	"This query on invoices is slow; fix it"	loads the skill. Adherence: the agent reads the query plan (`EXPLAIN`) before changing anything and proposes an index, not a blind rewrite
Should trigger	"Add row-level security so each customer only sees their own invoices"	loads the skill. Adherence: the tenant's identity comes from something the querying role cannot change, it is set per transaction, and the policy is tested with a second tenant
Near miss	"Restore the wedding database from last night's backup"	the agent follows the platform's recovery procedure and changes nothing in production without the documented approval; the skill does not drive this
Negative	a ClickHouse query in Coroot	does not load the skill

The row-level-security case is new. It is the check the rejected candidate would have failed.

🤖 Generated with Claude Code

🤖 Generated by the Agentic Engineer (interactive Claude Code session)

Activation cases to add to this suite: Prometheus (from #654)

promql and prometheus are prepared for adoption under #654. The suite has not landed yet, so these are the cases #654 asks for, ready to copy in:

Kind	Prompt	Expected
Should trigger	"Write an alert that fires when the gateway's 5-minute error ratio is above 2 %"	loads `promql` or `prometheus`. Adherence: the rule uses `rate()` over a range, guards against division by zero, and has a `for:` duration
Should not rely on a skill	"Add a trace pipeline to the OpenTelemetry collector"	no bundled OpenTelemetry skill exists; the agent looks the collector configuration up (for example through the `context7` documentation server) and does not point the pipeline at a hosted service
Near miss	"The Alertmanager HelmRelease is not ready"	loads `gitops-cluster-debug`, not `promql`
Over-trigger guard	a trivial question	loads no skill

Two changes from #654's wording, both following from the evaluation recorded there:

The alert case accepts either skill. promql owns the query and prometheus owns the rule file with its for: duration, so a correct run may load one or both.
The OpenTelemetry case no longer expects a skill to load, because the upstream's opentelemetry skill was rejected.

🤖 Generated with Claude Code

🤖 Generated by the Agentic Engineer (interactive Claude Code session)

One requirement handed over from #678. The authoring rules say each first-party skill keeps its acceptance scenarios where a suite can read them. The CI check built for #678 (#681) deliberately leaves that out, because this issue decides where scenarios live.

When the suite lands, please add the missing check here: every first-party skill has at least two scenarios in the suite's location, and CI fails when one has none. The check from #681 already lists the first-party skills (18 today), so the scenario check can reuse that list.

🤖 Generated with Claude Code

Navigation Menu

Repository navigation

Measure skill activation and adherence with a claude plugin eval suite #651

Problem

Proposal

Acceptance criteria

Activation cases to add to this suite: Svelte (from #652)

Activation cases to add to this suite: PostgreSQL (from #653)

Activation cases to add to this suite: Prometheus (from #654)

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Measure skill activation and adherence with a claude plugin eval suite #651

Description

Problem

Proposal

Acceptance criteria

Activity

coderabbitai commented on Nov 4, 2026

devantler commented on Nov 10, 2026

Activation cases to add to this suite: Svelte (from #652)

devantler commented on Nov 10, 2026

Activation cases to add to this suite: PostgreSQL (from #653)

devantler commented on Nov 10, 2026

Activation cases to add to this suite: Prometheus (from #654)

devantler commented on Nov 10, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions