Question about run-to-run consistency in skill evaluation

[blak0p]

I build tooling with a similar underlying concern in a different layer: a Git MCP server where an LLM never decides semantics directly — a deterministic layer classifies changes first, so the model can’t introduce inconsistency into decisions that matter.

Your train/test split + 3-run approach for skill evaluation caught my eye because it’s tackling the same root problem (LLM non-determinism) from the evaluation side instead. When two runs of the same skill produce different results on the same input, how does your system decide whether that’s the skill being poorly designed vs. just expected model noise? Trying to compare notes on how you draw that line.

[antongulin]

Great question — short version of how we draw the line:

Each eval query runs N times (default 3) and produces a trigger rate, not a boolean; a query passes when the rate clears the threshold (default 0.5). So expected model noise is absorbed by averaging. A mid-range rate (1/3, 2/3) is treated as a design signal — the description is borderline for that phrasing — and the fix is sharpening the description or the eval queries, not re-rolling. The optimize loop also holds out 40% of queries so improvements must generalize.

Since #29 (v0.2.22), trigger measurement is isolated from downstream execution flakiness by default (triggerOnly) — a run counts as triggered the moment the skill loads; set triggerOnly: false to make later workflow failures fail the run.

Your deterministic-classifier-first approach attacks the same problem from the other side; here we accept the nondeterminism and measure it statistically. The two compose nicely.

[blak0p]

Makes sense, thanks for walking through the threshold logic.

One thing I'm curious about: is the intention that a deterministic classifier and the statistical trigger-rate approach are meant to actually compose in the same pipeline — e.g. the classifier filters the clear-cut cases and the trigger-rate eval covers the ambiguous middle — or do you see them more as alternative strategies where someone picks one depending on their use case?

[antongulin]

Yeah, they work together — pretty much how you described it. The classifier handles the clear cases, and the eval covers the middle ones it can't call.

Only thing I'd add: don't let the classifier force a decision on the borderline ones. Those are the cases worth measuring — a score stuck in the middle usually just means the description needs work.