Observability for yoke: measure token cost and flow quality across prompt versions

[prineycom]

Observability for yoke: measure token cost and flow quality across prompt versions

Context

Yoke skills (/task, /plan, /do, /review, /fix, …) are prompts that evolve over time. Today we have two blind spots:

1. Cost is invisible. We don't know how many tokens a given skill run consumes, how that breaks down between skill instructions, project context (CLAUDE.md, yoke-context.md), user input, and tool results. We can't tell whether a skill version got cheaper or more expensive after an edit, nor which skills dominate the bill.

2. Quality is anecdotal. Model output is non-deterministic, so when we tweak a prompt the only feedback loop is "run it on my current task and see if it feels better." That means:

   • Regressions are caught late (or never).
   • We can't attribute a quality change to a specific prompt edit — was it the new paragraph in SKILL.md, the reordering of steps in a sub-agent, or simply model variance?
   • There's no baseline to compare against.

This issue bundles both problems because they share infrastructure: both need runs to be recorded with enough structure to diff across versions.

Original tickets: projectory-com/sp#41, projectory-com/sp#42.

Goals

G1 — Token cost accounting per run

For every skill invocation, capture:

• Total input / output / cache-read / cache-write tokens
• Per-turn breakdown (so we can see which tool result or agent exploded the bill)
• Attribution: how many input tokens came from
  • the skill itself (SKILL.md + referenced files)
  • project context (CLAUDE.md, yoke-context.md, AGENTS.md)
  • user input / ticket body
  • tool results (Read, Grep, Bash output, sub-agent returns)
• Skill + version identifier (commit SHA of the skill file at run time)

G2 — Flow quality evaluation against a fixed baseline

A deterministic-ish harness that:

• Runs a frozen set of baseline scenarios (small representative tasks) against the current yoke skills
• Produces a score per scenario (pass/fail + graded rubric: plan completeness, diff correctness, commit message quality, etc.)
• Stores the result with the skill version used
• Makes it trivial to compare two versions: "on the same 20 baselines, v1.2.0 scored 17/20, v1.3.0 scores 14/20 — here are the 3 regressions"

G3 — Change attribution

Given a regression, we must be able to ask: which prompt edit caused it?

• Each baseline result is pinned to the git SHA of every skill file involved
• Bisecting across skill commits must be possible (same baseline, walk skill history, find the commit where the score dropped)
• Per-turn cost/quality data lets us narrow down to the specific step that degraded

Proposed approach

Phase 1 — Telemetry collection (unlocks G1 and G3 cost side)

• Add a SessionStart / Stop hook pair (or PostToolUse on skill-invocations) that writes a JSON record per run to .claude/yoke-runs/<timestamp>-<skill>.json
• Leverage the Claude Code transcript format: usage counts are already emitted per message; the hook just aggregates them and tags with skill + git SHA
• Token attribution via provenance tags: when a skill loads a reference file or the harness loads CLAUDE.md, mark those tokens in the record
• Local-only by default (.gitignore the directory); opt-in export for team-wide analysis

Phase 2 — Baseline scenarios (unlocks G2)

• Create evals/ in the yoke repo with a handful of representative scenarios:
  • evals/task-from-ticket/ — ticket URL → expected task-file shape
  • evals/plan-from-task/ — task file → expected plan structure
  • evals/do-small-fix/ — plan + repo snapshot → expected diff
  • evals/review-clean-diff/ — clean diff → expected "no issues" review
  • evals/gca-commit-grouping/ — staged changes → expected commit grouping
• Each scenario ships with inputs, a rubric, and an "expected shape" (not byte-exact — structural/semantic checks)
• A yoke eval command (or skill) runs all scenarios N times, aggregates scores, writes a report

Phase 3 — Version diff + bisect (unlocks G3 quality side)

• yoke eval diff <sha1> <sha2> — run baselines against two skill versions, show per-scenario delta
• yoke eval bisect <good-sha> <bad-sha> <scenario> — git-bisect-style walk over skill commits to find the regression point
• Cost deltas reported alongside quality deltas (a "fix" that doubles tokens for 1% quality might not be worth it)

Open questions

• Scoring rubric design: how graded vs binary? LLM-as-judge (self-grading with a fixed judge prompt pinned to a model+version) or deterministic structural checks? Probably a mix — structural checks where possible, LLM judge for prose quality.
• Baseline drift: if models update (Opus 4.7 → 4.8), scores will shift even with unchanged prompts. Record the model version in every run; compare within a model version by default.
• Variance budget: how many runs per scenario to get a stable signal? Start with N=3, measure variance, tune.
• Storage: local JSON is fine for Phase 1; if we want team-wide analysis later, consider a lightweight central store (sqlite in the repo? external service?) — out of scope for the first cut.
• Privacy: task/ticket content may be sensitive. Local-by-default, explicit opt-in before any upload.

Success criteria

1. After a skill edit, a maintainer can answer in under a minute: "did this edit make the skill cheaper or more expensive on the baseline, and by how much?"
2. After a skill edit, a maintainer can answer in under 10 minutes: "did this edit improve or regress flow quality on the baseline, and on which scenarios?"
3. Given a known regression between two yoke versions, a maintainer can run yoke eval bisect and identify the exact commit that caused it.
4. Token cost reports distinguish skill tokens from project-context tokens from tool-result tokens, so we can see where the bill actually goes.

References

• Claude Code transcript / usage format: https://code.claude.com/docs/en/transcripts
• Claude Code hooks (for telemetry capture): https://code.claude.com/docs/en/hooks
• Prior art for prompt evals: Anthropic evals cookbook, Promptfoo, OpenAI evals

---

Migrated and expanded from projectory-com/sp#41 and projectory-com/sp#42.