diff --git a/.agents/skills/calibration-loop/SKILL.md b/.agents/skills/calibration-loop/SKILL.md new file mode 100644 index 0000000..09dd369 --- /dev/null +++ b/.agents/skills/calibration-loop/SKILL.md @@ -0,0 +1,61 @@ +--- +name: calibration-loop +description: "Route corpus sealing, candidate annotation, adjudication, and calibration through merged artifacts." +metadata: + author: TGPSKI + version: "1.0" +compatibility: "flow-indicator and local corpus, label, and run-record paths" +--- + +# Calibration loop + +## Prerequisites + +- Sync the upstream primary branch. Local changes, unmerged branches, and chat + history do not advance this workflow. +- Read docs/CALIBRATION.md and docs/COMMANDS.md before selecting a phase. +- Keep corpus and label artifacts private; they can contain transcript text. + +## Entry point + +Ask for one calibration root directory. Inspect its artifacts on the merged +upstream primary branch. + +## Progress detection + +| Merged artifact status | Progress | +|---|---| +| No corpus/manifest.json | Phase 1: corpus preparation. | +| corpus/manifest.json, no candidates/dev-*-*.jsonl | Phase 2: label and candidate annotation. | +| Candidate files, no labels/dev-*.jsonl | Phase 3: adjudication. | +| labels/dev-*.jsonl, no runs/dev-*.json | Phase 4: dev calibration and holdout review. | +| runs/dev-*.json present | Phase 4: inspect whether a reason supports a holdout opening. | + +## Determine phase + +| Status | Action | +|---|---| +| Artifact exists only in the working tree or an unmerged branch | Ignore it for routing; start from merged artifacts. | +| More than one incomplete phase has artifacts | Route to the earliest incomplete phase. | +| Artifact hash or manifest validation fails | Stop; preserve the artifact and repair its provenance before continuing. | +| Phase output is merged and complete | Route to the next phase. | + +## Design principles + +1. Ownership: the operator selects corpus, endpoint, adjudicator, classifier, + and any holdout reason; the workflow records those choices. +2. Source of truth: the sealed manifest, human label files, and calibration run + records decide progress and provenance. +3. Ask less, infer more: derive paths, splits, hashes, and prior outputs from + merged artifacts before asking. +4. Prefer simple: use dev, the heuristic classifier, and human labels until a + documented reason selects more. + +## Route to phase + +| Phase | Module | Outcome | +|---|---|---| +| 1 | @references/phase-01-corpus.md | A sealed manifest. | +| 2 | @references/phase-02-label-and-annotate.md | Blind units and model candidates. | +| 3 | @references/phase-03-adjudicate.md | Human ground-truth labels. | +| 4 | @references/phase-04-calibrate.md | Provenanced dev run and holdout decision. | diff --git a/.agents/skills/calibration-loop/references/phase-01-corpus.md b/.agents/skills/calibration-loop/references/phase-01-corpus.md new file mode 100644 index 0000000..5f5a1a7 --- /dev/null +++ b/.agents/skills/calibration-loop/references/phase-01-corpus.md @@ -0,0 +1,45 @@ +--- +name: calibration-loop-corpus +parent: calibration-loop +description: "Survey and seal a session-level calibration corpus." +--- + +# Phase 1: corpus preparation + +**Carry forward from prior phases**: calibration root. + +**Inspect**: + + flow-indicator corpus --harness claude-code --scan /selected/session/store + +| Status | Action | +|---|---| +| No merged manifest exists | Continue with the selected corpus root. | +| A merged manifest exists | Return to the router. | +| Survey has no eligible sessions | Stop; ask for another local session store. | + +**Decide**: + +Ask for the harness, scan directory, and corpus root. Present documented +selection bounds; do not select sessions by outcome. + +**Generate**: + + flow-indicator corpus --harness /selected/harness --scan /selected/store --build --out /calibration-root/corpus + +The manifest seals source hashes and assigns session-level dev and holdout +splits. Do not edit it after sealing. + +## Validate + + flow-indicator label --manifest /calibration-root/corpus/manifest.json --family obligation --split dev > /tmp/calibration-units.jsonl + +## PR Checkpoint + +**Title**: [calibration] Seal corpus + +**Files to include**: + +- /calibration-root/corpus/manifest.json + +**Next phase**: @phase-02-label-and-annotate.md diff --git a/.agents/skills/calibration-loop/references/phase-02-label-and-annotate.md b/.agents/skills/calibration-loop/references/phase-02-label-and-annotate.md new file mode 100644 index 0000000..18d2b6a --- /dev/null +++ b/.agents/skills/calibration-loop/references/phase-02-label-and-annotate.md @@ -0,0 +1,48 @@ +--- +name: calibration-loop-label-and-annotate +parent: calibration-loop +description: "Emit blind label units and record model candidate labels." +--- + +# Phase 2: label and candidate annotation + +**Carry forward from prior phases**: calibration root and sealed manifest path. + +**Inspect**: + + flow-indicator label --manifest /calibration-root/corpus/manifest.json --family /selected/family --split dev + +| Status | Action | +|---|---| +| Blind units and candidate files are absent | Continue. | +| Candidate files exist and human labels are absent | Preserve candidates; continue to Phase 3. | +| Human label files exist | Return to the router. | +| Manifest changed or cannot load | Stop; repair corpus provenance before emitting units. | + +**Decide**: + +Ask for family and sampling rate. Ask whether the operator selects an +OpenAI-compatible candidate endpoint. Candidate annotation is optional. + +**Generate**: + + flow-indicator label --manifest /calibration-root/corpus/manifest.json --family /selected/family --split dev --sample-per-mille /selected/rate > /calibration-root/units/dev-/selected/family.jsonl + flow-indicator annotate --manifest /calibration-root/corpus/manifest.json --out /calibration-root/candidates --family /selected/family --split dev --endpoint /selected/endpoint --model /selected/model + +The second command runs only on the operator-selected endpoint. Its two passes +write candidate files under candidates; model output is never ground truth. + +## Validate + + flow-indicator label --manifest /calibration-root/corpus/manifest.json --family /selected/family --split dev > /tmp/calibration-units.jsonl + +## PR Checkpoint + +**Title**: [calibration] Emit units and candidate labels + +**Files to include**: + +- /calibration-root/units/dev-/selected/family.jsonl +- /calibration-root/candidates/dev-/selected/family-*.jsonl + +**Next phase**: @phase-03-adjudicate.md diff --git a/.agents/skills/calibration-loop/references/phase-03-adjudicate.md b/.agents/skills/calibration-loop/references/phase-03-adjudicate.md new file mode 100644 index 0000000..20f8023 --- /dev/null +++ b/.agents/skills/calibration-loop/references/phase-03-adjudicate.md @@ -0,0 +1,47 @@ +--- +name: calibration-loop-adjudicate +parent: calibration-loop +description: "Create human-adjudicated labels without promoting model candidates." +--- + +# Phase 3: adjudication + +**Carry forward from prior phases**: manifest, family, blind units, and optional +candidate-label paths. + +**Inspect**: + +Read docs/CALIBRATION.md classes for the selected family. Compare candidates +only as a review aid after the human adjudicator records each unit. + +| Status | Action | +|---|---| +| Candidate passes agree | Record the agreement as codebook evidence; require human adjudication. | +| Candidate passes disagree | Send the unit to human adjudication; do not average a class. | +| Candidate files are absent | Adjudicate blind units directly. | +| Human label file is complete | Continue to Phase 4. | + +**Decide**: + +Ask for the human adjudicator and output label directory. Require the +adjudicator to record labels without the classifier verdict. + +**Generate**: + +Write human-adjudicated labels to /calibration-root/labels/dev-/selected/family.jsonl. +Keep model name, pass, and candidate provenance separate from human ground +truth. Preserve unlabelable only where the evidence cannot establish a class. + +## Validate + + flow-indicator calibrate --labels /calibration-root/labels --manifest /calibration-root/corpus/manifest.json --split dev --json + +## PR Checkpoint + +**Title**: [calibration] Adjudicate labels + +**Files to include**: + +- /calibration-root/labels/dev-/selected/family.jsonl + +**Next phase**: @phase-04-calibrate.md diff --git a/.agents/skills/calibration-loop/references/phase-04-calibrate.md b/.agents/skills/calibration-loop/references/phase-04-calibrate.md new file mode 100644 index 0000000..e09323a --- /dev/null +++ b/.agents/skills/calibration-loop/references/phase-04-calibrate.md @@ -0,0 +1,53 @@ +--- +name: calibration-loop-calibrate +parent: calibration-loop +description: "Score a sealed corpus with provenance and control holdout openings." +--- + +# Phase 4: calibration and holdout review + +**Carry forward from prior phases**: manifest, human labels, family, candidate +provenance, and prior calibration run records. + +**Inspect**: + + flow-indicator calibrate --labels /calibration-root/labels --manifest /calibration-root/corpus/manifest.json --split dev --classifier heuristic --json + +| Status | Action | +|---|---| +| No dev run record exists | Run the selected dev classifier and save its JSON result. | +| Dev record exists under the same manifest, ruleset, labels, and classifier identity | Compare it with the next run; continue only with a stated change. | +| Rule, profile, label set, manifest, or classifier identity changed | Record the changed provenance; treat scores as a new comparison. | +| Operator supplies a holdout reason | Run holdout once with --reason and save the record. | +| Operator supplies no holdout reason | Keep holdout sealed; do not run a holdout score. | +| A fixture is proposed as a tuning target | Stop; improve a general rule only on independently adjudicated dev evidence. | + +**Decide**: + +Ask for classifier path, the single change under evaluation, and a holdout +reason. A configured classifier needs endpoint and model provenance; heuristic +needs its rule hash. Do not ask for a holdout reason when dev evidence does not +justify an opening. + +**Generate**: + + flow-indicator calibrate --labels /calibration-root/labels --manifest /calibration-root/corpus/manifest.json --split dev --classifier /selected/classifier --json > /calibration-root/runs/dev-/selected/run.json + flow-indicator calibrate --labels /calibration-root/labels --manifest /calibration-root/corpus/manifest.json --split holdout --classifier /selected/classifier --reason "/operator/reason" --json > /calibration-root/runs/holdout-/selected/run.json + +The holdout command belongs only to the holdout-reason status row. Do not +change classifier rules in this workflow. + +## Validate + + flow-indicator calibrate --labels /calibration-root/labels --manifest /calibration-root/corpus/manifest.json --split dev + +## PR Checkpoint + +**Title**: [calibration] Record calibration review + +**Files to include**: + +- /calibration-root/runs/dev-/selected/run.json +- /calibration-root/runs/holdout-/selected/run.json + +**Next phase**: Return to @../SKILL.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 28b01b6..b65b060 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 configuration updates through privacy choices, optional profiles and semantic endpoints, session discovery, and a first replay or watch. +- .agents/skills/calibration-loop/ routes corpus sealing, candidate annotation, + human adjudication, and calibration through merged artifacts. + ## [0.4.1] - 2026-09-03 ### Added