Add calibration loop workflow

.agents/skills/calibration-loop/SKILL.md

@@ -0,0 +1,61 @@
+---
+name: calibration-loop
+description: "Route corpus sealing, candidate annotation, adjudication, and calibration through merged artifacts."
+metadata:
+  author: TGPSKI
+  version: "1.0"
+compatibility: "flow-indicator and local corpus, label, and run-record paths"
+---
+
+# Calibration loop
+
+## Prerequisites
+
+- Sync the upstream primary branch. Local changes, unmerged branches, and chat
+  history do not advance this workflow.
+- Read docs/CALIBRATION.md and docs/COMMANDS.md before selecting a phase.
+- Keep corpus and label artifacts private; they can contain transcript text.
+
+## Entry point
+
+Ask for one calibration root directory. Inspect its artifacts on the merged
+upstream primary branch.
+
+## Progress detection
+
+| Merged artifact status | Progress |
+|---|---|
+| No corpus/manifest.json | Phase 1: corpus preparation. |
+| corpus/manifest.json, no candidates/dev-*-*.jsonl | Phase 2: label and candidate annotation. |
+| Candidate files, no labels/dev-*.jsonl | Phase 3: adjudication. |
+| labels/dev-*.jsonl, no runs/dev-*.json | Phase 4: dev calibration and holdout review. |
+| runs/dev-*.json present | Phase 4: inspect whether a reason supports a holdout opening. |
+
+## Determine phase
+
+| Status | Action |
+|---|---|
+| Artifact exists only in the working tree or an unmerged branch | Ignore it for routing; start from merged artifacts. |
+| More than one incomplete phase has artifacts | Route to the earliest incomplete phase. |
+| Artifact hash or manifest validation fails | Stop; preserve the artifact and repair its provenance before continuing. |
+| Phase output is merged and complete | Route to the next phase. |
+
+## Design principles
+
+1. Ownership: the operator selects corpus, endpoint, adjudicator, classifier,
+   and any holdout reason; the workflow records those choices.
+2. Source of truth: the sealed manifest, human label files, and calibration run
+   records decide progress and provenance.
+3. Ask less, infer more: derive paths, splits, hashes, and prior outputs from
+   merged artifacts before asking.
+4. Prefer simple: use dev, the heuristic classifier, and human labels until a
+   documented reason selects more.
+
+## Route to phase
+
+| Phase | Module | Outcome |
+|---|---|---|
+| 1 | @references/phase-01-corpus.md | A sealed manifest. |
+| 2 | @references/phase-02-label-and-annotate.md | Blind units and model candidates. |
+| 3 | @references/phase-03-adjudicate.md | Human ground-truth labels. |
+| 4 | @references/phase-04-calibrate.md | Provenanced dev run and holdout decision. |

.agents/skills/calibration-loop/references/phase-01-corpus.md

@@ -0,0 +1,45 @@
+---
+name: calibration-loop-corpus
+parent: calibration-loop
+description: "Survey and seal a session-level calibration corpus."
+---
+
+# Phase 1: corpus preparation
+
+**Carry forward from prior phases**: calibration root.
+
+**Inspect**:
+
+    flow-indicator corpus --harness claude-code --scan /selected/session/store
+
+| Status | Action |
+|---|---|
+| No merged manifest exists | Continue with the selected corpus root. |
+| A merged manifest exists | Return to the router. |
+| Survey has no eligible sessions | Stop; ask for another local session store. |
+
+**Decide**:
+
+Ask for the harness, scan directory, and corpus root. Present documented
+selection bounds; do not select sessions by outcome.
+
+**Generate**:
+
+    flow-indicator corpus --harness /selected/harness --scan /selected/store --build --out /calibration-root/corpus
+
+The manifest seals source hashes and assigns session-level dev and holdout
+splits. Do not edit it after sealing.
+
+## Validate
+
+    flow-indicator label --manifest /calibration-root/corpus/manifest.json --family obligation --split dev > /tmp/calibration-units.jsonl
+
+## PR Checkpoint
+
+**Title**: [calibration] Seal corpus
+
+**Files to include**:
+
+- /calibration-root/corpus/manifest.json
+
+**Next phase**: @phase-02-label-and-annotate.md

.agents/skills/calibration-loop/references/phase-02-label-and-annotate.md

@@ -0,0 +1,48 @@
+---
+name: calibration-loop-label-and-annotate
+parent: calibration-loop
+description: "Emit blind label units and record model candidate labels."
+---
+
+# Phase 2: label and candidate annotation
+
+**Carry forward from prior phases**: calibration root and sealed manifest path.
+
+**Inspect**:
+
+    flow-indicator label --manifest /calibration-root/corpus/manifest.json --family /selected/family --split dev
+
+| Status | Action |
+|---|---|
+| Blind units and candidate files are absent | Continue. |
+| Candidate files exist and human labels are absent | Preserve candidates; continue to Phase 3. |
+| Human label files exist | Return to the router. |
+| Manifest changed or cannot load | Stop; repair corpus provenance before emitting units. |
+
+**Decide**:
+
+Ask for family and sampling rate. Ask whether the operator selects an
+OpenAI-compatible candidate endpoint. Candidate annotation is optional.
+
+**Generate**:
+
+    flow-indicator label --manifest /calibration-root/corpus/manifest.json --family /selected/family --split dev --sample-per-mille /selected/rate > /calibration-root/units/dev-/selected/family.jsonl
+    flow-indicator annotate --manifest /calibration-root/corpus/manifest.json --out /calibration-root/candidates --family /selected/family --split dev --endpoint /selected/endpoint --model /selected/model
+
+The second command runs only on the operator-selected endpoint. Its two passes
+write candidate files under candidates; model output is never ground truth.
+
+## Validate
+
+    flow-indicator label --manifest /calibration-root/corpus/manifest.json --family /selected/family --split dev > /tmp/calibration-units.jsonl
+
+## PR Checkpoint
+
+**Title**: [calibration] Emit units and candidate labels
+
+**Files to include**:
+
+- /calibration-root/units/dev-/selected/family.jsonl
+- /calibration-root/candidates/dev-/selected/family-*.jsonl
+
+**Next phase**: @phase-03-adjudicate.md

.agents/skills/calibration-loop/references/phase-03-adjudicate.md

@@ -0,0 +1,47 @@
+---
+name: calibration-loop-adjudicate
+parent: calibration-loop
+description: "Create human-adjudicated labels without promoting model candidates."
+---
+
+# Phase 3: adjudication
+
+**Carry forward from prior phases**: manifest, family, blind units, and optional
+candidate-label paths.
+
+**Inspect**:
+
+Read docs/CALIBRATION.md classes for the selected family. Compare candidates
+only as a review aid after the human adjudicator records each unit.
+
+| Status | Action |
+|---|---|
+| Candidate passes agree | Record the agreement as codebook evidence; require human adjudication. |
+| Candidate passes disagree | Send the unit to human adjudication; do not average a class. |
+| Candidate files are absent | Adjudicate blind units directly. |
+| Human label file is complete | Continue to Phase 4. |
+
+**Decide**:
+
+Ask for the human adjudicator and output label directory. Require the
+adjudicator to record labels without the classifier verdict.
+
+**Generate**:
+
+Write human-adjudicated labels to /calibration-root/labels/dev-/selected/family.jsonl.
+Keep model name, pass, and candidate provenance separate from human ground
+truth. Preserve unlabelable only where the evidence cannot establish a class.
+
+## Validate
+
+    flow-indicator calibrate --labels /calibration-root/labels --manifest /calibration-root/corpus/manifest.json --split dev --json
+
+## PR Checkpoint
+
+**Title**: [calibration] Adjudicate labels
+
+**Files to include**:
+
+- /calibration-root/labels/dev-/selected/family.jsonl
+
+**Next phase**: @phase-04-calibrate.md

.agents/skills/calibration-loop/references/phase-04-calibrate.md

@@ -0,0 +1,53 @@
+---
+name: calibration-loop-calibrate
+parent: calibration-loop
+description: "Score a sealed corpus with provenance and control holdout openings."
+---
+
+# Phase 4: calibration and holdout review
+
+**Carry forward from prior phases**: manifest, human labels, family, candidate
+provenance, and prior calibration run records.
+
+**Inspect**:
+
+    flow-indicator calibrate --labels /calibration-root/labels --manifest /calibration-root/corpus/manifest.json --split dev --classifier heuristic --json
+
+| Status | Action |
+|---|---|
+| No dev run record exists | Run the selected dev classifier and save its JSON result. |
+| Dev record exists under the same manifest, ruleset, labels, and classifier identity | Compare it with the next run; continue only with a stated change. |
+| Rule, profile, label set, manifest, or classifier identity changed | Record the changed provenance; treat scores as a new comparison. |
+| Operator supplies a holdout reason | Run holdout once with --reason and save the record. |
+| Operator supplies no holdout reason | Keep holdout sealed; do not run a holdout score. |
+| A fixture is proposed as a tuning target | Stop; improve a general rule only on independently adjudicated dev evidence. |
+
+**Decide**:
+
+Ask for classifier path, the single change under evaluation, and a holdout
+reason. A configured classifier needs endpoint and model provenance; heuristic
+needs its rule hash. Do not ask for a holdout reason when dev evidence does not
+justify an opening.
+
+**Generate**:
+
+    flow-indicator calibrate --labels /calibration-root/labels --manifest /calibration-root/corpus/manifest.json --split dev --classifier /selected/classifier --json > /calibration-root/runs/dev-/selected/run.json
+    flow-indicator calibrate --labels /calibration-root/labels --manifest /calibration-root/corpus/manifest.json --split holdout --classifier /selected/classifier --reason "/operator/reason" --json > /calibration-root/runs/holdout-/selected/run.json
+
+The holdout command belongs only to the holdout-reason status row. Do not
+change classifier rules in this workflow.
+
+## Validate
+
+    flow-indicator calibrate --labels /calibration-root/labels --manifest /calibration-root/corpus/manifest.json --split dev
+
+## PR Checkpoint
+
+**Title**: [calibration] Record calibration review
+
+**Files to include**:
+
+- /calibration-root/runs/dev-/selected/run.json
+- /calibration-root/runs/holdout-/selected/run.json
+
+**Next phase**: Return to @../SKILL.md

CHANGELOG.md

@@ -11,6 +11,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   configuration updates through privacy choices, optional profiles and semantic
   endpoints, session discovery, and a first replay or watch.
 
+- .agents/skills/calibration-loop/ routes corpus sealing, candidate annotation,
+  human adjudication, and calibration through merged artifacts.
+
 ## [0.4.1] - 2026-09-03
 
 ### Added