Read UK staging contract v3: a real blocked run status, block details and failure classes (dashboard and collector) #206

Why

UK builds report how they ended badly today:

• every failure is error_code: BUILD_FAILED with no failure_class;
• a build whose gates refused the candidate closes as completed, so this dashboard infers "blocked" from blocking_failure_count (build-monitor.ts, the comment at the top says so);
• a build refused at the preflight gates returns without any gate counts, so the dashboard shows it as passed;
• killed builds stay running.

A microcosm PR (stacked on PolicyEngine/microcosm#1099) fixes this at the source. It gives blocked runs a real blocked status, classifies failures, and moves the UK staging documents to schema_version 3. This issue is the dashboard and collector half. It has to be deployed and qualified before the microcosm PR merges: an old collector answers a blocked event with 422, and #1099's delivery queue retries that batch forever, so the run's terminal event would never arrive.

The v3 contract (what microcosm will write)

Staging run documents (run_manifest.json, progress.json, events.ndjson, calibration_progress.json):

• schema_version: 3 on all four documents of a run; schema_name values unchanged. v2 documents stay valid and must keep reading exactly as today.
• Run status: running | completed | blocked | failed.
• New top-level field block on run_manifest and progress: null, or exactly
  { "phase": <safe identifier>, "blocking_failure_count": <int ≥ 1>, "blocking_gate_ids": [<string>, ...] }.
  phase is one of preflight, terminal, assembled, transferred today; treat it as an open identifier.
  Rules: status == "blocked" requires block and failure: null; status == "failed" requires failure and block: null; any other status requires both null.
• failure gains a fifth key, failure_class: a lowercase identifier or null. Exactly five keys: error_code, error_type, message, local_diagnostic_reference, failure_class.
• Event status adds blocked. A blocked run ends with one terminal event:
  event_type: "stage", stage_id: "blocked", status: "blocked", details: { phase, blocking_failure_count, blocking_gate_ids, gate_statuses }.
  Completed and failed runs end as today (complete / failed).
• The delivery summary keeps contract_version: 2 (its shape does not change; microcosm's publisher pins it).

Error codes and classes microcosm will write on failed runs: INTERRUPTED / interrupted (Ctrl-C), TERMINATED / terminated (SIGTERM), OUT_OF_MEMORY / out_of_memory, GRAPH_NODE_FAILED / error, BUILD_FAILED / error (fallback). #1099's emitter also sends build_failure, unexpected_process_exit and dry_run_refusal (US).

Collector events (#1099's emitter): a blocked run sends event_type: "run", status: "blocked", stage_id: "blocked", details: { phase, blocking_failure_count, blocking_gate_ids, gate_statuses }. Failed run events add error_code to the existing error_type, failure_class, failed_during details.

What to change

Version handling: treat schema_version >= 2 as the structured contract. It is hard-coded to 2 in several places, and a v3 run would otherwise be parsed with the v1 transition logic or skip checks (line numbers at main 2da3bf3):

• frontend/lib/microcosm/build-monitor.ts:721 (schemaVersion = … === 2 ? 2 : 1);
• frontend/lib/microcosm/staging-artifact.ts:389, 436, 537-545 (run listing) and 857, 882, 891 (the diagnostics path falls back to the v1 layout and skips its digest check);
• frontend/lib/microcosm/staging-contract.ts:725 (event sequence check) and 766-773 (the mixed-version check, then if (!versions.has(2)) return skips every consistency check).

Validator (staging-contract.ts): accept v3 as specified above (status enums, block, five-key failure, the rules), keep v2 exact; mixed versions within one run stay an error.

blocked as a final status everywhere:

• frontend/lib/microcosm/build-runs.ts:134, 234 and staging-artifact.ts:399 (otherwise blocked runs are refetched forever with a short cache life);
• MARKER_STAGES and the terminal-stage handling in build-monitor.ts (~L38, 449-471);
• the tone mapping in frontend/components/microcosm/microcosm-staging-view.tsx:99.

Monitor logic (build-monitor.ts):

• runState: map blocked straight to the blocked state, with the failure line built from block (phase, count, gate ids); keep today's blocking_failure_count inference only for v2 runs;
• read failure_class from progress.failure (v3), falling back to the terminal event details;
• stopStatistics: count failure classes for blocked runs too (today only failed runs are counted, ~L1705);
• labels in build-monitor-view.tsx (~L88-103) for build_failure, unexpected_process_exit, dry_run_refusal.

Collector (telemetry-service/telemetry_collector/):

• models.TelemetryEvent.status: add blocked;
• storage._apply_event: add blocked to the terminal stage set (~L122); a run/blocked event sets run status blocked, ended_at, and keeps the block details; failed events keep error_code in failure;
• no database migration: status is Text and failure is JSONB.

Fixtures

The microcosm PR adds packages/microcosm-build/tests/fixtures/staging/v3/ (completed, calibration, failed with a class, blocked) with its own SHA256SUMS, generated by tools/generate_staging_contract_fixtures.py. Copy them byte for byte next to the v2 copy and pin their digest in staging-contract.test.ts; the v2 digest 0d54c630…455e5 must not change. Until they are frozen on the microcosm branch, test against hand-written v3 documents that follow the spec above; the branch link will be added here.

Done when

• bun test frontend/lib/microcosm and tsc --noEmit pass, with new cases: a v3 blocked run at preflight and at terminal reads as blocked with its gates; a v3 failed run carries its code and class into the stop statistics; v2 runs read exactly as before; a v3 document with a sixth failure key or block on a failed run is rejected.
• Collector pytest passes with a blocked run event case.
• Deployed, and a blocked event posted to the qualification collector returns 2xx. Then the microcosm PR can merge.