[anth-volk]

Fixes #263

Summary

The Staging page's Candidate tab now takes each run's status, stage and times from the hosted telemetry collector whenever the collector has a record of the run, as the Build progress tab already does. The Hugging Face values stay in the response, and the page says which source each state came from and what the Hugging Face copy still records when it has fallen behind.

What the two US runs now show

Checked against the production data from 2026-10-22, reproduced in the tests:

Run	Before (Hugging Face only)	After
…20261019T222143Z	Stalled; "Calibration, epoch 273 of 1500"	Failed, "Failed during Release gates", with the failure message. Status from: Telemetry collector. Hugging Face copy: "Running, Calibration, epoch 273 of 1500 · updated Oct 20, 06:09"
…20261021T160022Z	Stalled; "Calibration, epoch 935 of 1500"	Running, "Reform validation", with the last heartbeat. Hugging Face copy: "Running, Calibration, epoch 935 of 1500 · updated Oct 21, 18:59"

How it works

• Server. GET /api/microcosm/staging/runs and GET /api/microcosm/staging/run call the new loadCollectorTimelines in build-runs.ts.
  • It reads the same collector records the Build progress tab reads: the country's 60 most recent collector runs, through the same per-run cache, which is now shared as loadCollectorDocuments.
  • It derives each run's state with the same buildTimeline, so the two tabs agree on every run the collector knows, including the 5-minute heartbeat rule.
  • It fetches documents only for runs the Candidate tab lists.
• Overlay. lib/microcosm/staging-run-state.ts lays the collector's state over the Hugging Face summary:
  • status, stage (for a failed or refused run, the stage it stopped at), started_at, updated_at, ended_at, heartbeat_at and failure;
  • state_source (collector or hugging_face);
  • the Hugging Face values as hugging_face_status, hugging_face_stage and hugging_face_updated_at;
  • hugging_face_differs, which is true when the copy shows another state or is more than 15 minutes behind the collector. A healthy UK copy uploads every 5 minutes.
  • The detail route returns the same fields as state, next to the unchanged Hugging Face documents.
• Runs the collector doesn't know keep their Hugging Face state, derived with runState, which is now exported from build-monitor.ts. It applies the same 6-hour rule for silent runs that the view applied, so the view's own copy of that rule is removed.
• Collector not configured or failing. The routes return the Hugging Face states as before, plus collector_note, which the overview shows. One run whose record fails to load keeps its Hugging Face state, and the note names it. An invalid collector setting is caught the same way, so it can't hide the Hugging Face runs.
• Page.
  • The overview card, now the exported CandidateOverview, adds "Status from", "Last heartbeat", "Failure", and "Hugging Face copy" (shown only when the copy differs).
  • The run picker names each state's source in a tooltip and a footnote.
  • Run internals, which still read the Hugging Face copy, say so when it has fallen behind.
• Countries. Nothing depends on a country code. The routes only ask the collector about runs the country's Hugging Face staging repository lists, so a country without the staging capability never reaches the collector.

Runs only the collector knows: not listed

The Candidate tab doesn't add runs that exist only in the collector (55 UK runs today).

• The tab exists to review candidate files: calibration diagnostics, comparisons and the target map. Those runs have none in the staging repository, so they would open as empty candidates.
• The Build progress tab already lists them, with the same state.
• If they should appear here later, it should be after the planned hand-over of candidate files to the collector, so they open with something to review.

UK

• Runs both sources know. The 5 UK runs the collector knows (uk-frs-calibration-attempt-… and uk-local-candidate-…-20261018/19…) already agree, so they show the same state as before. Only "Status from: Telemetry collector" is added, with no Hugging Face copy row.
• The five schema 2 local-area candidates (…20261007T120029Z, …20261006T094003Z, …20260919T154920Z, …20260919T013428Z, …20260918T185154Z) show "Completed" on the Candidate tab and "Blocked" on Build progress.
  • This change does not align them. They come from Hugging Face only: the collector has no record of them among its 60 most recent UK runs.
  • Their Hugging Face status is completed. Build progress infers "Blocked" from blocking failures in their acceptance check events, which only the Build progress tab reads.
  • The Candidate tab now labels them "Finished", the Build progress tab's word for completed, instead of "Completed".
  • Aligning them would mean the Candidate list reads every run's events (four Hugging Face files per run) on the page's default tab, as Build progress does. I left that out. Version 3 builds record blocked themselves, so no new run is affected.

Environment variables

This adds no new variables. The Candidate tab's routes use the same three the Build progress routes use. They are in the same Next.js deployment, and Vercel applies a deployment's variables to every function in it.

Variable	Provider	Wiring	Status
MICROCOSM_TELEMETRY_COLLECTOR_URL	GitHub Production environment variable	deploy.yml passes vars.MICROCOSM_TELEMETRY_COLLECTOR_URL to deploy-dashboard.sh, which checks it's set and passes it to vercel deploy --env.	Confirmed: the variable exists in the Production environment, and the wiring is in the repository.
MICROCOSM_TELEMETRY_READ_SECRET_VERSION	Vercel project settings, naming a Google Secret Manager version	Not passed by the workflow. The server reads it at runtime and fetches the token itself (secret-manager.ts).	Not inspected directly. In effect: production Build progress read the collector today (heartbeat at 17:42 UTC).
MICROCOSM_TELEMETRY_GOOGLE_IDENTITY_PROVIDER	Vercel project settings	Read at runtime when VERCEL=1, which Vercel sets.	Same as above.

Both routes set runtime = "nodejs", which Secret Manager access needs. The list route also sets maxDuration = 120 to allow for the collector requests' 30-second time limits.

Tests

The tests were committed failing first, then the implementation (two pairs: server, then view).

• The two US runs: the failure at the release acceptance checks, and reform validation with the heartbeat. Covered at the overlay, the list route, the detail route and the overview card.
• Collector failures: not configured, a failed list request, one unreadable run record. Covered in loadCollectorTimelines, both routes and the card.
• Runs the collector doesn't know: a stale run shows as stalled, a finished run as finished, a run with no status as unknown.
• Collector-only runs are not listed.
• UK: a refused run the collector knows, and a local-area candidate it doesn't.
• loadCollectorTimelines asks for one list page of 60 runs and fetches documents only for the requested runs it knows.

Local results: bun test passed (800 tests), bun run lint (type checks) passed, and bun run build passed. I haven't viewed the page in a browser, because the private staging repository and the collector need credentials this machine doesn't have.

Overlaps

• Remove telemetry-service/ now that the collector deploys from its own repo #234 edits the "Sources" section of docs/build-monitor.md. This PR adds one sentence to the "Historical Hugging Face telemetry" bullet, outside Remove telemetry-service/ now that the collector deploys from its own repo #234's changes.
• Read Microcosm UK releases from releases.json and build folders only #241 edits comparison-release functions in staging-artifact.ts, and use-microcosm.ts.
  • This PR doesn't touch staging-artifact.ts.
  • In use-microcosm.ts it changes only the staging run response types and one import.

Known, separate issues (not changed here)

• The build monitor shows "unknown" uploads for runs read from the collector.
• The build monitor groups dry runs and spine builds with full builds of the same pipeline.
• The Microcosm US staging writer exceeds Hugging Face's hourly commit limit and switches its uploads off for the rest of the run, including the final failed or completed state. This PR changes no Microcosm code. Candidate diagnostics for such runs still reach this tab only through Hugging Face.

Review round 2026-10-26

Fixed

• An older collector record could hide a later end recorded on Hugging Face (medium). The overlay always took the collector's state. If the collector's record stopped before the run ended (running, then stalled once heartbeats stop) and the Hugging Face copy recorded passed, failed or blocked after the record's latest update, the tab showed the run as running or stalled. Before this branch it showed the end. The overlay now keeps the Hugging Face state in that case. A final collector state still wins, so schema 2 refusals the collector infers from acceptance check events stay. Failing test 4fc3acc, fix 7f7dbbb (also updates the README and docs/build-monitor.md).
• The two tabs now use one rule (approved follow-up). Build progress let any collector record replace the Hugging Face copy, so in the case above it still showed running or stalled. The rule now lives in collectorMissedTheEnd (build-monitor.ts). The Candidate overlay, Build progress's merged list (mergeHostedDocuments) and its run view (refreshHostedRun) all use it. Failing test 3cc2ee0 (one run seen through both tabs, plus the run view case), fix 7988977, which also updates the README and docs/build-monitor.md.
• Comment on HUGGING_FACE_LAG_MS (low). Staging writers upload only when the build records something (at most every 5 minutes for the UK), not every 5 minutes. So a healthy run in a long stage can also show the "Hugging Face copy" row; production had one 40 minutes behind today. 8f058eb.

Not changed, for follow-up

• The tabs agree for runs among the collector's 60 most recent runs for the country. For older runs the collector knows, the Build progress run view reads the collector directly, while the Candidate tab uses Hugging Face. Today the UK window covers about one week, and the US has 4 collector runs.
• The five schema 2 UK local-area candidates still show "Finished" here and "Blocked" on Build progress, as described above.
• Neither Candidate route caches the collector's run list. Each 30-second poll of each route makes one list request. A collector that doesn't answer delays the first load of the default tab by up to 30 seconds per request.

Checked: the three settings exist in the Vercel project's Production settings, and the collector address is also a GitHub Production variable that deploy.yml passes to vercel deploy --env. Production Build progress read collector records today. #234 and #241 merge cleanly with this branch. Local: bun test (807 pass), bun run lint and bun run build pass.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
calibration-diagnostics	Ready	Preview	Oct 26, 2026 12:37pm UTC