Publish UK execution graphs through a general build emitter service

README.md

@@ -44,16 +44,62 @@ until you restart the kernel, load it by path with
 `load_country_spec(Path(...))` (always re-read), or call
 `microcosm.build.country_spec._load_packaged_country_spec.cache_clear()`.
 
-## Staging build telemetry
+## Build progress and staging run files
+
+Supported US and UK build commands always start the local build publishing
+service. Its telemetry component reports live progress to the hosted collector when the operator's
+existing Hugging Face login is accepted; otherwise it retains the events
+locally and the build continues. This live event path is independent of the
+staging run files described below.
+
+The same process has a separate component for completed Orrery graph output.
+See [the service architecture](docs/build-emitter.md) for component boundaries
+and the extension interface.
+UK full, dense, and national builds export `graph.orrery.json` automatically,
+using the saved graph declaration and recorded execution phases. Before staging
+the artifact bundle on Hugging Face, the build preserves the exact graph and
+reviewed evidence, submits a durable publication job, waits at most 30 seconds
+for an initial result, and registers the graph files and `orrery.publication.json`
+receipt in the artifact manifest. The viewer is
+[Microcosm runs](https://microcosm-runs.vercel.app).
+
+Graph publication uses the operator's existing Hugging Face credential through
+the service's shared short-lived collector session. No new producer environment
+variable or shared upload secret is required. Publication IDs are independent
+of telemetry run IDs. Graph jobs are not discarded by telemetry retention or
+missing-credential rules. Their source files remain under the persistent
+attempt directory's `orrery-publication/`, and pending jobs resume when the
+service next starts. To retry one explicitly, including after restoring its
+preserved files on another machine:
 
-US fiscal refresh builds emit pre-release staging telemetry **by default**:
-progress JSON is uploaded to `policyengine/populace-us-staging` while the build
-runs (best-effort — a missing token or failed upload never fails the build), so
-every candidate shows up on the staging dashboard before it is published.
-Disable with `--no-staging`, or point elsewhere with `--staging-repo-id` /
-`POPULACE_STAGING_REPO_ID`. An *empty* `POPULACE_STAGING_REPO_ID` is ignored
-rather than read as off, and staging with no destination at all is an argparse
-error — `--no-staging` is the only way a build produces no telemetry.
+```bash
+microcosm-publish-graph --publication-id PUBLICATION_ID \
+  --directory /path/to/attempt/orrery-publication
+```
+
+Retries update `publication.status.json` in that preserved directory, never the
+original artifact bundle's receipt snapshot. A published ID cannot be reused
+for different bytes; changed output needs a new publication ID. Public uploads
+contain only graph declarations, native execution receipts, and reviewed
+aggregate summaries, not H5 files or content-store population payloads.
+
+`--staging-local-only`, `--no-staging`, and `--no-staged-dataset` export the graph
+without submitting a remote publication job. `--no-publish-orrery` independently
+disables graph submission; `--publish-orrery` explicitly enables it even for a
+local-only dataset build. A graph upload failure records a pending/failed receipt
+but does not prevent Hugging Face staging. If a build fails, its saved graph
+evidence is exported and preserved before temporary-directory cleanup; phases
+without recorded execution are not represented as completed.
+
+US fiscal refresh builds also write pre-release staging run files **by
+default**. Progress JSON is uploaded to `policyengine/populace-us-staging`
+while the build runs (best-effort — a missing token or failed upload never
+fails the build), so every candidate shows up on the staging dashboard before
+it is published. Disable these files with `--no-staging`, or point them
+elsewhere with `--staging-repo-id` / `POPULACE_STAGING_REPO_ID`. An *empty*
+`POPULACE_STAGING_REPO_ID` is ignored rather than read as off, and staging with
+no destination at all is an argparse error. `--no-staging` does not disable
+the local telemetry emitter service.
 
 The build manifest records what staging did: the run id, the destination, and
 how many files actually reached it, or an explicit `enabled: false` for a
@@ -75,8 +121,8 @@ The UK commands (`tools/build_uk_frs_spine.py`, a shim over the package's
 `uk_runtime.spine_build`, and `microcosm-build-uk` / `tools/build_uk_full.py`,
 whose `--release-role` builds either the national or the dense line;
 `tools/build_uk_rowwise_candidate.py` is a stub over the same driver) stage
-version 2 telemetry to `policyengine/populace-uk-staging` under the same
-switch. The build command also **stages the finished dataset bundle** it built,
+version 2 staging run files to `policyengine/populace-uk-staging` under the
+same switch. The build command also **stages the finished dataset bundle** it built,
 national, dense or exact-count, under `staged/<run_id>/` in the
 private `policyengine/populace-uk-private` repository so the team can inspect
 it without publishing it: `releases/` and `latest.json` are untouched, the
@@ -203,7 +249,8 @@ weights, calling the release tool's own gate functions (none is
 re-implemented), and exits `1` on any certain failure, `2` on AT-RISK only,
 and `0` when clean. An argparse error also exits `2` but writes no report, so
 the wrapper returns `64` whenever no report was written. It writes only its
-report: nothing under `--out`, no staging telemetry, no receipts. A base or
+report: nothing under `--out`, no staging run files, no receipts. The local
+telemetry emitter service still reports dry-run progress. A base or
 donor that the config does not name locally is still downloaded, into the same
 caches the release uses. A refusal before the stop point becomes the report's
 certain failure. A crash while grading is reported as the dry run's own error,

changelog.d/1080.added.md

@@ -0,0 +1 @@
+Add country-neutral recorded execution exports for Orrery, with independent execution/cache/gate states, phase history, digest-bound evidence links and aggregate calibration diagnostics. UK drivers preserve native evidence and presentation contracts, including checkpoint boundaries and the final national readback inventory. Orrery snapshots remain opt-in.

changelog.d/1099-telemetry-failure-reporting.fixed.md

@@ -0,0 +1 @@
+Report failed UK builds and active stages as failed, isolate telemetry in automated tests, and keep local telemetry setup errors from aborting builds. Rename the initial telemetry migration module so wheel validation accepts it without changing its revision identifier.

changelog.d/1144-uk-full-graph-export.fixed.md

@@ -0,0 +1 @@
+Allow the maintained full UK gate contracts to round-trip through compiler schema and Orrery export, while retaining bounded strings and document sizes.

changelog.d/always-on-telemetry-emitter.added.md

@@ -0,0 +1 @@
+Microcosm builds now start a local telemetry emitter service that reports authenticated live progress, process-tree CPU and memory, heartbeats, US engine-batch progress, and UK graph-node progress without making collector network requests in the build process. PolicyEngine members use their existing Hugging Face login automatically, while builds without an accepted organization credential remain local-only. The staging JSON writers are now named as run-bundle writers and operate independently from the hosted event emitter. SQLAlchemy ORM sessions now manage the local event spool, and Alembic applies and records its schema migrations.

changelog.d/uk-orrery-publication.added.md

@@ -0,0 +1,4 @@
+Export UK execution graphs automatically, publish immutable graph bundles through
+an independent component of the local build publishing service, and retain
+pending jobs and exact source bytes for explicit retry before staging receipts
+alongside the existing Hugging Face artifacts.

conftest.py

@@ -10,6 +10,9 @@
 import importlib.util
 import sys
 from pathlib import Path
+from urllib.parse import urlsplit
+
+import pytest
 
 _ROOT = str(Path(__file__).resolve().parent)
 if _ROOT not in sys.path:
@@ -65,3 +68,82 @@ def pytest_collection_modifyitems(items) -> None:
             item.add_marker("integration")
         if group.engine_module is not None:
             item.add_marker(f"requires_{group.country}")
+
+
+@pytest.fixture(autouse=True)
+def fake_telemetry_emitters(monkeypatch, tmp_path):
+    """Default every workspace test to in-memory telemetry, never production."""
+    from huggingface_hub import constants as hf_constants
+
+    from microcosm.build import telemetry_emitter
+    from microcosm.build.telemetry_emitter_service import collector
+    from test_support.microcosm_build.telemetry import FakeTelemetryEmitter
+
+    for name in ("HF_TOKEN", "HUGGING_FACE_HUB_TOKEN", "HUGGINGFACE_TOKEN"):
+        monkeypatch.delenv(name, raising=False)
+    isolated_hf = tmp_path / "telemetry-hf"
+    token_path = isolated_hf / "token"
+    monkeypatch.setenv("HF_HOME", str(isolated_hf))
+    monkeypatch.setenv("HF_TOKEN_PATH", str(token_path))
+    monkeypatch.setenv("XDG_CACHE_HOME", str(tmp_path / "telemetry-cache"))
+    # HF constants may have been imported before this fixture ran.
+    monkeypatch.setattr(hf_constants, "HF_HOME", str(isolated_hf))
+    monkeypatch.setattr(hf_constants, "HF_TOKEN_PATH", str(token_path))
+
+    emitters = []
+
+    def start(_cls, **kwargs):
+        fields = telemetry_emitter.TelemetryRun.__dataclass_fields__
+        run = telemetry_emitter.TelemetryRun(
+            **{name: value for name, value in kwargs.items() if name in fields}
+        )
+        emitter = FakeTelemetryEmitter(run)
+        emitters.append(emitter)
+        return emitter
+
+    monkeypatch.setattr(
+        telemetry_emitter.LocalTelemetryEmitter, "start", classmethod(start)
+    )
+    # Reset any cached handle left by a previously loaded build entrypoint.
+    for module in tuple(sys.modules.values()):
+        if isinstance(
+            getattr(module, "_ACTIVE_EMITTER", None),
+            telemetry_emitter.LocalTelemetryEmitter,
+        ):
+            monkeypatch.setattr(module, "_ACTIVE_EMITTER", None)
+
+    real_post = collector._http_post
+
+    def local_post(url, *args, **kwargs):
+        parsed = urlsplit(url)
+        collector._development_collector_url(f"{parsed.scheme}://{parsed.netloc}")
+        return real_post(url, *args, **kwargs)
+
+    monkeypatch.setattr(collector, "_http_post", local_post)
+    yield emitters
+    for emitter in emitters:
+        emitter.close()
+
+
+@pytest.fixture
+def real_local_telemetry(monkeypatch, fake_telemetry_emitters):
+    """Permit actual startup only with an explicit loopback collector."""
+    from microcosm.build import telemetry_emitter
+    from microcosm.build.telemetry_emitter_service.collector import (
+        _development_collector_url,
+    )
+
+    def start(cls, **kwargs):
+        _development_collector_url(kwargs.get("development_collector_url") or "")
+        return _REAL_TELEMETRY_START(cls, **kwargs)
+
+    monkeypatch.setattr(
+        telemetry_emitter.LocalTelemetryEmitter, "start", classmethod(start)
+    )
+
+
+# Save the underlying method before per-test patches so imported aliases and
+# class references all share the same default fake and explicit local opt-in.
+from microcosm.build.telemetry_emitter import LocalTelemetryEmitter  # noqa: E402
+
+_REAL_TELEMETRY_START = LocalTelemetryEmitter.start.__func__

docs/agent-guide.md

@@ -76,7 +76,30 @@ The Orrery parser compatibility assertion lives in
 engine-free runner installs the supported public Orrery range and locked Node
 dependency set under `tools/orrery-contract/`; the pytest test generates a
 document through Microcosm's public Python API and requires Orrery's public
-parser to accept it. It performs no browser rendering.
+parser to accept it. The recorded-evidence assertion in `test_graph_evidence.py`
+uses the same pinned parser for groups, statuses, activities and artifacts.
+These CI tests perform no browser rendering.
+
+Workspace tests use an in-memory telemetry emitter by default. The shared
+fixture isolates Hugging Face credentials and cache paths and blocks collector
+HTTP requests outside loopback. Tests of actual emitter startup must request
+`real_local_telemetry` and supply an explicit loopback development collector;
+that opt-in retains credential isolation. Other build tests can inspect
+`fake_telemetry_emitters` without creating sockets or service processes.
+
+The local build publishing process is `microcosm.build.emitter_service`.
+Its component-neutral runtime accepts independent publishers; the composition
+entrypoint installs telemetry and completed Orrery publication components with
+separate workers and queue policies. Shared short-lived authentication lives
+outside both components. The original telemetry imports remain compatibility
+APIs, and the existing SQLite location/migrations remain supported. Add a new
+publisher as a component rather than adding domain-specific branches to the
+runtime. Graph jobs have no telemetry-run foreign key and are not pruned with
+events. UK build completion exports and attempts graph publication before HF
+artifact staging, including preserved evidence for failed builds. Keep retries
+separate from the immutable artifact receipt; see README for the retry command
+and local-only controls. Test service composition and graph delivery in the
+existing engine-free category with fake HTTP, never against the runs app.
 
 New commits to a PR cancel older unfinished CI runs for that same PR.
 Each main-push run has a unique concurrency group, so all main-push runs

docs/build-emitter.md

@@ -0,0 +1,60 @@
+# Local build publishing service
+
+`microcosm.build.emitter_service` is one local process with a private Unix socket
+and independently scheduled publishing components. Its runtime does not know
+about event schemas, graph documents, collector registration, or upload APIs.
+
+The composition entrypoint installs two components:
+
+| Component | Responsibility | Failure and retention behavior |
+| --- | --- | --- |
+| Telemetry | Resource sampling, heartbeats, progress events, collector registration and delivery | Existing best-effort policy; event retention and missing-credential rules remain unchanged |
+| Orrery publication | Upload an exact preserved graph/evidence inventory and finalize its immutable publication | Independent durable jobs, bounded exponential retry, and explicit retry command; no telemetry registration or event pruning |
+
+Each component runs in its own worker. Slow graph I/O cannot delay heartbeat
+sampling or local acknowledgements for progress events. If a worker raises, the
+runtime restarts that worker without stopping other components. The service
+checks the build process's lifetime and notifies every component when it exits.
+Shutdown is bounded; unfinished graph jobs retain their bytes and database lease
+and become eligible again after the lease expires.
+Completion updates require the worker's current lease. Expired attempts cannot
+change a newer worker's result, and a published job cannot return to pending.
+
+The shared `CollectorSession` owns short-lived authentication, credential
+refresh, and synchronization between workers. Queue policies stay in the
+individual publishers. It obtains the existing HF credential and exchanges it
+with the collector; neither component receives a Vercel storage credential.
+
+## Add another publisher
+
+Implement `EmitterComponent` with a unique `name` and these methods:
+
+- `start()` initializes that component's local state.
+- `handle(message)` returns true only for an action it owns, after durable enqueue.
+- `run(stop)` performs delivery and observes the shared stop event.
+- `parent_exited()` records or preserves component-specific state.
+
+Add the instance to the composition entrypoint's component list. Do not add a
+publisher-specific branch to the socket runtime or put new jobs into the event
+queue merely to reuse its delivery loop. Keep independently meaningful retry
+and retention policies separate. A component may use the shared session but must
+not impose its rejection policy on another component's jobs.
+
+The service can start without `--registration-json` for graph-only publishing.
+The explicit `microcosm-publish-graph` retry command likewise requires only a
+publication ID and its preserved inventory, not a run ID. Existing
+`LocalTelemetryEmitter` imports remain compatible; the general public handle is
+also exported as `LocalBuildEmitter` by `microcosm.build.emitter`.
+
+The SQLite location and the original telemetry package's database/migration
+modules retain their existing names to preserve #1099 compatibility. They are
+storage compatibility details, not the service's architectural boundary.
+Graph jobs have their own table without a telemetry-run foreign key.
+
+## Verification
+
+Engine-free tests exercise a third synthetic component without changing the
+runtime, a blocked graph worker alongside a responsive publisher/socket, shared
+authentication, graph-only startup, queue persistence across restart, telemetry
+pruning, missing/rejected credentials, exact-byte upload, and explicit retry.
+Producer HTTP tests use fake responses and never import the runs app.