Add durable Orrery publication and graph readers to the local provider

.github/workflows/ci.yml

@@ -73,7 +73,7 @@ jobs:
       - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
         with:
           repository: PolicyEngine/calibration-diagnostics
-          ref: 80a2a4ac7654f586b2b4b8d4c6ec7d2379616722
+          ref: 280459c8d782658b3fb6098b27e9acba11af29d8
           path: .collector
           persist-credentials: false
       - uses: astral-sh/setup-uv@d0d8abe699bfb85fec6de9f7adb5ae17292296ff # v6
@@ -85,13 +85,27 @@ jobs:
           path: dist
       - run: bash tools/check-collector.sh
 
+  reader:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+      - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
+        with:
+          bun-version: "1.4.2"
+      - run: bash tools/check-reader.sh
+      - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+        with:
+          name: orrery-reader
+          path: reader-dist/*.tgz
+          if-no-files-found: error
+
   required:
     name: Required checks
     if: always()
-    needs: [quality, tests, build-packages, collector]
+    needs: [quality, tests, build-packages, collector, reader]
     runs-on: ubuntu-latest
     steps:
       - name: Require all validation jobs to succeed
         env:
           RESULTS: ${{ join(needs.*.result, ' ') }}
-        run: test "$RESULTS" = "success success success success"
+        run: test "$RESULTS" = "success success success success success"

.github/workflows/publish.yml

@@ -40,3 +40,22 @@ jobs:
       - uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1
         with:
           packages-dir: dist/
+
+  publish-reader:
+    needs: [verify, publish]
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      contents: read
+      id-token: write
+    steps:
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+      - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
+        with:
+          node-version: "24"
+          registry-url: "https://registry.npmjs.org"
+      - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
+        with:
+          name: orrery-reader
+          path: reader-dist
+      - run: bash tools/publish-reader.sh

.gitignore

@@ -7,4 +7,6 @@ dist/
 .coverage
 .collector/
 .test-artifacts/
-
+node_modules/
+reader-dist/
+typescript/orrery/dist/

AGENTS.md

@@ -2,6 +2,7 @@
 
 - Use uv, Python 3.13+, pytest, and Ruff. Keep workflows as single-command calls to Bash scripts when shell logic has multiple lines.
 - The generic runtime must not import telemetry or Microcosm. The telemetry client must not import service implementation, database, or authentication modules.
+- Graph adapters must not import service, database, authentication or HTTP code. Inventory hashing, durable enqueue, receipt lookup and retries run in the separate local emitter service; no direct-database fallback.
 - Publish only the `microcosm-emitter` distribution. Keep the generic host and required telemetry library internally separate under `microcosm_emitter`; do not introduce standalone telemetry packages or optional telemetry installation.
 - Python databases use SQLAlchemy declarative ORM and Alembic exclusively. No handwritten SQL, direct-driver persistence, create_all(), schema repair, or pre-Alembic adoption. TypeScript database work, if introduced, must use Drizzle ORM and Drizzle Kit migrations.
 - Tests use temporary databases created through real migrations, synthetic credentials, and loopback network services. Never use ambient credentials or contact the production collector in tests.

README.md

@@ -1,10 +1,13 @@
 # Microcosm emitter
 
-`microcosm-emitter` is one Python distribution containing the process host and a required telemetry library:
+`microcosm-emitter` is one Python distribution containing the process host, required telemetry library and completed-graph publication library:
 
 - `microcosm_emitter.host` owns subprocess startup, a private Unix socket, explicit module loading, parent-process monitoring, and shutdown. Its code does not import telemetry, database, authentication, or Microcosm modules.
 - `microcosm_emitter.telemetry.client` is the thin build adapter. It starts the local telemetry emitter service and sends build requests, without importing the service implementation, database, or authentication libraries.
 - `microcosm_emitter.telemetry.service` interprets those requests in the separate process. It owns stage tracking, event construction, credential redaction, resource sampling, SQLAlchemy ORM persistence, Alembic migrations, authentication, and collector delivery.
+- `microcosm_emitter.orrery.client` sends graph inventory, enqueue, receipt and retry requests over the same private socket. It does not hash files, open a database, authenticate or upload.
+- `microcosm_emitter.orrery.service` validates and hashes the selected graph files, commits publication jobs, and uploads exact bytes with retry and expiring worker reservations. A graph publication requires no telemetry run registration.
+- `microcosm_emitter.core` provides service-side authentication and SQLAlchemy/Alembic persistence shared by the telemetry and graph libraries. The generic host does not import it.
 
 Installing `microcosm-emitter` always installs the telemetry library and all its dependencies. Telemetry is not a standalone distribution or an optional extra. The package keeps import boundaries between the host, adapter, and service implementation; they do not need separate installation or publication.
 
@@ -16,9 +19,23 @@ The service interprets requests such as a stage change or build failure, timesta
 
 The adapter bounds socket calls and reports telemetry failures without failing the build. It converts Python arguments into JSON but does not process their telemetry meaning. Network requests run on the service worker, outside the message-handling lock.
 
-The service uses the existing Hugging Face credential from the environment or local login cache. The collector validates PolicyEngine membership before issuing a short-lived write credential. Missing or rejected credentials permanently mark that producer's events as local-only. Transient collector failures retain events for retry. The production destination belongs to the package; development overrides accept loopback addresses only.
+`block()` finishes the current stage and records that validation checks refused the build, including the phase, check identifiers, failure count and optional check statuses. `fail(error_code=...)` preserves the caller's failure code alongside the exception type and failure class. Microcosm classifies interruptions, process termination and memory failures; this package forwards those classifications rather than duplicating them. Both outcomes finish telemetry, so later heartbeat or parent-exit handling cannot replace them.
 
-The queue retains the current Microcosm Alembic revision, producer sequences, event IDs, and upload eligibility. Unversioned databases are rejected without schema modification. There is no raw SQL, automatic schema adoption, or parallel legacy implementation. The hosted collector and dashboard do not change.
+The service uses the existing Hugging Face credential from the environment or local login cache. The collector validates PolicyEngine membership before issuing a short-lived write credential. Missing or rejected credentials permanently mark that producer's events as local-only. Permanent HTTP client errors when registering a run or uploading events also keep only the affected producer's records local; HTTP 408, 429, server errors and network failures remain retryable. HTTP 401 invalidates the short-lived credential for renewal. The production destination belongs to the package; development overrides accept loopback addresses only.
+
+The queue upgrades the existing `20261007_01` Alembic revision to `20261008_02`, adding graph-job records without changing producer sequences, event IDs or upload eligibility. Unversioned databases are rejected without schema modification. There is no raw SQL, automatic schema adoption, or parallel legacy implementation. The hosted telemetry collector and calibration dashboard do not change.
+
+## Graph publication
+
+`GraphPublicationClient.start()` starts the graph module in a separate local emitter service process. `inventory()` asks that process to hash an explicit set of allowed graph files; `enqueue()` returns a pending receipt only after the job commits. `receipt()` reads its result and `retry()` makes a pending job eligible again. Startup or socket failures return an unavailable result, not a claim that a job was saved; the build adapter never writes a fallback database.
+
+Graph requests allow up to 30 seconds for bounded file validation and hashing; telemetry acknowledgements, readiness probes and close requests keep their existing short timeout. Very slow storage can still exceed the graph request limit, so a timeout is not proof that a late service-side commit did or did not occur. Repeating an enqueue with the same publication ID and inventory is idempotent.
+
+The host can also start telemetry and graph modules together with `start_services()`. Each module has an independent worker, so a slow graph upload does not block telemetry requests. Construct the telemetry adapter with `close_service_on_finish=False` when a build must publish graphs after recording completion, failure or refusal. Telemetry still finishes, but explicit outer cleanup closes the shared host after graph requests; standalone adapters close automatically by default. A module-specific graph client does not own shared-host shutdown. Both libraries share an existing HF credential and short-lived collector session, but graph jobs remain pending independently of telemetry event retention or local-only upload eligibility.
+
+Preserve the selected source files until publication succeeds. `microcosm-emitter-publish-graph` asks the service to retry a publication by ID and can re-enqueue a preserved directory on another machine. It never rewrites the original build receipt; the queue and separate latest-status file record retries.
+
+The existing TypeScript package `@policyengine/microcosm-provider-orrery` reads immutable published graph files on a server or in a browser. It remains separate from the Python distribution and does not change the hosted telemetry API.
 
 ## Development
 
@@ -29,14 +46,17 @@ uv sync --locked
 bash tools/check-quality.sh
 uv run --no-sync pytest -m 'not collector'
 bash tools/check-artifacts.sh
+bash tools/check-reader.sh
 ```
 
-Tests isolate ambient credentials and use synthetic identities. The artifact check builds one source archive and wheel, installs that wheel into a clean environment away from the source checkout, verifies that the host does not import telemetry or its dependencies, and exercises the real telemetry service process and packaged migrations.
+Tests isolate ambient credentials and use synthetic identities. The artifact check builds one source archive and wheel, installs that wheel into a clean environment away from the source checkout, verifies import boundaries, and exercises telemetry-only, graph-only and combined service startup with packaged migrations. Reader checks cover tests, formatting, types, build and an isolated tarball installation; they do not inspect visual styling.
 
-The collector compatibility job tests the installed wheel against a pinned dashboard collector and disposable PostgreSQL. It covers the HTTP exchange, event ingestion, dashboard run documents, rejected identities, and duplicate delivery after a lost acknowledgement. Local execution requires that pinned collector checkout and a loopback test database; `tools/check-collector.sh` verifies these inputs. Never point this test at a deployed database.
+The collector compatibility job tests the installed wheel against a pinned dashboard collector and disposable PostgreSQL. It covers the HTTP exchange, event ingestion, blocked outcomes and supplied failure codes in dashboard run documents, rejected identities, and duplicate delivery after a lost acknowledgement. Local execution requires that pinned collector checkout and a loopback test database; `tools/check-collector.sh` verifies these inputs. Never point this test at a deployed database. The pin verifies source compatibility, not which revision the hosted collector currently runs.
 
 ## Publication
 
-The project publishes only `microcosm-emitter`. A published GitHub release triggers version and main-branch ancestry checks, all CI checks, and artifact verification. Only then can the protected `pypi` environment approve publishing the exact tested wheel and source archive. The publish job uses PyPI Trusted Publishing; no long-lived publishing token is supplied through CI.
+The project publishes one Python distribution, `microcosm-emitter`. A published GitHub release triggers version and main-branch ancestry checks, all CI checks, and artifact verification. Only then can the protected `pypi` environment approve publishing the exact tested wheel and source archive. The publish job uses PyPI Trusted Publishing; no long-lived publishing token is supplied through CI. The separate TypeScript reader publication waits for Python publication and publishes the verified tarball through npm OIDC.
 
 First publication requires one PyPI pending trusted publisher for `microcosm-emitter`, attached to GitHub owner `PolicyEngine`, project `microcosm-emitter`, workflow `publish.yml`, and environment `pypi`. That external setup must be verified before releasing. No release is created or merged automatically by this project.
+
+The npm reader requires registry-owner first-publication and trusted-publisher setup; this code does not provision or verify those private account settings. No hosted-service deployment, registry publication or consumer activation follows from installing this package.

changelog.d/3-build-outcomes.fixed.md

@@ -0,0 +1 @@
+Preserve validation-refused build outcomes and caller-supplied failure codes in the telemetry service. Allow shared-host cleanup to wait until graph publication finishes. Retain only rejected producers' records locally after permanent collector HTTP client errors, and test these behaviors against the installed package and collector.

changelog.d/3-publication-failures.fixed.md

@@ -0,0 +1 @@
+Keep graph publication running when a preserved directory is removed or the local queue reports an error, and record the latest status only after the queue commit.

changelog.d/3-publication-types.changed.md

@@ -0,0 +1 @@
+Define explicit types for publication inventories and results, and return a frozen record when reserving a graph upload.

changelog.d/3-single-emitter.changed.md

@@ -0,0 +1 @@
+Integrate completed-graph publication into the single microcosm-emitter package. Keep graph adapters limited to socket requests and preserve shared SQLAlchemy/Alembic persistence, telemetry lifecycle behavior and separate TypeScript readers.

changelog.d/3.added.md

@@ -0,0 +1 @@
+Add independent completed-graph publication and verified Orrery document readers alongside telemetry in the local provider.

conftest.py

@@ -15,6 +15,7 @@ def isolate_credentials_and_network(tmp_path, monkeypatch):
     from huggingface_hub import constants
 
     monkeypatch.setattr(constants, "HF_TOKEN_PATH", str(hf_home / "token"))
+    from microcosm_emitter.core import auth
     from microcosm_emitter.telemetry.service import collector
 
     real_post = collector._http_post
@@ -25,3 +26,4 @@ def loopback_post(url, *args, **kwargs):
         return real_post(url, *args, **kwargs)
 
     monkeypatch.setattr(collector, "_http_post", loopback_post)
+    monkeypatch.setattr(auth, "_http_post", loopback_post)

package.json

@@ -0,0 +1,15 @@
+{
+  "name": "@policyengine/microcosm-provider-orrery",
+  "version": "0.1.0",
+  "private": true,
+  "type": "module",
+  "description": "Pinned-source development entry points; publish the built package from typescript/orrery",
+  "exports": {
+    "./contract": "./typescript/orrery/src/contract.ts",
+    "./browser": "./typescript/orrery/src/browser.ts",
+    "./server": "./typescript/orrery/src/server.ts"
+  },
+  "peerDependencies": {
+    "@axiom-foundation/orrery": "0.6.1"
+  }
+}

pyproject.toml

@@ -10,13 +10,18 @@ dependencies = [
     "sqlalchemy>=2,<3",
     "huggingface-hub>=0.20,<2",
     "psutil>=6,<8",
+    "httpx>=0.27,<1",
 ]
 
 [project.urls]
 Source = "https://github.com/PolicyEngine/microcosm-emitter"
 
 [project.entry-points."microcosm.emitter.modules"]
 telemetry = "microcosm_emitter.telemetry.service:create_module"
+orrery = "microcosm_emitter.orrery.service:create_module"
+
+[project.scripts]
+microcosm-emitter-publish-graph = "microcosm_emitter.orrery.cli:main"
 
 [build-system]
 requires = ["hatchling"]

src/microcosm_emitter/core/__init__.py

@@ -0,0 +1 @@
+"""Shared provider services; importing this package has no runtime side effects."""

src/microcosm_emitter/telemetry/service/alembic/env.py → src/microcosm_emitter/core/alembic/env.py

@@ -3,7 +3,7 @@
 from alembic import context
 from sqlalchemy.engine import Connection
 
-from microcosm_emitter.telemetry.service.models import SpoolModel
+from microcosm_emitter.core.models import SpoolModel
 
 _MISSING_CONNECTION_ERROR = (
     "telemetry spool migrations require a programmatically supplied connection"

src/microcosm_emitter/core/alembic/versions/v20261008_02_graph_publication.py

@@ -0,0 +1,27 @@
+"""Add durable graph publication jobs without changing event retention."""
+
+import sqlalchemy as sa
+from alembic import op
+
+revision = "20261008_02"
+down_revision = "20261007_01"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    op.create_table(
+        "graph_publication_jobs",
+        sa.Column("publication_id", sa.Text(), primary_key=True),
+        sa.Column("inventory", sa.Text(), nullable=False),
+        sa.Column("directory", sa.Text(), nullable=False),
+        sa.Column("status", sa.Text(), nullable=False),
+        sa.Column("attempts", sa.Integer(), nullable=False),
+        sa.Column("next_attempt_at", sa.Float(), nullable=False),
+        sa.Column("lease_until", sa.Float(), nullable=False),
+        sa.Column("receipt", sa.Text(), nullable=False),
+    )
+
+
+def downgrade() -> None:
+    op.drop_table("graph_publication_jobs")