[anth-volk]

Fixes #19

Third of four stacked pull requests (base: #22). Moves the transport of Microcosm's staging uploads into a new huggingface emitter module behind the interface. Microcosm keeps building and validating its staging documents and bundles; the emitter writes the bytes it is given, with the build's own credential. Nothing in Microcosm calls this yet (H2a and H2b switch it over).

What is ported

Microcosm code (41209b38c)	Interface call	Emitter behaviour
staging.py (contract version 1) and staging_v2.py uploads under runs/<run_id>/, through staging_storage.HuggingFaceDatasetStorage and BestEffortUploadSession	stage_run_files(StageRunFiles(...)), run_files_receipt, read_back_run_files	Same paths and bytes. Version 1's latest_staging.json is written at the root, and its run summary is merged into the repository's runs.json with Microcosm's ordering and formatting (staging._write_json).
staging_dataset.stage_bundle, used by uk_runtime/rowwise_staging.py and the Orrery evidence bundle	stage_dataset_bundle(StageDatasetBundle(...)), dataset_bundle_receipt	Same paths (staged/<run_id>/<file>), one commit pinned to the head it checked, the same commit message, and the same idempotency: a matching repository staged_manifest.json is already_staged with its own commit; a different one is REMOTE_DIFFERS and never overwritten. Each file is handed over with its role (dataset, calibration_diagnostics, build_manifest, staged_manifest, checksums, ...).
rowwise_staging.preflight_staged_dataset and HuggingFaceDatasetStorage.credential_can_write	check_write_access	Whether a credential is present, the repository is reachable, and the Hub reports a write scope.

Not ported: staging_dataset.fetch_bundle and tools/fetch_uk_staged_dataset.py (downloads, not hand-overs).

How the transport works

• Durable queue in its own hand-over database (SQLAlchemy ORM only), a SQLite file next to the spool database (events.sqlite3 → events.handovers.sqlite3) with its own Alembic history (handovers_20261020_01). The shared spool database stays at 20261008_02, which every released emitter accepts (see the second round of review fixes below). staged_run_files keeps the newest handed-over bytes per repository path; staged_run_deliveries keeps per-run counts and the pending runs.json entry; repository_commit_schedules spaces commits, backs off and leases each repository to one writer on the machine; handover_jobs holds bundles (and, in the next pull request, release changes) with leases, retries and receipts, like graph_publication_jobs; handover_files holds the bundle files copied at hand-over. Rows are kept after delivery and nothing deletes them yet, except the copied bundle bytes, which are deleted once the hand-over settles.
• Run files are copied, not referenced. The module reads each handed-over file and stores its bytes before answering, so the build may rewrite it immediately. Bytes equal to those last committed are not written again.
• One Hugging Face commit per repository per cycle. Each module tick (on the host's worker thread) commits, for each repository with run files due, every pending path in one commit; bundle uploads run on a worker thread of the module's own. An unforced hand-over is due one interval after the repository's last commit (the build's upload interval, for example Microcosm's 300 seconds for UK runs); a forced one (final states, artifacts) at the next cycle. Either way, two commits from this machine to one repository, of run files or bundles, are at least 10 seconds apart; only the last commit at shutdown ignores that spacing, and other machines' commits are not counted. Before, each file was its own upload_file commit, up to eight per upload cycle, each triggering the dataset repository's webhook. Each delivery leases its repository first, so this machine never commits to one repository twice at once.
• Best-effort for staging. Requests answer within seconds or report unavailable; nothing raises into the build. A failed commit keeps everything queued and backs off (2 s doubling up to 5 minutes), replacing BestEffortUploadSession's pause after three consecutive failures. Without a credential, work stays queued and one warning is printed. On shutdown the module commits every repository's pending run files once more, before it waits for any bundle upload; only a repository a bundle upload holds at that moment can miss that commit, if the upload outlasts the shutdown period. Warnings carry only error type names, never Hub messages.
• Concurrent writers. A runs.json merge commits with parent_commit; if another build commits first, the index is read and merged again. A bundle commit that loses a race is checked again (it may now be already_staged) and retried.
• Bundles: small files copied, large files verified. Before answering, the module copies the bundle's files smallest first, each up to 32 MiB and up to 64 MiB per hand-over, after checking each one's size and SHA-256 (FILE_SIZE_DIFFERS, FILE_DIGEST_DIFFERS); the receipt's copied_files names them, and those are uploaded as handed over. The worker re-hashes every other file before uploading; changed bytes are refused (PRESERVED_BYTES_CHANGED), missing files wait (PRESERVED_FILES_MISSING).
• Leases. Repository and job leases last 120 seconds and are renewed every 30 seconds while a delivery runs, so a long upload keeps them and a killed emitter blocks the repository for at most 120 seconds.
• Credential. Uploads use HfApi(token=…) with the token found in the same order as telemetry (core/auth.py), read again every cycle.
• Shared path rules (microcosm_emitter_interface.identifiers) are checked by both the interface and the module; staging never writes under releases/.

Behaviour differences to note

• runs.json: Microcosm started from an empty index on any download error, which could overwrite other runs. The module starts empty only when the Hub answers HTTP 404 for the file; every other error, including huggingface_hub's LocalEntryNotFoundError for connection errors, timeouts and 429/5xx responses, retries the cycle.
• read_back_run_files of a run with no file this emitter committed (a refused hand-over, or another spool database) is failed with NO_COMMITTED_FILES, never passed.
• Bundles: REPOSITORY_UNAVAILABLE and UPLOAD_FAILED were final after one attempt; they are now retried from the queue (the receipt keeps the last code while pending).
• Delivery counts are reported per run as files attempted and committed, and commits attempted and made, so Microcosm's staging delivery summary (upload_attempts, upload_successes) can be filled from the emitter's receipt in H2a.

Release

Three added fragments: merging releases the next minor version of both distributions.

Validation

Every test uses an in-memory Hugging Face API (tests/support/fake_hugging_face.py): no network, no real repositories.

Commit	Red (tests first)	Green
cd31388 Add the durable Hugging Face hand-over queue and repository calls	tests/huggingface/test_hub.py, test_queue.py: 29 of 29 fail (module missing). Modified tests/telemetry/test_persistence.py (new head revision and tables), run without the migration: 2 failed, 1 passed	32 passed; re-run: 32 passed
04f9805 Stage run files with one Hugging Face commit per repository per cycle	tests/huggingface/test_run_files.py, tests/interface/test_staging_client.py: 38 of 38 fail. Fixture tests with the new fixtures expected: 8 failed, 30 passed	38 passed; fixtures 38 passed; with test_package_names.py 80 passed; check-artifacts.sh exit 0; re-run: 80 passed
473b576 Stage dataset bundles with each file's role through the queue	tests/huggingface/test_bundles.py, tests/interface/test_bundle_client.py: 36 of 36 fail. Fixture tests: 5 failed, 36 passed	36 passed; fixtures 41 passed; re-run of tests/huggingface and tests/interface: 156 passed

For the next steps

• I2 (collector delivery): handover_jobs is keyed by hand-over and destination (today only huggingface), so the same hand-over can get a collector row without a new table; the bundle hand-over already carries each file's role.
• Jobs are claimed only for kinds this emitter delivers, so a job queued by a newer emitter sharing the database waits for it.

Overlap with open pull requests

Max's #5–#7 edit README.md, including the "Delivery and persistence" section, where this changes one sentence about migrations. #5 edits pyproject.toml and uv.lock.

Review fixes (2026-10-20)

Each fix was written red, then green; only the test files this pull request adds or changes were run. All tests use the in-memory Hub; none contacts a real repository.

Commit	Finding	Red (tests first)	Green
09cf78b Build staging and bundle messages inside the client's error handling	#22's fix carried up: stage_run_files, run_files_receipt, read_back_run_files, stage_dataset_bundle and check_write_access built their messages before the error handling. (When this branch was restacked, these call sites were first adapted to the new message-builder argument without changing behaviour; this commit makes them lazy.)	tests/interface/test_client_errors.py: 4 failed, 12 passed	39 passed with the staging and bundle client tests
bdd659c Start runs.json empty only when the Hub reports the file missing	is_absent matched any EntryNotFoundError/FileNotFoundError subclass, including LocalEntryNotFoundError, so a transient error wiped runs.json.	tests/huggingface/test_hub.py, test_run_files.py: 5 failed, 42 passed (with real huggingface_hub error classes)	47 passed
e67f60e Copy a bundle's small files into the queue when it is handed over	Microcosm's full_build_cli (at 41209b38c) rewrites the build manifest, sha256sums.txt and the national build record right after stage_dataset returns, so the queued upload failed with PRESERVED_BYTES_CHANGED.	test_bundles.py, test_persistence.py: 5 failed, 2 errors, 24 passed	31 passed; with the interface bundle, fixture and error tests: 76 passed
d13e186 Never pass a read-back of a run with no committed files	_read_back_run_files returned passed with files: 0.	test_run_files.py: 2 failed, 30 passed	43 passed with test_staging_client.py
6a277f7 Renew repository and job leases during deliveries; shorten them to 120 s	A 900-second lease outlived a killed emitter.	tests/huggingface/test_leases.py: 3 failed	tests/huggingface: 93 passed
04a612a Describe the runs.json merge, commit spacing and row retention exactly	README wording on runs.json and the 10-second spacing; HandoverQueue's docstring said rows are kept only until delivered.	Documentation only: no red step	—

Re-run after the last commit, every test file this pull request adds or changes: 182 passed. ruff check and ruff format --check pass.

Finding 9: the design, and what Microcosm's H2b must do

Design (emitter side). The bundle's small files are copied into the hand-over database when the build hands the bundle over, as run files are; only files too large to copy (the datasets, which Microcosm never rewrites) are referenced by path and re-hashed before upload. The copy is checked against the declared size and SHA-256 when it is taken, so the upload carries exactly the bytes Microcosm verified with StagedDatasetBundle.from_manifest and listed in its sidecars. tests/huggingface/test_bundles.py::test_files_microcosm_rewrites_after_staging_are_uploaded_as_handed_over follows full_build_cli's order: hand over, rewrite the manifest atomically as replace_manifest does, append to sha256sums.txt as refresh_sha256sums_entry does, rewrite the build record, then run the cycle; the commit carries the handed-over bytes. That is also what Hugging Face held before, because the synchronous stage_bundle uploaded before those rewrites.

What H2b must do (uk_runtime/rowwise_staging.stage_dataset, used by _close_attempt and _close_national_attempt):

1. Call client.stage_dataset_bundle(StageDatasetBundle(...)) where stage_bundle(bundle, storage=..., prefix=UK_STAGED_DATASET_PREFIX) is called today: after write_sidecars, inside stage_dataset, so before replace_manifest, refresh_sha256sums_entry and the national build-record rewrite. Do not rewrite any handed-over file before the call returns.
2. Hand over every bundle file with its role and the SHA-256 and size it has on disk at that moment: dataset, calibration_diagnostics, build_manifest (rowwise_candidate_manifest.json), staged_manifest, checksums (sha256sums.txt) and, in national builds, the build record as its own output role (for example build_record).
3. After the call, check that each file the close step rewrites (the manifest, sha256sums.txt, build_record.json) is in the receipt's copied_files. If one is not (a file above the copy limits, or an emitter that does not copy), wait for the hand-over to settle before rewriting it.
4. Keep every file not in copied_files unchanged and in place until the receipt settles (uploaded, already_staged or failed). On a machine that ends with the build (Modal), wait for that before the build exits (wait_seconds, or poll dataset_bundle_receipt): the queue lives in the machine's hand-over database, and a host started later elsewhere cannot finish it.
5. Map the receipt to staged_dataset's delivery record: revision from the receipt; pending is not yet staged; refused carries its code. Keep the preflight (preflight_staged_dataset) through check_write_access.

Review fixes, second round (2026-10-20)

Each fix is a commit that adds a test failing because of the bug, then a commit with the code. Only the test files this pull request adds or changes were run. All tests use the in-memory Hub; none contacts a real repository.

Commits (red → green)	Finding	Red (tests first)	Green
3330414 → 2709caf Keep hand-overs in their own database so the shared spool stays released	Migration 20261020_03 upgraded the shared spool database (by default ~/.cache/microcosm/telemetry/events.sqlite3). Released emitters 0.2.0 to 0.3.0, and #21's 0.4.0, refuse an Alembic revision they don't know, so a Microcosm build pinned to 0.2.1 on that machine would start with no telemetry and no Orrery queue.	tests/test_queue_compatibility.py, tests/huggingface/test_queue.py, tests/interface/test_staging_client.py: 6 failed, 25 passed (CommandError: Can't locate revision identified by '20261020_03')	31 passed; 188 across this pull request's changed test files
5071bcc → 904c1c0 Upload bundles on their own worker so other commits never wait	All repositories' commits ran one after another on the module's single worker thread, and the host runs the commit at shutdown on that thread after the current cycle. So a long bundle upload delayed the staging repository's run-file commits, the shutdown commit never ran if the build exited mid-upload, and a requested read-back answered pending.	tests/huggingface/test_delivery_workers.py (a host runtime with a 2-second shutdown period, and a bundle commit held on an event): 3 failed, 1 passed. Run files never reached the staging repository; the runtime reported LocalServiceError: Service worker failed (TimeoutError); the read-back answered pending. The passing test checks the held bundle still lands once released.	4 passed; 192 across this pull request's changed test files
cfcb2f6 → 0393d26 Reach the five-minute retry cap, recommit re-handed bytes, guard waits	Review code-quality items: retry_delay capped at 256 s, so MAX_RETRY_SECONDS (300) and the README's "up to five minutes" never applied; bytes equal to the last committed ones, handed over again while other bytes for the path were being committed, were left uncommitted; _wait_until_settled indexed receipt["handover_id"] outside the error handling.	test_queue.py, test_client_errors.py: 3 failed, 34 passed	37 passed

Design: a separate hand-over database (finding on the shared spool). The five tables this stack adds (staged_run_files, staged_run_deliveries, repository_commit_schedules, handover_jobs, handover_files; #24 and #27 add none) live in their own SQLite file with their own Alembic history. core/handover_models.py holds them on their own declarative base, core/handover_alembic holds their one migration (renamed handovers_20261020_01, so neither history accepts the other's file), and core/migrations.py applies either history through one function. The migration was unreleased, so it was moved and renamed, not kept for compatibility. The path is explicit: HuggingFaceSettings(handover_database_path=…) names the file; by default it is events.handovers.sqlite3 next to the spool database. The settings refuse the spool file itself (invalid_configuration), and the module opens only that file. The unreleased configuration fixture and its digest changed with it. Tests: a spool database at 20261008_02, opened by this branch's EventSpool with the module started on the same spool_path, still resolves in a migration history built from only the two released migration files and stays at 20261008_02; a new hand-over database starts empty and migrates on first open (alembic check passes); a spool database is refused as a hand-over database. The installed-interface smoke check confirms the wheel creates the file.

Design: bundle uploads on their own worker (finding on a slow upload). The host gives each module one worker thread and runs close (the commit at shutdown) on it only after the current tick returns, so nothing that can take minutes may run inside the tick. DeliveryWorker (huggingface/service/workers.py) runs one kind of delivery cycle on a thread of its own and can tell an upload in progress from an idle wait. The module starts one for dataset bundles; the tick commits run files only. close stops the workers, commits every repository's pending run files that no worker is uploading to, gives an upload in progress until the shutdown deadline, then commits that repository's run files too if time remains. A repository's lease still keeps two writers off it, so run files for the repository a bundle is uploading to wait for that upload. #24's release changes and #27's collector submissions get their own workers through the same class. background_workers=False keeps every delivery on the tick, for tests that drive the module by hand. The README's shutdown sentence now says which repository can miss the shutdown commit.

Re-run after the last commit, every test file this pull request adds or changes: 196 passed. tools/check-artifacts.sh exit 0; ruff check, ruff format --check and tools/check-changelog.sh pass.

Not changed here: copied bundle bytes kept while a job stays pending (retention is undecided); run-file commits recorded without rechecking the repository lease (the commit has landed, so recording it is right); the module's single request thread for read-backs and write-access checks.

Fix round 2 (2026-10-20)

Each fix is a commit that adds a test failing because of the bug, then a commit with the code. Only the test files this pull request adds or changes were run, against the in-memory Hub.

Commits (red → green)	Finding	Red (tests first)	Green
092b5d8 → 2a752c4 Start no bundle upload once shutdown begins	After close called the bundle worker's stop, the cycle in progress still leased, claimed and uploaded the other repositories on its list. When such an upload outlasted the shutdown period, the second shutdown commit was skipped, so the run files of a repository whose upload had finished in time missed it, which the README says cannot happen.	tests/huggingface/test_delivery_workers.py: 1 failed, 4 passed (AssertionError: an upload to the second repository started after shutdown began)	5 passed
a20a596 → 8d5a589 Check the commit spacing again once a repository is leased	The 10-second spacing between this machine's commits to a repository was checked only when a cycle listed the repository. With bundles on their own worker, the host's thread and the worker could commit to one repository back to back. (#24 already rechecked it; this makes #23 correct on its own.)	test_delivery_workers.py: 2 failed, 5 passed (the staging bundle was committed less than 10 seconds after its run files; the staging run files were committed less than 10 seconds after the bundle)	7 passed; 199 across this pull request's changed test files
0e810f2 Say how to repair a spool database an earlier build moved to 20261020_03	The repair note below, also in the README.	Documentation only: no red step	—

Design: no new upload once shutdown begins. DeliveryWorker now passes its stop check (stop_requested) to each cycle. The job cycle checks it before each repository on its list and again once that repository is leased, so after close calls stop the bundle worker starts no new upload. Only an upload already under way can then hold a repository during the commit at shutdown, which is the one exception the README names; that repository's run files get the second shutdown commit when the upload finishes in time. Cycles that no worker runs use never_stop.

Design: the spacing after the lease. Once a repository is leased, _claim_job claims no job and run_files_due reports no run files due until MIN_COMMIT_SPACING_SECONDS have passed since this machine's last commit to it. The helpers (_spacing_elapsed, commit_spacing_elapsed, _run_files_allowed) are the ones #24 uses, where release changes are exempt. The commit at shutdown still ignores the spacing.

Repair for a developer machine that ran an earlier, unreleased build of this branch. Before the hand-over database existed, this branch's migration 20261020_03 moved the shared spool database to that revision. No migration history contains that revision now, so on such a machine every emitter, this one and the released ones, refuses the spool database at startup (Can't locate revision identified by '20261020_03'), and builds run without telemetry or hand-overs. Nothing repairs it automatically. Stop every Microcosm build and emitter on the machine, then delete the spool database: $XDG_CACHE_HOME/microcosm/telemetry/events.sqlite3, or ~/.cache/microcosm/telemetry/events.sqlite3 when XDG_CACHE_HOME is unset (or the spool_path the build set), together with events.sqlite3-journal beside it if that file exists. Deleting it loses everything queued in it and not yet delivered: telemetry events not yet sent to the collector, graph publication jobs not yet published, and the run files and dataset bundles the earlier build handed over and had not committed. The next emitter creates a new spool database at 20261008_02. To keep the queued telemetry events and graph publication jobs instead, reset the revision with sqlite3 <spool file> "UPDATE alembic_version SET version_num = '20261008_02'": the earlier build added only the five hand-over tables, which every emitter then ignores, so their undelivered hand-overs are lost either way.

Re-run after the last commit, every test file this pull request adds or changes: 199 passed. tools/check-artifacts.sh exit 0; ruff check, ruff format --check and tools/check-changelog.sh pass.

Fix round 3 (2026-10-20)

Each code fix is a commit that adds a test failing because of the bug, then a commit with the code that leaves that test unchanged. Only the test files this pull request adds or changes were run, against the in-memory Hub. This section replaces the round-2 statements that the bundle worker "starts no new upload" and the round-2 repair command.

Commits (red → green)	Finding	Red (tests first)	Green
7ccbf7a → ebf18c9 Give back a job claimed after shutdown began instead of uploading it	The cycle checked for shutdown before each repository and once the repository was leased, but not after it claimed the job, so one upload could still start after shutdown began.	tests/huggingface/test_delivery_workers.py: 1 failed, 7 passed (AssertionError: a bundle upload started after shutdown began, once its job was claimed)	8 passed
817d017 Guard the spool repair to revision 20261020_03 and drop its tables	The repair note's UPDATE had no WHERE, kept the earlier build's tables, said "five tables", and said every emitter is affected.	Documentation only: no red step	—

Design: a check after the claim. _deliver_repository now checks for shutdown once more after it claims a job, immediately before the upload begins. A job claimed after shutdown began is given back with the new HandoverQueue.release_job, which clears its lease without counting an attempt, so it stays pending and due for the next emitter. The test begins shutdown from inside the job claim, after both earlier checks, so it covers this check alone. The README now states the exact guarantee: the worker checks before each repository, once the repository is leased, and once the job is claimed; during the commit at shutdown a repository can be held only by an upload that passed that last check before shutdown began, or, for the few database operations it takes to give it back, by a lease taken just before.

Repair note (replaces the round-2 note). On a developer machine that ran an earlier, unreleased build of this branch, the spool database is at revision 20261020_03, which no migration history contains now. An emitter started with the telemetry or orrery module, this one or a released one, refuses it at startup; an emitter started with only the huggingface module never opens the spool database and is not affected. After stopping every build and emitter on the machine, either delete the spool database (and events.sqlite3-journal beside it), losing everything queued in it, or keep the queued telemetry events and graph publication jobs with:

sqlite3 -bail <spool file> <<'SQL'
BEGIN;
-- Stop here, changing nothing, unless the file is at the earlier build's revision.
CREATE TEMP TABLE repair_guard (revision TEXT NOT NULL CHECK (revision = '20261020_03'));
INSERT INTO repair_guard VALUES ((SELECT version_num FROM alembic_version));
DROP TABLE IF EXISTS handover_files;
DROP TABLE IF EXISTS handover_jobs;
DROP TABLE IF EXISTS repository_commit_schedules;
DROP TABLE IF EXISTS staged_run_deliveries;
DROP TABLE IF EXISTS staged_run_files;
UPDATE alembic_version SET version_num = '20261008_02' WHERE version_num = '20261020_03';
COMMIT;
SQL

• The tables dropped are the four or five the earlier migration created (handover_files only in its later form), checked against that migration at every earlier head (f438369, 473b576, 04a612a); it created no index. No spool revision on any branch of the stack or in any release uses these names, so dropping them does what that migration's own downgrade() did.
• Unless the file's revision is 20261020_03, the guard row is refused, -bail stops sqlite3 at that error, and the open transaction is rolled back. So the command changes nothing in a spool database at any other revision, nor in the hand-over database (events.handovers.sqlite3), whose tables have the same names. I checked this with sqlite3 on scratch files at 20261020_03, at a later revision, at the hand-over revision 20261020_01, and with an empty version table.
• No repair code is added: the repository's rules exclude schema repair in code.

Re-run after the last commit, every test file this pull request adds or changes: 200 passed. tools/check-artifacts.sh exit 0; ruff check, ruff format --check and tools/check-changelog.sh pass.

🤖 Generated with Claude Code