[anth-volk]

Fixes #20

Last of four stacked pull requests (base: #23). Adds the release writer: each build is published side by side, and one releases.json per dataset repository is the only record of what is current. Nothing calls the writer yet; Microcosm switches to it in H2c.

What changes

microcosm_emitter_interface.releases (shared, so Microcosm, the emitter and a scheduled repository check run the same code):

• The record: schema_version 1, default (a release or null), and per release its role, current (a build or null) and each build's published_at and withdrawn, serialized in that order with releases and builds sorted.
• The changes: add_build (optionally making it current), make_current, withdraw_build.
• The checks every write must pass:
  • check_record: current names a listed build that is not withdrawn; default names a listed release;
  • check_change: existing build entries change only withdrawn (none removed);
  • check_repository: every listed build has its folder releases/<release>/<build>/ and a release_manifest.json naming the same release and build;
  • check_commit: no file inside an existing build folder is added, changed or removed, listed or not; no write creates a latest*.json file; current cannot change while any latest*.json file exists (the migration window before the old pointer files are deleted).

The writer (microcosm_emitter.huggingface.service.releases), queued as a release_change job:

• publish_release(PublishRelease(repository, release, build, release_role, make_current, files)): files with roles, including release_manifest.json. promote_build and withdraw_build change only releases.json.
• Every request carries a request_id, a new random one unless the caller gives one, and the emitter applies each request once: sending the same request again returns its receipt, while a new request is applied even when it repeats an earlier change (promoting a build back, or retrying a withdrawal or a refused change once its cause is gone). The module queues the validated request's own message, never extra keys.
• Every write: read releases.json and the file list at main's head, apply the change, run the checks, commit the new build folder and releases.json together with parent_commit set to that head. On a parent-commit conflict (HTTP 409 or 412), read the new head and apply the change again, up to five times; two concurrent publications both land.
• Publishing refuses a build whose folder exists (BUILD_FOLDER_EXISTS), except when the build is listed with a byte-identical manifest: that reports published with the commit that wrote it, even if another build was made current since (a retry after a commit that landed but raised). Publishing never promotes an already published build.
• A releases.json above 4 MiB or nested too deeply to decode is refused as INVALID_RECORD, and such a manifest as MANIFEST_MISMATCH; neither is retried.
• No Hugging Face tags or branches, no current/ folder, no mutable root file, no latest*.json; existing tags and files are untouched. release.py's staging branch, release tag and pointer-file commit are replaced by this layout, not ported.
• Preserved files are re-hashed before upload; Hub failures stay pending and are retried, for at most an hour (REQUEST_EXPIRED); a promotion, withdrawal or make-current publication that a newer such change overtook is refused (NEWER_CHANGE_LANDED); broken rules are refused with the rule's code and messages. Release changes run on a worker thread of their own (see Queue Hugging Face staging uploads with one commit per repository per cycle #23's DeliveryWorker). Release jobs skip the 10-second spacing between commits; bundles and run files keep it (they all still make at most one commit per repository per cycle).

Left open, with extension points (not decided here)

• Who may promote or withdraw: HuggingFaceModule(release_change_policy=...) takes a function of the typed change and a ReleaseCaller (the user the emitter runs as and the build process that started it) that returns a refusal reason or None. The module applies it when the change is handed over, so a refusal (NOT_AUTHORIZED) is answered at once and never queued. The default allows every change; nothing configures it yet.
• Whether the dense release has a current build: current may stay null; make_current=False publishes without promoting.
• How the dashboard shows withdrawn builds: the record only stores withdrawn.
• The US layout: nothing here depends on a country; release and build names are opaque identifiers.
• Also open: setting default (the writer never changes it; a new record starts with null), P1's migration write (it sets current while the pointer files still exist, which check_commit refuses, so it needs its own one-time path or a migration mode), and withdrawing the current build (refused; promote another build first). release_role is an addition to the plan's call: a new release needs its role for releases.json.

Release

Two added fragments: merging releases the next minor version of both distributions.

Validation

All tests use the in-memory Hugging Face API; nothing touches a real repository.

Commit	Red (tests first)	Green
244672a Add the releases.json record and its publication checks to the interface	tests/interface/test_releases_record.py: 26 of 26 fail (module missing)	26 passed; re-run: 26 passed
ca43761 Publish release builds side by side through the release writer	tests/huggingface/test_release_writer.py, tests/interface/test_release_client.py: 29 of 29 fail. Fixture tests with the three new release fixtures: 5 failed, 39 passed	29 passed; fixtures 44 passed; re-run of the four files: 73 passed

Before the review fixes, across the stack, every test file it adds or changes: 261 passed. bash tools/check-artifacts.sh exit 0; ruff check and ruff format --check pass.

Review fixes (2026-10-20)

Each fix was written red, then green; only the test files this pull request adds or changes were run. All tests use the in-memory Hub.

Commit	Finding	Red (tests first)	Green
f5c8daf Build release messages inside the client's error handling	#22's fix carried up to publish_release, promote_build and withdraw_build.	tests/interface/test_client_errors.py: 2 failed, 16 passed	29 passed with test_release_client.py
9ad61be Apply each requested release change; deduplicate only resent requests	Content-derived hand-over IDs made A → B → A, a withdrawal retried after another promotion, and a request repeated after its refusal's cause was gone do nothing.	release writer, release client, message and fixture tests: 14 failed, 73 passed	87 passed (the three unreleased release fixtures gain request_id; their digests are updated)
75ba8e9 Report a landed publication as published after another promotion	_already_published refused a landed publication whose build was no longer current.	test_release_writer.py: 2 failed, 23 passed	25 passed
21ba3d6 Refuse deeply nested or oversized release records instead of retrying	RecursionError from json.loads became a pending REPOSITORY_UNAVAILABLE, retried forever.	test_releases_record.py, test_release_writer.py: 5 failed, 51 passed	56 passed
1fc8fb9 Keep the commit spacing for every job except release changes	Every job skipped MIN_COMMIT_SPACING_SECONDS, so bundles could commit in bursts again.	test_release_writer.py: 1 failed	tests/huggingface: 121 passed
97a57f4 Queue typed release changes and pass the caller to the change policy	dict(message) queued; string statuses; RELEASE_CHANGE, MAX_PARENT_CONFLICT_ATTEMPTS and UPLOAD_FAILED defined twice or more; core.auth._huggingface_token imported; the policy had no caller identity.	test_boundaries.py, test_release_writer.py: 7 failed, 36 passed	43 passed; with tests/huggingface, tests/interface, tests/telemetry/test_delivery.py (its monkeypatches follow the renamed huggingface_token): 334 passed

Re-run after the last commit, every test file this pull request adds or changes: 183 passed. ruff check and ruff format --check pass. The other commits' hashes changed when this branch was restacked.

Not built here, as asked: P1's one-time migration path, which check_commit still refuses while pointer files exist.

Review fixes, second round (2026-10-20)

Each fix is a commit that adds a test failing because of the bug, then a commit with the code. Only the test files this pull request adds or changes were run. All tests use the in-memory Hub.

Commits (red → green)	Finding	Red (tests first)	Green
1287dd1 → 417d651 Expire queued release changes and refuse ones a newer change overtook	A release change that failed for any reason other than a broken rule (CREDENTIAL_MISSING, UPLOAD_FAILED including 401/403, REPOSITORY_UNAVAILABLE, PARENT_CONFLICT) stayed pending, was retried without limit and in backoff order, and was applied to whatever record it found. So it could land after a newer change from this or another machine and put back an older current, or land days later after hf auth login.	tests/huggingface/test_release_writer.py: 4 failed, 34 passed. The older build became current in both ordering cases; the change pending for two days was published; the expired change stayed pending with CREDENTIAL_MISSING. The two passing tests guard that changes handed over in order all apply and that a plain publication queued before a newer change still lands.	40 passed (two more tests added with the fix drive the recorded request time directly); 195 across this pull request's changed test files
a314547 → 0cb0a28 Name a caller without a user name by its numeric user ID	On Python 3.13 and later getpass.getuser() raises when no user name variable is set and the user ID has no password-database entry (a container run with an arbitrary user ID), so every release change came back unavailable. The policy itself is unchanged (allow-all by default).	test_release_writer.py: 1 failed, 40 passed (OSError: No username set in the environment)	41 passed
327be59 Say that release changes skip the commit spacing	The stale comment on MIN_COMMIT_SPACING_SECONDS and the queue docstring's "and later releases".	Documentation only: no red step	—
7233dc6 → baab6ed Accept only the integer schema version and keep written records readable	Review code-quality items: ReleasesRecord.parse accepted "schema_version": true or 1.0; the writer never checked that a record it writes stays within the 4 MiB it accepts when reading (check_change now refuses it with RECORD_TOO_LARGE).	tests/interface/test_releases_record.py: 3 failed, 29 passed	73 passed with test_release_writer.py
6bd4a45 Share the writer's refusal for broken rules; test running out of races	Review code-quality items: the violation messages were joined in two places; nothing tested a change that loses every parent-commit race (PARENT_CONFLICT).	No behaviour change and no red step: the new test guards existing behaviour	42 passed

The mechanism for queued release changes (also in releases.py's docstring and the README):

• Expiry. A release change still pending an hour after its hand-over (RELEASE_CHANGE_EXPIRY_SECONDS) becomes refused with REQUEST_EXPIRED. That holds without a credential too, so a later login never applies it.
• Order. An ordered change (a promotion, a withdrawal, or a publication that makes its build current) is refused with NEWER_CHANGE_LANDED when an ordered change requested after it has already landed on the repository's record. The writer learns that from two places:
  • this machine's queue, which also knows later changes that left the record unchanged;
  • the title of the commit that last changed releases.json. Every writer commit's title now ends [newest request <time>]: the newest request time of the ordered changes applied so far.
• Plain publications that only add a build are never refused for order and don't count as newer, because concurrent publications must both land (plan item 5). Their commits carry the recorded time forward.
• No empty commits. A change that leaves releases.json unchanged makes no commit and reports the head, which is what the Hub would do anyway.
• What the mechanism can't see. A newer change on another machine that left releases.json unchanged (promoting the build that is already current) makes no commit, so it records no time; only the expiry bounds how late an older change can still land after it. Request times come from each machine's clock.
• Why this design. Strict per-repository request order can't see other machines, and an expiry alone leaves an hour's window across machines. The record's shape is fixed by the plan, so the request time lives in the commit title, which the Hub returns with each file's last commit.

Restack onto #23's fixes. Some of this pull request's commits changed while it was rebased onto the second round of #23 fixes:

• 630afd4 (the release writer) adds a release-change worker beside Queue Hugging Face staging uploads with one commit per repository per cycle #23's bundle worker;
• 93a08d0 (commit spacing) merges with Queue Hugging Face staging uploads with one commit per repository per cycle #23's per-kind delivery cycles;
• the tests use handover_database_path, and background_workers=False where they drive the module by hand.

Re-run after the last commit, every test file this pull request adds or changes: 200 passed. ruff check, ruff format --check and tools/check-changelog.sh pass.

Unchanged, as open decisions: who may promote or withdraw (the policy stays allow-all), the manifest's build field, role names and file list (H2c), and AGENTS.md's publishing rule.

Fix round 2 (2026-10-20)

Restack onto #23's round-2 fixes. Every commit of this pull request has a new SHA; the titles are unchanged. The sections above use the earlier SHAs: 630afd4 is now 9c9c8da, 93a08d0 is b8bc990, 1287dd1 is 568fce8, 417d651 is 42e5686, a314547 is 65f7242, 0cb0a28 is 0b8b3a7, 327be59 is 69a3f48, 7233dc6 is e5eb811, baab6ed is 7998111 and 6bd4a45 is 7788f61. Conflict resolutions:

• 9c9c8da (the release writer) keeps Queue Hugging Face staging uploads with one commit per repository per cycle #23's spacing helpers, leaves out Queue Hugging Face staging uploads with one commit per repository per cycle #23's spacing check in _claim_job because this commit lets every job skip the spacing (as it did before), and gives the release-change worker Queue Hugging Face staging uploads with one commit per repository per cycle #23's stop check;
• b8bc990 (commit spacing) restores that check for every job except release changes;
• 69a3f48 merges its comment on MIN_COMMIT_SPACING_SECONDS with Queue Hugging Face staging uploads with one commit per repository per cycle #23's.

So the commits from 9c9c8da to ebd4893 fail #23's new test that a bundle waits for the spacing after a run-file commit, as the original 630afd4 let bundles skip it; b8bc990 and the head pass.

Commits (red → green)	Finding	Red (tests first)	Green
943913f → 35481d4 Keep release order per release; an overtaken publication keeps its build	417d651 compared request times across every release in a repository. A change to one release was refused because a later change to another release had landed first, and two publications that landed at once and both made their build current lost one build (plan §7, item 5).	tests/huggingface/test_release_writer.py: 4 failed, 44 passed. Each failure is the refusal ('refused', 'NEWER_CHANGE_LANDED') where ('published', None) is expected: the review's test (a national promotion requested before a dense promotion that landed first); its trigger A through the module's queue; its trigger B (concurrent publications to two releases, both making their build current); and concurrent publications to one release when the newer one lands first. The two passing tests guard the older-first order and that a stale promotion is still refused after another release's later commit.	50 passed (two tests added with the fix: the overtaken publication's message and title, and the window that drops old times); 208 across this pull request's changed test files

Design: order per release.

• Commit titles now end [newest requests: <release>=<time>; ...]: for each release, the newest request time among the ordered changes applied to it, carried forward by every writer commit. releases.json is unchanged.
• This machine's queue is grouped by release the same way.
• Refusals. A promotion or a withdrawal is refused with NEWER_CHANGE_LANDED only when a promotion, a withdrawal or a publication that makes its build current, to the same release and requested after it, has landed. Within one release the original fix holds.
• An overtaken publication that makes its build current is no longer refused. It lists its build, leaves current as the newer change set it, and reports published with a message saying so. Two publications that land at once, to one release or to two, therefore both keep their builds, and the newer request's build is current.
• Title size. A release's time leaves the titles once it is two hours (twice the expiry) older than the newest time listed. Every change requested before it has expired by then, unless two machines' clocks differ by more than an hour. Without that, repositories that give each build its own release would grow the titles without end.
• Why this design. The commit title is the only place, per change, that the writer reads back cheaply (the Hub returns it with each file's last commit), and the plan fixes releases.json's shape. Finding each release's last writer commit in the history instead would read the repository's whole commit history on every change.

The remaining one-hour window, left open. A newer change on another machine that leaves releases.json unchanged (for example, promoting the build that is already current) makes no commit, so it records no request time. An older change to the same release, queued on another machine, can then still land within one hour of its own hand-over; after that it is refused with REQUEST_EXPIRED. Closing this needs a change to the releases.json schema, which is your decision: the schema and the one-hour limit are unchanged here.

Re-run after the last commit, every test file this pull request adds or changes: 208 passed. ruff check, ruff format --check and tools/check-changelog.sh pass.

Fix round 3 (2026-10-20)

Restack onto #23's round-3 fixes. Every commit has a new SHA and the same title. Only 9c9c8da (now 5088f71) differs as a patch: Git merged #23's new README sentence about shutdown into it without a conflict. The sections above use the round-2 SHAs: aec7519 is now d2bf70a, 9c9c8da is 5088f71, 6fb10eb is a534280, b1a9fff is 69e49f5, 538a378 is 701c891, ebd4893 is c32cac9, b8bc990 is e3b50b4, 865fa7a is 3e3a2a6, 568fce8 is 43dbdfb, 42e5686 is 901269b, 65f7242 is 7c5f4d6, 0b8b3a7 is c97c065, 69a3f48 is f2bffe4, e5eb811 is 95f02d2, 7998111 is 6980d95, 7788f61 is 3b0182e, 943913f is 149f0d4 and 35481d4 is 5583e82. As before, the commits from 5088f71 to c32cac9 fail #23's test that a bundle waits for the spacing after a run-file commit; e3b50b4 and every later commit pass it. #23's new round-3 test passes at every commit of this pull request.

Commits (red → green)	Finding	Red (tests first)	Green
357c8d5 → 22fce01 Order only changes that set current; withdrawals are never overtaken	A withdrawal counted as a newer change to current, though it can never change current. A make-current publication requested before an unrelated withdrawal was listed but silently not made current, and promotions and withdrawals were refused with no conflict.	tests/huggingface/test_release_writer.py: 4 failed, 50 passed: a later withdrawal of another build kept the publication from becoming current; a promotion was refused for a later withdrawal of another build; a withdrawal was refused for a later promotion of another build; a queued promotion was refused for a later withdrawal of another build (on one machine, through the queue)	54 passed
57bd90d → cd73066 Name the release's current build in every release receipt	A publication that asked to become current but did not got status: published and error_code: None; only the prose message said so.	test_release_writer.py: 2 failed, 54 passed; tests/interface/test_release_client.py: 2 failed, 18 passed. Each: the receipt does not say which build is current	56 and 20 passed; 224 across this pull request's changed test files and #23's test_delivery_workers.py
dccca0b → d1fa450 Measure the title window from the Hub's date; ignore future times	One machine more than two hours fast dropped every other release's time from the commit titles (turning off the order check across machines for them), and its own time refused later changes to its release on other machines until real time caught up.	test_release_writer.py: 2 failed, 57 passed: a fast clock on another release's change turned off the order check for this release; a fast clock blocked a later promotion of its release on another machine. A guard (a clock one minute fast still orders changes) passes before and after.	59 passed; 244 across this pull request's changed test files, test_delivery_workers.py and test_hub.py

Design: only changes that set current are ordered. is_ordered becomes sets_current: true for a promotion and for a publication with make_current=True, false for a withdrawal. A withdrawal records no request time in the commit title, does not count among this machine's landed changes, and is never refused with NEWER_CHANGE_LANDED. It cannot conflict with a change to current: withdraw_build refuses the current build and make_current refuses a withdrawn one, so whichever of the two lands second for the same build is refused by those rules, and changes to different builds give the same record in either order. The title suffix keeps its form. This replaces the round-2 "Refusals" bullet above.

Design: the receipt's current field. ReleaseReceipt gains current: str | None:

• on a published receipt, the release's current build in releases.json once the change was applied (or, when nothing needed writing, as the writer found it), and None when the release has no current build;
• None on pending, refused and unavailable receipts.

So H2c can check receipt["status"] == "published" and receipt["current"] == request.build to know its build became current, without reading message. The writer fills it on its three published paths (a commit, a change that needed no commit, a build already published); the module's hand-over receipts and unavailable_release_receipt carry None. The publish_release docstring now states this and the one-hour expiry. The protocol fixtures record only request messages and the answer's top-level receipt key, so no fixture describes receipt fields, and none changed. The behaviour itself is unchanged and still awaits your acceptance (Decision 19): an overtaken make-current publication adds its build and leaves current with the newer request's build; its receipt now says so in current.

Design: a bound on what one fast clock can do. Every change is requested before its commit, and the Hub dates every commit with its own clock. The writer now reads that date from the same get_paths_info answer as the title (DatasetRepository.last_commit_title_and_date; no extra request).

• A time in the title more than CLOCK_TOLERANCE_SECONDS (5 minutes) after the Hub's date for that commit can only come from a fast clock. It is ignored and not carried into the next title; that machine's change is then ordered only by its own queue and the expiry.
• The two-hour window that drops old times now ends at the Hub's date for the last commit to releases.json, not at the newest time listed, so no machine's clock decides which other releases' times are dropped. A change requested before a dropped time is more than two hours older than a commit that has already landed, so it has expired, as before.
• When the Hub reports no date, both fall back to the round-2 behaviour. The in-memory Hub dates commits only when a test sets commit_clock.
• The releases.json schema and the one-hour limit are unchanged. Clocks that differ by less than the tolerance can still land changes requested closer together than that difference in the wrong order, as before. This replaces the round-2 "Title size" bullet above.

Re-run after the last commit, every test file this pull request adds or changes, plus every test file that uses the in-memory Hub (its support file changed): 333 passed. tools/check-artifacts.sh exit 0; ruff check, ruff format --check and tools/check-changelog.sh pass.

🤖 Generated with Claude Code