Skip to content

Document the artifact submission contract - #6

Merged
anth-volk merged 10 commits into
mainfrom
claude/submission-contract
Oct 27, 2026
Merged

anth-volk merged 10 commits into
mainfrom
claude/submission-contract

Conversation

@anth-volk

@anth-volk anth-volk commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #5

Summary

Adds docs/artifact-submissions.md, the contract for builds handing files to the collector. The collector's intake endpoints and worker framework (#12, stacked on this one) and the emitter's submission module (emitter#27) implement it.

  • Two kinds. calibration-diagnostics (roles calibration_diagnostics, plus optional build_manifest, release_manifest and demographics: the files the calibration-tree publisher reads for a build today) and orrery-inputs (schema, execution_overlay, any number of upstream, and native evidence files that are never published).
  • The emitter's roles. A table maps each bundle role emitter#27 sends (orrery_schema, execution_overlay, orrery_upstream, orrery_evidence and the four calibration roles) to the contract role it puts in files[].role. A second table maps emitter#22's earlier Orrery publication roles (PublicationFileRole: graph, schema, index, declaration, manifest, binding, summaries, upstream) for O7's re-submission of queued uploads: five of them become evidence, and graph has no contract role, because the collector projects its own Orrery artifact. That earlier path has no overlay file, which O7 has to obtain.
  • Provenance: the run ID, the build (the attempt ID), the country, the release (or null for a staged build), and the Hugging Face repo, commit and folder holding the same bytes. Each field has a character set and length (build_id, release and each folder segment use the emitter interface's safe_identifier rule: 128 characters, no colon). Null fields are hashed as JSON null and are never left out. The commit is the one that added the folder.
  • Identifiers are derived by the collector. The submission ID is a SHA-256 over the kind, provenance, licensed_data and the sorted inventory, so every request in the flow can be retried. The job ID also hashes a processing round, so a submission can be processed again. The digest of the files is files_sha256, which is a different value from the emitter receipt's inventory_sha256. files[].size is the emitter's bytes.
  • Bytes never pass through the API. The API accepts bodies of at most 1 MiB and runs with 512 MiB of memory and a 30 s timeout, so files go to signed upload URLs for a private bucket. Each upload instruction lists every header its signature covers, Host included. Completion checks each object's size and recorded digest from storage metadata; the worker job recomputes every SHA-256.
  • A collector without intake is recognisable. The token response states its scope. With intake switched off, the exchange for artifacts:submit and every submission path answer 404 with the code intake_not_enabled. A 404 means "unknown submission, create it again" only with the code submission_not_found. While intake is not available, the client keeps its files queued.
  • Every answer is specified: the create, completion and status bodies (including each upload instruction's fields, headers and URL expiry), each collector answer with its code and what the client does, 429, the maintenance-mode 503 the collector returns on every path, other 5xx answers, and the storage service's answers to an upload (an expired URL, a signature mismatch, 412).
  • Statuses: awaiting_upload, queued, processing, then one of succeeded, rejected or failed. A stored object with another size or SHA-256 rejects the submission at completion (stored_object_mismatch), because stored objects are never replaced.
  • Attempts, dispatches and recovery: 3 attempts and 3 dispatches per processing round. A recovery run every 10 minutes fails a job whose lease expired after its third attempt, dispatches again one whose attempt or execution ended early, and fails one that would need a fourth dispatch. An operator can reprocess a final submission in a new round.
  • Disclosure (decision D10, 2026-10-20): intake storage is private for every submission. Orrery outputs of licensed builds stay private until output clearance (D5). Calibration data derived from licensed data may be public. Native evidence is never published. licensed_data is required with no default.
  • Credential scope: the exchange issues an artifacts:submit token only when asked; existing clients still get telemetry:write tokens, and neither scope works on the other's endpoints.

Review fixes (2026-10-20)

Docs only, so there is no red/green step for any of these commits.

Commit Findings
397d555 Name the roles the emitter sends and the provenance syntax roles named as emitter#27 sends them, with both mapping tables; provenance character sets, lengths, null handling and the adding commit; inventory_sha256 renamed files_sha256; size documented as the emitter's bytes
cc798fa Specify every answer, and how a client recognises a collector without intake the token response's scope, intake_not_enabled and submission_not_found, the client rule to keep files queued; response bodies, upload instruction fields and headers (with Host), 429, 5xx and the maintenance-mode 503, storage answers; size and SHA-256 mismatches made final rejections
8e52d02 Reprocess submissions and recover jobs whose attempt ended without a result processing rounds in the job ID, numbered dispatches, recovery and reprocessing
673df3d Keep only licensed Orrery outputs private, per decision D10 the disclosure rules and the changelog fragment

Testing

tools/bump-version.py --check passes: with this fragment, the next release is 0.4.0 (main is at 0.3.0). CI runs the full checks.

Merging

Don't merge before the collector deployment switch-over is verified. A merge to main creates a release, and after the switch-over it also deploys.

🤖 Generated with Claude Code

anth-volk and others added 2 commits October 20, 2026 00:17
Builds will hand calibration diagnostics and Orrery inputs to the collector,
which stores them privately and processes them in worker jobs. This records
the contract both sides implement: the two kinds and their file roles, the
inventory and provenance fields, identifiers derived by the collector, the
statuses and receipt, and the limits.

File bytes go to signed upload URLs because the API accepts at most 1 MiB per
request body and runs with 512 MiB of memory and a 30 s timeout. For the same
reason, completion checks only sizes and recorded digests, and the worker
recomputes each file's SHA-256 from the stored bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Implementing the contract settled four details, recorded here: upload URLs
also sign x-goog-if-generation-match: 0, so an object is created once and a
repeated upload gets 412; a job waiting for another attempt shows as queued;
the collector's rejection codes are content_mismatch, object_missing,
worker_error and attempts_exhausted; and the default handler's result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in the Orrery projection (#9) and the 0.3.0 release commit, so the
branch is up to date with main as the ruleset requires.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The emitter builds submissions from the roles a staged bundle names. The
calibration roles have the same names in both places; the Orrery inputs
carry an orrery_ prefix in a bundle (except execution_overlay), which the
emitter maps to schema, upstream and evidence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk and others added 4 commits October 20, 2026 16:53
Map each bundle role emitter#27 sends to its contract role, and each of
emitter#22's earlier Orrery publication roles to the contract role a
re-submission would use. Give build_id, release, the Hugging Face repo and
folder a character set and length, say that null provenance fields are
hashed as JSON null, and that the commit is the one that added the folder.
Rename inventory_sha256 to files_sha256, since the emitter's receipt field
of that name is computed differently, and say that size is the emitter's
bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… intake

The token response now states its scope, an exchange or submission path on
a collector with intake switched off answers 404 with the code
intake_not_enabled, and a 404 means an unknown submission only with the
code submission_not_found. A client keeps its files queued while intake is
not available. Document the create and status bodies, each upload
instruction with every signed header (Host included) and its expiry, 429,
the maintenance-mode 503 and other 5xx answers, and the storage service's
answers to an upload. A stored object with another size or SHA-256 now
rejects the submission at completion, since it can never be replaced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…result

The job ID now hashes a processing round, so an operator can process a
final submission again: after K4 or O4 registers a handler, or when the
projection or Orrery changes. Dispatches are numbered in each Cloud Task
name. A recovery run every 10 minutes fails a job whose lease expired after
its third attempt, dispatches again one whose attempt or execution ended
early, and fails one that would need a fourth dispatch, so every job
reaches a final status within its configured attempts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Calibration data derived from licensed UK data may stay publicly readable
(D10). Orrery outputs of licensed builds stay private until output
clearance (D5), and native evidence is never published.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk and others added 2 commits October 22, 2026 21:48
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk
anth-volk merged commit 35680a2 into main Oct 27, 2026
4 checks passed
@anth-volk
anth-volk deleted the claude/submission-contract branch October 27, 2026 11:29
anth-volk pushed a commit that referenced this pull request Oct 27, 2026
Brings in #6 (the submission contract, released as 0.4.0), #4, #11 and #23.
No conflicts: README.md, pyproject.toml and uv.lock merged automatically, and
the consumed fragment changelog.d/artifact-submission-contract.added.md is
removed, leaving only this branch's artifact-intake.added.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Define the artifact submission contract

1 participant