Skip to content

Certify Microcosm builds published side by side in their own release folders #682

Open

Description

@anth-volk

Problem

Microcosm is moving its dataset repositories to a new publication layout:

  • each build is published once, in its own folder, releases/<release>/<build>/, and that folder never changes afterwards;
  • one releases.json at the repository root records which build of each release is current;
  • no new Hugging Face tags are created;
  • each build's release_manifest.json names its release and build as two strings (the build is the attempt ID Microcosm already assigns, for example uk-frs-calibration-attempt-20261019T083323Z-6cba5d71), and no longer carries cut_tag.

The format of releases.json and the publication checks are in microcosm-emitter#24 (microcosm_emitter_interface.releases). Microcosm switches to the layout in a later change, after a one-time migration of the existing UK builds.

Certification and the import-time compatibility check currently name a data build with release_build_id, which prefers build.cut_tag (#615). They also assume build is an object. A manifest in the new layout:

  • would fail to parse, because its build is a string;
  • has no tag to pin, so a certificate has to pin the build by its folder path instead;
  • keeps its files inside the build folder, while today's dataset artifact paths are relative to the repository root.

Proposed change

  • Parse a manifest whose release and build are strings, and name its build by build, so two builds of one release never match.
  • Certify such a build from its own folder: the manifest URI must be releases/<release>/<build>/release_manifest.json, and the build's files are pinned at releases/<release>/<build>/<file> at the revision the manifest was certified at.
  • At import, refuse a remote manifest that names a different release or build than the folder it was read from.
  • Keep manifests published before the layout (with cut_tag) behaving exactly as they do now.

Relates to #553 and #615.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions