feat(catalog): mirror artifact WASM binaries for CORS-enabled fetch (closes #304)

.github/workflows/ci.yml

@@ -260,6 +260,9 @@ jobs:
       - name: Generate static capability pages + sitemap.xml (registry#131)
         run: python3 scripts/ci/generate_catalog_pages.py catalog/catalog.json https://registry.traverse-framework.com catalog
 
+      - name: Mirror artifact WASM binaries for CORS-enabled fetch (registry#304)
+        run: python3 scripts/ci/mirror_artifacts.py catalog
+
       - name: Upload Pages artifact
         uses: actions/upload-pages-artifact@v5
         with:

docs/decision-log.md

@@ -392,3 +392,11 @@ Also updated `CONTRIBUTING.md`'s existing entry-62 guidance to state the new saf
 - **Execution boundary**: this entry only. No code, schema, or CI change -- the fields, their emptiness, and the lack of enforcement are all already correct as-is.
 
 **Execution boundary**: the spec, the CI check + its tests, the CI workflow tooling change, the `CONTRIBUTING.md` section, this entry, and the two tracking issues (#301, #302) are the full scope of this entry. Separately in this same session but tracked under their own PRs: the corruption fix (`#299`) and the two-crate backfill (`#300`) the audit above surfaced. Not attempted: backfilling any of the 15 or 18 tracked capabilities (registry#301/#302's own call, paced as ordinary backlog / a cross-team ask respectively).
+
+66. **Published artifact WASM binaries mirrored onto the CORS-enabled GitHub Pages catalog site; contract.json's GitHub Release URL stays the sole authoritative pointer (2026-08-22, closes registry#304)**: external report (registry#304) confirmed by direct `curl`: `catalog.json` on `registry.traverse-framework.com` serves with `access-control-allow-origin: *`, but every capability's `artifact.url` (a GitHub Release asset) 302-redirects to a signed Azure Blob URL that returns no CORS header at all on the final 200 -- a browser-side consumer can discover a capability via the (CORS-enabled) catalog but its `fetch()` of the artifact bytes the catalog points at fails outright.
+
+- **Feasibility checked before scoping, not assumed**: `gh api repos/traverse-framework/registry/releases --paginate` summed across all `artifacts/*` releases -- 79 unique artifacts (dedup by URL; several capability versions share a digest per spec 007's documented edge case), 1.6MB total, ~17KB average. Well inside GitHub Pages' size/bandwidth limits; the storage-cost concern that would have justified pausing for owner sign-off doesn't apply.
+- **Chosen approach**: new `scripts/ci/mirror_artifacts.py`, run as a `build-catalog` CI step (push-to-main only, same job that already builds `catalog.json`) after static-page generation and before the Pages upload. Walks the whole `capabilities/**/contract.json` tree (deprecated versions included -- a yanked version's artifact must stay fetchable too, same non-retroactive-filtering stance `gather_catalog_data.py` already takes), downloads each unique `artifact.url`, re-verifies the bytes against `artifact.digest` (`sha256:` mismatch fails the CI run rather than silently serving wrong bytes -- this repo has never had *any* runtime digest verification of artifact bytes, format-only checks in `capability_validation.py`; this is the first), and writes to `catalog/artifacts/<id>-<version>/<asset-name>` -- exactly the path suffix `artifact.url` already carries after `.../releases/download/`, so the CORS-enabled mirror URL is always a fixed prefix swap (`https://github.com/.../releases/download/` -> `https://registry.traverse-framework.com/`), never a new piece of state. `generate_catalog_pages.py` renders this derived URL as a second "Artifact (CORS mirror)" link on every capability's detail page, computed with the identical regex `mirror_artifacts.py` uses (kept in explicit lockstep via a code comment on both sides, no shared module -- a one-line transform duplicated twice with a cross-reference beats a new abstraction for two call sites).
+- **Why not touch `contract.json` or spec 007**: the mirror is a convenience read-path over already-public bytes, never itself referenced by a contract -- it carries none of spec 007's immutability obligations and is regenerated fresh on every `build-catalog` run, exactly like `catalog.json` itself. `artifact.digest`/`artifact.url` remain the sole authoritative record; no spec amendment needed, same `001-registry-foundation` governing-spec citation the rest of the catalog pipeline already uses (catalog build files have never been governed by a dedicated spec, same gap `capability-src/` had before decision 38).
+- **Not attempted**: exposing the mirror URL as a new `contract.json` or `catalog.json` schema field (the fixed-prefix derivation makes that redundant); mirroring anything for the 18 Callweave capabilities differently from the other 98 (they publish through the identical `artifacts/*` release convention, so the same script covers them with no special-casing).
+- **Verified, not assumed**: ran `mirror_artifacts.py` against this repo's real, live release assets (not fixtures) -- 79/79 mirrored, digests matched, 0 skipped. Ran `generate_catalog_pages.py` against a representative fixture and confirmed the rendered page's mirror link resolves to the exact path the mirror script writes to. New unit tests: `scripts/ci/tests/test_mirror_artifacts.py` (URL-recognition, digest-match write path, digest-mismatch fail-closed path, all mocked at the network boundary) and `scripts/ci/tests/test_generate_catalog_pages.py` (first test file for that script -- scoped to the new `artifact_mirror_url` helper only, not full-module coverage). Whole-tree `capability_validation.py` and the full `scripts/ci/tests` suite (91 tests) both pass clean on this branch.

scripts/ci/generate_catalog_pages.py

@@ -48,6 +48,7 @@
 
 import html
 import json
+import re
 import sys
 from pathlib import Path
 from typing import Optional
@@ -58,6 +59,21 @@ def esc(value) -> str:
     return html.escape(str(value), quote=True)
 
 
+# Same spec 007-artifact-hosting tag-scheme prefix mirror_artifacts.py
+# matches against -- kept in lockstep with that script's ARTIFACT_URL_RE.
+# mirror_artifacts.py copies every artifact to this exact relative path
+# under the catalog output dir, so the CORS-enabled mirror URL is always
+# this fixed prefix swap, never new state to keep in sync (registry#304).
+ARTIFACT_URL_PREFIX_RE = re.compile(
+    r"^https://github\.com/traverse-framework/registry/releases/download/(artifacts/[^/]+/[^/]+)$"
+)
+
+
+def artifact_mirror_url(base_url: str, artifact_url: str) -> Optional[str]:
+    match = ARTIFACT_URL_PREFIX_RE.match(artifact_url)
+    return f"{base_url}/{match.group(1)}" if match else None
+
+
 def field_row(label: str, value) -> str:
     if not value:
         return ""
@@ -384,6 +400,9 @@ def render_capability_page(
         field_rows += f'<div class="field-row"><span class="field-label">Artifact digest</span><span class="field-value t-mono">{esc(artifact["digest"])}</span></div>'
     if artifact.get("url"):
         field_rows += f'<div class="field-row"><span class="field-label">Artifact</span><span class="field-value"><a href="{esc(artifact["url"])}">{esc(artifact["url"])}</a></span></div>'
+        mirror_url = artifact_mirror_url(base_url, artifact["url"])
+        if mirror_url:
+            field_rows += f'<div class="field-row"><span class="field-label">Artifact (CORS mirror)</span><span class="field-value"><a href="{esc(mirror_url)}">{esc(mirror_url)}</a></span></div>'
 
     use_cases = contract.get("use_cases") or []
     use_cases_html = (

scripts/ci/mirror_artifacts.py

@@ -0,0 +1,112 @@
+#!/usr/bin/env python3
+"""Mirror published capability WASM artifacts into the discovery catalog's
+GitHub Pages output so they're fetchable with CORS (registry#304).
+
+GitHub Release asset downloads (this repo's own artifacts/<id>-<version>
+releases, spec 007-artifact-hosting) redirect to a signed Azure Blob URL
+with no access-control-allow-origin header -- any browser-side consumer
+that discovers a capability via catalog.json (which IS CORS-enabled, same
+Pages site) cannot then fetch the artifact bytes catalog.json points at.
+Confirmed by direct curl against both endpoints; see registry#304.
+
+This script re-hosts a read-only copy of each artifact directly under the
+same Pages site catalog-builder already writes to, at the exact path
+suffix its own artifact.url already uses after ".../releases/download/"
+(artifacts/<id>-<version>/<asset-name>) -- so the CORS-enabled mirror URL
+is always a fixed prefix swap of the authoritative artifact.url, not a new
+piece of state to keep in sync (generate_catalog_pages.py derives it the
+same way when rendering the link). Mirrored bytes are re-verified against
+artifact.digest before being written; a mismatch fails CI rather than
+silently serving corrupted bytes.
+
+contract.json's artifact.digest/url remain the sole authoritative record
+per spec 007 -- this mirror is a convenience read-path, never referenced by
+a contract, so it carries none of spec 007's immutability obligations. It
+is regenerated fresh on every catalog build, like catalog.json itself, and
+walks the whole capabilities/ tree (deprecated versions included, same as
+gather_catalog_data.py) since a yanked version's artifact must stay
+fetchable too.
+
+Usage: mirror_artifacts.py <catalog_output_dir>
+"""
+
+import hashlib
+import json
+import re
+import sys
+import urllib.request
+from pathlib import Path
+from typing import Optional
+
+ROOT = Path(__file__).resolve().parents[2]
+
+# Kept in exact lockstep with capability_validation.py's ARTIFACT_RELEASE_URL_RE
+# -- both encode the same spec 007 tag scheme.
+ARTIFACT_URL_PREFIX = "https://github.com/traverse-framework/registry/releases/download/"
+ARTIFACT_URL_RE = re.compile(r"^" + re.escape(ARTIFACT_URL_PREFIX) + r"(artifacts/[^/]+/[^/]+)$")
+
+
+def mirror_relpath_for_url(url: str) -> Optional[str]:
+    """The path under the catalog output dir a mirrored artifact lives at,
+    or None if `url` isn't a recognized this-repo release-asset URL."""
+    match = ARTIFACT_URL_RE.match(url)
+    return match.group(1) if match else None
+
+
+def fetch(url: str) -> bytes:
+    with urllib.request.urlopen(url, timeout=60) as response:  # noqa: S310 (fixed, validated host)
+        return response.read()
+
+
+def main(argv) -> int:
+    if len(argv) != 2:
+        print("Usage: mirror_artifacts.py <catalog_output_dir>", file=sys.stderr)
+        return 2
+    out_dir = Path(argv[1])
+
+    seen_urls = set()
+    mirrored = 0
+    skipped = 0
+    for contract_path in sorted(ROOT.glob("capabilities/*/*/*/contract.json")):
+        contract = json.loads(contract_path.read_text())
+        artifact = contract.get("artifact") or {}
+        url = artifact.get("url")
+        digest = artifact.get("digest")
+        if not url or not digest:
+            continue
+        if url in seen_urls:
+            continue
+        seen_urls.add(url)
+
+        relpath = mirror_relpath_for_url(url)
+        if relpath is None:
+            print(f"SKIP (unrecognized artifact URL host/shape): {url} ({contract_path})", file=sys.stderr)
+            skipped += 1
+            continue
+
+        dest = out_dir / relpath
+        if dest.exists():
+            continue
+
+        body = fetch(url)
+        actual_digest = f"sha256:{hashlib.sha256(body).hexdigest()}"
+        if actual_digest != digest:
+            print(
+                f"FATAL: digest mismatch mirroring {url}\n"
+                f"  contract digest:   {digest}\n"
+                f"  downloaded digest: {actual_digest}\n"
+                f"  ({contract_path})",
+                file=sys.stderr,
+            )
+            return 1
+
+        dest.parent.mkdir(parents=True, exist_ok=True)
+        dest.write_bytes(body)
+        mirrored += 1
+
+    print(f"Mirrored {mirrored} artifact(s) into {out_dir}/artifacts/ ({skipped} skipped) (registry#304)")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main(sys.argv))

scripts/ci/tests/test_generate_catalog_pages.py

@@ -0,0 +1,45 @@
+#!/usr/bin/env python3
+"""Unit tests for the artifact CORS-mirror link helper (registry#304)."""
+
+import importlib.util
+import os
+import unittest
+from pathlib import Path
+
+REPO_ROOT = Path(__file__).resolve().parents[3]
+MODULE_PATH = REPO_ROOT / "scripts" / "ci" / "generate_catalog_pages.py"
+os.chdir(REPO_ROOT)
+
+
+def load_module():
+    spec = importlib.util.spec_from_file_location("generate_catalog_pages", MODULE_PATH)
+    module = importlib.util.module_from_spec(spec)
+    assert spec.loader is not None
+    spec.loader.exec_module(module)
+    return module
+
+
+class ArtifactMirrorUrlTests(unittest.TestCase):
+    def setUp(self):
+        self.mod = load_module()
+
+    def test_recognized_release_url_gets_mirror(self):
+        url = self.mod.artifact_mirror_url(
+            "https://registry.traverse-framework.com",
+            "https://github.com/traverse-framework/registry/releases/download/artifacts/core.foo-1.0.0/core-foo.wasm",
+        )
+        self.assertEqual(
+            url,
+            "https://registry.traverse-framework.com/artifacts/core.foo-1.0.0/core-foo.wasm",
+        )
+
+    def test_unrecognized_url_returns_none(self):
+        url = self.mod.artifact_mirror_url(
+            "https://registry.traverse-framework.com",
+            "https://example.com/some/other/path.wasm",
+        )
+        self.assertIsNone(url)
+
+
+if __name__ == "__main__":
+    unittest.main()

scripts/ci/tests/test_mirror_artifacts.py

@@ -0,0 +1,93 @@
+#!/usr/bin/env python3
+"""Unit tests for the artifact-mirroring CI step (registry#304)."""
+
+import importlib.util
+import os
+import sys
+import unittest
+from pathlib import Path
+from unittest import mock
+
+REPO_ROOT = Path(__file__).resolve().parents[3]
+MODULE_PATH = REPO_ROOT / "scripts" / "ci" / "mirror_artifacts.py"
+os.chdir(REPO_ROOT)
+
+
+def load_module():
+    spec = importlib.util.spec_from_file_location("mirror_artifacts", MODULE_PATH)
+    module = importlib.util.module_from_spec(spec)
+    assert spec.loader is not None
+    spec.loader.exec_module(module)
+    return module
+
+
+class MirrorRelpathForUrlTests(unittest.TestCase):
+    def setUp(self):
+        self.mod = load_module()
+
+    def test_recognized_release_url(self):
+        url = "https://github.com/traverse-framework/registry/releases/download/artifacts/core.foo-1.0.0/core-foo.wasm"
+        self.assertEqual(
+            self.mod.mirror_relpath_for_url(url),
+            "artifacts/core.foo-1.0.0/core-foo.wasm",
+        )
+
+    def test_wrong_host_rejected(self):
+        url = "https://example.com/releases/download/artifacts/core.foo-1.0.0/core-foo.wasm"
+        self.assertIsNone(self.mod.mirror_relpath_for_url(url))
+
+    def test_missing_asset_segment_rejected(self):
+        url = "https://github.com/traverse-framework/registry/releases/download/artifacts/core.foo-1.0.0"
+        self.assertIsNone(self.mod.mirror_relpath_for_url(url))
+
+
+class MainDigestVerificationTests(unittest.TestCase):
+    def setUp(self):
+        self.mod = load_module()
+
+    def _contract(self, tmp_path, digest, url):
+        contract_dir = tmp_path / "capabilities" / "core" / "core.foo" / "1.0.0"
+        contract_dir.mkdir(parents=True)
+        contract_path = contract_dir / "contract.json"
+        contract_path.write_text('{"artifact": {"digest": "%s", "url": "%s"}}' % (digest, url))
+        return contract_path
+
+    def test_matching_digest_writes_mirror(self):
+        import tempfile
+
+        with tempfile.TemporaryDirectory() as tmp:
+            tmp_path = Path(tmp)
+            body = b"fake wasm bytes"
+            import hashlib
+
+            digest = f"sha256:{hashlib.sha256(body).hexdigest()}"
+            url = "https://github.com/traverse-framework/registry/releases/download/artifacts/core.foo-1.0.0/core-foo.wasm"
+            self._contract(tmp_path, digest, url)
+
+            out_dir = tmp_path / "catalog"
+            with mock.patch.object(self.mod, "ROOT", tmp_path), mock.patch.object(self.mod, "fetch", return_value=body):
+                rc = self.mod.main(["mirror_artifacts.py", str(out_dir)])
+
+            self.assertEqual(rc, 0)
+            mirrored = out_dir / "artifacts" / "core.foo-1.0.0" / "core-foo.wasm"
+            self.assertTrue(mirrored.exists())
+            self.assertEqual(mirrored.read_bytes(), body)
+
+    def test_digest_mismatch_fails_closed(self):
+        import tempfile
+
+        with tempfile.TemporaryDirectory() as tmp:
+            tmp_path = Path(tmp)
+            url = "https://github.com/traverse-framework/registry/releases/download/artifacts/core.foo-1.0.0/core-foo.wasm"
+            self._contract(tmp_path, "sha256:" + "0" * 64, url)
+
+            out_dir = tmp_path / "catalog"
+            with mock.patch.object(self.mod, "ROOT", tmp_path), mock.patch.object(self.mod, "fetch", return_value=b"different bytes"):
+                rc = self.mod.main(["mirror_artifacts.py", str(out_dir)])
+
+            self.assertEqual(rc, 1)
+            self.assertFalse((out_dir / "artifacts" / "core.foo-1.0.0" / "core-foo.wasm").exists())
+
+
+if __name__ == "__main__":
+    unittest.main()