diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b4a91d5..f99431c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -260,6 +260,9 @@ jobs: - name: Generate static capability pages + sitemap.xml (registry#131) run: python3 scripts/ci/generate_catalog_pages.py catalog/catalog.json https://registry.traverse-framework.com catalog + - name: Mirror artifact WASM binaries for CORS-enabled fetch (registry#304) + run: python3 scripts/ci/mirror_artifacts.py catalog + - name: Upload Pages artifact uses: actions/upload-pages-artifact@v5 with: diff --git a/docs/decision-log.md b/docs/decision-log.md index 30611ff..a714a8d 100644 --- a/docs/decision-log.md +++ b/docs/decision-log.md @@ -392,3 +392,11 @@ Also updated `CONTRIBUTING.md`'s existing entry-62 guidance to state the new saf - **Execution boundary**: this entry only. No code, schema, or CI change -- the fields, their emptiness, and the lack of enforcement are all already correct as-is. **Execution boundary**: the spec, the CI check + its tests, the CI workflow tooling change, the `CONTRIBUTING.md` section, this entry, and the two tracking issues (#301, #302) are the full scope of this entry. Separately in this same session but tracked under their own PRs: the corruption fix (`#299`) and the two-crate backfill (`#300`) the audit above surfaced. Not attempted: backfilling any of the 15 or 18 tracked capabilities (registry#301/#302's own call, paced as ordinary backlog / a cross-team ask respectively). + +66. **Published artifact WASM binaries mirrored onto the CORS-enabled GitHub Pages catalog site; contract.json's GitHub Release URL stays the sole authoritative pointer (2026-08-22, closes registry#304)**: external report (registry#304) confirmed by direct `curl`: `catalog.json` on `registry.traverse-framework.com` serves with `access-control-allow-origin: *`, but every capability's `artifact.url` (a GitHub Release asset) 302-redirects to a signed Azure Blob URL that returns no CORS header at all on the final 200 -- a browser-side consumer can discover a capability via the (CORS-enabled) catalog but its `fetch()` of the artifact bytes the catalog points at fails outright. + +- **Feasibility checked before scoping, not assumed**: `gh api repos/traverse-framework/registry/releases --paginate` summed across all `artifacts/*` releases -- 79 unique artifacts (dedup by URL; several capability versions share a digest per spec 007's documented edge case), 1.6MB total, ~17KB average. Well inside GitHub Pages' size/bandwidth limits; the storage-cost concern that would have justified pausing for owner sign-off doesn't apply. +- **Chosen approach**: new `scripts/ci/mirror_artifacts.py`, run as a `build-catalog` CI step (push-to-main only, same job that already builds `catalog.json`) after static-page generation and before the Pages upload. Walks the whole `capabilities/**/contract.json` tree (deprecated versions included -- a yanked version's artifact must stay fetchable too, same non-retroactive-filtering stance `gather_catalog_data.py` already takes), downloads each unique `artifact.url`, re-verifies the bytes against `artifact.digest` (`sha256:` mismatch fails the CI run rather than silently serving wrong bytes -- this repo has never had *any* runtime digest verification of artifact bytes, format-only checks in `capability_validation.py`; this is the first), and writes to `catalog/artifacts/-/` -- exactly the path suffix `artifact.url` already carries after `.../releases/download/`, so the CORS-enabled mirror URL is always a fixed prefix swap (`https://github.com/.../releases/download/` -> `https://registry.traverse-framework.com/`), never a new piece of state. `generate_catalog_pages.py` renders this derived URL as a second "Artifact (CORS mirror)" link on every capability's detail page, computed with the identical regex `mirror_artifacts.py` uses (kept in explicit lockstep via a code comment on both sides, no shared module -- a one-line transform duplicated twice with a cross-reference beats a new abstraction for two call sites). +- **Why not touch `contract.json` or spec 007**: the mirror is a convenience read-path over already-public bytes, never itself referenced by a contract -- it carries none of spec 007's immutability obligations and is regenerated fresh on every `build-catalog` run, exactly like `catalog.json` itself. `artifact.digest`/`artifact.url` remain the sole authoritative record; no spec amendment needed, same `001-registry-foundation` governing-spec citation the rest of the catalog pipeline already uses (catalog build files have never been governed by a dedicated spec, same gap `capability-src/` had before decision 38). +- **Not attempted**: exposing the mirror URL as a new `contract.json` or `catalog.json` schema field (the fixed-prefix derivation makes that redundant); mirroring anything for the 18 Callweave capabilities differently from the other 98 (they publish through the identical `artifacts/*` release convention, so the same script covers them with no special-casing). +- **Verified, not assumed**: ran `mirror_artifacts.py` against this repo's real, live release assets (not fixtures) -- 79/79 mirrored, digests matched, 0 skipped. Ran `generate_catalog_pages.py` against a representative fixture and confirmed the rendered page's mirror link resolves to the exact path the mirror script writes to. New unit tests: `scripts/ci/tests/test_mirror_artifacts.py` (URL-recognition, digest-match write path, digest-mismatch fail-closed path, all mocked at the network boundary) and `scripts/ci/tests/test_generate_catalog_pages.py` (first test file for that script -- scoped to the new `artifact_mirror_url` helper only, not full-module coverage). Whole-tree `capability_validation.py` and the full `scripts/ci/tests` suite (91 tests) both pass clean on this branch. diff --git a/scripts/ci/generate_catalog_pages.py b/scripts/ci/generate_catalog_pages.py index c32945f..8909689 100644 --- a/scripts/ci/generate_catalog_pages.py +++ b/scripts/ci/generate_catalog_pages.py @@ -48,6 +48,7 @@ import html import json +import re import sys from pathlib import Path from typing import Optional @@ -58,6 +59,21 @@ def esc(value) -> str: return html.escape(str(value), quote=True) +# Same spec 007-artifact-hosting tag-scheme prefix mirror_artifacts.py +# matches against -- kept in lockstep with that script's ARTIFACT_URL_RE. +# mirror_artifacts.py copies every artifact to this exact relative path +# under the catalog output dir, so the CORS-enabled mirror URL is always +# this fixed prefix swap, never new state to keep in sync (registry#304). +ARTIFACT_URL_PREFIX_RE = re.compile( + r"^https://github\.com/traverse-framework/registry/releases/download/(artifacts/[^/]+/[^/]+)$" +) + + +def artifact_mirror_url(base_url: str, artifact_url: str) -> Optional[str]: + match = ARTIFACT_URL_PREFIX_RE.match(artifact_url) + return f"{base_url}/{match.group(1)}" if match else None + + def field_row(label: str, value) -> str: if not value: return "" @@ -384,6 +400,9 @@ def render_capability_page( field_rows += f'
Artifact digest{esc(artifact["digest"])}
' if artifact.get("url"): field_rows += f'
Artifact{esc(artifact["url"])}
' + mirror_url = artifact_mirror_url(base_url, artifact["url"]) + if mirror_url: + field_rows += f'
Artifact (CORS mirror){esc(mirror_url)}
' use_cases = contract.get("use_cases") or [] use_cases_html = ( diff --git a/scripts/ci/mirror_artifacts.py b/scripts/ci/mirror_artifacts.py new file mode 100644 index 0000000..c749e0c --- /dev/null +++ b/scripts/ci/mirror_artifacts.py @@ -0,0 +1,112 @@ +#!/usr/bin/env python3 +"""Mirror published capability WASM artifacts into the discovery catalog's +GitHub Pages output so they're fetchable with CORS (registry#304). + +GitHub Release asset downloads (this repo's own artifacts/- +releases, spec 007-artifact-hosting) redirect to a signed Azure Blob URL +with no access-control-allow-origin header -- any browser-side consumer +that discovers a capability via catalog.json (which IS CORS-enabled, same +Pages site) cannot then fetch the artifact bytes catalog.json points at. +Confirmed by direct curl against both endpoints; see registry#304. + +This script re-hosts a read-only copy of each artifact directly under the +same Pages site catalog-builder already writes to, at the exact path +suffix its own artifact.url already uses after ".../releases/download/" +(artifacts/-/) -- so the CORS-enabled mirror URL +is always a fixed prefix swap of the authoritative artifact.url, not a new +piece of state to keep in sync (generate_catalog_pages.py derives it the +same way when rendering the link). Mirrored bytes are re-verified against +artifact.digest before being written; a mismatch fails CI rather than +silently serving corrupted bytes. + +contract.json's artifact.digest/url remain the sole authoritative record +per spec 007 -- this mirror is a convenience read-path, never referenced by +a contract, so it carries none of spec 007's immutability obligations. It +is regenerated fresh on every catalog build, like catalog.json itself, and +walks the whole capabilities/ tree (deprecated versions included, same as +gather_catalog_data.py) since a yanked version's artifact must stay +fetchable too. + +Usage: mirror_artifacts.py +""" + +import hashlib +import json +import re +import sys +import urllib.request +from pathlib import Path +from typing import Optional + +ROOT = Path(__file__).resolve().parents[2] + +# Kept in exact lockstep with capability_validation.py's ARTIFACT_RELEASE_URL_RE +# -- both encode the same spec 007 tag scheme. +ARTIFACT_URL_PREFIX = "https://github.com/traverse-framework/registry/releases/download/" +ARTIFACT_URL_RE = re.compile(r"^" + re.escape(ARTIFACT_URL_PREFIX) + r"(artifacts/[^/]+/[^/]+)$") + + +def mirror_relpath_for_url(url: str) -> Optional[str]: + """The path under the catalog output dir a mirrored artifact lives at, + or None if `url` isn't a recognized this-repo release-asset URL.""" + match = ARTIFACT_URL_RE.match(url) + return match.group(1) if match else None + + +def fetch(url: str) -> bytes: + with urllib.request.urlopen(url, timeout=60) as response: # noqa: S310 (fixed, validated host) + return response.read() + + +def main(argv) -> int: + if len(argv) != 2: + print("Usage: mirror_artifacts.py ", file=sys.stderr) + return 2 + out_dir = Path(argv[1]) + + seen_urls = set() + mirrored = 0 + skipped = 0 + for contract_path in sorted(ROOT.glob("capabilities/*/*/*/contract.json")): + contract = json.loads(contract_path.read_text()) + artifact = contract.get("artifact") or {} + url = artifact.get("url") + digest = artifact.get("digest") + if not url or not digest: + continue + if url in seen_urls: + continue + seen_urls.add(url) + + relpath = mirror_relpath_for_url(url) + if relpath is None: + print(f"SKIP (unrecognized artifact URL host/shape): {url} ({contract_path})", file=sys.stderr) + skipped += 1 + continue + + dest = out_dir / relpath + if dest.exists(): + continue + + body = fetch(url) + actual_digest = f"sha256:{hashlib.sha256(body).hexdigest()}" + if actual_digest != digest: + print( + f"FATAL: digest mismatch mirroring {url}\n" + f" contract digest: {digest}\n" + f" downloaded digest: {actual_digest}\n" + f" ({contract_path})", + file=sys.stderr, + ) + return 1 + + dest.parent.mkdir(parents=True, exist_ok=True) + dest.write_bytes(body) + mirrored += 1 + + print(f"Mirrored {mirrored} artifact(s) into {out_dir}/artifacts/ ({skipped} skipped) (registry#304)") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv)) diff --git a/scripts/ci/tests/test_generate_catalog_pages.py b/scripts/ci/tests/test_generate_catalog_pages.py new file mode 100644 index 0000000..b339002 --- /dev/null +++ b/scripts/ci/tests/test_generate_catalog_pages.py @@ -0,0 +1,45 @@ +#!/usr/bin/env python3 +"""Unit tests for the artifact CORS-mirror link helper (registry#304).""" + +import importlib.util +import os +import unittest +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parents[3] +MODULE_PATH = REPO_ROOT / "scripts" / "ci" / "generate_catalog_pages.py" +os.chdir(REPO_ROOT) + + +def load_module(): + spec = importlib.util.spec_from_file_location("generate_catalog_pages", MODULE_PATH) + module = importlib.util.module_from_spec(spec) + assert spec.loader is not None + spec.loader.exec_module(module) + return module + + +class ArtifactMirrorUrlTests(unittest.TestCase): + def setUp(self): + self.mod = load_module() + + def test_recognized_release_url_gets_mirror(self): + url = self.mod.artifact_mirror_url( + "https://registry.traverse-framework.com", + "https://github.com/traverse-framework/registry/releases/download/artifacts/core.foo-1.0.0/core-foo.wasm", + ) + self.assertEqual( + url, + "https://registry.traverse-framework.com/artifacts/core.foo-1.0.0/core-foo.wasm", + ) + + def test_unrecognized_url_returns_none(self): + url = self.mod.artifact_mirror_url( + "https://registry.traverse-framework.com", + "https://example.com/some/other/path.wasm", + ) + self.assertIsNone(url) + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/ci/tests/test_mirror_artifacts.py b/scripts/ci/tests/test_mirror_artifacts.py new file mode 100644 index 0000000..1046b37 --- /dev/null +++ b/scripts/ci/tests/test_mirror_artifacts.py @@ -0,0 +1,93 @@ +#!/usr/bin/env python3 +"""Unit tests for the artifact-mirroring CI step (registry#304).""" + +import importlib.util +import os +import sys +import unittest +from pathlib import Path +from unittest import mock + +REPO_ROOT = Path(__file__).resolve().parents[3] +MODULE_PATH = REPO_ROOT / "scripts" / "ci" / "mirror_artifacts.py" +os.chdir(REPO_ROOT) + + +def load_module(): + spec = importlib.util.spec_from_file_location("mirror_artifacts", MODULE_PATH) + module = importlib.util.module_from_spec(spec) + assert spec.loader is not None + spec.loader.exec_module(module) + return module + + +class MirrorRelpathForUrlTests(unittest.TestCase): + def setUp(self): + self.mod = load_module() + + def test_recognized_release_url(self): + url = "https://github.com/traverse-framework/registry/releases/download/artifacts/core.foo-1.0.0/core-foo.wasm" + self.assertEqual( + self.mod.mirror_relpath_for_url(url), + "artifacts/core.foo-1.0.0/core-foo.wasm", + ) + + def test_wrong_host_rejected(self): + url = "https://example.com/releases/download/artifacts/core.foo-1.0.0/core-foo.wasm" + self.assertIsNone(self.mod.mirror_relpath_for_url(url)) + + def test_missing_asset_segment_rejected(self): + url = "https://github.com/traverse-framework/registry/releases/download/artifacts/core.foo-1.0.0" + self.assertIsNone(self.mod.mirror_relpath_for_url(url)) + + +class MainDigestVerificationTests(unittest.TestCase): + def setUp(self): + self.mod = load_module() + + def _contract(self, tmp_path, digest, url): + contract_dir = tmp_path / "capabilities" / "core" / "core.foo" / "1.0.0" + contract_dir.mkdir(parents=True) + contract_path = contract_dir / "contract.json" + contract_path.write_text('{"artifact": {"digest": "%s", "url": "%s"}}' % (digest, url)) + return contract_path + + def test_matching_digest_writes_mirror(self): + import tempfile + + with tempfile.TemporaryDirectory() as tmp: + tmp_path = Path(tmp) + body = b"fake wasm bytes" + import hashlib + + digest = f"sha256:{hashlib.sha256(body).hexdigest()}" + url = "https://github.com/traverse-framework/registry/releases/download/artifacts/core.foo-1.0.0/core-foo.wasm" + self._contract(tmp_path, digest, url) + + out_dir = tmp_path / "catalog" + with mock.patch.object(self.mod, "ROOT", tmp_path), mock.patch.object(self.mod, "fetch", return_value=body): + rc = self.mod.main(["mirror_artifacts.py", str(out_dir)]) + + self.assertEqual(rc, 0) + mirrored = out_dir / "artifacts" / "core.foo-1.0.0" / "core-foo.wasm" + self.assertTrue(mirrored.exists()) + self.assertEqual(mirrored.read_bytes(), body) + + def test_digest_mismatch_fails_closed(self): + import tempfile + + with tempfile.TemporaryDirectory() as tmp: + tmp_path = Path(tmp) + url = "https://github.com/traverse-framework/registry/releases/download/artifacts/core.foo-1.0.0/core-foo.wasm" + self._contract(tmp_path, "sha256:" + "0" * 64, url) + + out_dir = tmp_path / "catalog" + with mock.patch.object(self.mod, "ROOT", tmp_path), mock.patch.object(self.mod, "fetch", return_value=b"different bytes"): + rc = self.mod.main(["mirror_artifacts.py", str(out_dir)]) + + self.assertEqual(rc, 1) + self.assertFalse((out_dir / "artifacts" / "core.foo-1.0.0" / "core-foo.wasm").exists()) + + +if __name__ == "__main__": + unittest.main()