diff --git a/changelog.d/1291-orrery-reconstruction-marker.added.md b/changelog.d/1291-orrery-reconstruction-marker.added.md new file mode 100644 index 0000000000..64c2f3bdd5 --- /dev/null +++ b/changelog.d/1291-orrery-reconstruction-marker.added.md @@ -0,0 +1 @@ +Add `microcosm.graph.reconstruction`, which rebuilds a declaration-only graph schema from a build's recorded `graph.json` and marks it with the `microcosm.reconstruction` extension (protocol `microcosm.graph.reconstruction.v1`). The marker names the revision that declared the graph (`source_revision`), the revision of the code that performed the reconstruction (`reconstruction_revision`, read by `code_revision` from the checkout that tracks that code, ignoring inherited repository-selecting git variables such as `GIT_DIR`), and every input file by digest. Only the declaration's bytes come from the build: the schema's `compiled`, `fields` and `input_bindings` sections and the Orrery projection come from the reconstructing code. The marker travels in `graph.schema.json` into the Orrery document's metadata. A declaration that the current serializer would not reproduce byte for byte is refused, and no operation is presented as executed. diff --git a/changelog.d/1291-uk-orrery-reconstruction.added.md b/changelog.d/1291-uk-orrery-reconstruction.added.md new file mode 100644 index 0000000000..258227a587 --- /dev/null +++ b/changelog.d/1291-uk-orrery-reconstruction.added.md @@ -0,0 +1 @@ +Add `python -m microcosm.build.uk_runtime.orrery_reconstruction`, which turns the output directory of a UK dense or national build from before #1144 into a declaration-only Orrery publication. It writes a new directory with the normal publication layout (`graph.schema.json`, `graph.orrery.json`, `orrery.upload.json`) and a summary report. The declaration comes from the build's recorded `graph.json` and the revision from its recorded code pin. The schema's compiled sections and the Orrery document are computed by the code running the command, whose revision the marker records beside the build's. Every input is checked against the digests the build recorded. The one exception is the candidate manifest digest that dense builds closed by #901's driver left in `build.json` beside the note that staging receipts follow: it is skipped and reported, and the manifest is checked against `sha256sums.txt` instead. Builds that already record Orrery evidence are refused, as is an output directory inside the build directory, which is only read. Nothing is uploaded. diff --git a/docs/agent-guide.md b/docs/agent-guide.md index b14084800d..a033b16845 100644 --- a/docs/agent-guide.md +++ b/docs/agent-guide.md @@ -93,7 +93,9 @@ engine-free runner installs the supported public Orrery range and locked Node dependency set under `tools/orrery-contract/`; the pytest test generates a document through Microcosm's public Python API and requires Orrery's public parser to accept it. The recorded-evidence assertion in `test_graph_evidence.py` -uses the same pinned parser for groups, statuses, activities and artifacts. +uses the same pinned parser for groups, statuses, activities and artifacts, and +the one in `test_graph_reconstruction.py` uses it for the reconstruction marker +of a declaration-only document. These CI tests perform no browser rendering. Workspace tests use an in-memory telemetry emitter by default. The shared diff --git a/docs/orrery-adapter.md b/docs/orrery-adapter.md index 469ab3e36d..e9422e6ae8 100644 --- a/docs/orrery-adapter.md +++ b/docs/orrery-adapter.md @@ -314,6 +314,190 @@ mismatches, missing required bytes and corrupt files refuse export. Unavailable gate-blocked outputs, omitted operations and unconfigured summary providers are explicit. Earlier phases not supplied to the exporter remain unknown. +## Reconstructed declarations + +A build from before #1144 recorded its graph declaration, but not the run +binding that execution export validates, so its execution cannot be shown. +`microcosm.graph.reconstruction` rebuilds that build's schema from the recorded +declaration alone and marks it as reconstructed: + +```python +from microcosm.graph.reconstruction import ( + code_revision, + reconstructed_schema, + reconstruction_marker, +) + +marker = reconstruction_marker( + git_commit=recorded_commit, + git_dirty=False, + revision_origin="recorded", + reconstruction_revision=code_revision(__file__), + inputs=inputs, # one {name, role, sha256, bytes, verification} per file read +) +schema = reconstructed_schema(declaration_bytes, marker) +``` + +Only the declaration comes from the build. The schema's `graph` section and +`graph_sha256` reproduce the recorded file byte for byte. Its `compiled`, +`fields` and `input_bindings` sections, and the Orrery document projected from +it, are computed by the code running the reconstruction. A reconstruction made +at another revision can therefore differ in those parts, which is why the +marker records that revision separately from the build's. + +The marker is the producer extension `microcosm.reconstruction`: + +```json +{ + "protocol": "microcosm.graph.reconstruction.v1", + "reconstructed": true, + "basis": "declaration_only", + "source_revision": { + "git_commit": "", + "git_dirty": false, + "origin": "recorded" + }, + "reconstruction_revision": { + "git_commit": "", + "git_dirty": false + }, + "inputs": [ + {"name": "graph.json", "role": "declaration", "sha256": "", + "bytes": 4096, "verification": "no_recorded_digest"}, + {"name": "rowwise_candidate_manifest.json", "role": "code_pin", + "sha256": "", "bytes": 2048, "verification": "recorded_digest_matched"} + ], + "scope": "Rebuilt after the build from its recorded graph declaration, the only part taken from the build: ..." +} +``` + +- **`source_revision`:** the code that declared the graph. `origin` is + `recorded` when the build wrote the revision itself, and `caller_asserted` + when the build recorded none and the caller supplied one. `git_dirty` is + `null` when the build did not record it. +- **`reconstruction_revision`:** the code that performed the reconstruction, + and so computed every section except the declaration. `code_revision(path)` + reads it from the git checkout that tracks `path` (the reconstructing + module's own file), not from the current directory, and runs git without + the repository-selecting variables of `git rev-parse --local-env-vars` + (`GIT_DIR`, `GIT_WORK_TREE` and the rest), so an inherited one cannot point + it at another checkout. Both fields are `null` + when no checkout tracks that file, as for an installed wheel. `git_dirty` is + `null` alone when git could not check the working tree. +- **`inputs`:** every file the reconstruction read, in name order, by digest + and size. Exactly one input is the declaration, and its digest equals the + schema's `graph_sha256`. `verification` compares the file with the digests + the build itself recorded for it: + + | Value | Meaning | + | --- | --- | + | `recorded_digest_matched` | Every recorded digest matched. | + | `fallback_digest_matched` | The build marked one recorded digest as taken before it rewrote the file. That digest was skipped, and another recorded digest matched. | + | `only_stale_digest_recorded` | Every recorded digest is marked as taken before the rewrite, so nothing the build recorded checks the file. | + | `no_recorded_digest` | The build recorded no digest for the file. | + + In the last two cases the file is bound only by the digest in the marker. +- **Exact bytes:** `reconstructed_schema` refuses a declaration that the + current serializer would not write byte for byte. The schema's `graph` + section is therefore the recorded file, not a reinterpretation of it. A file + that is not a graph declaration at all, such as a JSON array, is also refused + with a `ValueError`. +- **Where it travels:** the marker is saved in `graph.schema.json`, so any + later projection of the saved schema keeps it. Orrery documents show it at + `metadata.microcosm.extensions["microcosm.reconstruction"]`. + `validate_reconstruction(schema)` returns the marker, or `None` for a schema + that was not reconstructed. +- **No execution claims:** a reconstructed document has no activities, + artifacts, statuses or `metadata.execution`. No operation appears as + executed, cached or gated. CI parses one with Orrery's public parser + (`tools/orrery-contract/verify-reconstruction.mjs`). + +### UK builds before #1144 + +UK dense and national builds from before #1144 wrote `graph.json` beside their +outputs, and recorded their code revision in `rowwise_candidate_manifest.json`. +This command turns such a build's output directory into a reconstructed +publication: + +```sh +uv run python -m microcosm.build.uk_runtime.orrery_reconstruction \ + --build path/to/past-build \ + --output path/to/reconstructed-publication +``` + +The declaration is the build's own `graph.json`. It is not rebuilt by running +the UK graph builders at the current revision. The declaration format has not +changed since the UK graph driver, so a recorded file reproduces byte for byte; +one that does not is refused. Everything derived from it (the schema's +compiled sections and the Orrery document) is computed by the code running the +command, and the marker records that code's revision as +`reconstruction_revision`. + +The build directory is only read. The command: + +- **Reads:** `graph.json`, which is required, and also `build.json`, + `rowwise_candidate_manifest.json` and `sha256sums.txt` where present. Links, + devices and inputs over 32 MiB are refused. +- **Checks recorded digests:** a file whose bytes disagree with a digest the + build recorded for it is refused. `build.json` binds the candidate manifest, + and `sha256sums.txt` binds the files it lists. One exception covers the dense + driver merged with #901 (2026-09-29). It recorded the manifest's digest in + `build.json` with the note `staging receipts are appended after + publication`, then appended `staging_delivery` and `staged_dataset` to the + manifest and refreshed only `sha256sums.txt`. `build.json` was reissued over + the final bytes, without the note, only from #1057 (2026-09-30). A + `build.json` digest with that exact note that no longer matches is skipped + and reported, and the manifest must then match `sha256sums.txt` + (`fallback_digest_matched`). A build run with `--no-staging` or + `--no-staged-dataset` wrote no digest list, so its manifest is reported as + `only_stale_digest_recorded`, with a warning. +- **Resolves the revision:** it takes the candidate manifest's `git_commit` + and `git_dirty`, which must agree with the copy under `identity.code`. + `--revision` must match that commit, or supplies one when the build recorded + none; that revision is marked `caller_asserted`. With neither, it refuses. +- **Refuses later builds:** a build that records Orrery evidence + (`graph.schema.json` or `execution.evidence.json`) is exported with + `python -m microcosm.graph.orrery` instead. +- **Refuses missing declarations:** a directory without `graph.json` is refused, + and the refusal names the build records that were found. Staged bundles never + carried `graph.json`, and builds from before the UK graph driver never wrote + it. + +It writes a new directory with the layout of a normal publication, and moves the +directory into place only after every file is written. The output must not +exist, either as given (a symbolic link counts, even one whose target does not +exist) or once resolved. It must not be the build directory or lie inside it +once `..` and links are resolved. The command writes to that resolved path, so +a path that passes through a missing folder inside the build creates nothing +there, and a read-only build can be reconstructed. It prints the resolved path, +because the path as given need not reach it: + +| File | Contents | +| --- | --- | +| `graph.schema.json` | Compiler schema of the recorded declaration, with the reconstruction marker | +| `graph.orrery.json` | Orrery document, titled `UK · reconstructed declaration ()` | +| `orrery.upload.json` | Publication inventory (roles `graph` and `schema`); its publication ID is derived from the files' digests | +| `orrery.reconstruction.json` | Summary report; not in the inventory | + +The summary report, of which the command prints a short version, lists: + +- the inputs and how each was verified; +- the source revision and the reconstruction revision; +- the recorded digests skipped as stale, each with its note and the records the + file was checked against instead; +- the expected files that are missing; +- evidence found but not bound to operations: run manifests, + `evidence-index.json`, `operations.json` and the content store; +- warnings, for a dirty or unrecorded working tree in the build or the + reconstruction, an unknown reconstruction revision, a caller-asserted + revision, a skipped stale digest, or a declaration the build recorded no + digest for. Before #1144, the last one applies to every build. + +Output is deterministic: reconstructing the same build twice from the same +code revision gives identical bytes and the same publication ID. The command +uploads nothing and starts no emitter service, so publishing the directory +remains a separate step. + ## Orrery evidence panels | Surface | Exported evidence | diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/orrery_reconstruction.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/orrery_reconstruction.py new file mode 100644 index 0000000000..9b5dd8b628 --- /dev/null +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/orrery_reconstruction.py @@ -0,0 +1,647 @@ +"""Rebuild a declaration-only Orrery publication for a UK build before #1144. + +UK dense and national builds from before microcosm#1144 wrote their graph +declaration (``graph.json``) and their code pin, but no run binding, so their +recorded execution cannot be validated. This module projects the recorded +declaration alone into the normal publication layout, marked as reconstructed +by :mod:`microcosm.graph.reconstruction`. Only the declaration's bytes come +from the build: the schema's compiled sections and the Orrery document are +computed by the code running this module, whose revision the marker records +beside the build's. It reads only the build directory, executes no kernel and +uploads nothing: the publication directory is submitted later, if at all, +through the usual publication path. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import shutil +import stat +import uuid +from collections.abc import Mapping, Sequence +from dataclasses import dataclass +from pathlib import Path +from typing import Any, NoReturn + +from microcosm.build.staging_dataset import SHA256SUMS_FILENAME, parse_sha256sums +from microcosm.graph.canonical import canonical_json +from microcosm.graph.orrery import orrery_json_from_schema +from microcosm.graph.reconstruction import ( + DECLARATION_ONLY, + DECLARATION_ROLE, + GIT_COMMIT_PATTERN, + code_revision, + reconstructed_schema, + reconstruction_marker, +) + +from .orrery_publication import INVENTORY_NAME +from .rowwise_cli import MANIFEST_FILENAME, json_text + +__all__ = [ + "GRAPH_NAME", + "REPORT_NAME", + "REPORT_PROTOCOL", + "SCHEMA_NAME", + "main", + "reconstruct_uk_publication", + "summary_text", +] + +DECLARATION_NAME = "graph.json" +COMPLETION_NAME = "build.json" +SCHEMA_NAME = "graph.schema.json" +EVIDENCE_INDEX_NAME = "execution.evidence.json" +GRAPH_NAME = "graph.orrery.json" +REPORT_NAME = "orrery.reconstruction.json" +REPORT_PROTOCOL = "microcosm.uk.orrery-reconstruction-report.v1" +PUBLICATION_ID_PREFIX = "reconstructed-" +# The bound the Orrery command applies to a declaration or schema input. +_MAX_INPUT_BYTES = 32 * 1024 * 1024 +# Every build file the reconstruction reads, with its role in the marker. +_INPUT_ROLES = { + DECLARATION_NAME: DECLARATION_ROLE, + COMPLETION_NAME: "completion_record", + MANIFEST_FILENAME: "code_pin", + SHA256SUMS_FILENAME: "digest_list", +} +_ABSENT_REASONS = { + COMPLETION_NAME: ( + "completion record; without it the candidate manifest's recorded " + "digest cannot be checked" + ), + MANIFEST_FILENAME: "candidate manifest, where the build recorded its code revision", + SHA256SUMS_FILENAME: "digest list, written only when the build staged its bundle", +} +# Evidence a pre-#1144 build left that a declaration-only reconstruction finds +# but deliberately does not bind to operations. +_UNUSED_REASONS = { + ".graph-store": ( + "content store; with no run binding there is nothing to verify it against" + ), + "evidence-index.json": ( + "index of materialized artifacts; not linked to operations in a " + "declaration-only reconstruction" + ), + "operations.json": ( + "operation inventory written beside the declaration; not needed to project it" + ), +} +_RUN_MANIFEST_SUFFIX = ".graph.json" +_RUN_MANIFEST_REASON = ( + "run manifest; its receipts are not bound to operations in a " + "declaration-only reconstruction" +) +_BUILD_FIELDS = ("build_kind", "release_role", "release_id", "build_id", "created_at") +# The note the UK drivers wrote beside the candidate manifest's digest in +# build.json. The close step then appended the staging receipts +# (``staging_delivery``, ``staged_dataset``) to the manifest and refreshed only +# sha256sums.txt, so the digest beside the note is of the manifest as first +# written. ``_refresh_completion`` (cb0d39045, merged with #1057 on 2026-09-30) +# reissues build.json over the final bytes and removes the note; dense builds +# closed by the driver merged with #901 (2026-09-29) until then kept it. +STAGING_RECEIPTS_NOTE = "staging receipts are appended after publication" + + +@dataclass(frozen=True) +class _RecordedDigest: + """One digest a build file recorded for another file.""" + + recorder: str + sha256: str + size_bytes: int | None = None + # Set when the recorder says the digest was taken before the file was + # rewritten, so a mismatch is expected rather than evidence of tampering. + stale_note: str | None = None + + +def _refuse(message: str) -> NoReturn: + raise ValueError(f"UK Orrery reconstruction: {message}.") + + +def _require(condition: bool, message: str) -> None: + if not condition: + _refuse(message) + + +def _read_input(directory: Path, name: str) -> bytes: + """Read one regular input file, refusing links, devices and oversized files.""" + + path = directory / name + _require(not path.is_symlink(), f"{name} must be a regular file, not a link") + descriptor = os.open( + path, os.O_RDONLY | os.O_NOFOLLOW | getattr(os, "O_NONBLOCK", 0) + ) + with os.fdopen(descriptor, "rb") as stream: + info = os.fstat(stream.fileno()) + _require(stat.S_ISREG(info.st_mode), f"{name} must be a regular file") + _require(info.st_size <= _MAX_INPUT_BYTES, f"{name} exceeds the input limit") + payload = stream.read(_MAX_INPUT_BYTES + 1) + _require(len(payload) <= _MAX_INPUT_BYTES, f"{name} exceeds the input limit") + return payload + + +def _unique_object(pairs: list[tuple[str, object]]) -> dict[str, object]: + result: dict[str, object] = {} + for key, value in pairs: + _require(key not in result, f"duplicate JSON key {key!r}") + result[key] = value + return result + + +def _reject_constant(constant: str) -> NoReturn: + _refuse(f"non-finite JSON number {constant}") + + +def _json_object(payload: bytes, name: str) -> dict[str, object]: + try: + value = json.loads( + payload.decode("utf-8"), + object_pairs_hook=_unique_object, + parse_constant=_reject_constant, + ) + except (UnicodeDecodeError, json.JSONDecodeError) as error: + raise ValueError( + f"UK Orrery reconstruction: {name} is not valid JSON." + ) from error + _require(type(value) is dict, f"{name} must contain a JSON object") + return value + + +def _recorded_digests( + payloads: Mapping[str, bytes], records: Mapping[str, dict[str, object]] +) -> dict[str, list[_RecordedDigest]]: + """Digests the build itself recorded, by the name of the file they cover.""" + + recorded: dict[str, list[_RecordedDigest]] = {} + if SHA256SUMS_FILENAME in payloads: + listing = payloads[SHA256SUMS_FILENAME].decode("utf-8") + for digest, name in parse_sha256sums(listing): + recorded.setdefault(name, []).append( + _RecordedDigest(SHA256SUMS_FILENAME, digest) + ) + completion = records.get(COMPLETION_NAME, {}) + manifest_record = completion.get("rowwise_candidate_manifest") + if isinstance(manifest_record, dict) and "sha256" in manifest_record: + _require( + manifest_record.get("filename", MANIFEST_FILENAME) == MANIFEST_FILENAME, + f"{COMPLETION_NAME} records an unexpected candidate manifest name", + ) + size = manifest_record.get("size_bytes") + note = manifest_record.get("note") + recorded.setdefault(MANIFEST_FILENAME, []).append( + _RecordedDigest( + COMPLETION_NAME, + str(manifest_record["sha256"]), + size if type(size) is int else None, + note if note == STAGING_RECEIPTS_NOTE else None, + ) + ) + return recorded + + +def _verification( + name: str, payload: bytes, records: Sequence[_RecordedDigest] +) -> tuple[str, list[dict[str, object]]]: + """Check ``payload`` against its recorded digests: ``(state, stale records)``. + + A mismatch is refused, except against a record that says it was taken + before the file was rewritten. Such a record is skipped and reported, and + the file counts as verified only if another recorded digest matches it. + """ + + digest, size = hashlib.sha256(payload).hexdigest(), len(payload) + matched, skipped = [], [] + for record in records: + if record.sha256 == digest and record.size_bytes in (None, size): + matched.append(record.recorder) + elif record.stale_note is not None: + skipped.append(record) + else: + _refuse(f"{name} does not match the digest recorded in {record.recorder}") + stale = [ + { + "name": name, + "recorded_in": record.recorder, + "sha256": record.sha256, + "bytes": record.size_bytes, + "note": record.stale_note, + "checked_against": matched, + } + for record in skipped + ] + if not records: + return "no_recorded_digest", stale + if not skipped: + return "recorded_digest_matched", stale + return ( + "fallback_digest_matched" if matched else "only_stale_digest_recorded" + ), stale + + +def _recorded_code(manifest: Mapping[str, object]) -> tuple[str | None, bool | None]: + """The code pin a candidate manifest records, top level and under identity.""" + + top_level = { + key: manifest[key] for key in ("git_commit", "git_dirty") if key in manifest + } + identity = manifest.get("identity") + code = identity.get("code") if isinstance(identity, dict) else None + nested = ( + {key: code[key] for key in ("git_commit", "git_dirty") if key in code} + if isinstance(code, dict) + else {} + ) + _require( + all(top_level[key] == nested[key] for key in top_level.keys() & nested.keys()), + f"{MANIFEST_FILENAME} records two different code revisions", + ) + pin = {**nested, **top_level} + commit, dirty = pin.get("git_commit"), pin.get("git_dirty") + _require( + commit is None + or (type(commit) is str and GIT_COMMIT_PATTERN.fullmatch(commit) is not None), + f"{MANIFEST_FILENAME} records a malformed code revision", + ) + _require( + type(dirty) in (bool, type(None)), + f"{MANIFEST_FILENAME} records a malformed working-tree flag", + ) + return commit, dirty # type: ignore[return-value] + + +def _stale_digest_warning(stale: Mapping[str, Any]) -> str: + """Name a recorded digest skipped as stale, and what checked the file instead.""" + + skipped = ( + f"{stale['recorded_in']} records the digest {stale['name']} had before the " + "staging receipts were appended, as its note says" + ) + if stale["checked_against"]: + return ( + f"{skipped}, so {stale['name']} was checked against " + f"{', '.join(stale['checked_against'])} instead." + ) + return ( + f"{skipped}, and the build recorded no later digest, so nothing the build " + f"recorded checks {stale['name']}." + ) + + +def _source_revision( + build: Path, recorded: tuple[str | None, bool | None], revision: str | None +) -> tuple[str, bool | None, str]: + """Resolve ``(git_commit, git_dirty, origin)`` from the build and the caller.""" + + if revision is not None: + _require( + GIT_COMMIT_PATTERN.fullmatch(revision) is not None, + "--revision must be a full lowercase git commit", + ) + recorded_commit, recorded_dirty = recorded + if recorded_commit is not None: + _require( + revision in (None, recorded_commit), + f"--revision {revision} does not match the recorded revision " + f"{recorded_commit}", + ) + return recorded_commit, recorded_dirty, "recorded" + _require( + revision is not None, + f"{build} records no code revision; pass --revision with the commit the " + "build ran from", + ) + assert revision is not None + return revision, None, "caller_asserted" + + +def _build_identity(*records: Mapping[str, object]) -> dict[str, str]: + """Identifiers the build recorded, from the first record that names each.""" + + identity = {} + for field in _BUILD_FIELDS: + value = next((record[field] for record in records if field in record), None) + if type(value) is str: + identity[field] = value + return identity + + +def _within(directory: Path, candidate: Path) -> bool: + """Whether ``candidate`` is ``directory`` or lies beneath it, links resolved. + + Each existing ancestor of the resolved path is compared with ``directory`` + as a file, so ``..``, symbolic links and case-insensitive file systems + cannot disguise the build directory. + """ + + resolved = candidate.resolve() + return any( + ancestor.exists() and os.path.samefile(ancestor, directory) + for ancestor in (resolved, *resolved.parents) + ) + + +def _unused_evidence(build: Path) -> list[dict[str, str]]: + found = [] + for name in sorted(os.listdir(build)): + if name in _UNUSED_REASONS: + found.append({"name": name, "reason": _UNUSED_REASONS[name]}) + elif name.endswith(_RUN_MANIFEST_SUFFIX): + found.append({"name": name, "reason": _RUN_MANIFEST_REASON}) + return found + + +def _published_file(name: str, role: str, payload: bytes) -> dict[str, object]: + return { + "name": name, + "role": role, + "bytes": len(payload), + "sha256": hashlib.sha256(payload).hexdigest(), + } + + +def _write_publication(output: Path, files: Mapping[str, bytes]) -> None: + """Write every file into a hidden sibling, then move it into place whole.""" + + output.parent.mkdir(parents=True, exist_ok=True) + staging = output.parent / f".{output.name}.{uuid.uuid4().hex}.partial" + staging.mkdir() + try: + for name, payload in files.items(): + with (staging / name).open("xb") as stream: + stream.write(payload) + _require(not os.path.lexists(output), f"output {output} already exists") + os.rename(staging, output) + except BaseException: + shutil.rmtree(staging, ignore_errors=True) + raise + + +def reconstruct_uk_publication( + build: Path, output: Path, *, revision: str | None = None +) -> dict[str, object]: + """Write a declaration-only publication for ``build`` into a new ``output``. + + ``build`` is a UK build's output directory from before run bindings; it is + only read. ``revision`` must equal the code revision the build recorded, + or supplies one (marked as caller-asserted) when the build recorded none. + The returned summary report is also written as ``orrery.reconstruction.json`` + beside the publication files, outside the upload inventory. + """ + + build, requested_output = Path(build), Path(output) + # Check and write the same resolved path. Otherwise a path that enters a + # missing folder inside the build and leaves it again with ``..`` passes + # the check, and creating its parents makes that folder in the build. + output = requested_output.resolve() + _require(build.is_dir(), f"build directory {build} does not exist") + _require( + not _within(build, output), + f"output {requested_output} is inside the build directory {build}, which " + "is only read", + ) + _require( + not os.path.lexists(requested_output) and not os.path.lexists(output), + f"output {requested_output} already exists", + ) + recorded_evidence = [ + name + for name in (SCHEMA_NAME, EVIDENCE_INDEX_NAME) + if os.path.lexists(build / name) + ] + _require( + not recorded_evidence, + f"{build} records Orrery evidence ({', '.join(recorded_evidence)}); export " + "it with python -m microcosm.graph.orrery instead", + ) + present = [name for name in sorted(_INPUT_ROLES) if os.path.lexists(build / name)] + _require( + DECLARATION_NAME in present, + f"{build} has no recorded graph declaration ({DECLARATION_NAME}); staged " + "bundles and builds from before the UK graph driver never wrote one. " + + (f"Found {', '.join(present)}" if present else "No build record found"), + ) + payloads = {name: _read_input(build, name) for name in present} + records = { + name: _json_object(payloads[name], name) + for name in (COMPLETION_NAME, MANIFEST_FILENAME) + if name in payloads + } + recorded = _recorded_digests(payloads, records) + verification: dict[str, str] = {} + stale_digests: list[dict[str, object]] = [] + for name, payload in payloads.items(): + verification[name], stale_records = _verification( + name, payload, recorded.get(name, []) + ) + stale_digests.extend(stale_records) + manifest = records.get(MANIFEST_FILENAME, {}) + git_commit, git_dirty, origin = _source_revision( + build, _recorded_code(manifest), revision + ) + # The code computing the schema's sections and the Orrery projection, + # identified from this module's file rather than the working directory. + reconstructing = code_revision(Path(__file__)) + marker = reconstruction_marker( + git_commit=git_commit, + git_dirty=git_dirty, + revision_origin=origin, + reconstruction_revision=reconstructing, + inputs=[ + { + "name": name, + "role": _INPUT_ROLES[name], + "sha256": hashlib.sha256(payload).hexdigest(), + "bytes": len(payload), + "verification": verification[name], + } + for name, payload in payloads.items() + ], + ) + schema = reconstructed_schema(payloads[DECLARATION_NAME], marker) + country = str(schema["country"]) + schema_bytes = canonical_json(schema) + graph_bytes = orrery_json_from_schema( + schema, + title=f"{country.upper()} · reconstructed declaration ({git_commit[:12]})", + ).encode("utf-8") + published = [ + _published_file(GRAPH_NAME, "graph", graph_bytes), + _published_file(SCHEMA_NAME, "schema", schema_bytes), + ] + # Derived from the published bytes, so reconstructing the same build twice + # yields the same publication rather than a second one. + publication_id = ( + PUBLICATION_ID_PREFIX + + hashlib.sha256(canonical_json(published)).hexdigest()[:32] + ) + inventory = {"version": 1, "publication_id": publication_id, "files": published} + warnings = [] + if origin == "caller_asserted": + warnings.append( + "The build records no code revision; the revision was supplied with " + "--revision and cannot be checked against the build." + ) + elif git_dirty is True: + warnings.append( + "The build ran from a working tree with uncommitted changes, so its " + "recorded revision does not fully identify the code that declared " + "the graph." + ) + elif git_dirty is None: + warnings.append("The build did not record whether its working tree was clean.") + if reconstructing["git_commit"] is None: + warnings.append( + "The code running this reconstruction is not a file tracked by a git " + "checkout, so the marker records no reconstruction revision." + ) + elif reconstructing["git_dirty"] is None: + warnings.append( + "git could not say whether the working tree running this " + "reconstruction was clean." + ) + elif reconstructing["git_dirty"]: + warnings.append( + "This reconstruction ran from a working tree with uncommitted changes, " + "so its reconstruction revision does not fully identify the code that " + "compiled the schema and projected the Orrery document." + ) + warnings.extend(_stale_digest_warning(stale) for stale in stale_digests) + if verification[DECLARATION_NAME] == "no_recorded_digest": + warnings.append( + f"The build recorded no digest for {DECLARATION_NAME}; it is bound only " + "by the digest in the reconstruction marker." + ) + graph: Any = schema["graph"] + report: dict[str, object] = { + "protocol": REPORT_PROTOCOL, + "status": "reconstructed", + "basis": DECLARATION_ONLY, + "publication_id": publication_id, + "published_files": [item["name"] for item in published] + [INVENTORY_NAME], + "source_revision": marker["source_revision"], + "reconstruction_revision": marker["reconstruction_revision"], + "build": _build_identity(manifest, records.get(COMPLETION_NAME, {})), + "declaration": { + "name": DECLARATION_NAME, + "country": country, + "graph_sha256": schema["graph_sha256"], + "operations": len(graph["nodes"]), + "sources": len(graph["sources"]), + }, + "inputs": marker["inputs"], + "stale_recorded_digests": stale_digests, + "missing": [ + {"name": name, "reason": _ABSENT_REASONS[name]} + for name in sorted(_ABSENT_REASONS) + if name not in payloads + ], + "unused_evidence": _unused_evidence(build), + "warnings": warnings, + } + _write_publication( + output, + { + SCHEMA_NAME: schema_bytes, + GRAPH_NAME: graph_bytes, + INVENTORY_NAME: canonical_json(inventory), + REPORT_NAME: json_text(report).encode("utf-8"), + }, + ) + return report + + +def _count(number: int, noun: str) -> str: + return f"{number} {noun}{'' if number == 1 else 's'}" + + +def summary_text(report: Mapping[str, Any], output: Path) -> str: + """A short human-readable account of one reconstruction.""" + + revision, declaration = report["source_revision"], report["declaration"] + origin = ( + "recorded by the build" + if revision["origin"] == "recorded" + else "asserted with --revision" + ) + reconstructing = report["reconstruction_revision"] + if reconstructing["git_commit"] is None: + reconstructed_by = "unknown revision" + else: + tree = {False: "clean", True: "uncommitted changes"}.get( + reconstructing["git_dirty"], "working tree not checked" + ) + reconstructed_by = f"{reconstructing['git_commit']} ({tree})" + lines = [ + f"Wrote a reconstructed Orrery publication (declaration only) to {output}", + f" revision: {revision['git_commit']} ({origin})", + f" reconstructed by: {reconstructed_by}", + f" declaration: {declaration['name']}, " + f"{_count(declaration['operations'], 'operation')}, " + f"{_count(declaration['sources'], 'source')}", + " inputs: " + + ", ".join( + f"{item['name']} ({item['verification'].replace('_', ' ')})" + for item in report["inputs"] + ), + ] + for label, key in (("missing", "missing"), ("found, not bound", "unused_evidence")): + if report[key]: + names = ", ".join(item["name"] for item in report[key]) + lines.append(f" {label}: {names}") + lines.extend(f" warning: {warning}" for warning in report["warnings"]) + lines.append(f" report: {output / REPORT_NAME}") + return "\n".join(lines) + + +def main(argv: list[str] | None = None) -> int: + """Write a declaration-only Orrery publication for a UK build before #1144.""" + + parser = argparse.ArgumentParser( + prog="python -m microcosm.build.uk_runtime.orrery_reconstruction", + description=main.__doc__, + ) + parser.add_argument( + "--build", + type=Path, + required=True, + help="the past build's output directory, which is only read", + ) + parser.add_argument( + "--output", + type=Path, + required=True, + help=( + "new publication directory, outside the build directory; it must not exist" + ), + ) + parser.add_argument( + "--revision", + help=( + "full git commit the build ran from; it must match the revision the " + "build recorded, or supplies one when the build recorded none" + ), + ) + args = parser.parse_args(argv) + try: + # The publication is written at the resolved path, which the given + # spelling need not reach: ``/missing/../../publication`` + # resolves beside the build, but ``missing`` cannot be entered. + # Resolve before the write, as ``reconstruct_uk_publication`` does, + # and print that path. The given spelling is still what is passed, so + # that the checks on it run. + written_output = args.output.resolve() + report = reconstruct_uk_publication( + args.build, args.output, revision=args.revision + ) + except (OSError, ValueError, RecursionError) as error: + parser.error(str(error)) + print(summary_text(report, written_output)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_orrery_reconstruction.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_orrery_reconstruction.py new file mode 100644 index 0000000000..ede92261ca --- /dev/null +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_orrery_reconstruction.py @@ -0,0 +1,828 @@ +"""Declaration-only Orrery publications for UK builds from before run bindings. + +Fixtures replay the files a UK dense or national build wrote before +microcosm#1144, in the order that era's ``full_build_cli`` wrote them: the +declaration, its operation inventory and run manifests, the candidate manifest +carrying the code pin, the completion record binding that manifest by digest, +for a staged build ``sha256sums.txt``, and then the close step, which appends +the staging receipts to the manifest. +""" + +from __future__ import annotations + +import hashlib +import json +import os +import subprocess +from pathlib import Path + +import pytest +from microcosm_emitter.orrery.service.queue import publication_inventory + +from microcosm.build.country_spec import load_country_spec +from microcosm.build.staging_dataset import refresh_sha256sums_entry +from microcosm.build.uk_runtime import orrery_reconstruction +from microcosm.build.uk_runtime.graph import uk_spine_endpoint, uk_spine_graph +from microcosm.build.uk_runtime.graph_build import UKFullBuildConfig, uk_full_graph +from microcosm.build.uk_runtime.graph_calibration import UKGraphCalibrationConfig +from microcosm.build.uk_runtime.graph_evidence import add_uk_spine_gate_nodes +from microcosm.build.uk_runtime.graph_terminal import append_uk_full_gate_nodes +from microcosm.build.uk_runtime.orrery_publication import INVENTORY_NAME +from microcosm.build.uk_runtime.orrery_reconstruction import ( + GRAPH_NAME, + REPORT_NAME, + SCHEMA_NAME, + main, + reconstruct_uk_publication, +) +from microcosm.build.uk_runtime.rowwise_cli import MANIFEST_FILENAME, json_text +from microcosm.graph import Graph, Node, Owned, SourceRef, StructuralDelta +from microcosm.graph.canonical import canonical_json +from microcosm.graph.reconstruction import ( + RECONSTRUCTION_EXTENSION, + validate_reconstruction, +) +from microcosm.graph.schema import validate_graph_schema +from microcosm.graph.serialize import graph_to_json +from test_support.paths import paths_for + +_RECONSTRUCTION_VERIFY = ( + paths_for("microcosm-build").repository + / "tools" + / "orrery-contract" + / "verify-reconstruction.mjs" +) +RECORDED_COMMIT = "4bf7af406a1b2c3d4e5f60718293a4b5c6d7e8f9" +OTHER_COMMIT = "7f235941c0000000000000000000000000000000" +NATIONAL_BUILD_ID = "uk-national-20260930T120000Z-s42" +# The completion-record layouts UK builds wrote before #1144: +# - DENSE_STALE_COMPLETION: the dense driver merged in #901 (430187fbc, merged +# 2026-09-29). ``build.json`` binds the candidate manifest as first written, +# with STAGING_RECEIPTS_NOTE; the close step then rewrites the manifest and +# refreshes only ``sha256sums.txt``, so that digest is stale. +# - DENSE_REFRESHED_COMPLETION: from cb0d39045 (#1057, merged 2026-09-30), +# ``_refresh_completion`` reissues ``build.json`` over the final bytes and +# removes the note. +# - NATIONAL_COMPLETION: the national graph path, which arrived in #1057 with +# the refresh, so its ``build.json`` (``uk_national_build_completion``) is +# always refreshed, its ``build_record`` entry included. +DENSE_STALE_COMPLETION = "dense_stale_completion" +DENSE_REFRESHED_COMPLETION = "dense_refreshed_completion" +NATIONAL_COMPLETION = "national_completion" +STAGING_RECEIPTS_NOTE = "staging receipts are appended after publication" + + +def sha256_hex(payload: bytes) -> str: + return hashlib.sha256(payload).hexdigest() + + +def pre_1144_declaration() -> Graph: + return Graph( + "uk", + (SourceRef("uk_spine", "raw-bytes-v1", "Recorded FRS spine"),), + ( + Node( + "uk.full.spine_checkpoint", + "fixture.checkpoint@1", + sources=("uk_spine",), + structural=StructuralDelta.CREATE, + outputs=(Owned("household", "household_weight", "float64"),), + description="Load the recorded spine checkpoint", + ), + Node( + "uk.full.gates.calibrated", + "fixture.gates@1", + population="uk.full.spine_checkpoint", + description="Evaluate the calibrated gate battery", + ), + ), + ) + + +def file_entry(path: Path) -> dict[str, object]: + """A completion-record entry, as ``artifact_files.file_artifact`` wrote it.""" + + payload = path.read_bytes() + return { + "filename": path.name, + "sha256": sha256_hex(payload), + "size_bytes": len(payload), + } + + +def output_entry(path: Path) -> dict[str, object]: + """A candidate manifest ``outputs`` entry, as both drivers wrote it.""" + + payload = path.read_bytes() + return {"path": str(path), "sha256": sha256_hex(payload), "bytes": len(payload)} + + +def write_pre_1144_build( + directory: Path, + *, + graph: Graph | None = None, + git_commit: str | None = RECORDED_COMMIT, + git_dirty: bool | None = False, + layout: str = DENSE_REFRESHED_COMPLETION, + staged: bool = True, +) -> Path: + """Replay the files a UK build left in its output directory before #1144. + + The order is the driver's: the candidate manifest, ``build.json`` over it + (noting that staging receipts follow), the staging sidecars when the + bundle was staged, then the close step, which appends ``staging_delivery`` + and ``staged_dataset`` to the manifest whether or not it staged. Only the + refreshed layouts then reissue ``build.json``. + """ + + national = layout == NATIONAL_COMPLETION + release_role = "national" if national else "dense" + release_id = f"microcosm-uk-2025-{release_role}" + directory.mkdir(parents=True) + declared = pre_1144_declaration() if graph is None else graph + (directory / "graph.json").write_bytes(graph_to_json(declared).encode()) + (directory / "operations.json").write_bytes( + canonical_json([{"id": node.id} for node in declared.nodes]) + ) + (directory / "build.graph.json").write_text('{"schema_version":3}') + (directory / "preflight.graph.json").write_text('{"schema_version":3}') + (directory / "evidence-index.json").write_text("{}") + (directory / ".graph-store").mkdir() + dataset = directory / ( + "microcosm_uk_2024_25.h5" if national else "microcosm_uk_2024_25_local.h5" + ) + dataset.write_bytes(b"licensed population") + outputs = {"dataset": output_entry(dataset)} + code = {"git_commit": git_commit, "git_dirty": git_dirty} + manifest: dict[str, object] = { + "schema_version": 4, + "release_role": release_role, + "release_id": release_id, + "created_at": "2026-09-30T12:00:00+00:00", + **code, + "identity": {"code": code}, + "outputs": outputs, + } + if national: + build_record = directory / "build_record.json" + build_record.write_text(json_text({"build_id": NATIONAL_BUILD_ID})) + outputs["build_record"] = output_entry(build_record) + manifest.update( + build_kind="uk_national_calibrated_candidate", build_id=NATIONAL_BUILD_ID + ) + else: + manifest.update(build_kind="uk_rowwise_calibrated_candidate") + manifest_path = directory / MANIFEST_FILENAME + manifest_path.write_text(json_text(manifest)) + manifest_entry = {**file_entry(manifest_path), "note": STAGING_RECEIPTS_NOTE} + if national: + completion: dict[str, object] = { + "schema_version": 1, + "kind": "uk_national_build_completion", + "release_role": release_role, + "release_id": release_id, + "build_id": NATIONAL_BUILD_ID, + "dataset": {**file_entry(dataset), "path": str(dataset)}, + "build_record": {**file_entry(build_record), "path": str(build_record)}, + "rowwise_candidate_manifest": manifest_entry, + "graph": {"artifacts": {}, "readback": {"passed": True}}, + "release_authorized": False, + } + else: + candidate = directory / "candidate.json" + candidate.write_bytes(canonical_json({"readback_passed": True})) + completion = { + "readback_passed": True, + "kind": "uk_full_build_completion", + "release_role": release_role, + "candidate_manifest": file_entry(candidate), + "rowwise_candidate_manifest": manifest_entry, + } + (directory / "build.json").write_bytes(canonical_json(completion)) + sums = directory / "sha256sums.txt" + if staged: + # ``write_sidecars``: every output, the manifest as staged and the staged + # manifest, sorted by name. + (directory / "staged_manifest.json").write_text('{"run_id":"fixture"}') + listed = sorted( + [Path(str(entry["path"])).name for entry in outputs.values()] + + [MANIFEST_FILENAME, "staged_manifest.json"] + ) + sums.write_text( + "".join( + f"{sha256_hex((directory / name).read_bytes())} {name}\n" + for name in listed + ) + ) + # The close step (``_close_attempt`` / ``_close_national_attempt``). + manifest["staging_delivery"] = {"mode": "hub" if staged else "disabled"} + manifest["staged_dataset"] = {"status": "uploaded" if staged else "disabled"} + rewritten = {"rowwise_candidate_manifest": manifest_path} + if national: + build_record.write_text( + json_text( + {"build_id": NATIONAL_BUILD_ID, "staging_delivery": {"mode": "hub"}} + ) + ) + outputs["build_record"] = output_entry(build_record) + rewritten["build_record"] = build_record + manifest_path.write_text(json_text(manifest)) + if sums.is_file(): + for path in rewritten.values(): + refresh_sha256sums_entry(directory, path.name) + if layout != DENSE_STALE_COMPLETION: + # ``_refresh_completion``: drop the note and bind the final bytes. + for role, path in rewritten.items(): + entry = dict(completion[role]) # type: ignore[arg-type] + entry.pop("note", None) + completion[role] = {**entry, **file_entry(path)} + (directory / "build.json").write_bytes(canonical_json(completion)) + return directory + + +def read_json(path: Path) -> dict: + return json.loads(path.read_bytes()) + + +def inputs_by_name(marker: dict) -> dict[str, dict]: + return {item["name"]: item for item in marker["inputs"]} + + +def test_dense_build_reconstructs_the_normal_publication_layout(tmp_path): + build = write_pre_1144_build(tmp_path / "build") + output = tmp_path / "publication" + + reconstruct_uk_publication(build, output) + + assert sorted(path.name for path in output.iterdir()) == sorted( + [GRAPH_NAME, SCHEMA_NAME, INVENTORY_NAME, REPORT_NAME] + ) + inventory = read_json(output / INVENTORY_NAME) + # The emitter recomputes the inventory before it queues a directory; the + # reconstructed one must be exactly what it would compute. + assert inventory == publication_inventory( + output, + {GRAPH_NAME: "graph", SCHEMA_NAME: "schema"}, + publication_id=inventory["publication_id"], + ) + schema = validate_graph_schema(read_json(output / SCHEMA_NAME)) + assert (output / SCHEMA_NAME).read_bytes() == canonical_json(schema) + document = read_json(output / GRAPH_NAME) + assert document["schemaVersion"] == "graph-explorer/v1" + assert document["metadata"]["microcosm"]["graph_sha256"] == sha256_hex( + (build / "graph.json").read_bytes() + ) + assert "reconstructed declaration" in document["title"] + assert not {"activities", "artifacts"} & document.keys() + assert all("statuses" not in node for node in document["nodes"]) + # Nothing from the build other than the declaration's projection is copied. + assert not {path.name for path in build.iterdir()} & { + path.name for path in output.iterdir() + } + + +def test_marker_names_the_recorded_revision_and_every_input_digest(tmp_path): + build = write_pre_1144_build(tmp_path / "build") + output = tmp_path / "publication" + + reconstruct_uk_publication(build, output) + + schema = read_json(output / SCHEMA_NAME) + marker = validate_reconstruction(schema) + assert marker is not None + document = read_json(output / GRAPH_NAME) + assert ( + document["metadata"]["microcosm"]["extensions"][RECONSTRUCTION_EXTENSION] + == marker + ) + assert marker["source_revision"] == { + "git_commit": RECORDED_COMMIT, + "git_dirty": False, + "origin": "recorded", + } + inputs = inputs_by_name(marker) + assert set(inputs) == { + "build.json", + "graph.json", + MANIFEST_FILENAME, + "sha256sums.txt", + } + for name, item in inputs.items(): + payload = (build / name).read_bytes() + assert item["sha256"] == sha256_hex(payload) + assert item["bytes"] == len(payload) + assert inputs["graph.json"]["role"] == "declaration" + assert inputs["graph.json"]["verification"] == "no_recorded_digest" + assert inputs[MANIFEST_FILENAME]["role"] == "code_pin" + assert inputs[MANIFEST_FILENAME]["verification"] == "recorded_digest_matched" + + +def test_marker_records_the_code_that_ran_the_reconstruction_beside_the_source( + tmp_path, monkeypatch +): + looked_up = [] + + def reconstructing_checkout(path): + looked_up.append(Path(path)) + return {"git_commit": OTHER_COMMIT, "git_dirty": True} + + monkeypatch.setattr(orrery_reconstruction, "code_revision", reconstructing_checkout) + build = write_pre_1144_build(tmp_path / "build") + + report = reconstruct_uk_publication(build, tmp_path / "publication") + + marker = validate_reconstruction(read_json(tmp_path / "publication" / SCHEMA_NAME)) + # The revision is looked up from the reconstructing module's own file, not + # from whatever directory the command runs in. + assert looked_up == [Path(orrery_reconstruction.__file__)] + assert marker["source_revision"] == { + "git_commit": RECORDED_COMMIT, + "git_dirty": False, + "origin": "recorded", + } + assert marker["reconstruction_revision"] == { + "git_commit": OTHER_COMMIT, + "git_dirty": True, + } + assert report["reconstruction_revision"] == marker["reconstruction_revision"] + assert any( + "reconstruction ran from a working tree with uncommitted changes" in warning + for warning in report["warnings"] + ) + + +def test_unknown_reconstruction_revision_is_recorded_as_unknown( + tmp_path, monkeypatch, capsys +): + monkeypatch.setattr( + orrery_reconstruction, + "code_revision", + lambda path: {"git_commit": None, "git_dirty": None}, + ) + build = write_pre_1144_build(tmp_path / "build") + output = tmp_path / "publication" + + assert main(["--build", str(build), "--output", str(output)]) == 0 + + report = read_json(output / REPORT_NAME) + assert report["reconstruction_revision"] == {"git_commit": None, "git_dirty": None} + assert any( + "no reconstruction revision" in warning for warning in report["warnings"] + ) + assert "reconstructed by: unknown revision" in capsys.readouterr().out + + +@pytest.mark.parametrize("layout", [DENSE_REFRESHED_COMPLETION, NATIONAL_COMPLETION]) +def test_candidate_manifest_disagreeing_with_a_refreshed_build_record_is_refused( + tmp_path, layout +): + build = write_pre_1144_build(tmp_path / "build", layout=layout, staged=False) + completion = read_json(build / "build.json") + completion["rowwise_candidate_manifest"]["sha256"] = "0" * 64 + (build / "build.json").write_bytes(canonical_json(completion)) + + with pytest.raises( + ValueError, match="does not match the digest recorded in build.json" + ): + reconstruct_uk_publication(build, tmp_path / "publication") + assert not (tmp_path / "publication").exists() + + +def test_only_the_staging_receipts_note_marks_a_completion_digest_as_stale(tmp_path): + build = write_pre_1144_build(tmp_path / "build", layout=DENSE_STALE_COMPLETION) + completion = read_json(build / "build.json") + completion["rowwise_candidate_manifest"]["note"] = "checked by hand" + (build / "build.json").write_bytes(canonical_json(completion)) + + with pytest.raises( + ValueError, match="does not match the digest recorded in build.json" + ): + reconstruct_uk_publication(build, tmp_path / "publication") + + +@pytest.mark.parametrize("layout", [DENSE_REFRESHED_COMPLETION, DENSE_STALE_COMPLETION]) +def test_candidate_manifest_disagreeing_with_sha256sums_is_refused(tmp_path, layout): + # A stale completion digest is skipped, but the digest list still binds. + build = write_pre_1144_build(tmp_path / "build", layout=layout) + (build / "sha256sums.txt").write_text(f"{'0' * 64} {MANIFEST_FILENAME}\n") + + with pytest.raises( + ValueError, match="does not match the digest recorded in sha256sums.txt" + ): + reconstruct_uk_publication(build, tmp_path / "publication") + assert not (tmp_path / "publication").exists() + + +def test_dense_build_closed_before_the_completion_refresh_is_checked_against_the_digest_list( + tmp_path, +): + build = write_pre_1144_build(tmp_path / "build", layout=DENSE_STALE_COMPLETION) + stale_entry = read_json(build / "build.json")["rowwise_candidate_manifest"] + # What #901's driver left: the note, beside the digest of the manifest as it + # was before the close step appended the staging receipts. + assert stale_entry["note"] == STAGING_RECEIPTS_NOTE + assert stale_entry["sha256"] != sha256_hex((build / MANIFEST_FILENAME).read_bytes()) + + report = reconstruct_uk_publication(build, tmp_path / "publication") + + marker = validate_reconstruction(read_json(tmp_path / "publication" / SCHEMA_NAME)) + assert marker["source_revision"]["git_commit"] == RECORDED_COMMIT + manifest_input = inputs_by_name(marker)[MANIFEST_FILENAME] + assert manifest_input["verification"] == "fallback_digest_matched" + assert report["inputs"] == marker["inputs"] + assert report["stale_recorded_digests"] == [ + { + "name": MANIFEST_FILENAME, + "recorded_in": "build.json", + "sha256": stale_entry["sha256"], + "bytes": stale_entry["size_bytes"], + "note": STAGING_RECEIPTS_NOTE, + "checked_against": ["sha256sums.txt"], + } + ] + assert any( + "build.json" in warning and "checked against sha256sums.txt" in warning + for warning in report["warnings"] + ) + + +def test_stale_completion_digest_without_a_digest_list_leaves_the_manifest_unchecked( + tmp_path, +): + # With staging disabled the driver wrote no sidecars, but the close step + # still appended the receipts, so no current digest covers the manifest. + build = write_pre_1144_build( + tmp_path / "build", layout=DENSE_STALE_COMPLETION, staged=False + ) + + report = reconstruct_uk_publication(build, tmp_path / "publication") + + marker = validate_reconstruction(read_json(tmp_path / "publication" / SCHEMA_NAME)) + manifest_input = inputs_by_name(marker)[MANIFEST_FILENAME] + assert manifest_input["verification"] == "only_stale_digest_recorded" + assert [item["checked_against"] for item in report["stale_recorded_digests"]] == [ + [] + ] + assert any( + "nothing the build recorded checks" in warning for warning in report["warnings"] + ) + + +def test_national_build_reconstructs_from_its_completion_record(tmp_path): + build = write_pre_1144_build(tmp_path / "build", layout=NATIONAL_COMPLETION) + completion = read_json(build / "build.json") + assert completion["kind"] == "uk_national_build_completion" + assert "note" not in completion["rowwise_candidate_manifest"] + assert completion["build_record"]["sha256"] == sha256_hex( + (build / "build_record.json").read_bytes() + ) + + report = reconstruct_uk_publication(build, tmp_path / "publication") + + marker = validate_reconstruction(read_json(tmp_path / "publication" / SCHEMA_NAME)) + inputs = inputs_by_name(marker) + assert set(inputs) == { + "build.json", + "graph.json", + MANIFEST_FILENAME, + "sha256sums.txt", + } + assert inputs[MANIFEST_FILENAME]["verification"] == "recorded_digest_matched" + assert report["stale_recorded_digests"] == [] + assert report["build"] == { + "build_kind": "uk_national_calibrated_candidate", + "release_role": "national", + "release_id": "microcosm-uk-2025-national", + "build_id": NATIONAL_BUILD_ID, + "created_at": "2026-09-30T12:00:00+00:00", + } + + +@pytest.mark.parametrize( + "recorded_file", ["graph.schema.json", "execution.evidence.json"] +) +def test_build_with_recorded_orrery_evidence_is_refused(tmp_path, recorded_file): + build = write_pre_1144_build(tmp_path / "build") + (build / recorded_file).write_text("{}") + + with pytest.raises(ValueError, match="python -m microcosm.graph.orrery"): + reconstruct_uk_publication(build, tmp_path / "publication") + + +def test_staged_bundle_without_a_declaration_is_refused_naming_what_is_missing( + tmp_path, +): + # A staged bundle (staged//) never carried graph.json before #1144. + bundle = write_pre_1144_build(tmp_path / "bundle") + (bundle / "graph.json").unlink() + + with pytest.raises(ValueError, match="graph.json") as refusal: + reconstruct_uk_publication(bundle, tmp_path / "publication") + assert MANIFEST_FILENAME in str(refusal.value) + assert not (tmp_path / "publication").exists() + + +def test_caller_revision_must_match_the_recorded_revision(tmp_path): + build = write_pre_1144_build(tmp_path / "build") + + with pytest.raises(ValueError, match="does not match the recorded revision"): + reconstruct_uk_publication( + build, tmp_path / "publication", revision=OTHER_COMMIT + ) + reconstruct_uk_publication(build, tmp_path / "confirmed", revision=RECORDED_COMMIT) + marker = validate_reconstruction(read_json(tmp_path / "confirmed" / SCHEMA_NAME)) + assert marker["source_revision"]["origin"] == "recorded" + + +def test_caller_revision_supplies_an_unrecorded_revision_as_asserted(tmp_path): + build = write_pre_1144_build(tmp_path / "build", git_commit=None, git_dirty=None) + + report = reconstruct_uk_publication( + build, tmp_path / "publication", revision=OTHER_COMMIT + ) + + marker = validate_reconstruction(read_json(tmp_path / "publication" / SCHEMA_NAME)) + assert marker["source_revision"] == { + "git_commit": OTHER_COMMIT, + "git_dirty": None, + "origin": "caller_asserted", + } + assert any("--revision" in warning for warning in report["warnings"]) + + +@pytest.mark.parametrize("malformed_revision", ["4bf7af406", RECORDED_COMMIT.upper()]) +def test_malformed_caller_revision_is_refused(tmp_path, malformed_revision): + build = write_pre_1144_build(tmp_path / "build") + + with pytest.raises(ValueError, match="full lowercase git commit"): + reconstruct_uk_publication( + build, tmp_path / "publication", revision=malformed_revision + ) + + +def test_build_without_any_revision_is_refused(tmp_path): + build = write_pre_1144_build(tmp_path / "build", staged=False) + (build / MANIFEST_FILENAME).unlink() + (build / "build.json").unlink() + + with pytest.raises(ValueError, match="pass --revision"): + reconstruct_uk_publication(build, tmp_path / "publication") + + +def test_summary_reports_unused_evidence_missing_files_and_warnings(tmp_path): + build = write_pre_1144_build( + tmp_path / "build", git_dirty=True, layout=NATIONAL_COMPLETION, staged=False + ) + + report = reconstruct_uk_publication(build, tmp_path / "publication") + + assert report == read_json(tmp_path / "publication" / REPORT_NAME) + assert report["status"] == "reconstructed" + assert report["basis"] == "declaration_only" + assert report["build"]["release_role"] == "national" + assert report["declaration"]["operations"] == 2 + assert report["declaration"]["sources"] == 1 + assert [item["name"] for item in report["missing"]] == ["sha256sums.txt"] + assert [item["name"] for item in report["unused_evidence"]] == [ + ".graph-store", + "build.graph.json", + "evidence-index.json", + "operations.json", + "preflight.graph.json", + ] + assert all(item["reason"] for item in report["missing"] + report["unused_evidence"]) + assert any("uncommitted changes" in warning for warning in report["warnings"]) + assert any("graph.json" in warning for warning in report["warnings"]) + published = { + item["name"] + for item in read_json(tmp_path / "publication" / INVENTORY_NAME)["files"] + } + assert REPORT_NAME not in published + + +def test_reconstruction_is_deterministic(tmp_path): + build = write_pre_1144_build(tmp_path / "build") + + reconstruct_uk_publication(build, tmp_path / "first") + reconstruct_uk_publication(build, tmp_path / "second") + + for name in (GRAPH_NAME, SCHEMA_NAME, INVENTORY_NAME, REPORT_NAME): + assert (tmp_path / "first" / name).read_bytes() == ( + tmp_path / "second" / name + ).read_bytes() + + +def test_existing_output_is_refused_and_failure_leaves_no_partial_output( + tmp_path, monkeypatch +): + build = write_pre_1144_build(tmp_path / "build") + existing = tmp_path / "existing" + existing.mkdir() + + with pytest.raises(ValueError, match="already exists"): + reconstruct_uk_publication(build, existing) + assert list(existing.iterdir()) == [] + + def interrupted_rename(source, destination): + raise OSError("interrupted before the publication was moved into place") + + # Every file is written, then the move into place fails: nothing remains. + monkeypatch.setattr(os, "rename", interrupted_rename) + with pytest.raises(OSError, match="interrupted"): + reconstruct_uk_publication(build, tmp_path / "publication") + assert sorted(path.name for path in tmp_path.iterdir()) == ["build", "existing"] + + +@pytest.mark.parametrize( + "output_parts", + [ + ("build",), + ("build", "publication"), + ("elsewhere", "..", "build", "publication"), + ("link-to-build", "publication"), + ], + ids=["the_build_itself", "inside", "through_a_parent_reference", "through_a_link"], +) +def test_output_inside_the_build_directory_is_refused(tmp_path, output_parts): + build = write_pre_1144_build(tmp_path / "build") + (tmp_path / "elsewhere").mkdir() + (tmp_path / "link-to-build").symlink_to(build, target_is_directory=True) + listing = sorted(path.name for path in build.iterdir()) + + with pytest.raises(ValueError, match="inside the build directory"): + reconstruct_uk_publication(build, tmp_path.joinpath(*output_parts)) + + # The build is only read: nothing was written into it, not even staging. + assert sorted(path.name for path in build.iterdir()) == listing + + +def _tree_snapshot(directory: Path) -> dict[str, bytes | None]: + """Every path under ``directory``, with each file's bytes (``None`` for a folder).""" + + return { + path.relative_to(directory).as_posix(): ( + None if path.is_dir() else path.read_bytes() + ) + for path in sorted(directory.rglob("*")) + } + + +@pytest.mark.parametrize( + "build_is_writable", [True, False], ids=["writable_build", "read_only_build"] +) +def test_output_leaving_a_missing_folder_inside_the_build_creates_nothing_there( + tmp_path, build_is_writable +): + # It names a folder beside the build, so it is accepted; the folders it + # passes through on the way must not be created inside the build. + build = write_pre_1144_build(tmp_path / "build") + output = build / "missing" / ".." / ".." / "publication" + before = _tree_snapshot(build) + original_mode = build.stat().st_mode + if not build_is_writable: + if os.geteuid() == 0: + pytest.skip("root ignores directory permissions") + build.chmod(0o555) + try: + reconstruct_uk_publication(build, output) + finally: + build.chmod(original_mode) + + assert _tree_snapshot(build) == before + assert sorted(path.name for path in (tmp_path / "publication").iterdir()) == ( + sorted([GRAPH_NAME, SCHEMA_NAME, INVENTORY_NAME, REPORT_NAME]) + ) + + +def test_output_named_by_a_dangling_link_leading_outside_the_build_is_refused( + tmp_path, +): + # The given name exists, as a symbolic link, but the path it resolves to + # does not. Checking only the resolved path would accept it and write the + # publication wherever the link points. + build = write_pre_1144_build(tmp_path / "build") + (tmp_path / "elsewhere").mkdir() + link_target = tmp_path / "elsewhere" / "publication" + dangling_link = tmp_path / "publication-link" + dangling_link.symlink_to(link_target, target_is_directory=True) + + with pytest.raises(ValueError, match="already exists"): + reconstruct_uk_publication(build, dangling_link) + + assert dangling_link.is_symlink() + assert not os.path.lexists(link_target) + assert list((tmp_path / "elsewhere").iterdir()) == [] + + +def test_declaration_behind_a_symbolic_link_is_refused(tmp_path): + build = write_pre_1144_build(tmp_path / "build") + elsewhere = tmp_path / "elsewhere.json" + (build / "graph.json").rename(elsewhere) + (build / "graph.json").symlink_to(elsewhere) + + with pytest.raises(ValueError, match="regular file"): + reconstruct_uk_publication(build, tmp_path / "publication") + + +def test_cli_writes_the_publication_and_prints_a_summary(tmp_path, capsys): + build = write_pre_1144_build(tmp_path / "build") + output = tmp_path / "publication" + + assert main(["--build", str(build), "--output", str(output)]) == 0 + + printed = capsys.readouterr().out + assert str(output) in printed + assert RECORDED_COMMIT in printed + assert "reconstructed by: " in printed + assert "declaration only" in printed + assert (output / GRAPH_NAME).is_file() + with pytest.raises(SystemExit) as refusal: + main(["--build", str(build), "--output", str(output)]) + assert refusal.value.code == 2 + assert "already exists" in capsys.readouterr().err + + +def test_cli_prints_the_resolved_output_path_it_wrote(tmp_path, capsys): + # The given spelling enters a folder the build does not have and leaves + # it with "..", so it names no reachable path; the publication is written + # at its resolved path beside the build, which is the one to print. + build = write_pre_1144_build(tmp_path / "build") + given_output = build / "missing" / ".." / ".." / "cli_publication" + written_output = tmp_path.resolve() / "cli_publication" + + assert main(["--build", str(build), "--output", str(given_output)]) == 0 + + printed = capsys.readouterr().out + assert ( + f"Wrote a reconstructed Orrery publication (declaration only) to " + f"{written_output}\n" + ) in printed + assert f" report: {written_output / REPORT_NAME}" in printed.splitlines() + assert str(given_output) not in printed + assert (written_output / REPORT_NAME).is_file() + assert not os.path.lexists(given_output) + + +def test_cli_refuses_a_declaration_that_is_not_a_json_object(tmp_path, capsys): + build = write_pre_1144_build(tmp_path / "build", git_commit=None, git_dirty=None) + (build / "graph.json").write_bytes(b"[]") + output = tmp_path / "publication" + + arguments = ["--build", str(build), "--output", str(output)] + with pytest.raises(SystemExit) as refusal: + main([*arguments, "--revision", OTHER_COMMIT]) + + assert refusal.value.code == 2 + assert "recorded declaration is not a graph" in capsys.readouterr().err + assert not output.exists() + + +def test_cli_reconstructs_a_build_closed_before_the_completion_refresh( + tmp_path, capsys +): + build = write_pre_1144_build(tmp_path / "build", layout=DENSE_STALE_COMPLETION) + output = tmp_path / "publication" + + arguments = ["--build", str(build), "--output", str(output)] + assert main([*arguments, "--revision", RECORDED_COMMIT]) == 0 + + printed = capsys.readouterr().out + assert f"{MANIFEST_FILENAME} (fallback digest matched)" in printed + + +def test_maintained_uk_dense_declaration_reconstructs_within_orrery_limits(tmp_path): + spine = add_uk_spine_gate_nodes( + uk_spine_graph(source_mode="split"), + spec=load_country_spec("uk"), + engine_identity="declaration-only", + ) + full = uk_full_graph( + UKFullBuildConfig( + calibration_year=2025, + geography_assignment="legacy", + calibration=UKGraphCalibrationConfig(), + ), + spine=spine, + ) + declared = append_uk_full_gate_nodes( + full.graph, + calibration=full.calibration, + spine_stage_names=uk_spine_endpoint(spine).stage_names, + engine_identity="declaration-only", + review_date="2026-10-08", + ) + build = write_pre_1144_build(tmp_path / "build", graph=declared) + output = tmp_path / "publication" + + report = reconstruct_uk_publication(build, output) + + assert report["declaration"]["operations"] == len(declared.nodes) + result = subprocess.run( + ["node", str(_RECONSTRUCTION_VERIFY)], + input=(output / GRAPH_NAME).read_text(encoding="utf-8"), + text=True, + capture_output=True, + check=False, + ) + assert result.returncode == 0, result.stderr + assert result.stdout == "Orrery accepted the reconstructed declaration.\n" diff --git a/packages/microcosm-graph/src/microcosm/graph/reconstruction.py b/packages/microcosm-graph/src/microcosm/graph/reconstruction.py new file mode 100644 index 0000000000..40db0f6f60 --- /dev/null +++ b/packages/microcosm-graph/src/microcosm/graph/reconstruction.py @@ -0,0 +1,364 @@ +"""Declaration-only graph schemas rebuilt after a build from its recorded files. + +A build that recorded its graph declaration but no run binding cannot have its +execution verified, yet its declared graph is still exact. A reconstructed +schema projects that recorded declaration alone and carries a marker in its +producer extensions: the marker says the schema was rebuilt afterwards and +binds every input file by digest. It names two code revisions: the one that +declared the graph (``source_revision``) and the one that performed the +reconstruction (``reconstruction_revision``). Only the declaration's bytes come +from the build. The schema's ``compiled``, ``fields`` and ``input_bindings`` +sections, and any Orrery projection of it, are computed by the reconstructing +code. Because the marker lives in ``graph.schema.json``, any later projection +of the saved schema keeps it, and Orrery documents show it under +``metadata.microcosm.extensions``. Nothing here presents an operation as +executed; no receipt, cache or gate state is attached. +""" + +from __future__ import annotations + +import hashlib +import os +import re +import subprocess +from collections.abc import Mapping, Sequence +from pathlib import Path + +from .decl import compile_graph +from .schema import _plain_json, graph_schema +from .serialize import graph_from_json + +__all__ = [ + "DECLARATION_ONLY", + "DECLARATION_ROLE", + "GIT_COMMIT_PATTERN", + "INPUT_VERIFICATIONS", + "RECONSTRUCTION_EXTENSION", + "RECONSTRUCTION_PROTOCOL", + "REVISION_ORIGINS", + "code_revision", + "reconstructed_schema", + "reconstruction_marker", + "validate_reconstruction", +] + +RECONSTRUCTION_EXTENSION = "microcosm.reconstruction" +RECONSTRUCTION_PROTOCOL = "microcosm.graph.reconstruction.v1" +DECLARATION_ONLY = "declaration_only" +DECLARATION_ROLE = "declaration" +# ``recorded``: the build wrote the revision itself. ``caller_asserted``: the +# build recorded none, and whoever ran the reconstruction supplied it. +REVISION_ORIGINS = frozenset({"recorded", "caller_asserted"}) +# How an input's bytes compare with the digests the build itself recorded: +# - ``recorded_digest_matched``: every recorded digest matched. +# - ``fallback_digest_matched``: the build marked one recorded digest as taken +# before it rewrote the file. That digest was skipped, and another digest the +# build recorded matched. +# - ``only_stale_digest_recorded``: every digest the build recorded for the file +# is marked as taken before the rewrite, so nothing it recorded checks the +# current bytes. +# - ``no_recorded_digest``: the build recorded none. +# In the last two cases the file is bound only by the digest listed here. +INPUT_VERIFICATIONS = frozenset( + { + "recorded_digest_matched", + "fallback_digest_matched", + "only_stale_digest_recorded", + "no_recorded_digest", + } +) +RECONSTRUCTION_SCOPE = ( + "Rebuilt after the build from its recorded graph declaration, the only part " + "taken from the build: the compiled sections and this projection come from " + "the code at reconstruction_revision. Operations carry no execution, cache " + "or gate state; nothing here was observed running." +) +_MARKER_FIELDS = frozenset( + { + "protocol", + "reconstructed", + "basis", + "source_revision", + "reconstruction_revision", + "inputs", + "scope", + } +) +_REVISION_FIELDS = frozenset({"git_commit", "git_dirty", "origin"}) +_RECONSTRUCTION_REVISION_FIELDS = frozenset({"git_commit", "git_dirty"}) +_INPUT_FIELDS = frozenset({"name", "role", "sha256", "bytes", "verification"}) +# A full lowercase SHA-1 or SHA-256 git object name; abbreviations are refused. +GIT_COMMIT_PATTERN = re.compile(r"[0-9a-f]{40}(?:[0-9a-f]{24})?") +_SHA256 = re.compile(r"[0-9a-f]{64}") + + +def _require(condition: bool, message: str) -> None: + if not condition: + raise ValueError(f"Graph reconstruction: {message}.") + + +def _text(value: object, label: str) -> str: + _require(type(value) is str and bool(value.strip()), f"{label} must be text") + return value # type: ignore[return-value] + + +def _file_name(value: object) -> str: + name = _text(value, "input name") + _require( + name not in {".", ".."} + and not any(separator in name for separator in ("/", "\\")) + and not any(ord(character) < 32 for character in name), + "input names must be plain file names", + ) + return name + + +def _validated_marker(value: object) -> dict[str, object]: + marker = _plain_json(value) + _require(type(marker) is dict, "marker must be an object") + assert isinstance(marker, dict) + _require(set(marker) == _MARKER_FIELDS, "marker fields") + _require(marker["protocol"] == RECONSTRUCTION_PROTOCOL, "marker protocol") + _require(marker["reconstructed"] is True, "marker must say reconstructed") + _require(marker["basis"] == DECLARATION_ONLY, "marker basis") + _text(marker["scope"], "marker scope") + revision = marker["source_revision"] + _require( + type(revision) is dict and set(revision) == _REVISION_FIELDS, + "source revision fields", + ) + _require( + type(revision["git_commit"]) is str + and GIT_COMMIT_PATTERN.fullmatch(revision["git_commit"]) is not None, + "source revision must be a full lowercase git commit", + ) + _require( + type(revision["git_dirty"]) in (bool, type(None)), + "source revision dirty flag must be true, false or null", + ) + _require(revision["origin"] in REVISION_ORIGINS, "source revision origin") + reconstruction = marker["reconstruction_revision"] + _require( + type(reconstruction) is dict + and set(reconstruction) == _RECONSTRUCTION_REVISION_FIELDS, + "reconstruction revision fields", + ) + commit = reconstruction["git_commit"] + _require( + commit is None + or (type(commit) is str and GIT_COMMIT_PATTERN.fullmatch(commit) is not None), + "reconstruction revision must be a full lowercase git commit or null", + ) + _require( + type(reconstruction["git_dirty"]) in (bool, type(None)) + and (commit is not None or reconstruction["git_dirty"] is None), + "reconstruction revision dirty flag must be true, false or, without a " + "commit, null", + ) + inputs = marker["inputs"] + _require(type(inputs) is list and bool(inputs), "marker inputs") + names = [] + for item in inputs: + _require(type(item) is dict and set(item) == _INPUT_FIELDS, "input fields") + names.append(_file_name(item["name"])) + _text(item["role"], "input role") + _require( + type(item["sha256"]) is str + and _SHA256.fullmatch(item["sha256"]) is not None, + "input digests must be lowercase SHA-256", + ) + _require( + type(item["bytes"]) is int and item["bytes"] > 0, + "input sizes must be positive integers", + ) + _require( + item["verification"] in INPUT_VERIFICATIONS, "input verification state" + ) + _require(len(set(names)) == len(names), "duplicate input name") + _require(names == sorted(names), "inputs must be listed in name order") + _require( + sum(item["role"] == DECLARATION_ROLE for item in inputs) == 1, + "exactly one input must be the declaration", + ) + return marker + + +def _declaration_input(marker: Mapping[str, object]) -> Mapping[str, object]: + inputs = marker["inputs"] + assert isinstance(inputs, list) + return next(item for item in inputs if item["role"] == DECLARATION_ROLE) + + +def _git_environment() -> dict[str, str] | None: + """This process's environment without the variables that choose git's repository. + + The names removed are the installed git's own list of variables local to + one repository (``git rev-parse --local-env-vars``): ``GIT_DIR``, + ``GIT_WORK_TREE``, ``GIT_INDEX_FILE``, ``GIT_COMMON_DIR``, + ``GIT_OBJECT_DIRECTORY``, the ``git -c`` carriers ``GIT_CONFIG_PARAMETERS`` + and ``GIT_CONFIG_COUNT``, and the rest. ``None`` when git cannot list them. + ``microcosm.frame.adapters.axiom`` strips the same list for its checkout + check; this module does not import that adapter, so that it stays light. + """ + + try: + result = subprocess.run( + ["git", "rev-parse", "--local-env-vars"], + check=False, + capture_output=True, + text=True, + ) + except OSError: + return None + if result.returncode != 0: + return None + local = frozenset(result.stdout.split()) + return {name: value for name, value in os.environ.items() if name not in local} + + +def code_revision(path: str | Path) -> dict[str, object]: + """The git commit and working-tree state of the checkout that tracks ``path``. + + Pass the file of the code performing the reconstruction. git runs in that + file's directory, not the current one, and the file must be tracked there: + an installed copy that happens to sit inside some checkout (a virtual + environment within a repository, say) identifies no revision. Both values + are ``None`` when no checkout tracks the file, so an unknown revision is + never recorded as a clean one. git runs without the caller's + repository-selecting variables (``GIT_DIR``, ``GIT_WORK_TREE`` and the rest), + so an outer git command, hook or shell cannot point it at another checkout. + """ + + location = Path(path).resolve() + unknown: dict[str, object] = {"git_commit": None, "git_dirty": None} + environment = _git_environment() + if environment is None: + return unknown + + def git(*arguments: str) -> str | None: + try: + result = subprocess.run( + ["git", "-C", str(location.parent), *arguments], + check=False, + capture_output=True, + text=True, + env=environment, + ) + except OSError: + return None + return result.stdout if result.returncode == 0 else None + + if git("ls-files", "--error-unmatch", "--", location.name) is None: + return unknown + head = git("rev-parse", "HEAD") + commit = None if head is None else head.strip() + if commit is None or GIT_COMMIT_PATTERN.fullmatch(commit) is None: + return unknown + changes = git("status", "--porcelain", "--untracked-files=no") + return { + "git_commit": commit, + "git_dirty": None if changes is None else bool(changes.strip()), + } + + +def reconstruction_marker( + *, + git_commit: str, + git_dirty: bool | None, + revision_origin: str, + reconstruction_revision: Mapping[str, object], + inputs: Sequence[Mapping[str, object]], +) -> dict[str, object]: + """Return a validated marker naming both revisions and every input read. + + ``git_commit``, ``git_dirty`` and ``revision_origin`` describe the code that + declared the graph. ``reconstruction_revision`` is :func:`code_revision` of + the code performing the reconstruction, which computes every section of the + schema except the declaration itself. Each input is a mapping with ``name`` + (a plain file name), ``role``, ``sha256``, ``bytes`` and ``verification``; + exactly one has role ``declaration``. Inputs are listed in name order. + """ + + ordered = sorted( + (dict(item) for item in inputs), key=lambda item: str(item.get("name")) + ) + return _validated_marker( + { + "protocol": RECONSTRUCTION_PROTOCOL, + "reconstructed": True, + "basis": DECLARATION_ONLY, + "source_revision": { + "git_commit": git_commit, + "git_dirty": git_dirty, + "origin": revision_origin, + }, + "reconstruction_revision": dict(reconstruction_revision), + "inputs": ordered, + "scope": RECONSTRUCTION_SCOPE, + } + ) + + +def validate_reconstruction(schema: Mapping[str, object]) -> dict[str, object] | None: + """Return a schema's validated reconstruction marker, or ``None`` if absent. + + The marker must name the declaration embedded in the schema: its + declaration input's digest equals the schema's ``graph_sha256``. + """ + + extensions = schema["extensions"] + _require(isinstance(extensions, Mapping), "schema extensions must be an object") + assert isinstance(extensions, Mapping) + if RECONSTRUCTION_EXTENSION not in extensions: + return None + marker = _validated_marker(extensions[RECONSTRUCTION_EXTENSION]) + _require( + _declaration_input(marker)["sha256"] == schema["graph_sha256"], + "marker does not name this declaration", + ) + return marker + + +def reconstructed_schema( + declaration: bytes, marker: Mapping[str, object] +) -> dict[str, object]: + """Project recorded declaration bytes into a marked compiler schema. + + ``declaration`` is the exact ``graph.json`` a build wrote. The marker must + name those bytes, and the current serializer must reproduce them exactly; + otherwise the projection would describe a different declaration from the + one that was recorded, and the declaration is refused. + """ + + _require(type(declaration) is bytes, "declaration must be bytes") + checked = _validated_marker(marker) + named = _declaration_input(checked) + declaration_sha256 = hashlib.sha256(declaration).hexdigest() + _require( + named["sha256"] == declaration_sha256 and named["bytes"] == len(declaration), + "marker does not name this declaration", + ) + try: + text = declaration.decode("utf-8") + except UnicodeDecodeError as error: + raise ValueError( + "Graph reconstruction: declaration must be UTF-8 JSON." + ) from error + try: + declared = graph_from_json(text) + except TypeError as error: + # The serializer reports JSON of the wrong shape (an array where the + # graph object belongs, say) as a TypeError; recorded files are data. + raise ValueError( + f"Graph reconstruction: the recorded declaration is not a graph ({error})." + ) from error + schema = graph_schema( + compile_graph(declared), + extensions={RECONSTRUCTION_EXTENSION: checked}, + ) + _require( + schema["graph_sha256"] == declaration_sha256, + "the declaration does not reproduce its recorded bytes through the " + "current serializer", + ) + return schema diff --git a/packages/microcosm-graph/tests/engine_free/shared/test_graph_reconstruction.py b/packages/microcosm-graph/tests/engine_free/shared/test_graph_reconstruction.py new file mode 100644 index 0000000000..7da4f87af8 --- /dev/null +++ b/packages/microcosm-graph/tests/engine_free/shared/test_graph_reconstruction.py @@ -0,0 +1,416 @@ +"""Declaration-only schemas rebuilt for builds that recorded no run binding.""" + +from __future__ import annotations + +import copy +import hashlib +import json +import subprocess + +import pytest + +from microcosm.graph import graph_schema, graph_to_json +from microcosm.graph.canonical import canonical_json +from microcosm.graph.orrery import orrery_document_from_schema, orrery_json_from_schema +from microcosm.graph.reconstruction import ( + DECLARATION_ONLY, + RECONSTRUCTION_EXTENSION, + RECONSTRUCTION_PROTOCOL, + code_revision, + reconstructed_schema, + reconstruction_marker, + validate_reconstruction, +) +from microcosm.graph.schema import validate_graph_schema +from test_support.microcosm_graph.schema import ( + compiled_default_population_graph, + compiled_graph, +) +from test_support.paths import paths_for + +_TEST_PATHS = paths_for("microcosm-graph") +_RECONSTRUCTION_VERIFY = ( + _TEST_PATHS.repository / "tools" / "orrery-contract" / "verify-reconstruction.mjs" +) +RECORDED_COMMIT = "0123456789abcdef0123456789abcdef01234567" +# The revision of the code that ran the reconstruction, later than the build's. +RECONSTRUCTING_REVISION = { + "git_commit": "89abcdef0123456789abcdef0123456789abcdef", + "git_dirty": False, +} + + +def recorded_declaration() -> bytes: + """The bytes a build wrote to graph.json for its declared graph.""" + + return graph_to_json(compiled_graph().graph).encode("utf-8") + + +def input_record(name: str, role: str, payload: bytes, verification: str) -> dict: + return { + "name": name, + "role": role, + "sha256": hashlib.sha256(payload).hexdigest(), + "bytes": len(payload), + "verification": verification, + } + + +def marker_for(declaration: bytes, **revision_overrides) -> dict: + revision = { + "git_commit": RECORDED_COMMIT, + "git_dirty": False, + "revision_origin": "recorded", + "reconstruction_revision": RECONSTRUCTING_REVISION, + **revision_overrides, + } + return reconstruction_marker( + **revision, + inputs=[ + input_record( + "rowwise_candidate_manifest.json", + "code_pin", + b'{"git_commit":"recorded"}', + "recorded_digest_matched", + ), + input_record( + "graph.json", "declaration", declaration, "no_recorded_digest" + ), + ], + ) + + +def test_reconstructed_schema_carries_the_marker_into_orrery_metadata(): + declaration = recorded_declaration() + marker = marker_for(declaration) + + schema = reconstructed_schema(declaration, marker) + + assert schema["extensions"] == {RECONSTRUCTION_EXTENSION: marker} + assert schema["graph_sha256"] == hashlib.sha256(declaration).hexdigest() + assert marker["protocol"] == RECONSTRUCTION_PROTOCOL + assert marker["reconstructed"] is True + assert marker["basis"] == DECLARATION_ONLY + assert marker["source_revision"] == { + "git_commit": RECORDED_COMMIT, + "git_dirty": False, + "origin": "recorded", + } + # Inputs are listed once each, in name order, whatever order they came in. + assert [item["name"] for item in marker["inputs"]] == [ + "graph.json", + "rowwise_candidate_manifest.json", + ] + assert marker["scope"].strip() + document = orrery_document_from_schema(schema) + carried = document["metadata"]["microcosm"]["extensions"][RECONSTRUCTION_EXTENSION] + assert carried == marker + + +def test_marker_records_the_reconstructing_code_apart_from_the_source_revision(): + declaration = recorded_declaration() + + marker = marker_for(declaration) + unknown = marker_for( + declaration, + reconstruction_revision={"git_commit": None, "git_dirty": None}, + ) + + # The declaration's bytes come from the source revision; the compiled + # sections and the Orrery projection come from the reconstructing code. + assert marker["source_revision"]["git_commit"] == RECORDED_COMMIT + assert marker["reconstruction_revision"] == RECONSTRUCTING_REVISION + assert unknown["reconstruction_revision"] == {"git_commit": None, "git_dirty": None} + assert "reconstruction_revision" in marker["scope"] + schema = reconstructed_schema(declaration, marker) + assert validate_reconstruction(schema)["reconstruction_revision"] == ( + RECONSTRUCTING_REVISION + ) + + +def run_git(checkout, *arguments) -> str: + return subprocess.run( + [ + "git", + "-C", + str(checkout), + "-c", + "user.name=Reconstruction fixture", + "-c", + "user.email=fixture@example.invalid", + "-c", + "commit.gpgsign=false", + *arguments, + ], + check=True, + capture_output=True, + text=True, + ).stdout.strip() + + +def test_code_revision_names_the_checkout_that_tracks_the_file(tmp_path): + checkout = tmp_path / "checkout" + checkout.mkdir() + run_git(checkout, "init", "--quiet") + reconstructing_code = checkout / "reconstruct.py" + reconstructing_code.write_text("VERSION = 1\n") + run_git(checkout, "add", "reconstruct.py") + run_git(checkout, "commit", "--quiet", "-m", "Add the reconstructing code") + head = run_git(checkout, "rev-parse", "HEAD") + + assert code_revision(reconstructing_code) == { + "git_commit": head, + "git_dirty": False, + } + reconstructing_code.write_text("VERSION = 2\n") + assert code_revision(reconstructing_code) == {"git_commit": head, "git_dirty": True} + # A file no checkout tracks (an installed copy that happens to sit inside + # a checkout, or one outside any) identifies no revision. + untracked_copy = checkout / "installed.py" + untracked_copy.write_text("VERSION = 2\n") + outside_any_checkout = tmp_path / "outside.py" + outside_any_checkout.write_text("VERSION = 2\n") + unknown = {"git_commit": None, "git_dirty": None} + assert code_revision(untracked_copy) == unknown + assert code_revision(outside_any_checkout) == unknown + + +def _checkout_tracking(checkout, files: dict[str, str]) -> str: + """Commit ``files`` to a new repository at ``checkout``; return its HEAD.""" + + checkout.mkdir() + run_git(checkout, "init", "--quiet") + for name, text in files.items(): + (checkout / name).write_text(text) + run_git(checkout, "add", *files) + run_git(checkout, "commit", "--quiet", "-m", "Add the fixture files") + return run_git(checkout, "rev-parse", "HEAD") + + +def test_code_revision_ignores_git_variables_that_select_another_checkout( + tmp_path, monkeypatch +): + reconstructing = tmp_path / "reconstructing" + reconstructing_head = _checkout_tracking( + reconstructing, + {"reconstruct.py": "VERSION = 1\n", "only_here.py": "VERSION = 1\n"}, + ) + other = tmp_path / "other" + other_head = _checkout_tracking(other, {"reconstruct.py": "VERSION = 2\n"}) + assert other_head != reconstructing_head + # An outer git command, a hook or a caller's shell can export these, and + # each one points git at the other checkout. + monkeypatch.setenv("GIT_DIR", str(other / ".git")) + monkeypatch.setenv("GIT_WORK_TREE", str(other)) + monkeypatch.setenv("GIT_INDEX_FILE", str(other / ".git" / "index")) + monkeypatch.setenv("GIT_COMMON_DIR", str(other / ".git")) + + expected = {"git_commit": reconstructing_head, "git_dirty": False} + # A file both checkouts track by the same name ... + assert code_revision(reconstructing / "reconstruct.py") == expected + # ... and one only the reconstructing checkout tracks. + assert code_revision(reconstructing / "only_here.py") == expected + + +def test_reconstructed_document_claims_no_execution(): + declaration = recorded_declaration() + + document = orrery_document_from_schema( + reconstructed_schema(declaration, marker_for(declaration)) + ) + + assert not {"activities", "artifacts", "receipts"} & document.keys() + assert all("statuses" not in node for node in document["nodes"]) + assert "execution" not in document["metadata"] + assert sum(node["kind"] == "operation" for node in document["nodes"]) == len( + compiled_graph().graph.nodes + ) + + +def test_declaration_that_does_not_reproduce_its_recorded_bytes_is_refused(): + # The same graph, re-indented: it parses, but the current serializer would + # not write these bytes, so the projection would not be the recorded file. + reformatted = json.dumps(json.loads(recorded_declaration()), indent=2).encode() + + with pytest.raises(ValueError, match="does not reproduce its recorded bytes"): + reconstructed_schema(reformatted, marker_for(reformatted)) + + +@pytest.mark.parametrize( + "declaration", + [ + b"[]", + b'"graph"', + # A graph object whose node list has the wrong type. + b'{"country":"uk","nodes":{},"sources":[]}', + ], + ids=["array", "string", "node_list_of_wrong_type"], +) +def test_declaration_that_is_not_a_graph_is_refused(declaration): + with pytest.raises(ValueError, match="recorded declaration is not a graph"): + reconstructed_schema(declaration, marker_for(declaration)) + + +def test_marker_must_name_the_declaration_it_was_built_from(): + other_declaration = graph_to_json(compiled_default_population_graph().graph).encode( + "utf-8" + ) + + with pytest.raises(ValueError, match="does not name this declaration"): + reconstructed_schema(other_declaration, marker_for(recorded_declaration())) + + +def _short_commit(marker: dict) -> None: + marker["source_revision"]["git_commit"] = "0123456" + + +def _uppercase_commit(marker: dict) -> None: + marker["source_revision"]["git_commit"] = RECORDED_COMMIT.upper() + + +def _guessed_origin(marker: dict) -> None: + marker["source_revision"]["origin"] = "guessed" + + +def _textual_dirty_flag(marker: dict) -> None: + marker["source_revision"]["git_dirty"] = "no" + + +def _no_declaration_input(marker: dict) -> None: + for item in marker["inputs"]: + item["role"] = "code_pin" + + +def _two_declaration_inputs(marker: dict) -> None: + for item in marker["inputs"]: + item["role"] = "declaration" + + +def _duplicate_input_names(marker: dict) -> None: + marker["inputs"][1]["name"] = marker["inputs"][0]["name"] + + +def _unsorted_inputs(marker: dict) -> None: + marker["inputs"].reverse() + + +def _nested_input_path(marker: dict) -> None: + marker["inputs"][1]["name"] = "../rowwise_candidate_manifest.json" + + +def _unknown_verification(marker: dict) -> None: + marker["inputs"][1]["verification"] = "trusted" + + +def _negative_input_size(marker: dict) -> None: + marker["inputs"][1]["bytes"] = -1 + + +def _unknown_marker_field(marker: dict) -> None: + marker["recorded_execution"] = True + + +def _recorded_basis(marker: dict) -> None: + marker["basis"] = "recorded_execution" + + +def _not_reconstructed(marker: dict) -> None: + marker["reconstructed"] = False + + +def _other_protocol(marker: dict) -> None: + marker["protocol"] = "microcosm.graph.reconstruction.v0" + + +def _no_reconstruction_revision(marker: dict) -> None: + del marker["reconstruction_revision"] + + +def _short_reconstruction_commit(marker: dict) -> None: + marker["reconstruction_revision"]["git_commit"] = "89abcde" + + +def _textual_reconstruction_dirty_flag(marker: dict) -> None: + marker["reconstruction_revision"]["git_dirty"] = "no" + + +def _dirty_flag_without_a_reconstruction_commit(marker: dict) -> None: + marker["reconstruction_revision"] = {"git_commit": None, "git_dirty": False} + + +def _reconstruction_revision_with_an_origin(marker: dict) -> None: + marker["reconstruction_revision"]["origin"] = "recorded" + + +@pytest.mark.parametrize( + "corrupt", + [ + _short_commit, + _uppercase_commit, + _guessed_origin, + _textual_dirty_flag, + _no_declaration_input, + _two_declaration_inputs, + _duplicate_input_names, + _unsorted_inputs, + _nested_input_path, + _unknown_verification, + _negative_input_size, + _unknown_marker_field, + _recorded_basis, + _not_reconstructed, + _other_protocol, + _no_reconstruction_revision, + _short_reconstruction_commit, + _textual_reconstruction_dirty_flag, + _dirty_flag_without_a_reconstruction_commit, + _reconstruction_revision_with_an_origin, + ], + ids=lambda corrupt: corrupt.__name__.strip("_"), +) +def test_marker_rejects_malformed_provenance(corrupt): + declaration = recorded_declaration() + marker = copy.deepcopy(marker_for(declaration)) + corrupt(marker) + + with pytest.raises(ValueError, match="Graph reconstruction"): + reconstructed_schema(declaration, marker) + + +def test_saved_reconstructed_schema_keeps_its_marker_when_reprojected(): + declaration = recorded_declaration() + marker = marker_for(declaration, git_dirty=None, revision_origin="caller_asserted") + schema = reconstructed_schema(declaration, marker) + + # A later projection (a collector re-projecting stored inputs, say) reads + # only the saved graph.schema.json bytes; the marker travels inside them. + restored = validate_graph_schema(json.loads(canonical_json(schema))) + + assert validate_reconstruction(restored) == marker + assert orrery_json_from_schema(restored) == orrery_json_from_schema(schema) + assert validate_reconstruction(graph_schema(compiled_graph())) is None + relabelled = graph_schema( + compiled_default_population_graph(), + extensions={RECONSTRUCTION_EXTENSION: marker}, + ) + with pytest.raises(ValueError, match="does not name this declaration"): + validate_reconstruction(relabelled) + + +def test_public_orrery_parser_accepts_reconstructed_document(): + declaration = recorded_declaration() + payload = orrery_json_from_schema( + reconstructed_schema(declaration, marker_for(declaration)), + title="INVENTED · reconstructed declaration", + ) + + result = subprocess.run( + ["node", str(_RECONSTRUCTION_VERIFY)], + input=payload, + text=True, + capture_output=True, + check=False, + ) + + assert result.returncode == 0, result.stderr + assert result.stdout == "Orrery accepted the reconstructed declaration.\n" diff --git a/tools/orrery-contract/verify-reconstruction.mjs b/tools/orrery-contract/verify-reconstruction.mjs new file mode 100644 index 0000000000..e21447651a --- /dev/null +++ b/tools/orrery-contract/verify-reconstruction.mjs @@ -0,0 +1,23 @@ +import assert from 'node:assert/strict'; +import { parseGraphDocument } from '@axiom-foundation/orrery'; + +const chunks = []; +for await (const chunk of process.stdin) chunks.push(chunk); +const document = parseGraphDocument(JSON.parse(Buffer.concat(chunks).toString('utf8'))); +const marker = document.metadata.microcosm.extensions['microcosm.reconstruction']; +assert.equal(marker.protocol, 'microcosm.graph.reconstruction.v1'); +assert.equal(marker.reconstructed, true); +assert.equal(marker.basis, 'declaration_only'); +assert.match(marker.source_revision.git_commit, /^[0-9a-f]{40}([0-9a-f]{24})?$/); +assert.equal(marker.inputs.filter(input => input.role === 'declaration').length, 1); +assert.equal( + marker.inputs.find(input => input.role === 'declaration').sha256, + document.metadata.microcosm.graph_sha256, +); +assert.ok(document.nodes.some(node => node.kind === 'operation')); +// A declaration-only reconstruction never presents an operation as executed. +assert.ok(!document.nodes.some(node => node.statuses?.length)); +assert.ok(!document.activities?.length); +assert.ok(!document.artifacts?.length); +assert.ok(!('execution' in document.metadata)); +process.stdout.write('Orrery accepted the reconstructed declaration.\n');