Move review TSVs to LFS and guard large Git blobs

.gitattributes

@@ -8,3 +8,6 @@ knowledge/** text eol=lf
 # Large complete Machine KB files; hydrate before reading or building.
 knowledge/claims/claims.jsonl filter=lfs diff=lfs merge=lfs -text
 knowledge/inputs/claim-records.json filter=lfs diff=lfs merge=lfs -text
+
+# Preserve review tables byte-for-byte; hydrate before examining their evidence.
+project/review/**/*.tsv filter=lfs diff=lfs merge=lfs -text

.github/workflows/encyclopedia-compliance.yml

@@ -18,6 +18,34 @@ permissions:
   statuses: read
 
 jobs:
+  git-storage:
+    name: Git storage policy
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Check out Git objects and comparison history
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+          lfs: false
+
+      - uses: actions/setup-python@v5
+        with: {python-version: "3.12"}
+
+      - name: Test Git storage guard
+        run: python -m unittest discover -s project/review/checks/tests -p test_git_storage.py -v
+
+      - name: Reject oversized new Git blobs and ordinary review TSVs
+        env:
+          STORAGE_BASE: ${{ github.event.pull_request.base.sha || github.event.before }}
+        shell: bash
+        run: |
+          args=(--head HEAD)
+          if [[ -n "$STORAGE_BASE" && "$STORAGE_BASE" != 0000000000000000000000000000000000000000 ]]; then
+            args+=(--base "$STORAGE_BASE")
+          fi
+          python project/review/checks/check_git_storage.py "${args[@]}"
+
   mechanical-compliance:
     name: Mechanical ESG and ECV compliance
     runs-on: ubuntu-latest

project/review/checks/README.md

@@ -26,6 +26,41 @@ also emitting reproducible evidence probes.
 
 These checks are suitable as required continuous-integration status checks.
 
+## Git storage policy
+
+`check_git_storage.py` rejects ordinary Git blobs larger than 5 MiB and requires
+every tracked TSV under `project/review/` to be stored as a canonical Git LFS
+pointer. It examines Git objects, so hydrated LFS payloads may exceed the limit.
+The staged index is checked by default; changing only the working file or
+adding an LFS attribute without staging the conversion cannot satisfy the guard.
+
+The reviewed [legacy baseline](git-storage-baseline.json) records nine existing
+large KB files at their exact paths and Git object IDs. These unchanged bytes
+are retained to keep this migration bounded. A new version or copy at another
+path receives no exemption and must use appropriate storage. Review changes to
+this baseline as storage-policy changes, not routine generated updates.
+
+After staging changes, run:
+
+```text
+python project/review/checks/check_git_storage.py --base origin/main
+python -m unittest discover -s project/review/checks/tests -p test_git_storage.py -v
+```
+
+For a committed candidate, add `--head HEAD`. CI checks both the final tree and
+new objects reachable between the PR base and its merge checkout, or between
+the previous and current pushed commit. Adding a large blob and deleting it in
+a later commit still fails. Existing history is not rescanned as new material.
+The standalone CI job needs Git objects and comparison history, not LFS payloads.
+It does not rewrite history, change tags, or verify remote LFS payload availability.
+
+Use Git LFS for review tables and other large public text artifacts. Retain
+private and manufacturer archives in their established external storage. Before
+examining review TSV evidence, hydrate it with `git lfs pull`; an LFS pointer is
+not the table. The TSV migration preserves paths and exact bytes, including
+line endings. Existing large historical blobs remain reachable; new oversized
+ordinary blobs are blocked without changing published history or tags.
+
 ## Advisory review checks
 
 `audit_epistemic_drift.py` searches for candidates that may indicate
@@ -84,6 +119,7 @@ ruleset or branch protection to require these status checks before merging:
 
 - `Mechanical ESG and ECV compliance`
 - `Source integrity and reproducibility`
+- `Git storage policy`
 
 `Advisory epistemic review` should run on every pull request but must not be a
 required factual gate. Its findings require evidence-aware adjudication and can

project/review/checks/check_git_storage.py

@@ -0,0 +1,131 @@
+#!/usr/bin/env python3
+"""Reject large ordinary Git blobs without rewriting or rescanning old history."""
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+import re
+import subprocess
+import sys
+
+MAX_BLOB_BYTES = 5 * 1024 * 1024
+BASELINE_PATH = "project/review/checks/git-storage-baseline.json"
+POINTER = re.compile(
+    rb"version https://git-lfs.github.com/spec/v1\n"
+    rb"oid sha256:[0-9a-f]{64}\nsize (?:0|[1-9][0-9]*)\n"
+)
+
+
+def git(root: Path, *args: str, data: bytes | None = None) -> bytes:
+    return subprocess.run(["git", *args], cwd=root, input=data,
+                          stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=True).stdout
+
+
+def commit(root: Path, ref: str) -> str:
+    return git(root, "rev-parse", "--verify", "--end-of-options", ref + "^{commit}").decode().strip()
+
+
+def snapshot(root: Path, head: str | None) -> dict[str, str]:
+    command = ("ls-tree", "-r", "-z", "--full-tree", head) if head else ("ls-files", "--stage", "-z")
+    result = {}
+    for row in git(root, *command).split(b"\0"):
+        if not row:
+            continue
+        metadata, path = row.split(b"\t", 1)
+        mode, middle, last = metadata.split()
+        if not head and last != b"0":
+            raise ValueError("Unmerged index entries must be resolved before checking storage")
+        if mode == b"160000":  # A submodule commit is not a file blob.
+            continue
+        result[path.decode("utf-8", "surrogateescape")] = (last if head else middle).decode()
+    return result
+
+
+def object_sizes(root: Path, oids: set[str]) -> dict[str, int]:
+    if not oids:
+        return {}
+    output = git(root, "cat-file", "--batch-check=%(objectname) %(objecttype) %(objectsize)",
+                 data=("\n".join(sorted(oids)) + "\n").encode())
+    result = {}
+    for row in output.decode().splitlines():
+        oid, kind, size = row.split()
+        if kind == "blob":
+            result[oid] = int(size)
+    return result
+
+
+def legacy_blobs(root: Path, files: dict[str, str], sizes: dict[str, int]) -> dict[str, str]:
+    oid = files.get(BASELINE_PATH)
+    if oid is None or sizes[oid] > 64 * 1024:
+        raise ValueError("Missing or oversized reviewed Git storage baseline")
+    value = json.loads(git(root, "cat-file", "blob", oid))
+    if (set(value) != {"format_version", "baseline_commit", "reason", "ordinary_blobs"}
+            or value["format_version"] != 1
+            or not re.fullmatch(r"[0-9a-f]{40}", value["baseline_commit"])
+            or not isinstance(value["reason"], str)
+            or not isinstance(value["ordinary_blobs"], list)):
+        raise ValueError("Invalid reviewed Git storage baseline")
+    result = {}
+    for item in value["ordinary_blobs"]:
+        if (set(item) != {"path", "git_blob", "size_bytes"}
+                or not isinstance(item["path"], str) or item["path"] in result
+                or item["path"].startswith("/") or ".." in item["path"].split("/")
+                or not re.fullmatch(r"[0-9a-f]{40}", item["git_blob"])
+                or type(item["size_bytes"]) is not int or item["size_bytes"] <= MAX_BLOB_BYTES):
+            raise ValueError("Invalid or duplicated legacy Git blob exception")
+        if object_sizes(root, {item["git_blob"]}).get(item["git_blob"]) != item["size_bytes"]:
+            raise ValueError("Legacy Git blob exception has an incorrect size")
+        result[item["path"]] = item["git_blob"]
+    return result
+
+
+def check_storage(root: Path, head: str | None = None, base: str | None = None) -> list[str]:
+    revision = commit(root, head) if head else None
+    files = snapshot(root, revision)
+    sizes = object_sizes(root, set(files.values()))
+    legacy = legacy_blobs(root, files, sizes)
+    errors = []
+    for path, oid in sorted(files.items()):
+        if sizes[oid] > MAX_BLOB_BYTES and legacy.get(path) != oid:
+            errors.append(f"{path!r}: ordinary Git blob is {sizes[oid]} bytes; limit is {MAX_BLOB_BYTES}")
+        if path.startswith("project/review/") and path.endswith(".tsv"):
+            if sizes[oid] > 1024 or not POINTER.fullmatch(git(root, "cat-file", "blob", oid)):
+                errors.append(f"{path!r}: review TSV must be a canonical Git LFS pointer in Git")
+    if base:
+        start, end = commit(root, base), revision or commit(root, "HEAD")
+        try:
+            git(root, "merge-base", "--is-ancestor", start, end)
+        except subprocess.CalledProcessError as error:
+            raise ValueError("Storage comparison base must be an ancestor of the checked commit") from error
+        oids = set(git(root, "rev-list", "--objects", "--no-object-names", end, "^" + start).decode().splitlines())
+        for oid, size in sorted(object_sizes(root, oids).items()):
+            if size > MAX_BLOB_BYTES:
+                errors.append(f"New history blob {oid}: {size} bytes exceeds {MAX_BLOB_BYTES}; "
+                              "removing it in a later commit does not remove its storage cost")
+    return errors
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--head", help="Check this committed tree instead of the staged Git index")
+    parser.add_argument("--base", help="Also reject oversized new blobs in this ancestor-to-head history range")
+    args = parser.parse_args()
+    try:
+        root = Path(subprocess.check_output(["git", "rev-parse", "--show-toplevel"], text=True).strip())
+        errors = check_storage(root, args.head, args.base)
+    except (ValueError, KeyError, TypeError, subprocess.CalledProcessError) as error:
+        print(f"Git storage check failed: {error}", file=sys.stderr)
+        return 1
+    if errors:
+        for error in errors:
+            print(error, file=sys.stderr)
+        print("Use Git LFS for large public text artifacts; keep private/vendor archives in their established storage.",
+              file=sys.stderr)
+        return 1
+    print("Git storage check passed: size limit, exact legacy exceptions and review TSV pointers")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

project/review/checks/git-storage-baseline.json

@@ -0,0 +1,52 @@
+{
+  "format_version": 1,
+  "baseline_commit": "7102cffdcee8f3fd0f66aaca956472143508133f",
+  "reason": "Existing large ordinary KB blobs retained unchanged from current main before this storage policy is introduced. New versions must use LFS; this is not a path exemption or permission to rewrite history.",
+  "ordinary_blobs": [
+    {
+      "path": "knowledge/id-registry.json",
+      "git_blob": "f7dcfc8b285f086cd23c732bd2676e570c48324b",
+      "size_bytes": 9507857
+    },
+    {
+      "path": "knowledge/inputs/claim-context.json",
+      "git_blob": "e78c52731f7a02b0f1f2c93f5919531dbe67a1c3",
+      "size_bytes": 37977091
+    },
+    {
+      "path": "knowledge/inputs/device-unit-review.json",
+      "git_blob": "e86dbe68513a4d7b38f3646248852f4965ba08d0",
+      "size_bytes": 26525024
+    },
+    {
+      "path": "knowledge/inputs/id-registry.json",
+      "git_blob": "f7dcfc8b285f086cd23c732bd2676e570c48324b",
+      "size_bytes": 9507857
+    },
+    {
+      "path": "knowledge/inputs/reference-records.json",
+      "git_blob": "bc61870a8d858fd57caa0459813466ad08008eae",
+      "size_bytes": 26031286
+    },
+    {
+      "path": "knowledge/llm/llm-corpus.md",
+      "git_blob": "f05707cee5b36699a3ce60b7eccf52a8136f1420",
+      "size_bytes": 10414882
+    },
+    {
+      "path": "knowledge/reference/entities.jsonl",
+      "git_blob": "8beedbfb34513a7c151a52110f36137d067ff94b",
+      "size_bytes": 19505727
+    },
+    {
+      "path": "knowledge/reference/relationships.jsonl",
+      "git_blob": "e2197eea6a18b8dc8e95223fe03b49ac82438d28",
+      "size_bytes": 20202064
+    },
+    {
+      "path": "knowledge/retrieval/chunks.jsonl",
+      "git_blob": "7e9315ce1c124592c9e9cc1d32a2bc03662503bf",
+      "size_bytes": 18820617
+    }
+  ]
+}