perf(control-plane): streamed integrity snapshots, planner snapshot chain, package-repository memo (#227 E1#1/#4)

CHANGELOG.md

@@ -54,6 +54,12 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Version
 
 - **New payload binaries are linked with debug information stripped.** `-s -w` joins `ARTIFACT_LDFLAGS` in `scripts/build-agent-handoff-session-start.sh` ([#228](https://github.com/L3DigitalNet/project-standards/issues/228) lever 1), taking the launcher from 3,906,258 to 2,646,142 bytes (−32.3%). `.gopclntab` is retained, so panic traces still carry function names and line numbers; DWARF inspection of the shipped file is recovered by rebuilding from the same script. Published payload bytes stay unstripped, so a size step between a retained version and its successor is expected. Consumers running a `pre-commit` `check-added-large-files` guard still need the documented exemption for the hook path — the launcher remains far above a typical `--maxkb=1024`.
 
+- **The provider integrity guard no longer retains the bytes it only compares, and repeated package-repository builds in one process are memoized** ([#227](https://github.com/L3DigitalNet/project-standards/issues/227)). Every provider invocation captures a whole-repository snapshot before and after the call to prove the provider changed no declared live path; those two captures read each declared target in full, held every byte in memory, and hashed each one twice. They now stream: the precondition hash is seeded from the pre-read mode and fed chunk by chunk, no bytes are retained, and the content digest is computed only when a caller asked for content. `RepositorySnapshot.capture` gained `retain_content` for that, and `assert_current` uses it too. On this repository `project-standards reconcile --check` falls from 17.5 s to 13.7 s (median of three) and peak RSS from 185 MB to 117 MB. `build_package_repository` additionally serves a repeated build of an unchanged tree from an in-process memo keyed on the size and mtime of every file under `standards/` and `catalogs/` — a cache over parse results, not an integrity guard, which is why `stat` metadata suffices there while the control plane's snapshot still reads and hashes bytes. A second build of this repository costs 0.05 s instead of 2.75 s; the fingerprint itself costs 0.05 s, which a single-build command now pays once.
+- **`provider_snapshot_chain()` lets a planning pass share one integrity snapshot between consecutive providers.** Inside that window the AFTER snapshot of one provider becomes the BEFORE snapshot of the next provider declaring the identical target set, so N invocations cost N+1 captures instead of 2N (measured on this repository: 201 captures to 102, and 12.5 s to 10.4 s in-process). Every AFTER capture is still a fresh full read, so a provider that changes a declared live path is still refused with `CP-PROVIDER-INTEGRITY`. The window is opt-in because it carries an obligation: the caller promises that nothing but the providers themselves touches a declared path between two invocations, so publication, recovery, or an operator edit between two plans in a long-lived process is never misattributed to the next provider.
+
+- **The provider integrity guard no longer retains the bytes it only compares, and repeated package-repository builds in one process are memoized** ([#227](https://github.com/L3DigitalNet/project-standards/issues/227)). Every provider invocation captures a whole-repository snapshot before and after the call to prove the provider changed no declared live path; those two captures read each declared target in full, held every byte in memory, and hashed each one twice. They now stream: the precondition hash is seeded from the pre-read mode and fed chunk by chunk, no bytes are retained, and the content digest is computed only when a caller asked for content. `RepositorySnapshot.capture` gained `retain_content` for that, and `assert_current` uses it too. Combined with the snapshot chain below, `project-standards reconcile --check` falls from 17.5 s to 9.5 s on this repository (median of three) and peak RSS from 185 MB to 117 MB. `build_package_repository` additionally serves a repeated build of an unchanged tree from an in-process memo keyed on the size and mtime of every file under `standards/` and `catalogs/` — a cache over parse results, not an integrity guard, which is why `stat` metadata suffices there while the control plane's snapshot still reads and hashes bytes. A second build of this repository costs 0.05 s instead of 2.75 s; the fingerprint itself costs 0.05 s, which a single-build command now pays once.
+- **`plan_reconciliation` now shares one integrity snapshot between consecutive providers.** Inside the new `provider_snapshot_chain()` window the AFTER snapshot of one provider becomes the BEFORE snapshot of the next provider declaring the identical target set, so N invocations cost N+1 captures instead of 2N — 201 captures to 102 on this repository. Every AFTER capture is still a fresh full read, so a provider that changes a declared live path is still refused with `CP-PROVIDER-INTEGRITY`. The window is opt-in and planning is the only pass that enters it: it invokes every provider and writes nothing itself, so between two invocations only a provider could have touched a declared path. Publication and the executor's post-publication verification providers stay outside, because a window spanning a write would report the control plane's own bytes as the next provider's violation.
+
 ### Fixed
 
 - **`Agent Handoff 1.17` stops an untrusted checkout from executing a command during session start.** The `session-start` launcher ran its Git reads with the full inherited process environment and no configuration isolation, so a repository-local or ancestor `core.fsmonitor` setting named a hook that Git ran — unconditionally, before the operator had seen anything — as soon as that checkout was opened as a session-start target ([#235](https://github.com/L3DigitalNet/project-standards/issues/235)). Every read now runs with an explicit minimal environment (`PATH` and `HOME` only, so no `GIT_DIR`, `GIT_WORK_TREE`, or `GIT_CONFIG_*` value from the harness can redirect it) and passes `-c core.fsmonitor=`, which outranks every configuration file, plus `--no-optional-locks` so the read cannot race a concurrent write. The injected session context is byte-identical to 1.16; reconcile replaces the installed hook because its digest moved. Catalog 5 promotes `agent-handoff@1.17` and retains 1.16.

src/project_standards/control_plane/planner.py

@@ -77,6 +77,7 @@
     ProviderInvocation,
     ProviderResult,
     invoke_provider,
+    provider_snapshot_chain,
     resolve_referenced_inputs,
 )
 from project_standards.control_plane.resolution import (
@@ -2894,6 +2895,19 @@ def _alias_analysis(
 
 def plan_reconciliation(request: PlannerRequest) -> ReconciliationPlan:
     """Build one deterministic, complete, and read-only reconciliation plan."""
+    # Planning is the one pass that satisfies the snapshot chain's obligation:
+    # it invokes every render, validate, and transform provider in turn and
+    # writes nothing itself, so between two invocations only a provider could
+    # have touched a declared path — which is exactly what the chained BEFORE
+    # snapshot then proves it did not. Publication and the executor's
+    # post-publication verification providers stay outside the window; a window
+    # spanning a write would report the control plane's own bytes as a
+    # CP-PROVIDER-INTEGRITY violation by the next provider.
+    with provider_snapshot_chain():
+        return _plan_reconciliation(request)
+
+
+def _plan_reconciliation(request: PlannerRequest) -> ReconciliationPlan:
     original_request = request
     resolution = resolve_packages(request.resolution)
     payloads = _payload_map(request.payloads)

src/project_standards/control_plane/providers.py

@@ -10,8 +10,9 @@
 import stat
 import sys
 import tempfile
-from collections.abc import Iterator, Mapping, Sequence
-from contextlib import redirect_stderr, redirect_stdout
+import threading
+from collections.abc import Generator, Iterator, Mapping, Sequence
+from contextlib import contextmanager, redirect_stderr, redirect_stdout
 from dataclasses import dataclass, replace
 from pathlib import Path
 from types import MappingProxyType
@@ -44,7 +45,10 @@
     safe_failure_detail,
 )
 from project_standards.control_plane.schemas import MutationPlanSchema, ProviderInputSchema
-from project_standards.control_plane.snapshot import RepositorySnapshot
+from project_standards.control_plane.snapshot import (
+    RepositorySnapshot,
+    canonical_targets,
+)
 from project_standards.package_contract.paths import (
     PackageVersion,
     SafeRelativePath,
@@ -488,9 +492,83 @@ def _declared_snapshot_paths(snapshots: JsonObject) -> tuple[SafeRelativePath, .
         raise ControlPlaneError("provider snapshot declares an invalid repository path") from exc
 
 
+@dataclass(frozen=True, slots=True)
+class _ChainedSnapshot:
+    """The AFTER snapshot of the most recent provider, offered to the next one."""
+
+    root: Path
+    targets: tuple[SafeRelativePath, ...]
+    snapshot: RepositorySnapshot
+
+
+# Chaining state is per thread: two threads planning against one repository see
+# each other's writes, and a slot armed by one of them describes a tree the
+# other never observed.
+_chain_state = threading.local()
+
+
+def _chain_slot() -> _ChainedSnapshot | None:
+    return cast("_ChainedSnapshot | None", getattr(_chain_state, "slot", None))
+
+
+@contextmanager
+def provider_snapshot_chain() -> Generator[None]:
+    """Allow consecutive provider invocations to share one integrity snapshot.
+
+    Inside this window, the AFTER snapshot the CP-PROVIDER-INTEGRITY guard takes
+    for one provider becomes the BEFORE snapshot of the next provider that
+    declares the identical target set, so N invocations cost N+1 captures rather
+    than 2N. Every declared path is still read in full once per invocation and
+    every AFTER capture is still a fresh read, so a provider that changes a
+    declared live path is caught exactly as before.
+
+    The window carries one obligation, and it is the reason chaining is opt-in
+    rather than always on: the caller promises that nothing but the providers
+    themselves touches a declared path between two invocations. Publication,
+    recovery, a spec fix that rewrites the file it just linted, or an operator
+    editing the tree between two plans in a long-lived process would otherwise
+    be attributed to the next provider as a CP-PROVIDER-INTEGRITY violation.
+    Open the window around one planning pass — never around a pass that writes,
+    and never across a repository mutation the process performed itself.
+
+    Nested windows are permitted and share the single slot; leaving any window
+    discards it, so a chained snapshot never outlives the pass that vouched
+    for it — including the outer pass, which resumes with no slot rather than
+    with the reading it held before the inner pass ran.
+    """
+    previous_active = getattr(_chain_state, "active", False)
+    _chain_state.active = True
+    _chain_state.slot = None
+    try:
+        yield
+    finally:
+        _chain_state.active = previous_active
+        _chain_state.slot = None
+
+
+def _arm_snapshot_chain(after: RepositorySnapshot) -> None:
+    if getattr(_chain_state, "active", False):
+        _chain_state.slot = _ChainedSnapshot(after.root, after.targets, after)
+
+
+def _capture_declared_paths(
+    root: Path,
+    targets: tuple[SafeRelativePath, ...],
+) -> RepositorySnapshot:
+    """Take this invocation's BEFORE snapshot, reusing a chained one when offered."""
+    ordered = canonical_targets(targets)
+    slot = _chain_slot()
+    # Consumed at most once: the reference for the next invocation is whatever
+    # this invocation's own AFTER capture observes, never an older reading.
+    _chain_state.slot = None
+    if slot is not None and slot.root == root and slot.targets == ordered:
+        return slot.snapshot
+    return RepositorySnapshot.capture(root, ordered, retain_content=False)
+
+
 def _assert_declared_paths_unchanged(before: RepositorySnapshot) -> None:
     try:
-        after = RepositorySnapshot.capture(before.root, before.targets)
+        after = RepositorySnapshot.capture(before.root, before.targets, retain_content=False)
     except ControlPlaneError as exc:
         raise ControlPlaneError(
             "CP-PROVIDER-INTEGRITY: provider made a declared live path unsafe"
@@ -500,6 +578,9 @@ def _assert_declared_paths_unchanged(before: RepositorySnapshot) -> None:
             raise ControlPlaneError(
                 f"CP-PROVIDER-INTEGRITY: provider changed live path {expected.path.original}"
             )
+    # Armed only after the comparison succeeded: a snapshot that already failed
+    # the guard must never become the next invocation's reference.
+    _arm_snapshot_chain(after)
 
 
 def _output_notice(stdout: _OutputSink, stderr: _OutputSink) -> str | None:
@@ -921,7 +1002,7 @@ def _invoke_command_provider(
             extensions=payload.manifest.extensions,
         ),
     )
-    before = RepositorySnapshot.capture(
+    before = _capture_declared_paths(
         root,
         _declared_snapshot_paths(effective_invocation.snapshots),
     )
@@ -1013,7 +1094,7 @@ def invoke_provider_in_child(invocation: ProviderInvocation) -> ProviderResult:
     _validate_json_schema(prepared.input_schema, input_value, kind="input")
     frozen_input = _deep_freeze(input_value)
     frozen_resources = MappingProxyType(prepared.resources)
-    before = RepositorySnapshot.capture(
+    before = _capture_declared_paths(
         root,
         _declared_snapshot_paths(effective_invocation.snapshots),
     )
@@ -1078,7 +1159,7 @@ def invoke_provider(invocation: ProviderInvocation) -> ProviderResult:
             extensions=payload.manifest.extensions,
         ),
     )
-    before = RepositorySnapshot.capture(
+    before = _capture_declared_paths(
         root,
         _declared_snapshot_paths(effective_invocation.snapshots),
     )