Verify retained tests across two committed project revisions

README.md

@@ -179,6 +179,14 @@ The default model IDs come from the Token Factory docs' August 2026 deprecation
 
 ## Eval (scripted stub)
 
+To apply the failed-before/passed-after check to your own committed project,
+use `testpilot regression --repo PROJECT --report REPORT --before REF --after REF --out NEW_DIRECTORY`.
+It executes the same retained tests on both revisions without model calls or
+checkout changes, and requires a matched native failure-to-pass witness.
+Timeouts, collection errors and mismatched generated cases remain inconclusive.
+See [paired revision regression checks](docs/REGRESSION.md) for interpreter setup,
+source limits and the complete result contract.
+
 Each case has a repo with one injected bug, the PR diff, a hidden `fix/` used only as an oracle, and canned model replies. The scripts are written to exercise every loop path: a first-try diagnosis, an import-error repair followed by a diagnosis, a timeout, exhausting the round limit, and weak tests that miss the bug.
 
 * **bug-revealing**: the final tests fail on the buggy repo and pass with `fix/` applied.

docs/REGRESSION.md

@@ -0,0 +1,120 @@
+# Check a retained regression test before and after a fix
+
+`regression` runs the exact tests saved in one TestPilot generation or recheck
+report on two committed versions of your Python project. It answers whether a
+retained test actually failed on the chosen before revision and passed on the
+chosen after revision. It makes zero model calls and does not repair tests or
+apply the proposed patch.
+
+Use your project's existing interpreter with pytest and its dependencies
+installed. Both Git revisions must already be available locally. The command
+does not fetch objects or install dependencies.
+
+```bash
+python -m testpilot regression \
+  --repo /path/to/project \
+  --report /path/to/saved-run/report.json \
+  --before BUGGY_COMMIT \
+  --after FIXED_COMMIT \
+  --python /path/to/project/.venv/bin/python \
+  --timeout 60 \
+  --out /path/to/new-regression-check
+```
+
+`--repo` is the root of a local Git worktree. The output directory must be new,
+outside that repository, and have an existing parent directory. Each pytest
+phase receives the specified positive, finite timeout. The default interpreter
+is the one running TestPilot; a selected virtual environment is preserved.
+
+The two refs are resolved once, before execution, to full commit and tree IDs.
+They need not be ancestors. Dirty files, staged changes, untracked files and the
+current branch do not supply the source being tested. They are left untouched.
+Each revision is materialized in a temporary directory directly from its Git
+blobs, retaining regular-file bytes and executable modes. Git replacement refs,
+archive attributes, smudge filters and caller Git-directory environment settings
+cannot silently substitute the selected source.
+
+The exact saved test bytes are admitted once by the existing retained-test
+reader. Both revision snapshots and test placements are validated before either
+pytest run starts. If a retained path exists in a commit, its bytes must exactly
+match the saved test; different bytes or a file/directory conflict refuse the
+operation. Absent retained paths are inserted only into the disposable runner
+copy. The generation report's previous status is recorded as context, never
+used as evidence that a current execution passed.
+
+## Understand the result
+
+The new directory contains `regression.md` and `regression.json`. The JSON schema
+identifier is `testpilot.regression/1`; it is a paired execution result, separate
+from generation and recheck results.
+
+| Status | Meaning | Exit |
+| --- | --- | --- |
+| `verified_regression` | At least one exact generated case failed before and passed after, and the after run is successful. | 0 |
+| `not_regression` | The comparable generated cases already passed before; an unrelated existing-test failure does not establish a witness. | 1 |
+| `after_failed` | The comparable after execution still fails, including a remaining existing-suite failure. | 1 |
+| `inconclusive` | Execution or generated-case comparability is incomplete. | 1 |
+| Admission/setup/output refusal | A result could not be produced; existing output paths are preserved. | 2 |
+
+Comparison uses each actual native `(generated_file, nodeid)` identity. Every
+retained file must collect at least one generated case; identities must be
+unique and exactly match across phases. Both native runs must finish, provide
+JUnit, return 0 or 1, and contain no native errors. All generated after cases
+must pass. A timeout, collection/import/setup error, missing JUnit, missing file
+collection, changed or duplicate identity, or generated skip is inconclusive.
+Existing-suite skips retain the current native runner's semantics.
+
+`comparison.witnesses` contains the matched failed-before/passed-after cases.
+`case_pairs`, `unmatched_before` and `unmatched_after` make identity differences
+explicit. Both `before.result` and `after.result` retain the native sandbox
+result, case outcomes, generated counts, timeout and JUnit availability. No
+result is inferred from an older report or from exit status alone.
+
+The result also records full requested refs, commit/tree IDs, selected Python,
+source file/byte totals, input-report SHA-256, exact retained test contents and
+their hashes, zero model calls and execution timestamps. `snapshot_sha256`
+binds the admitted original tree inventory: a UTF-8 JSON array sorted by UTF-8
+path bytes; each record has keys in order `path`, `mode`, `blob`, `bytes`;
+`ensure_ascii=True`, separators `(',', ':')`, and no trailing newline. `mode`
+and `blob` are Git strings and `bytes` is an integer. File content is checked
+against each original Git blob identity when materialized.
+
+## Boundaries
+
+This version admits at most 10,000 regular files, 8 MiB per file and 64 MiB per
+revision. Git symlinks, submodules/gitlinks, unavailable objects and unsupported
+paths are refused rather than omitted. Paths must be UTF-8 relative Git paths;
+backslashes, colons, traversal components and `.git` components are unsupported.
+Git must support its `--no-lazy-fetch` option. A dependency that is only present
+in your working checkout will not appear in a committed snapshot.
+
+The established TestPilot sandbox, collection rules, ignored directories and
+generated-case collector remain in force. Coverage is disabled for both phases
+so optional instrumentation does not change the paired test invocation. No
+operating-system or network isolation is added: project tests retain the
+existing local execution boundary and can access ordinary process resources.
+
+A passing pair is evidence about these two revisions and this exact suite. It
+does not establish why a test failed, which intervening change fixed it, the
+completeness of the tests, or whether another environment will behave the same.
+
+## Python API
+
+```python
+from testpilot.regression import RegressionError, check_regression
+
+result = check_regression(
+    "/path/to/project",
+    "/path/to/saved-run/report.json",
+    before="BUGGY_COMMIT",
+    after="FIXED_COMMIT",
+    timeout_s=60.0,
+    python="/path/to/project/.venv/bin/python",
+)
+print(result["status"], result["comparison"]["witnesses"])
+```
+
+The API returns the same dictionary without publishing files. Invalid input,
+Git admission, snapshot preparation and execution setup raise `RegressionError`
+(a `ValueError` subclass). Native test failures and inconclusive executions
+return their honest comparison status instead of raising.

docs/receiving/paired-regression-e82707f2bc62/README.md

@@ -0,0 +1,72 @@
+# Paired committed-revision regression receiving
+
+Source claim: https://github.com/Jacob-Met/testpilot/issues/44
+Owner: estate-e82707f2bc62 / coordination_discovery.
+
+The new command executes one admitted retained suite on two explicit Git
+commits. A result is verified only when a uniquely matched generated case
+actually fails before and passes after, with successful after execution and
+complete comparable generated identities. No saved historical status, model
+response, unrelated existing failure, missing JUnit, timeout, error or skip is
+substituted for this evidence. The public API and limitations are described in
+[REGRESSION.md](../../REGRESSION.md).
+
+## Exact native evidence
+
+The original source is 191cca4e416286a9e1fa4b5d5daf5fc30b936db6. A real CLI
+workflow uses the maintained calc_clamp fixture, synthetic local Git commits,
+and actual generation plus rechecks. The identical four retained cases produce
+three failures and one pass before, then four passes after; both rechecks use
+zero model calls. Original paired invocation refuses as unsupported with exit2
+and no output. All original input/source bytes remain unchanged.
+
+The new command executes that same saved report on those same commits and
+returns verified_regression, exit0, with three exact failed-before/passed-after
+witnesses. Both native count records match the original controls. The original
+report, complete fixture Git metadata, native stdout/stderr/results, source
+capsule and source/input guards are retained in author-evidence.zip.
+
+Fourteen frozen authored methods produce 2 failures and12 missing-module errors
+on original source, then14 passes on the candidate. An explicit inventory sort
+was added to make the documented signature serialization stable; the same14
+methods pass again. These are separate recorded source phases, not a claim that
+the original source implemented the feature. Both logs, exact test and final
+received source are in the62-member verified archive.
+
+Independent receiving was frozen before candidate access. Six scenarios are
+accepted across the documented phases: actual non-ancestor CLI execution with
+raw Git assets/executable modes/archive attributes and dirty-tree preservation;
+reverse after-failure exit1; native report admission; unrelated-suite failure
+exclusion; generated-node identity drift; and admission of both trees before
+execution plus output preservation. The first three candidate successes were
+retained while a receiver-only missing import was corrected. The unchanged
+remaining three then encountered ENOSPC before product execution and passed
+after the receiver compressed its own metadata. The independent README/receipt
+and24-member archive preserve every predecessor. This is not mislabeled as a
+single six-pass successor run. The portable test and literal fixture are added
+to normal native test discovery.
+
+Root independently read the complete module, tests and guide and reversed the
+additive CLI/README spans to the exact received parent. Its source review is
+included; it makes no independent execution claim.
+
+## Current-parent composition and limits
+
+The received production module is Git9d877deed683c0e38d61fcf5c7320d8f5dd672c9,
+SHA256a802c332d4f0adaf8f49306b2686d4f04050325d60a25367af18da3241940b89.
+The exact current parent and newer runtime changes are in current-composition.json.
+Only the new command's CLI and README spans are applied to those current files;
+removing them restores the full current bytes. Existing parser, timeout,
+process custody, recheck, collector, model, history and comparison owners are
+preserved. All other parent leaves remain unchanged by the complete-tree overlay.
+
+The unchanged repository CI must qualify the actual composed PR checkout.
+Hosted qualification and merge are pending when this packet is published;
+neither is inferred from local source presence. No installed service,
+provider/account operation or deployment is claimed.
+
+Local execution used Python3.12.14 with genuine existing pytest9.1.1 through a
+small no-pip environment. Failed storage/setup attempts and the empty failed
+document write remain in the author packet. No successful dependency install
+or broad shared cleanup is claimed. Exact source, fixture and test data are
+synthetic or already public project source; no participant data is used.

docs/receiving/paired-regression-e82707f2bc62/current-composition.json

@@ -0,0 +1,20 @@
+{
+  "parent": "513f17454af378ee042ccd29dc930ef332a48734",
+  "tree": "ffded624f1e44bed9e10872dfb795dae2dda1a46",
+  "preserved_runtime_advances_since_received_parent": [
+    {
+      "path": "testpilot/__main__.py",
+      "before": "44f485af279e66f75c6e4ef881b5a75e5d0cdef2",
+      "after": "53b9436ca572c0cb9e9a10e32252197bdd9ffd98"
+    },
+    {
+      "path": "testpilot/diff.py",
+      "before": "29dd325ab288ac1d43fd70f33b207281f5375d4b",
+      "after": "ab4d562bdad9e5e2ecd977df8245562d16f1cbd9"
+    }
+  ],
+  "shared_cli_and_readme_additions_reverse_byte_exactly": true,
+  "received_regression_blob": "9d877deed683c0e38d61fcf5c7320d8f5dd672c9",
+  "parent_leaf_count": 1426,
+  "current_composed_hosted_gate": "pending at publication; local source-identical cases not relabeled"
+}

docs/receiving/paired-regression-e82707f2bc62/independent/README.md

@@ -0,0 +1,11 @@
+# Independent paired-revision receiving
+
+This receiver uses authored negative-bucket and identity-drift inputs with real local Git repositories, actual CLI/API calls, and the existing retained-suite pytest collector. It was frozen from the public contract before candidate source access.
+
+The actual CLI proves two generated fail-to-pass witnesses across non-ancestor commits, while preserving raw committed export-ignore/export-subst data, executable mode, dirty files, index, refs and saved report bytes. Reversing those commits yields after_failed. Separate API cases show that an existing-suite failure is not a generated witness, changed parametrized node identities are inconclusive, and a conflicting second tree is refused before the first tree can execute. Existing/inside-repository output destinations are preserved.
+
+The first test had a receiver-only missing explicit import of importlib.util. Its original baseline and candidate logs remain unchanged: the candidate's three CLI/report cases passed, while three API methods stopped at that import. The successor adds only that import; removing the line recovers the exact original test. Its affected baseline methods fail because the capability is absent. The first successor candidate attempt stopped on ENOSPC during fixture creation; this is retained as an environment boundary. After compressing only the receiver's own verified metadata copies, the same three methods passed. All six scenarios are qualified across those original and affected runs, not misrepresented as one full six-pass successor execution.
+
+The portable test derives source from imported testpilot, uses disposable temporary directories and a sibling fixtures/regression_receiving.json file. An optional TESTPILOT_RECEIVING_PYTHON selects a genuinely installed pytest interpreter; otherwise it uses the current interpreter. TESTPILOT_RECEIVING_EVIDENCE optionally retains full results. Repository CI should run the complete successor with its declared dependencies.
+
+receiving.json pins source/test/fixture identities and precise outcomes. original-peer-evidence.zip contains every original protocol, test, fixture, run log, source freeze and raw result, including receiver/environment failures. Each archived member was read back byte for byte. No production edits, model invocation, provider interaction or security-isolation claim is made.