diff --git a/README.md b/README.md index df07258..c7bad7a 100644 --- a/README.md +++ b/README.md @@ -179,6 +179,14 @@ The default model IDs come from the Token Factory docs' August 2026 deprecation ## Eval (scripted stub) +To apply the failed-before/passed-after check to your own committed project, +use `testpilot regression --repo PROJECT --report REPORT --before REF --after REF --out NEW_DIRECTORY`. +It executes the same retained tests on both revisions without model calls or +checkout changes, and requires a matched native failure-to-pass witness. +Timeouts, collection errors and mismatched generated cases remain inconclusive. +See [paired revision regression checks](docs/REGRESSION.md) for interpreter setup, +source limits and the complete result contract. + Each case has a repo with one injected bug, the PR diff, a hidden `fix/` used only as an oracle, and canned model replies. The scripts are written to exercise every loop path: a first-try diagnosis, an import-error repair followed by a diagnosis, a timeout, exhausting the round limit, and weak tests that miss the bug. * **bug-revealing**: the final tests fail on the buggy repo and pass with `fix/` applied. diff --git a/docs/REGRESSION.md b/docs/REGRESSION.md new file mode 100644 index 0000000..af7de6f --- /dev/null +++ b/docs/REGRESSION.md @@ -0,0 +1,120 @@ +# Check a retained regression test before and after a fix + +`regression` runs the exact tests saved in one TestPilot generation or recheck +report on two committed versions of your Python project. It answers whether a +retained test actually failed on the chosen before revision and passed on the +chosen after revision. It makes zero model calls and does not repair tests or +apply the proposed patch. + +Use your project's existing interpreter with pytest and its dependencies +installed. Both Git revisions must already be available locally. The command +does not fetch objects or install dependencies. + +```bash +python -m testpilot regression \ + --repo /path/to/project \ + --report /path/to/saved-run/report.json \ + --before BUGGY_COMMIT \ + --after FIXED_COMMIT \ + --python /path/to/project/.venv/bin/python \ + --timeout 60 \ + --out /path/to/new-regression-check +``` + +`--repo` is the root of a local Git worktree. The output directory must be new, +outside that repository, and have an existing parent directory. Each pytest +phase receives the specified positive, finite timeout. The default interpreter +is the one running TestPilot; a selected virtual environment is preserved. + +The two refs are resolved once, before execution, to full commit and tree IDs. +They need not be ancestors. Dirty files, staged changes, untracked files and the +current branch do not supply the source being tested. They are left untouched. +Each revision is materialized in a temporary directory directly from its Git +blobs, retaining regular-file bytes and executable modes. Git replacement refs, +archive attributes, smudge filters and caller Git-directory environment settings +cannot silently substitute the selected source. + +The exact saved test bytes are admitted once by the existing retained-test +reader. Both revision snapshots and test placements are validated before either +pytest run starts. If a retained path exists in a commit, its bytes must exactly +match the saved test; different bytes or a file/directory conflict refuse the +operation. Absent retained paths are inserted only into the disposable runner +copy. The generation report's previous status is recorded as context, never +used as evidence that a current execution passed. + +## Understand the result + +The new directory contains `regression.md` and `regression.json`. The JSON schema +identifier is `testpilot.regression/1`; it is a paired execution result, separate +from generation and recheck results. + +| Status | Meaning | Exit | +| --- | --- | --- | +| `verified_regression` | At least one exact generated case failed before and passed after, and the after run is successful. | 0 | +| `not_regression` | The comparable generated cases already passed before; an unrelated existing-test failure does not establish a witness. | 1 | +| `after_failed` | The comparable after execution still fails, including a remaining existing-suite failure. | 1 | +| `inconclusive` | Execution or generated-case comparability is incomplete. | 1 | +| Admission/setup/output refusal | A result could not be produced; existing output paths are preserved. | 2 | + +Comparison uses each actual native `(generated_file, nodeid)` identity. Every +retained file must collect at least one generated case; identities must be +unique and exactly match across phases. Both native runs must finish, provide +JUnit, return 0 or 1, and contain no native errors. All generated after cases +must pass. A timeout, collection/import/setup error, missing JUnit, missing file +collection, changed or duplicate identity, or generated skip is inconclusive. +Existing-suite skips retain the current native runner's semantics. + +`comparison.witnesses` contains the matched failed-before/passed-after cases. +`case_pairs`, `unmatched_before` and `unmatched_after` make identity differences +explicit. Both `before.result` and `after.result` retain the native sandbox +result, case outcomes, generated counts, timeout and JUnit availability. No +result is inferred from an older report or from exit status alone. + +The result also records full requested refs, commit/tree IDs, selected Python, +source file/byte totals, input-report SHA-256, exact retained test contents and +their hashes, zero model calls and execution timestamps. `snapshot_sha256` +binds the admitted original tree inventory: a UTF-8 JSON array sorted by UTF-8 +path bytes; each record has keys in order `path`, `mode`, `blob`, `bytes`; +`ensure_ascii=True`, separators `(',', ':')`, and no trailing newline. `mode` +and `blob` are Git strings and `bytes` is an integer. File content is checked +against each original Git blob identity when materialized. + +## Boundaries + +This version admits at most 10,000 regular files, 8 MiB per file and 64 MiB per +revision. Git symlinks, submodules/gitlinks, unavailable objects and unsupported +paths are refused rather than omitted. Paths must be UTF-8 relative Git paths; +backslashes, colons, traversal components and `.git` components are unsupported. +Git must support its `--no-lazy-fetch` option. A dependency that is only present +in your working checkout will not appear in a committed snapshot. + +The established TestPilot sandbox, collection rules, ignored directories and +generated-case collector remain in force. Coverage is disabled for both phases +so optional instrumentation does not change the paired test invocation. No +operating-system or network isolation is added: project tests retain the +existing local execution boundary and can access ordinary process resources. + +A passing pair is evidence about these two revisions and this exact suite. It +does not establish why a test failed, which intervening change fixed it, the +completeness of the tests, or whether another environment will behave the same. + +## Python API + +```python +from testpilot.regression import RegressionError, check_regression + +result = check_regression( + "/path/to/project", + "/path/to/saved-run/report.json", + before="BUGGY_COMMIT", + after="FIXED_COMMIT", + timeout_s=60.0, + python="/path/to/project/.venv/bin/python", +) +print(result["status"], result["comparison"]["witnesses"]) +``` + +The API returns the same dictionary without publishing files. Invalid input, +Git admission, snapshot preparation and execution setup raise `RegressionError` +(a `ValueError` subclass). Native test failures and inconclusive executions +return their honest comparison status instead of raising. diff --git a/docs/receiving/paired-regression-e82707f2bc62/README.md b/docs/receiving/paired-regression-e82707f2bc62/README.md new file mode 100644 index 0000000..6d6af0b --- /dev/null +++ b/docs/receiving/paired-regression-e82707f2bc62/README.md @@ -0,0 +1,72 @@ +# Paired committed-revision regression receiving + +Source claim: https://github.com/Jacob-Met/testpilot/issues/44 +Owner: estate-e82707f2bc62 / coordination_discovery. + +The new command executes one admitted retained suite on two explicit Git +commits. A result is verified only when a uniquely matched generated case +actually fails before and passes after, with successful after execution and +complete comparable generated identities. No saved historical status, model +response, unrelated existing failure, missing JUnit, timeout, error or skip is +substituted for this evidence. The public API and limitations are described in +[REGRESSION.md](../../REGRESSION.md). + +## Exact native evidence + +The original source is 191cca4e416286a9e1fa4b5d5daf5fc30b936db6. A real CLI +workflow uses the maintained calc_clamp fixture, synthetic local Git commits, +and actual generation plus rechecks. The identical four retained cases produce +three failures and one pass before, then four passes after; both rechecks use +zero model calls. Original paired invocation refuses as unsupported with exit2 +and no output. All original input/source bytes remain unchanged. + +The new command executes that same saved report on those same commits and +returns verified_regression, exit0, with three exact failed-before/passed-after +witnesses. Both native count records match the original controls. The original +report, complete fixture Git metadata, native stdout/stderr/results, source +capsule and source/input guards are retained in author-evidence.zip. + +Fourteen frozen authored methods produce 2 failures and12 missing-module errors +on original source, then14 passes on the candidate. An explicit inventory sort +was added to make the documented signature serialization stable; the same14 +methods pass again. These are separate recorded source phases, not a claim that +the original source implemented the feature. Both logs, exact test and final +received source are in the62-member verified archive. + +Independent receiving was frozen before candidate access. Six scenarios are +accepted across the documented phases: actual non-ancestor CLI execution with +raw Git assets/executable modes/archive attributes and dirty-tree preservation; +reverse after-failure exit1; native report admission; unrelated-suite failure +exclusion; generated-node identity drift; and admission of both trees before +execution plus output preservation. The first three candidate successes were +retained while a receiver-only missing import was corrected. The unchanged +remaining three then encountered ENOSPC before product execution and passed +after the receiver compressed its own metadata. The independent README/receipt +and24-member archive preserve every predecessor. This is not mislabeled as a +single six-pass successor run. The portable test and literal fixture are added +to normal native test discovery. + +Root independently read the complete module, tests and guide and reversed the +additive CLI/README spans to the exact received parent. Its source review is +included; it makes no independent execution claim. + +## Current-parent composition and limits + +The received production module is Git9d877deed683c0e38d61fcf5c7320d8f5dd672c9, +SHA256a802c332d4f0adaf8f49306b2686d4f04050325d60a25367af18da3241940b89. +The exact current parent and newer runtime changes are in current-composition.json. +Only the new command's CLI and README spans are applied to those current files; +removing them restores the full current bytes. Existing parser, timeout, +process custody, recheck, collector, model, history and comparison owners are +preserved. All other parent leaves remain unchanged by the complete-tree overlay. + +The unchanged repository CI must qualify the actual composed PR checkout. +Hosted qualification and merge are pending when this packet is published; +neither is inferred from local source presence. No installed service, +provider/account operation or deployment is claimed. + +Local execution used Python3.12.14 with genuine existing pytest9.1.1 through a +small no-pip environment. Failed storage/setup attempts and the empty failed +document write remain in the author packet. No successful dependency install +or broad shared cleanup is claimed. Exact source, fixture and test data are +synthetic or already public project source; no participant data is used. diff --git a/docs/receiving/paired-regression-e82707f2bc62/author-evidence.zip b/docs/receiving/paired-regression-e82707f2bc62/author-evidence.zip new file mode 100644 index 0000000..2a30e44 Binary files /dev/null and b/docs/receiving/paired-regression-e82707f2bc62/author-evidence.zip differ diff --git a/docs/receiving/paired-regression-e82707f2bc62/current-composition.json b/docs/receiving/paired-regression-e82707f2bc62/current-composition.json new file mode 100644 index 0000000..9caa293 --- /dev/null +++ b/docs/receiving/paired-regression-e82707f2bc62/current-composition.json @@ -0,0 +1,20 @@ +{ + "parent": "513f17454af378ee042ccd29dc930ef332a48734", + "tree": "ffded624f1e44bed9e10872dfb795dae2dda1a46", + "preserved_runtime_advances_since_received_parent": [ + { + "path": "testpilot/__main__.py", + "before": "44f485af279e66f75c6e4ef881b5a75e5d0cdef2", + "after": "53b9436ca572c0cb9e9a10e32252197bdd9ffd98" + }, + { + "path": "testpilot/diff.py", + "before": "29dd325ab288ac1d43fd70f33b207281f5375d4b", + "after": "ab4d562bdad9e5e2ecd977df8245562d16f1cbd9" + } + ], + "shared_cli_and_readme_additions_reverse_byte_exactly": true, + "received_regression_blob": "9d877deed683c0e38d61fcf5c7320d8f5dd672c9", + "parent_leaf_count": 1426, + "current_composed_hosted_gate": "pending at publication; local source-identical cases not relabeled" +} diff --git a/docs/receiving/paired-regression-e82707f2bc62/independent/README.md b/docs/receiving/paired-regression-e82707f2bc62/independent/README.md new file mode 100644 index 0000000..79f3e72 --- /dev/null +++ b/docs/receiving/paired-regression-e82707f2bc62/independent/README.md @@ -0,0 +1,11 @@ +# Independent paired-revision receiving + +This receiver uses authored negative-bucket and identity-drift inputs with real local Git repositories, actual CLI/API calls, and the existing retained-suite pytest collector. It was frozen from the public contract before candidate source access. + +The actual CLI proves two generated fail-to-pass witnesses across non-ancestor commits, while preserving raw committed export-ignore/export-subst data, executable mode, dirty files, index, refs and saved report bytes. Reversing those commits yields after_failed. Separate API cases show that an existing-suite failure is not a generated witness, changed parametrized node identities are inconclusive, and a conflicting second tree is refused before the first tree can execute. Existing/inside-repository output destinations are preserved. + +The first test had a receiver-only missing explicit import of importlib.util. Its original baseline and candidate logs remain unchanged: the candidate's three CLI/report cases passed, while three API methods stopped at that import. The successor adds only that import; removing the line recovers the exact original test. Its affected baseline methods fail because the capability is absent. The first successor candidate attempt stopped on ENOSPC during fixture creation; this is retained as an environment boundary. After compressing only the receiver's own verified metadata copies, the same three methods passed. All six scenarios are qualified across those original and affected runs, not misrepresented as one full six-pass successor execution. + +The portable test derives source from imported testpilot, uses disposable temporary directories and a sibling fixtures/regression_receiving.json file. An optional TESTPILOT_RECEIVING_PYTHON selects a genuinely installed pytest interpreter; otherwise it uses the current interpreter. TESTPILOT_RECEIVING_EVIDENCE optionally retains full results. Repository CI should run the complete successor with its declared dependencies. + +receiving.json pins source/test/fixture identities and precise outcomes. original-peer-evidence.zip contains every original protocol, test, fixture, run log, source freeze and raw result, including receiver/environment failures. Each archived member was read back byte for byte. No production edits, model invocation, provider interaction or security-isolation claim is made. diff --git a/docs/receiving/paired-regression-e82707f2bc62/independent/original-peer-evidence.zip b/docs/receiving/paired-regression-e82707f2bc62/independent/original-peer-evidence.zip new file mode 100644 index 0000000..c350630 Binary files /dev/null and b/docs/receiving/paired-regression-e82707f2bc62/independent/original-peer-evidence.zip differ diff --git a/docs/receiving/paired-regression-e82707f2bc62/independent/receiving.json b/docs/receiving/paired-regression-e82707f2bc62/independent/receiving.json new file mode 100644 index 0000000..ae2928b --- /dev/null +++ b/docs/receiving/paired-regression-e82707f2bc62/independent/receiving.json @@ -0,0 +1,106 @@ +{ + "schema": "testpilot-paired-independent-receiving/1", + "reviewer": "estate-e82707f2bc62/runtime_discovery", + "protocol": { + "bytes": 4268, + "sha256": "3f6c304189a4dd88d21933cb95f3e37fc759e4a70215e553e6f79dde31d489ee", + "git_blob": "1319036843142c722165d9e7ae2fe5671f62c949" + }, + "fixture": { + "bytes": 1870, + "sha256": "22a80950602655564bce2e19f89010a73d83356b802c80b3604480b910418305", + "git_blob": "7a48df341b451330f70ed9c22f63d93454a9213c" + }, + "original_test": { + "bytes": 13441, + "sha256": "33122c130acf1eac90f2f2a078f611fc18f5f4b81d052f8493eeec49b21e9f60", + "git_blob": "2c335654ca02093c464d286ad3613beeeb7ef770" + }, + "active_native_test": { + "bytes": 13463, + "sha256": "4d6bf9da8db377daa13047d0de8f6a53b8d96ea4690a96fab1e09f7f97bd07e4", + "git_blob": "215a7ad4d7ac73aa20c63a2243a49cb5755e4f97" + }, + "candidate_source": { + "README.md": "02ab6703742d6a5465c27b22a12405b99ba88bba192b10742fdf7f5102b691a7", + "pyproject.toml": "5503eab7b6c145d4de241e5be805a4fb8f30828053cbe39dc05a43c2896642eb", + "docs/REGRESSION.md": "43a4d9c0c9bbf1b6d05f64bc55170cd4259c7c6cfc34409fc8ba3b516de95d7e", + "tests/test_regression.py": "b3b7c8892c24db5cc7368b4960e334aee1b51835293a6045306c847819af3578", + "testpilot/regression.py": "a802c332d4f0adaf8f49306b2686d4f04050325d60a25367af18da3241940b89", + "testpilot/sandbox.py": "8d5aadafff5188473ec45eb1cd8f01fb562836ff05bf12a27939cde83b33472a", + "testpilot/html_report.py": "f2b65ee9201f9e91ad183fcb8f6b8bf6e69c77b1a066fcd90bb431951e41a2f6", + "testpilot/__init__.py": "65c63b80d7da34a83a6049a79c24b8335878cd49cc79951fdefc5f1532ae841f", + "testpilot/recheck.py": "3eab07bd639994f897f76c0cd2b941d5e294c6d7c8132c82deb800e44d7079e6", + "testpilot/model.py": "f6ae697f1f5d686ffdcb96f5451d1a9600d7d41c7bcf324d7e8c276a107e8673", + "testpilot/diff.py": "2cf38707ea175bb1f04d55335de344ae943458658055332b4aa68f27763214c0", + "testpilot/targets.py": "35add23585b0e88246288572f49bd857ebf4af83a221b9129e784e9b456c1ab0", + "testpilot/__main__.py": "92c5dc62290e02c6c7041038322018ff2a2337a9c9bdee8beb17b5603b95301a", + "testpilot/_pytest_generated.py": "d65352d7bade965d7b9dc73efe5d4d01844fa8c7474a1bdcdb3b708f6decf768", + "testpilot/loop.py": "7e2336ff4d3e88aa8f575e4d7c04f3fe1cc1bf3023f0d230941ab63cead675a8", + "testpilot/recheck_html.py": "6cc81b1d1e63282708a4cad448224bac5359519c4955d82c6244afa80738e633", + "eval/cases/calc_clamp/change.diff": "524dd8551c7043beb01153c54c90beda78e2f2b4a7ae3807533f0bfe24af1f3f", + "eval/cases/calc_clamp/case.json": "9e997cb92c4c15437f44c11a42fbf3d3a325d6d456e5ae80aaafdf8b2582fea4", + "eval/cases/calc_clamp/repo/calc.py": "1ef33b0ae046ffc6a81683a141651f8cb64902ecaaa3c7fde8d012f0c37b40e3", + "eval/cases/calc_clamp/script/03_repair.md": "81d17160bf1cafc05b69d7db223ffffcee47bc58d8519344c9bfc318ebfb5809", + "eval/cases/calc_clamp/script/02_tests.md": "7d9b403d6dac6e2280d44ea7b7d4a3c2ab83b75d62f479c44a53b6e34e1efcd6", + "eval/cases/calc_clamp/script/01_plan.md": "9204691f94b5bfadce3da3d8c2590cf3c60518f6890439004b8d80ac6f0c29bf", + "eval/cases/calc_clamp/fix/calc.py": "d8e853f3e9de197268cbef9422cea093804b0d4e8b2ebe05829f41901a56f004", + "eval/cases/calc_clamp/repo/tests/test_existing.py": "47b2ebb38283eb3cc54a9e6bfe63f93ad9b73e797d89deef92215daa0063903d", + ".github/workflows/ci.yml": "236fd600bea41471b373b91d8e389aa12c58cba79c83efa071f6ab805f1ebf59" + }, + "original_baseline": { + "pyproject.toml": "5503eab7b6c145d4de241e5be805a4fb8f30828053cbe39dc05a43c2896642eb", + "README.md": "a446169268324fbe3e1753fd8838c03ef6fc7f791bcf62cf2143418342f5c0b4", + "testpilot/recheck_html.py": "6cc81b1d1e63282708a4cad448224bac5359519c4955d82c6244afa80738e633", + "testpilot/loop.py": "7e2336ff4d3e88aa8f575e4d7c04f3fe1cc1bf3023f0d230941ab63cead675a8", + "testpilot/_pytest_generated.py": "d65352d7bade965d7b9dc73efe5d4d01844fa8c7474a1bdcdb3b708f6decf768", + "testpilot/__main__.py": "b9c8268797fc34bed1dc9a6a89b1fd0cb44a885ee63595cf531bfa601b33990a", + "testpilot/targets.py": "35add23585b0e88246288572f49bd857ebf4af83a221b9129e784e9b456c1ab0", + "testpilot/diff.py": "2cf38707ea175bb1f04d55335de344ae943458658055332b4aa68f27763214c0", + "testpilot/model.py": "f6ae697f1f5d686ffdcb96f5451d1a9600d7d41c7bcf324d7e8c276a107e8673", + "testpilot/recheck.py": "3eab07bd639994f897f76c0cd2b941d5e294c6d7c8132c82deb800e44d7079e6", + "testpilot/__init__.py": "65c63b80d7da34a83a6049a79c24b8335878cd49cc79951fdefc5f1532ae841f", + "testpilot/html_report.py": "f2b65ee9201f9e91ad183fcb8f6b8bf6e69c77b1a066fcd90bb431951e41a2f6", + "testpilot/sandbox.py": "8d5aadafff5188473ec45eb1cd8f01fb562836ff05bf12a27939cde83b33472a", + ".github/workflows/ci.yml": "236fd600bea41471b373b91d8e389aa12c58cba79c83efa071f6ab805f1ebf59", + "eval/cases/calc_clamp/case.json": "9e997cb92c4c15437f44c11a42fbf3d3a325d6d456e5ae80aaafdf8b2582fea4", + "eval/cases/calc_clamp/change.diff": "524dd8551c7043beb01153c54c90beda78e2f2b4a7ae3807533f0bfe24af1f3f", + "eval/cases/calc_clamp/fix/calc.py": "d8e853f3e9de197268cbef9422cea093804b0d4e8b2ebe05829f41901a56f004", + "eval/cases/calc_clamp/script/01_plan.md": "9204691f94b5bfadce3da3d8c2590cf3c60518f6890439004b8d80ac6f0c29bf", + "eval/cases/calc_clamp/script/02_tests.md": "7d9b403d6dac6e2280d44ea7b7d4a3c2ab83b75d62f479c44a53b6e34e1efcd6", + "eval/cases/calc_clamp/script/03_repair.md": "81d17160bf1cafc05b69d7db223ffffcee47bc58d8519344c9bfc318ebfb5809", + "eval/cases/calc_clamp/repo/calc.py": "1ef33b0ae046ffc6a81683a141651f8cb64902ecaaa3c7fde8d012f0c37b40e3", + "eval/cases/calc_clamp/repo/tests/test_existing.py": "47b2ebb38283eb3cc54a9e6bfe63f93ad9b73e797d89deef92215daa0063903d" + }, + "original_first_runs": { + "baseline": "1 pass, 2 absent-CLI failures, 3 receiver importlib.util errors", + "candidate": "3 pass (real forward CLI, reverse CLI, original report control), 3 identical receiver errors" + }, + "import_only_successor": { + "change": "Add explicit import importlib.util; removing that line recovers byte-identical original test; all methods/fixtures/assertions unchanged.", + "baseline": "Affected 3 methods: 3 absent-module failures, no errors", + "candidate_first": "Affected 3 stopped during fixture creation on ENOSPC; 2 setup failures and 1 setup error, no product conclusion", + "candidate_space_recovered": "Same 3 affected methods pass in 2.485s after preserving and compressing only own metadata", + "test_sha256": "4d6bf9da8db377daa13047d0de8f6a53b8d96ea4690a96fab1e09f7f97bd07e4" + }, + "aggregate": "All six scenarios qualified across original 3-pass execution and affected successor 3-pass execution. No claim of a single local six-pass successor run.", + "native_stack": { + "python": "/dev/shm/e827-testpilot-revisions/runtime/bin/python", + "pytest": "9.1.1 genuine retained installed package", + "execution": "actual Git blobs/commits, subprocess CLI, native unchanged pytest collector; no fake runner/model/API" + }, + "source_and_input_preservation": "Every frozen baseline/candidate source snapshot rechecked unchanged; per-case source repository/index/refs/dirty files/report byte hashes unchanged.", + "lint": "Ruff0.16.10 check active native receiver passes", + "archive": { + "bytes": 30193, + "sha256": "f5ab53553d4445cfaa3e5fd8ee3053d6af300246050846abf9cfe89179a86fd2", + "git_blob": "c350630cd8a2da4735804b94ce72b7483b46cf22", + "members": 24, + "every_member_readback_exact": true + }, + "limits": [ + "No security-sandbox claim; actual native subprocess boundary applies.", + "Source annotation or current-parent changes require hosted native qualification.", + "No production code edited by independent receiver." + ] +} diff --git a/docs/receiving/paired-regression-e82707f2bc62/root-source-review-v2.json b/docs/receiving/paired-regression-e82707f2bc62/root-source-review-v2.json new file mode 100644 index 0000000..e484009 --- /dev/null +++ b/docs/receiving/paired-regression-e82707f2bc62/root-source-review-v2.json @@ -0,0 +1,74 @@ +{ + "reviewer": "estate-e82707f2bc62/root", + "reviewed_at": "2026-10-08T19:10:23.022717+00:00", + "source_freeze": "source-freeze-v2.json", + "status": "Source accepted, independent native and actual current-parent hosted receiving remain required", + "files": [ + { + "path": "testpilot/regression.py", + "git_blob": "9d877deed683c0e38d61fcf5c7320d8f5dd672c9", + "sha256": "a802c332d4f0adaf8f49306b2686d4f04050325d60a25367af18da3241940b89" + }, + { + "path": "testpilot/__main__.py", + "git_blob": "cfb8ad4a6e3b9c30dbd5fc55c6028167c9fbc106", + "sha256": "92c5dc62290e02c6c7041038322018ff2a2337a9c9bdee8beb17b5603b95301a" + }, + { + "path": "tests/test_regression.py", + "git_blob": "bbd21beaccac0fccea92bfbb1c9b4c5bf87e9998", + "sha256": "b3b7c8892c24db5cc7368b4960e334aee1b51835293a6045306c847819af3578" + }, + { + "path": "docs/REGRESSION.md", + "git_blob": "af7de6f54af478d5d5056a3d403211ffca3d3ce8", + "sha256": "43a4d9c0c9bbf1b6d05f64bc55170cd4259c7c6cfc34409fc8ba3b516de95d7e" + }, + { + "path": "README.md", + "git_blob": "da0a4bba9bbc6680e0a16a875fcf8ad96bda51ad", + "sha256": "02ab6703742d6a5465c27b22a12405b99ba88bba192b10742fdf7f5102b691a7" + } + ], + "public_contract_sha256": "85a093ae885223e26db05b090d217ca37a4bffb0e6d72fbe4ef469b97d689ad2", + "exact_parent": "cec50d8df4499ba509615e179d82ef3861379bd2", + "additive_composition": [ + { + "path": "testpilot/__main__.py", + "parent_blob": "44f485af279e66f75c6e4ef881b5a75e5d0cdef2", + "inserted_spans_chars": [ + 47, + 860, + 72 + ], + "reverse_exact": true + }, + { + "path": "README.md", + "parent_blob": "9eaab06ffbfd831cfdd2916fff4dd15198cb8f3e", + "inserted_spans_chars": [ + 543 + ], + "reverse_exact": true + } + ], + "protected_local_runtime_files_exact_to_freeze": 10, + "reviewed_behaviors": [ + "Resolve both refs to original complete commit/tree identities before execution; no replacement, lazy fetch, checkout or hooks", + "Admit regular-file trees, original blob identities/modes and both retained-test placements before either pytest phase", + "Reuse unchanged saved-suite reader and run_pytest with exact_files=True and coverage=False", + "Pair actual generated_file/nodeid values; duplicates, mismatches, missing retained files, generated skips and native errors remain inconclusive", + "Require successful after execution plus at least one genuine failed-before/passed-after generated witness", + "Preserve dirty worktree/index/ref state and existing output; publish only new external destination", + "Document existing sandbox execution boundary and absence of causal or test-completeness claims" + ], + "reviewed_source": "Complete336-line module,246-line author receiver,public contract,guide,CLI/README additions and unchanged retained-reader/native-sandbox boundaries", + "root_execution_claim": false, + "native_receiving_attribution": { + "author_original": "14 methods:2fail/12missing-module errors", + "author_candidate_v1": "14pass", + "author_candidate_v2": "same14pass", + "original_retained_clamp": "Separate original witness and actual candidate successor retained by author" + }, + "current_main_followup": "Author reports newer e5b7ac4a malformed-diff PR48; final publication must preserve its actual CLI/parser and all unrelated leaves. This review does not claim that later composition was executed." +} diff --git a/testpilot/__main__.py b/testpilot/__main__.py index 53b9436..7e5c1b6 100644 --- a/testpilot/__main__.py +++ b/testpilot/__main__.py @@ -14,6 +14,7 @@ from .loop import TestPilot, render_report, write_outputs from .model import ModelError, RoutingConfig, make_client from .recheck import run_recheck_command +from .regression import run_regression_command from .targets import TargetSelectionError, resolve_targets, selection_record @@ -132,8 +133,20 @@ def main(argv: list[str] | None = None) -> int: recheck.add_argument("--timeout", type=float, default=60.0, help="pytest timeout, seconds (default 60)") recheck.add_argument("--python", type=_python_executable, metavar="EXECUTABLE", help="project Python; path or PATH command (default: this interpreter)") + regression = sub.add_parser("regression", help="check retained tests on two committed revisions") + regression.add_argument("--repo", required=True, help="root of the local Git worktree") + regression.add_argument("--report", required=True, help="saved report.json or recheck.json") + regression.add_argument("--before", required=True, help="explicit before Git revision") + regression.add_argument("--after", required=True, help="explicit after Git revision") + regression.add_argument("--out", required=True, help="new result directory outside the repository") + regression.add_argument("--timeout", type=float, default=60.0, help="pytest timeout per revision") + regression.add_argument("--python", type=_python_executable, metavar="EXECUTABLE", + help="project Python for both runs (default: this interpreter)") a = ap.parse_args(argv) + if a.cmd == "regression": + return run_regression_command(a) + if a.cmd == "recheck": return run_recheck_command(a) diff --git a/testpilot/regression.py b/testpilot/regression.py new file mode 100644 index 0000000..9d877de --- /dev/null +++ b/testpilot/regression.py @@ -0,0 +1,336 @@ +"""Execute one retained suite against two immutable local Git revisions.""" +from __future__ import annotations + +import hashlib +import json +import math +import os +from pathlib import Path, PurePosixPath +import re +import shutil +import subprocess +import sys +import tempfile +from datetime import datetime, timezone +import xml.etree.ElementTree as ET + +from .recheck import read_saved_tests +from .sandbox import SandboxResult, run_pytest + +SCHEMA = "testpilot.regression/1" +MAX_SOURCE_FILES = 10_000 +MAX_SOURCE_FILE_BYTES = 8 * 1024 * 1024 +MAX_SOURCE_BYTES = 64 * 1024 * 1024 +_OID = re.compile(r"(?:[0-9a-f]{40}|[0-9a-f]{64})\Z") +_GIT_TIMEOUT = 30 +_MAX_TREE_OUTPUT = 16 * 1024 * 1024 + + +class RegressionError(ValueError): + """The requested paired execution could not be admitted or completed.""" + + +def _sha(data: bytes) -> str: + return hashlib.sha256(data).hexdigest() + + +def _now() -> str: + return datetime.now(timezone.utc).isoformat() + + +def _git(repo: Path, *args: str, limit: int = 4096) -> bytes: + # Reading original objects must not follow caller GIT_DIR/WORK_TREE, + # replacements, optional locks, or a promisor's on-demand network fetch. + env = {k: v for k, v in os.environ.items() if not k.upper().startswith("GIT_")} + env.update(GIT_CONFIG_NOSYSTEM="1", GIT_CONFIG_GLOBAL=os.devnull, + GIT_TERMINAL_PROMPT="0") + try: + result = subprocess.run( + ["git", "--no-replace-objects", "--no-lazy-fetch", "--no-optional-locks", + "-C", str(repo), *args], capture_output=True, env=env, timeout=_GIT_TIMEOUT, + ) + except (OSError, subprocess.TimeoutExpired) as exc: + raise RegressionError(f"cannot read local Git objects: {exc}") from exc + if result.returncode: + detail = result.stderr.decode("utf-8", errors="replace").strip()[-2000:] + raise RegressionError(f"Git {args[0]} refused: {detail or result.returncode}") + if len(result.stdout) > limit: + raise RegressionError(f"Git {args[0]} output exceeds the admitted size limit") + return result.stdout + + +def _repository(repo: str | Path) -> Path: + try: + project = Path(repo).expanduser().resolve(strict=True) + except (OSError, RuntimeError, TypeError, ValueError) as exc: + raise RegressionError(f"cannot open repository root: {exc}") from exc + if not project.is_dir(): + raise RegressionError("repository must be an existing directory") + if (_git(project, "rev-parse", "--is-inside-work-tree") != b"true\n" + or _git(project, "rev-parse", "--show-prefix") != b"\n"): + raise RegressionError("--repo must be the root of a local Git worktree") + return project + + +def _revision(repo: Path, ref: str) -> dict: + if not isinstance(ref, str) or not ref or "\0" in ref: + raise RegressionError("before and after must be non-empty Git revision strings") + commit = _git(repo, "rev-parse", "--verify", "--end-of-options", ref + "^{commit}") + commit = commit.decode("ascii").strip() + if not _OID.fullmatch(commit): + raise RegressionError("Git did not return one complete commit identity") + tree = _git(repo, "rev-parse", "--verify", "--end-of-options", commit + "^{tree}") + tree = tree.decode("ascii").strip() + if not _OID.fullmatch(tree): + raise RegressionError("Git did not return one complete tree identity") + return {"requested_ref": ref, "commit": commit, "tree": tree} + + +def _tree_files(repo: Path, tree: str) -> list[dict]: + raw = _git(repo, "ls-tree", "--full-tree", "-r", "-l", "-z", tree, + limit=_MAX_TREE_OUTPUT) + if raw and not raw.endswith(b"\0"): + raise RegressionError("Git tree listing was incomplete") + records = raw[:-1].split(b"\0") if raw else [] + if len(records) > MAX_SOURCE_FILES: + raise RegressionError(f"revision exceeds {MAX_SOURCE_FILES} source files") + entries = [] + paths = set() + total = 0 + for record in records: + try: + header, path_bytes = record.split(b"\t", 1) + mode, kind, oid, size_raw = header.split() + path = path_bytes.decode("utf-8") + oid = oid.decode("ascii") + mode = mode.decode("ascii") + except (ValueError, UnicodeError) as exc: + raise RegressionError("unsupported or malformed Git tree path/record") from exc + if kind != b"blob" or mode not in {"100644", "100755"}: + raise RegressionError(f"unsupported Git entry {path!r}: mode {mode}; " + "symlinks and submodules are not materialized") + parts = path.split("/") + if (not path or PurePosixPath(path).is_absolute() + or any(p in {"", ".", ".."} or p.casefold() == ".git" for p in parts) + or "\\" in path or ":" in path or path in paths or not _OID.fullmatch(oid)): + raise RegressionError(f"unsupported or unsafe Git path: {path!r}") + try: + size = int(size_raw) + except ValueError as exc: + raise RegressionError(f"missing Git blob size for {path!r}") from exc + if size < 0 or size > MAX_SOURCE_FILE_BYTES: + raise RegressionError(f"source file {path!r} exceeds {MAX_SOURCE_FILE_BYTES} bytes") + total += size + if total > MAX_SOURCE_BYTES: + raise RegressionError(f"revision exceeds {MAX_SOURCE_BYTES} source bytes") + paths.add(path) + entries.append({"path": path, "mode": mode, "blob": oid, "bytes": size}) + return sorted(entries, key=lambda entry: entry["path"].encode("utf-8")) + + +def _materialize(repo: Path, entries: list[dict], destination: Path) -> None: + destination.mkdir() + for entry in entries: + data = _git(repo, "cat-file", "blob", entry["blob"], limit=entry["bytes"]) + algorithm = hashlib.sha1 if len(entry["blob"]) == 40 else hashlib.sha256 + actual = algorithm(b"blob " + str(len(data)).encode("ascii") + b"\0" + data).hexdigest() + if len(data) != entry["bytes"] or actual != entry["blob"]: + raise RegressionError(f"Git blob content/identity changed for {entry['path']!r}") + target = destination.joinpath(*entry["path"].split("/")) + target.parent.mkdir(parents=True, exist_ok=True) + with target.open("xb") as stream: + stream.write(data) + target.chmod(0o755 if entry["mode"] == "100755" else 0o644) + + +def _admit_tests(snapshot: Path, files: dict[str, str]) -> None: + for path, content in files.items(): + target = snapshot / path + for parent in target.parents: + if parent == snapshot: + break + if parent.exists() and not parent.is_dir(): + raise RegressionError(f"retained test parent conflicts with committed source: {path}") + if target.exists() and (not target.is_file() or target.read_bytes() != content.encode("utf-8")): + raise RegressionError(f"retained test differs from committed source: {path}") + + +def _comparison(before: SandboxResult, after: SandboxResult, files: dict[str, str]) -> tuple[str, dict]: + result = {"comparable": False, "reason": "", "case_pairs": [], "witnesses": [], + "unmatched_before": [], "unmatched_after": []} + maps = [] + for phase in (before, after): + mapping = {} + for case in phase.generated_cases: + identity = (case.generated_file, case.nodeid) + if identity in mapping: + result["reason"] = "duplicate_generated_case_identity" + return "inconclusive", result + mapping[identity] = case.outcome + maps.append(mapping) + old, new = maps + + def identity_record(key): + return {"generated_file": key[0], "nodeid": key[1]} + + result["unmatched_before"] = [identity_record(k) for k in sorted(old.keys() - new.keys())] + result["unmatched_after"] = [identity_record(k) for k in sorted(new.keys() - old.keys())] + result["case_pairs"] = [{**identity_record(k), "before": old[k], "after": new[k]} + for k in sorted(old.keys() & new.keys())] + if any(p.timed_out or not p.junit_available or p.returncode not in (0, 1) or p.errors + for p in (before, after)): + result["reason"] = "incomplete_native_run" + elif any({key[0] for key in mapping} != set(files) for mapping in maps): + result["reason"] = "missing_generated_file_collection" + elif old.keys() != new.keys(): + result["reason"] = "generated_case_identity_mismatch" + elif any(outcome not in {"passed", "failed"} for mapping in maps for outcome in mapping.values()): + result["reason"] = "skipped_or_unknown_generated_cases" + else: + result["comparable"] = True + if not after.ok or any(outcome != "passed" for outcome in new.values()): + result["reason"] = "after_did_not_pass" + return "after_failed", result + result["witnesses"] = [pair for pair in result["case_pairs"] + if pair["before"] == "failed" and pair["after"] == "passed"] + if result["witnesses"]: + result["reason"] = "failed_before_passed_after" + return "verified_regression", result + result["reason"] = "no_generated_failure_before" + return "not_regression", result + return "inconclusive", result + + +def check_regression(repo: str | Path, report_path: str | Path, *, before: str, after: str, + timeout_s: float = 60.0, python: str | None = None) -> dict: + """Run exact saved tests on two commits, without changing the worktree/index.""" + try: + valid_timeout = (isinstance(timeout_s, (int, float)) and not isinstance(timeout_s, bool) + and math.isfinite(timeout_s) and timeout_s > 0) + except OverflowError: + valid_timeout = False + if not valid_timeout: + raise RegressionError("timeout must be a finite positive number") + if python is not None and (not isinstance(python, str) or not python): + raise RegressionError("python must identify an executable interpreter") + chosen = shutil.which(os.path.expanduser(python or sys.executable)) + if chosen is None: + raise RegressionError(f"Python interpreter is unavailable: {python!r}") + executable = str(Path(chosen).parent.resolve() / Path(chosen).name) + started = _now() + project = _repository(repo) + try: + saved = read_saved_tests(report_path) + files = dict(saved.tests) + phases = [_revision(project, before), _revision(project, after)] + listings = [_tree_files(project, phase["tree"]) for phase in phases] + for phase, entries in zip(phases, listings): + encoded = json.dumps(entries, ensure_ascii=True, separators=(",", ":")).encode("utf-8") + phase.update(source_files=len(entries), source_bytes=sum(e["bytes"] for e in entries), + snapshot_sha256=_sha(encoded)) + with tempfile.TemporaryDirectory(prefix="testpilot-regression-") as temporary: + snapshots = [Path(temporary) / label for label in ("before", "after")] + # A bad after tree/test placement must refuse before the before run. + for entries, snapshot in zip(listings, snapshots): + _materialize(project, entries, snapshot) + _admit_tests(snapshot, files) + native = [run_pytest(snapshot, dict(files), timeout=timeout_s, python=executable, + coverage=False, exact_files=True) for snapshot in snapshots] + status, comparison = _comparison(native[0], native[1], files) + for phase, result in zip(phases, native): + phase["result"] = result.to_dict() + except (OSError, ValueError, ET.ParseError) as exc: + if isinstance(exc, RegressionError): + raise + raise RegressionError(f"cannot execute retained revision check: {exc}") from exc + return { + "schema": SCHEMA, "status": status, "ok": status == "verified_regression", + "repo": str(project), "python": executable, "started_at": started, "finished_at": _now(), + "source_report": {"path": saved.path, "sha256": saved.sha256, + "recorded_status": saved.source_status}, + "test_files": files, + "retained_tests": [{"path": path, "bytes": len(content.encode("utf-8")), + "sha256": _sha(content.encode("utf-8"))} + for path, content in saved.tests], + "before": phases[0], "after": phases[1], "comparison": comparison, "model_calls": 0, + "scope": "Two explicit committed trees; exact retained tests; existing local pytest " + "sandbox and collection policy; no model, patch, checkout, or dependency install.", + } + + +def _destination(repo: Path, value: str | Path) -> Path: + target = Path(value).expanduser() + if target.exists() or target.is_symlink(): + raise RegressionError("output must be a new directory; existing paths are not replaced") + target = target.resolve() + if target == repo or target.is_relative_to(repo): + raise RegressionError("output must be outside the checked repository") + if not target.parent.is_dir(): + raise RegressionError("output parent directory must already exist") + return target + + +def render_regression(result: dict) -> str: + """Readable observation summary; exact native evidence stays in the JSON.""" + lines = ["# TestPilot retained-suite regression check", "", f"Status: **{result['status']}**", "", + f"Reason: {result['comparison']['reason']}", "", "| Phase | Commit | Native result |", + "| --- | --- | --- |"] + for label in ("before", "after"): + phase = result[label] + lines.append(f"| {label} | `{phase['commit']}` | {phase['result']['summary']} |") + lines += ["", f"Matched failure-to-pass witnesses: {len(result['comparison']['witnesses'])}", + "", "```json", json.dumps(result["comparison"]["witnesses"], indent=2) + .replace("`", "\\u0060").replace("<", "\\u003c"), "```", "", + "Source report SHA-256: `" + result["source_report"]["sha256"] + "`", "", + "Model calls: 0. These are observations on two explicit revisions; they do not " + "identify which intervening change caused a result.", "", + "Project tests run within TestPilot's existing local execution boundary. " + "This command adds no operating-system or network isolation.", ""] + return "\n".join(lines) + + +def run_regression_command(args) -> int: + """CLI boundary: preserve prior destinations and publish only real results.""" + created = [] + target = None + owns_directory = False + try: + project = _repository(args.repo) + target = _destination(project, args.out) + result = check_regression(project, args.report, before=args.before, after=args.after, + timeout_s=args.timeout, python=args.python) + outputs = {"regression.json": json.dumps(result, indent=2, allow_nan=False) + "\n", + "regression.md": render_regression(result)} + target.mkdir() + owns_directory = True + for name, text in outputs.items(): + path = target / name + with path.open("x", encoding="utf-8", newline="\n") as stream: + created.append(path) + stream.write(text) + print(f"{result['status']}: {result['comparison']['reason']}", flush=True) + print(f"wrote {target / 'regression.json'}, {target / 'regression.md'}", flush=True) + return 0 if result["ok"] else 1 + except (OSError, ValueError, RuntimeError) as exc: + # Remove only files reserved by this invocation. Never remove a prior + # directory or a concurrent writer's entry. + for path in created: + try: + path.unlink() + except OSError: + pass + if owns_directory: + try: + target.rmdir() + except OSError: + pass + if isinstance(exc, BrokenPipeError): + try: + sys.stdout.close() + except OSError: + pass + try: + print(f"testpilot: cannot check regression: {exc}", file=sys.stderr) + except OSError: + pass + return 2 diff --git a/tests/fixtures/regression_receiving.json b/tests/fixtures/regression_receiving.json new file mode 100644 index 0000000..7a48df3 --- /dev/null +++ b/tests/fixtures/regression_receiving.json @@ -0,0 +1,30 @@ +{ + "schema": "testpilot-paired-independent-fixtures/1", + "modules": { + "before": "def bucket(value, width):\n return int(value / width)\n", + "after": "def bucket(value, width):\n return value // width\n", + "unchanged": "def answer():\n return 42\n", + "drift_before": "CASES = ['before-only']\n", + "drift_after": "CASES = ['after-only']\n" + }, + "retained": { + "regression": "from pathlib import Path\nimport os\nimport pytest\nfrom buckets import bucket\n\n@pytest.mark.parametrize(('value', 'expected'), [(-1, -1), (-11, -2), (20, 2)], ids=['negative-edge', 'negative-next', 'positive-control'])\ndef test_partition(value, expected):\n assert bucket(value, 10) == expected\n\ndef test_original_committed_assets():\n assert Path('payload.txt').read_bytes() == 'Literal café \\n'.encode()\n assert Path('build_info.txt').read_text() == '$Format:%H$\\n'\n assert os.access('tool.py', os.X_OK)\n", + "unchanged": "from fixed import answer\n\ndef test_retained_answer():\n assert answer() == 42\n", + "drift": "import pytest\nfrom identities import CASES\n\n@pytest.mark.parametrize('label', CASES)\ndef test_identity(label):\n assert label.endswith('-only')\n", + "placement": "def test_no_execution_on_refusal():\n assert True\n" + }, + "assets": { + ".gitattributes": "payload.txt export-ignore\nbuild_info.txt export-subst\n", + "payload.txt": "Literal café \n", + "build_info.txt": "$Format:%H$\n", + "tool.py": "#!/usr/bin/env python3\nprint('fixture only')\n" + }, + "existing": { + "fail": "def test_existing():\n assert False, 'unrelated existing failure'\n", + "pass": "def test_existing():\n assert True\n" + }, + "report_context": { + "status": "recorded-generation-failed", + "diagnosis": "Historical context only; independently execute both supplied refs." + } +} diff --git a/tests/test_regression.py b/tests/test_regression.py new file mode 100644 index 0000000..bbd21be --- /dev/null +++ b/tests/test_regression.py @@ -0,0 +1,246 @@ +"""Real Git/native pytest controls for the paired retained-suite command.""" +from __future__ import annotations + +import hashlib +import importlib +import json +import os +from pathlib import Path +import shutil +import subprocess +import sys +import tempfile +import unittest +from unittest.mock import patch + +import testpilot + +SOURCE = Path(testpilot.__file__).resolve().parents[1] +RETAINED = "from subject import answer\n\ndef test_retained_answer():\n assert answer() == 7\n" + + +class RegressionTests(unittest.TestCase): + def setUp(self): + self.temporary = tempfile.TemporaryDirectory(prefix="testpilot-regression-test-") + self.addCleanup(self.temporary.cleanup) + self.root = Path(self.temporary.name) + self.repo = self.root / "project with spaces" + self.repo.mkdir() + self.env = {k: v for k, v in os.environ.items() if not k.upper().startswith("GIT_")} + self.env.update(GIT_CONFIG_NOSYSTEM="1", GIT_CONFIG_GLOBAL=os.devnull, + GIT_AUTHOR_NAME="TestPilot synthetic author", + GIT_AUTHOR_EMAIL="tests@example.invalid", + GIT_COMMITTER_NAME="TestPilot synthetic author", + GIT_COMMITTER_EMAIL="tests@example.invalid", + PYTHONDONTWRITEBYTECODE="1") + self.git("init", "--quiet", "--template=", "--initial-branch=main") + self.write("subject.py", "def answer():\n return 2\n") + self.write("tests/test_existing.py", "def test_existing():\n assert True\n") + self.before = self.commit("before") + self.write("subject.py", "def answer():\n return 7\n") + self.after = self.commit("after") + self.report = self.root / "saved.json" + self.save_report({"tests/test_retained.py": RETAINED}) + + def git(self, *args): + return subprocess.run(["git", "-C", str(self.repo), *args], env=self.env, + check=True, capture_output=True, text=True).stdout.strip() + + def write(self, name, content): + path = self.repo / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(content, encoding="utf-8", newline="") + return path + + def commit(self, message): + self.git("add", "--all") + self.git("-c", "core.hooksPath=" + os.devnull, "commit", "--quiet", "-m", message) + return self.git("rev-parse", "HEAD") + + def save_report(self, files, **extra): + self.report.write_text(json.dumps({"status": "verified", "test_files": files, **extra})) + + def module(self): + return importlib.import_module("testpilot.regression") + + def run_pair(self, **kwargs): + options = {"before": self.before, "after": self.after, + "python": sys.executable, "timeout_s": 10} + options.update(kwargs) + return self.module().check_regression(self.repo, self.report, **options) + + def protected(self): + return {str(p.relative_to(self.repo)): hashlib.sha256(p.read_bytes()).hexdigest() + for p in self.repo.rglob("*") if p.is_file()} + + def cli(self, *extra, output=None): + output = output or self.root / "paired" + env = dict(self.env, PYTHONPATH=str(SOURCE)) + return subprocess.run([sys.executable, "-B", "-m", "testpilot", "regression", + "--repo", str(self.repo), "--report", str(self.report), + "--before", self.before, "--after", self.after, + "--python", sys.executable, "--timeout", "10", + "--out", str(output), *extra], env=env, + capture_output=True, text=True, timeout=45) + + def test_real_cli_uses_committed_bytes_and_preserves_dirty_worktree(self): + self.write("subject.py", "raise RuntimeError('dirty source must not run')\n") + self.write("untracked.txt", "keep exactly\n") + original = self.protected() + report_bytes = self.report.read_bytes() + process = self.cli() + self.assertEqual(process.returncode, 0, process.stderr) + result = json.loads((self.root / "paired/regression.json").read_text()) + self.assertEqual(result["schema"], "testpilot.regression/1") + self.assertEqual(result["status"], "verified_regression") + self.assertTrue(result["ok"]) + self.assertEqual(result["before"]["commit"], self.before) + self.assertEqual(result["after"]["commit"], self.after) + self.assertEqual(result["before"]["tree"], self.git("rev-parse", self.before + "^{tree}")) + self.assertEqual(result["before"]["result"]["generated"]["failed"], 1) + self.assertEqual(result["after"]["result"]["generated"]["passed"], 1) + self.assertEqual(result["model_calls"], 0) + self.assertEqual(result["test_files"], {"tests/test_retained.py": RETAINED}) + self.assertEqual(len(result["comparison"]["witnesses"]), 1) + self.assertEqual(result["source_report"]["sha256"], hashlib.sha256(report_bytes).hexdigest()) + self.assertIn("verified_regression", (self.root / "paired/regression.md").read_text()) + self.assertEqual(self.protected(), original) + self.assertEqual(self.report.read_bytes(), report_bytes) + + def test_same_revision_and_old_recorded_status_do_not_create_proof(self): + result = self.run_pair(before=self.after) + self.assertEqual(result["status"], "not_regression") + self.assertFalse(result["ok"]) + self.assertEqual(result["source_report"]["recorded_status"], "verified") + self.assertEqual(result["comparison"]["witnesses"], []) + self.assertTrue(result["before"]["result"]["ok"]) + + def test_after_assertion_failure_is_after_failed(self): + result = self.run_pair(before=self.after, after=self.before) + self.assertEqual(result["status"], "after_failed") + self.assertTrue(result["comparison"]["comparable"]) + self.assertEqual(result["comparison"]["witnesses"], []) + + def test_existing_test_failure_cannot_prove_generated_regression(self): + self.write("tests/test_existing.py", "def test_existing():\n assert False\n") + old = self.commit("existing failure; generated already correct") + self.write("tests/test_existing.py", "def test_existing():\n assert True\n") + new = self.commit("existing correction") + result = self.run_pair(before=old, after=new) + self.assertEqual(result["before"]["result"]["failed"], 1) + self.assertEqual(result["before"]["result"]["generated"]["failed"], 0) + self.assertEqual(result["status"], "not_regression") + + def test_generated_skip_is_inconclusive(self): + self.save_report({"tests/test_retained.py": "import pytest\n" + RETAINED.replace( + "def test_retained_answer():", "@pytest.mark.skip(reason='no execution')\ndef test_retained_answer():")}) + result = self.run_pair() + self.assertEqual(result["status"], "inconclusive") + self.assertFalse(result["comparison"]["comparable"]) + self.assertEqual(result["after"]["result"]["generated"]["skipped"], 1) + + def test_collection_error_is_inconclusive(self): + self.save_report({"tests/test_retained.py": "import absent_testpilot_receiving_module\n" + RETAINED}) + result = self.run_pair() + self.assertEqual(result["status"], "inconclusive") + self.assertGreater(result["before"]["result"]["errors"], 0) + + def test_each_retained_file_must_collect(self): + self.save_report({"tests/test_retained.py": RETAINED, + "tests/test_no_cases.py": "# This saved file has no runnable tests.\n"}) + result = self.run_pair() + self.assertEqual(result["status"], "inconclusive") + self.assertEqual(result["comparison"]["reason"], "missing_generated_file_collection") + + def test_native_timeout_remains_inconclusive(self): + self.save_report({"tests/test_retained.py": "import time\ndef test_slow():\n time.sleep(4)\n"}) + result = self.run_pair(timeout_s=0.4) + self.assertEqual(result["status"], "inconclusive") + self.assertTrue(result["before"]["result"]["timed_out"]) + self.assertTrue(result["after"]["result"]["timed_out"]) + + def test_git_archive_attributes_do_not_remove_or_substitute_source(self): + self.write(".gitattributes", "subject.py export-ignore\nmetadata.txt export-subst\n") + self.write("metadata.txt", "$Format:%H$\n") + attributes = self.commit("original blobs retained") + self.save_report({"tests/test_retained.py": RETAINED + + "\ndef test_raw_metadata():\n from pathlib import Path\n" + " assert Path('metadata.txt').read_text() == '$Format:%H$\\n'\n"}) + result = self.run_pair(before=attributes, after=attributes) + self.assertEqual(result["status"], "not_regression") + self.assertEqual(result["after"]["result"]["generated"]["passed"], 2) + + def test_git_replacement_and_caller_routing_do_not_change_requested_source(self): + self.git("replace", self.before, self.after) + protected = self.protected() + with patch.dict(os.environ, {"GIT_DIR": str(self.root / "not-the-repository"), + "GIT_WORK_TREE": str(self.root)}): + result = self.run_pair() + self.assertEqual(result["status"], "verified_regression") + self.assertEqual(self.protected(), protected) + + def test_committed_retained_file_must_match_and_both_preflight_before_pytest(self): + self.write("tests/test_retained.py", "def test_wrong():\n assert True\n") + conflict = self.commit("conflicting retained path") + module = self.module() + with patch.object(module, "run_pytest", side_effect=AssertionError("must not execute")) as runner: + with self.assertRaisesRegex(module.RegressionError, "retained test differs"): + self.run_pair(after=conflict) + runner.assert_not_called() + self.write("tests/test_retained.py", RETAINED) + exact = self.commit("exact retained bytes") + result = self.run_pair(after=exact) + self.assertEqual(result["status"], "verified_regression") + + def test_unsupported_tree_mode_refuses_before_execution(self): + self.git("update-index", "--add", "--cacheinfo", "160000," + self.before + ",vendor") + self.git("-c", "core.hooksPath=" + os.devnull, "commit", "--quiet", "-m", "gitlink") + unsupported = self.git("rev-parse", "HEAD") + module = self.module() + with patch.object(module, "run_pytest", side_effect=AssertionError("must not execute")) as runner: + with self.assertRaisesRegex(module.RegressionError, "unsupported Git entry"): + self.run_pair(after=unsupported) + runner.assert_not_called() + + def test_limits_and_invalid_inputs_refuse_before_execution(self): + module = self.module() + protected = self.protected() + with patch.object(module, "run_pytest", side_effect=AssertionError("must not execute")) as runner: + for value in (0, -1, float("nan"), float("inf"), True): + with self.assertRaisesRegex(module.RegressionError, "finite positive"): + self.run_pair(timeout_s=value) + with self.assertRaises(module.RegressionError): + self.run_pair(after="does-not-exist") + with patch.object(module, "MAX_SOURCE_FILES", 1): + with self.assertRaisesRegex(module.RegressionError, "source files"): + self.run_pair() + with patch.object(module, "MAX_SOURCE_FILE_BYTES", 1): + with self.assertRaisesRegex(module.RegressionError, "source file"): + self.run_pair() + self.report.write_text('{"test_files": {}, "test_files": {}}') + with self.assertRaises(module.RegressionError): + self.run_pair() + runner.assert_not_called() + self.assertEqual(self.protected(), protected) + + def test_cli_refusals_preserve_existing_outputs_and_source(self): + prior = self.root / "existing" + prior.mkdir() + (prior / "regression.json").write_bytes(b"prior evidence\n") + protected = self.protected() + result = self.cli(output=prior) + self.assertEqual(result.returncode, 2, result.stderr) + self.assertNotIn("invalid choice", result.stderr) + self.assertEqual((prior / "regression.json").read_bytes(), b"prior evidence\n") + internal = self.repo / "output" + result = self.cli(output=internal) + self.assertEqual(result.returncode, 2) + self.assertFalse(internal.exists()) + result = self.cli("--after", "missing-reference") + self.assertEqual(result.returncode, 2) + self.assertFalse((self.root / "paired").exists()) + self.assertEqual(self.protected(), protected) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_regression_receiving.py b/tests/test_regression_receiving.py new file mode 100644 index 0000000..215a7ad --- /dev/null +++ b/tests/test_regression_receiving.py @@ -0,0 +1,246 @@ +"""Independent real Git/pytest receiving for the paired retained-suite command.""" +from __future__ import annotations + +import hashlib +import importlib +import importlib.util +import json +import os +import stat +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +import testpilot +from testpilot.recheck import read_saved_tests + +SOURCE = Path(testpilot.__file__).resolve().parent.parent +FIXTURE = json.loads((Path(__file__).parent / "fixtures" / "regression_receiving.json").read_text()) + + +class PairedRevisionReceiving(unittest.TestCase): + def setUp(self): + self.temporary = tempfile.TemporaryDirectory(prefix="testpilot-paired-peer-") + self.addCleanup(self.temporary.cleanup) + self.home = Path(self.temporary.name) + self.python = os.environ.get("TESTPILOT_RECEIVING_PYTHON", sys.executable) + self.env = dict(os.environ) + self.env.update( + PYTHONDONTWRITEBYTECODE="1", GIT_CONFIG_NOSYSTEM="1", + GIT_CONFIG_GLOBAL=os.devnull, GIT_AUTHOR_NAME="Native receiver", + GIT_AUTHOR_EMAIL="receiver@example.invalid", GIT_COMMITTER_NAME="Native receiver", + GIT_COMMITTER_EMAIL="receiver@example.invalid", + GIT_AUTHOR_DATE="2026-10-08T12:00:00Z", GIT_COMMITTER_DATE="2026-10-08T12:00:00Z", + ) + for name in list(self.env): + if name.startswith("GIT_") and name not in { + "GIT_CONFIG_NOSYSTEM", "GIT_CONFIG_GLOBAL", "GIT_AUTHOR_NAME", + "GIT_AUTHOR_EMAIL", "GIT_COMMITTER_NAME", "GIT_COMMITTER_EMAIL", + "GIT_AUTHOR_DATE", "GIT_COMMITTER_DATE", + }: + self.env.pop(name) + + def git(self, repo, *args): + run = subprocess.run( + ["git", "-C", str(repo), *args], env=self.env, capture_output=True, check=False, + ) + self.assertEqual(run.returncode, 0, run.stderr.decode()) + return run.stdout + + def repository(self): + repo = self.home / "source" + repo.mkdir() + self.git(repo, "init", "-q", "--initial-branch=main") + self.git(repo, "config", "core.autocrlf", "false") + return repo + + def commit(self, repo, files, ref): + for name, content in files.items(): + path = repo / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(content.encode()) + if name == "tool.py": + path.chmod(0o755) + self.git(repo, "add", "--all") + tree = self.git(repo, "write-tree").decode().strip() + # Deliberately no -p: the public comparison must admit non-ancestor refs. + commit = self.git(repo, "commit-tree", tree, "-m", ref).decode().strip() + self.git(repo, "update-ref", "refs/heads/" + ref, commit) + return commit, tree + + def report(self, retained): + path = self.home / "saved-report.json" + data = {**FIXTURE["report_context"], "test_files": retained} + path.write_bytes((json.dumps(data, ensure_ascii=False, indent=2) + "\n").encode()) + return path + + def inventory(self, repo): + return { + path.relative_to(repo).as_posix(): (stat.S_IMODE(path.stat().st_mode), hashlib.sha256(path.read_bytes()).hexdigest()) + for path in repo.rglob("*") if path.is_file() + } + + def signature(self, repo, commit): + entries = [] + for row in self.git(repo, "ls-tree", "-rz", "--full-tree", commit).split(b"\0"): + if not row: + continue + metadata, path = row.split(b"\t", 1) + mode, kind, blob = metadata.split() + self.assertEqual(kind, b"blob") + contents = self.git(repo, "cat-file", "blob", blob.decode()) + entries.append({"path": path.decode(), "mode": mode.decode(), "blob": blob.decode(), "bytes": len(contents)}) + entries.sort(key=lambda item: item["path"].encode()) + encoded = json.dumps(entries, ensure_ascii=True, separators=(",", ":")).encode() + return hashlib.sha256(encoded).hexdigest(), len(entries), sum(item["bytes"] for item in entries) + + def module(self): + self.assertIsNotNone(importlib.util.find_spec("testpilot.regression"), "original source has no paired regression capability") + return importlib.import_module("testpilot.regression") + + def cli(self, repo, report, before, after, out): + env = {**self.env, "PYTHONPATH": str(SOURCE)} + return subprocess.run( + [self.python, "-B", "-m", "testpilot", "regression", "--repo", str(repo), + "--report", str(report), "--before", before, "--after", after, + "--out", str(out), "--python", self.python, "--timeout", "20"], + cwd=self.home, env=env, capture_output=True, check=False, timeout=55, + ) + + def save_evidence(self, name, result): + destination = os.environ.get("TESTPILOT_RECEIVING_EVIDENCE") + if destination: + path = Path(destination) + path.mkdir(parents=True, exist_ok=True) + (path / (name + ".json")).write_text(json.dumps(result, indent=2) + "\n") + + def bucket_pair(self): + repo = self.repository() + assets = FIXTURE["assets"] + before = self.commit(repo, {**assets, "buckets.py": FIXTURE["modules"]["before"]}, "main") + after = self.commit(repo, {"buckets.py": FIXTURE["modules"]["after"]}, "after") + report = self.report({"tests/test_generated.py": FIXTURE["retained"]["regression"]}) + return repo, before, after, report + + def test_original_saved_report_control(self): + report = self.report({"tests/test_z.py": "def test_z():\n assert True\n", + "tests/test_a.py": FIXTURE["retained"]["unchanged"]}) + saved = read_saved_tests(report) + self.assertEqual([name for name, _ in saved.tests], ["tests/test_a.py", "tests/test_z.py"]) + self.assertEqual(saved.sha256, hashlib.sha256(report.read_bytes()).hexdigest()) + self.assertEqual(saved.source_status, "recorded-generation-failed") + + def test_actual_cli_uses_raw_nonancestor_commits_and_preserves_inputs(self): + repo, before, after, report = self.bucket_pair() + (repo / "buckets.py").write_text("raise RuntimeError('dirty checkout must not run')\n") + (repo / "untracked-note.txt").write_text("Keep my unfinished note.\n") + snapshot, original = self.inventory(repo), report.read_bytes() + out = self.home / "paired-output" + run = self.cli(repo, report, "main", "after", out) + self.assertEqual(run.returncode, 0, run.stderr.decode()) + result = json.loads((out / "regression.json").read_bytes()) + self.save_evidence("committed-cli-witness", result) + self.assertEqual(result["schema"], "testpilot.regression/1") + self.assertEqual(result["status"], "verified_regression") + self.assertIs(result["ok"], True) + self.assertEqual(result["model_calls"], 0) + self.assertEqual(result["test_files"], json.loads(original)["test_files"]) + self.assertEqual(result["source_report"]["sha256"], hashlib.sha256(original).hexdigest()) + self.assertEqual(result["source_report"]["recorded_status"], "recorded-generation-failed") + self.assertIs(result["comparison"]["comparable"], True) + witnesses = result["comparison"]["witnesses"] + self.assertEqual(len(witnesses), 2) + self.assertEqual({item["nodeid"].split("[")[-1] for item in witnesses}, {"negative-edge]", "negative-next]"}) + self.assertTrue(all(item["generated_file"] == "tests/test_generated.py" and item["before"] == "failed" and item["after"] == "passed" for item in witnesses)) + self.assertEqual(len(result["comparison"]["case_pairs"]), 4) + for phase, requested, expected in [("before", "main", before), ("after", "after", after)]: + received = result[phase] + signature, count, size = self.signature(repo, expected[0]) + self.assertEqual((received["commit"], received["tree"]), expected) + self.assertEqual(received["requested_ref"], requested) + self.assertEqual((received["snapshot_sha256"], received["source_files"], received["source_bytes"]), (signature, count, size)) + self.assertIs(received["result"]["junit_available"], True) + self.assertEqual(received["result"]["generated"]["collected"], 4) + self.assertIn("verified_regression", (out / "regression.md").read_text()) + self.assertEqual(report.read_bytes(), original) + self.assertEqual(self.inventory(repo), snapshot) + + def test_existing_suite_failure_never_becomes_a_generated_witness(self): + module = self.module() + repo = self.repository() + self.commit(repo, {"fixed.py": FIXTURE["modules"]["unchanged"], "tests/test_existing.py": FIXTURE["existing"]["fail"]}, "main") + self.commit(repo, {"tests/test_existing.py": FIXTURE["existing"]["pass"]}, "after") + report = self.report({"tests/test_generated.py": FIXTURE["retained"]["unchanged"]}) + snapshot, original = self.inventory(repo), report.read_bytes() + result = module.check_regression(repo, report, before="main", after="after", timeout_s=20, python=self.python) + self.save_evidence("existing-failure-not-witness", result) + self.assertEqual(result["status"], "not_regression") + self.assertIs(result["ok"], False) + self.assertIs(result["comparison"]["comparable"], True) + self.assertEqual(result["comparison"]["witnesses"], []) + self.assertEqual(result["before"]["result"]["failed"], 1) + self.assertEqual([(item["before"], item["after"]) for item in result["comparison"]["case_pairs"]], [("passed", "passed")]) + self.assertEqual((self.inventory(repo), report.read_bytes()), (snapshot, original)) + + def test_real_parametrized_identity_drift_is_inconclusive(self): + module = self.module() + repo = self.repository() + self.commit(repo, {"identities.py": FIXTURE["modules"]["drift_before"]}, "main") + self.commit(repo, {"identities.py": FIXTURE["modules"]["drift_after"]}, "after") + report = self.report({"tests/test_generated.py": FIXTURE["retained"]["drift"]}) + snapshot = self.inventory(repo) + result = module.check_regression(repo, report, before="main", after="after", timeout_s=20, python=self.python) + self.save_evidence("generated-identity-drift", result) + self.assertEqual(result["status"], "inconclusive") + self.assertIs(result["comparison"]["comparable"], False) + self.assertEqual(result["comparison"]["witnesses"], []) + self.assertIn("before-only", json.dumps(result["comparison"]["unmatched_before"])) + self.assertIn("after-only", json.dumps(result["comparison"]["unmatched_after"])) + self.assertEqual(self.inventory(repo), snapshot) + + def test_actual_reverse_cli_reports_after_failure(self): + repo, _, _, report = self.bucket_pair() + snapshot, original = self.inventory(repo), report.read_bytes() + out = self.home / "reversed-output" + run = self.cli(repo, report, "after", "main", out) + self.assertEqual(run.returncode, 1, run.stderr.decode()) + result = json.loads((out / "regression.json").read_bytes()) + self.save_evidence("reverse-after-failed", result) + self.assertEqual(result["status"], "after_failed") + self.assertIs(result["ok"], False) + self.assertIs(result["comparison"]["comparable"], True) + self.assertEqual(result["comparison"]["witnesses"], []) + self.assertEqual(result["after"]["result"]["generated"]["failed"], 2) + self.assertEqual((self.inventory(repo), report.read_bytes()), (snapshot, original)) + + def test_both_trees_are_admitted_before_execution_and_output_is_preserved(self): + module = self.module() + repo = self.repository() + marker = self.home / "collection-must-not-start" + conftest = f"from pathlib import Path\nPath({str(marker)!r}).write_text('pytest started')\n" + self.commit(repo, {"conftest.py": conftest}, "main") + self.commit(repo, {"tests/test_generated.py": "def test_conflicting():\n assert False\n"}, "after") + report = self.report({"tests/test_generated.py": FIXTURE["retained"]["placement"]}) + snapshot, original = self.inventory(repo), report.read_bytes() + with self.assertRaises(module.RegressionError): + module.check_regression(repo, report, before="main", after="after", timeout_s=20, python=self.python) + self.assertFalse(marker.exists()) + existing = self.home / "existing-output" + existing.mkdir() + (existing / "keep.txt").write_text("original output") + run = self.cli(repo, report, "main", "after", existing) + self.assertEqual(run.returncode, 2, run.stderr.decode()) + self.assertEqual(sorted(path.name for path in existing.iterdir()), ["keep.txt"]) + self.assertEqual((existing / "keep.txt").read_text(), "original output") + internal = repo / "forbidden-output" + refused = self.cli(repo, report, "main", "after", internal) + self.assertEqual(refused.returncode, 2, refused.stderr.decode()) + self.assertFalse(internal.exists()) + self.assertFalse(marker.exists()) + self.assertEqual((self.inventory(repo), report.read_bytes()), (snapshot, original)) + + +if __name__ == "__main__": + unittest.main(verbosity=2)