diff --git a/.agent-loop/initiatives/WS-QUAL-001-backend-coverage-floor/CHUNK_MAP.md b/.agent-loop/initiatives/WS-QUAL-001-backend-coverage-floor/CHUNK_MAP.md index 35e2de8c..6937f2a7 100644 --- a/.agent-loop/initiatives/WS-QUAL-001-backend-coverage-floor/CHUNK_MAP.md +++ b/.agent-loop/initiatives/WS-QUAL-001-backend-coverage-floor/CHUNK_MAP.md @@ -23,8 +23,8 @@ remain stopped historical experiments. Do not resume them. | `WS-QUAL-001-PLAN3` | Replace percentage-only closure with behavior/mutation assurance | L1 | Merged PR #272; late review corrected by PLAN3R1 | | `WS-QUAL-001-PLAN3R1` | Resolve five valid late CodeRabbit findings from PR #272 | L1 | Merged PR #278 | | `WS-QUAL-001-04P` | Establish protected hash-verified mutation dependency authority | L1 | Merged PR #281 | -| `WS-QUAL-001-04M` | Pilot pinned changed-scope mutation evidence without a score gate | L1 | Implemented and internally reviewed; hosted PR-head evidence pending | -| `WS-QUAL-001-05M` | Add calibrated blocking behavior-mutation policy | L1 | Proposed only after accepted 04M hosted evidence and explicit instruction | +| `WS-QUAL-001-04M` | Pilot pinned changed-scope mutation evidence without a score gate | L1 | Merged PR #285 as `7f395d47`; hosted calibration accepted | +| `WS-QUAL-001-05M` | Add calibrated blocking behavior-mutation policy | L1 | Active by explicit human instruction | ## Dependency rule diff --git a/.agent-loop/initiatives/WS-QUAL-001-backend-coverage-floor/STATUS.md b/.agent-loop/initiatives/WS-QUAL-001-backend-coverage-floor/STATUS.md index 17cca589..d832b3ea 100644 --- a/.agent-loop/initiatives/WS-QUAL-001-backend-coverage-floor/STATUS.md +++ b/.agent-loop/initiatives/WS-QUAL-001-backend-coverage-floor/STATUS.md @@ -19,21 +19,22 @@ findings. `WS-QUAL-001-04P` merged through PR #281 and established the protected, exactly pinned, hash-verified mutation dependency authority. -`WS-QUAL-001-04M` is implemented and internally reviewed. Its rebased exact-head -local pilot at `f8b59eec` completed in 254.672 seconds with 1,091 generated -mutants fully reconciled: 84 killed, 59 survived, 948 excluded, and zero error, -timeout, or suspicious outcomes. Strong calibration killed two representative -mutants and deliberately weak calibration left two alive. The focused policy -module is at 90.11 percent coverage. Hosted PR-head evidence remains required -before the human calibration checkpoint. +`WS-QUAL-001-04M` merged through PR #285 as `7f395d47`. Its final exact-head +hosted pilot on `0c25acec8fb3326e68169512e829711a0790b190` completed mutation +execution in 34.886 seconds and the hosted job in 53 seconds. It reconciled +2,493 generated mutants: 149 killed, 89 survived, 2,255 excluded, and zero +error, timeout, or suspicious outcomes. Strong calibration killed two +representative mutants and the deliberately weak calibration left two alive. +The human accepted this evidence and explicitly started `WS-QUAL-001-05M`. The corrected proposal remains two-stage: -1. `04M` — bounded, pinned, changed-scope mutation pilot with complete evidence - and no blocking score. -2. Human calibration checkpoint. -3. `05M` — separately approved blocking survivor policy for eligible changed - logic and explicit test-only behavior claims. +1. `04M` — merged bounded, pinned, changed-scope mutation pilot with complete + evidence and no blocking score. +2. Human calibration checkpoint — accepted. +3. `05M` — implemented and internally reviewed bounded blocking survivor + policy for eligible changed logic and explicit test-only behavior claims; + exact-head hosted CI and external review remain before human merge. The mutation score remains observational. Existing Backend semantic lanes, global 78-percent coverage, and protected 90-percent subsystem floors remain @@ -41,5 +42,4 @@ unchanged and blocking on their existing terms. ## Stop condition -Stop after the 04M review and PR. Do not start 05M without accepted exact hosted -pilot evidence, a human calibration checkpoint, and a new human instruction. +Stop after the 05M PR is merge-ready. Do not start another QUAL chunk. diff --git a/.agent-loop/initiatives/WS-QUAL-001-backend-coverage-floor/chunks/WS-QUAL-001-05M-blocking-behavior-mutation-gate.md b/.agent-loop/initiatives/WS-QUAL-001-backend-coverage-floor/chunks/WS-QUAL-001-05M-blocking-behavior-mutation-gate.md index 60556b38..4c7fe425 100644 --- a/.agent-loop/initiatives/WS-QUAL-001-backend-coverage-floor/chunks/WS-QUAL-001-05M-blocking-behavior-mutation-gate.md +++ b/.agent-loop/initiatives/WS-QUAL-001-backend-coverage-floor/chunks/WS-QUAL-001-05M-blocking-behavior-mutation-gate.md @@ -6,21 +6,19 @@ ## Goal -After accepted pilot evidence and separate human approval, make complete -mutation outcomes blocking for eligible changed production logic and explicit +Convert the accepted 04M changed-scope pilot into a required, fail-closed +behavior-mutation check for eligible Backend production changes and explicit test-only behavior claims. -## Why this chunk exists +## Accepted calibration input -The pilot measures feasibility. This separate chunk converts only calibrated, -deterministic evidence into contributor protection. - -## Approved plan reference - -- INTENT: `.agent-loop/initiatives/WS-QUAL-001-backend-coverage-floor/INTENT.md` -- PLAN: `.agent-loop/initiatives/WS-QUAL-001-backend-coverage-floor/PLAN.md` -- CHUNK_MAP: `.agent-loop/initiatives/WS-QUAL-001-backend-coverage-floor/CHUNK_MAP.md` -- Required input: accepted exact hosted `WS-QUAL-001-04M` pilot evidence. +PR #285 merged as `7f395d47`. Its final exact-head hosted pilot on +`0c25acec8fb3326e68169512e829711a0790b190` completed mutation execution in +34.886 seconds and the hosted job in 53 seconds. It reconciled 2,493 generated +mutants: 149 killed, 89 survived, 2,255 excluded, and zero timeout, suspicious, +or error outcomes. Strong calibration killed two representative mutants and +the deliberately weak control left two representative mutants alive. The +human accepted this calibration and explicitly started 05M. ## Risk class @@ -32,64 +30,172 @@ P2. ## Allowed files -Exact files must be refreshed from the merged 04M implementation before start. -Expected ownership is limited to its mutation policy/tests, independent -workflow, backend testing operations guide, Agent Gate invariant, and QUAL -initiative evidence. Test-only inputs use only schema-v1 -`.ci/behavior-claims/.json` files validated by the merged policy. -The refreshed allowed list must explicitly include `CONTRIBUTING.md`, -`.ci/behavior-claims/README.md`, the policy-owned schema, and a copyable example -so external contributors see the exact blocking workflow before it is enabled. +```text +backend/pyproject.toml +backend/scripts/mutation_policy.py +backend/tests/test_mutation_policy.py +scripts/behavior-claim.schema.json +scripts/mutation-requirements.in +scripts/mutation-requirements.txt +scripts/test_lightweight_agent_gates.py +.ci/behavior-claims/README.md +.ci/behavior-claims/WS-QUAL-001-05M.json +.ci/behavior-claims/example.behavior-claim.json +.github/workflows/mutation-pilot.yml +CONTRIBUTING.md +docs/operations_backend_testing.md +.agent-loop/initiatives/WS-QUAL-001-backend-coverage-floor/** +``` ## Not allowed ```text -start without accepted 04M evidence and explicit human instruction -global mutation percentage -change to 78-percent global or protected 90-percent coverage floors -free-form exemptions, source mutation pragmas, silent timeout/error success +backend/app/** or backend/alembic/** product changes +global mutation score or percentage threshold +change to the global 78-percent or protected 90-percent coverage floors +free-form exemptions, source mutation pragmas, or survivor allowlists +silent success for survivor, timeout, suspicious, error, stale, or unknown status full-repository mutation on ordinary PRs -production behavior, migration, or dependency changes +production dependency changes unrelated to the protected mutation runner +Backend semantic-lane, fan-in, coverage, or test-inventory weakening +pull_request_target, privileged PR execution, writable token, checkout credentials, +secrets in mutation execution, or unpinned Actions ``` +## Policy design + +- Eligible changed Python targets under the merged policy remain mandatory. +- One canonical schema-v1 claim file supplies bounded callable and owning-test + scope. The workflow discovers it from the exact base/head delta; workflow + inputs, labels, environment variables, and PR prose cannot choose it. +- For eligible production changes, the protected evaluator maps every added or + modified executable hunk to its current qualified function, async function, + or method and requires the claim to own every derived callable. Unmapped + module/class executable changes, deleted-only executable hunks, omitted + callables, and unrelated-callable claims fail closed rather than silently + narrowing mutation scope. +- Disposable mutmut configuration is derived from the validated exact-head + selection. Contributors do not edit a second target/test list in + `backend/pyproject.toml`. +- `killed` passes. `survived`, `timeout`, `suspicious`, and `error` block. + `excluded` passes only as an engine result outside the reviewed callable + filters and remains fully enumerated in evidence. Unknown or missing statuses + block. +- The only allowed surviving classification is the repository-owned + `calibration_control` for mutants belonging to the exact + `_weak_calibration` callable. It is derived and verified by policy, never + supplied as contributor prose or an allowlist. +- Changes with no eligible target and no behavior claim are deterministically + `not_applicable`; they do not install or run the mutation engine. An explicit + test-only behavior claim remains additive and cannot remove changed targets. +- Ordinary pull requests execute selection, config generation, evidence + validation, outcome classification, and final verdict with the evaluator + and every imported policy helper archived from the protected base revision. + Protected evaluation must not import PR-head policy helpers. Candidate policy + and workflow changes therefore cannot become their own authority. The 05M bootstrap PR is + reviewed under merged 04M plus all internal/external checks; its first + protected-main run must prove the new evaluator before it becomes authority + for later PRs. GitHub permissions, required checks, and explicit human merge + remain the authority for changes to the evaluator/workflow themselves. + ## Acceptance criteria -- [ ] Eligibility and evidence grammar are unchanged from accepted pilot proof - unless a separately reviewed correction is explicit. -- [ ] Policy enumerates every engine status, including killed, survived, - suspicious, timeout, error, and excluded. Each status blocks or maps - to an independently verified, typed policy classification. -- [ ] Any allowed classification is narrow, typed, evidence-bound, and tested; - missing, stale, broad, free-form, or unrecognized classifications fail - closed, and no status passes implicitly. -- [ ] Test-only behavior/coverage claims cannot bypass target mutation. -- [ ] `CONTRIBUTING.md` and the canonical claim README/schema/example explain - when a claim is required, the permitted typed non-behavioral cases, local - verification, evidence interpretation, and repair of surviving mutants. -- [ ] Non-eligible maintenance/docs/generated changes do not run irrelevant - mutants. -- [ ] Hosted p95 and critical-path impact satisfy the accepted pilot bound. -- [ ] Backend, Agent Gates, 78-percent global floor, and protected 90-percent - floors remain authoritative and green. +- [ ] The pull-request workflow always emits one stable required check; it does + not use workflow-level path filters that leave skipped required checks + pending. Internal preflight returns typed `not_applicable` before mutation + dependency installation/execution for unrelated changes. +- [ ] Exactly one changed canonical claim is discovered when mutation is + applicable; missing, multiple, unsafe, stale, symlinked, or mismatched + claims fail closed. +- [ ] Eligibility and schema-v1 behavior ownership remain compatible with 04M; + any 05M discovery/configuration correction is explicit and tested. +- [ ] Every added/modified executable hunk in an eligible production target is + mapped to and covered by the exact claimed callable. Omitted or unrelated + callables, nested methods, async functions, decorators, module/class-level + executable changes, renames, and deleted-only hunks have explicit + deterministic pass/block behavior and cannot escape mutation silently. +- [ ] Disposable mutmut configuration is deterministically generated from the + validated selection and bound into exact-head evidence; mutable static + target/test duplication is removed. +- [ ] Every engine status is enumerated. Survivor, timeout, suspicious, error, + unknown, incomplete, and stale evidence block without implicit success. +- [ ] `excluded` passes only when the exact mutant is outside the reviewed + callable filters; an excluded mutant matching a selected filter blocks. +- [ ] The only allowed survivor is a policy-derived exact weak-calibration + control; contributor-authored classifications and free-form exemptions + are rejected. +- [ ] Test-only behavior claims cannot replace or narrow mandatory changed + targets. No-target/no-claim changes return typed `not_applicable` rather + than fabricating mutation evidence. +- [ ] Baseline failure, target escape, source-tree mutation, custody failure, + dependency drift, and malformed evidence remain blocking. +- [ ] `CONTRIBUTING.md`, the canonical claim README/schema/example, and the + Backend operations guide explain applicability, claim creation, local + verification, evidence interpretation, and repair of survivors. +- [ ] The mutation toolchain comes only from the trusted-base hash-locked + manifest. Eligible owning tests run with production/dev dependencies + resolved only from the trusted-base `backend/uv.lock` and + `backend/pyproject.toml`; PR-head dependency metadata cannot become gate + authority. +- [ ] The one-time 04M-to-05M bootstrap may use the reviewed head mutation + manifest because protected 04M does not contain the locked `uv` runner; + after merge, ordinary PRs and protected-main runs use only the protected + base manifest and backend lock metadata. +- [ ] Ordinary PR verdicts use the protected-base evaluator, not PR-head policy + code or helpers. Workflow invariants prove the evaluator and all imported + policy helpers come from the protected base. The bootstrap transition is + explicit and a successful protected `main` run is required before 05M is + considered operational. +- [ ] Required mutation execution remains independent and within the accepted + 15-minute job cap, retains a hard 720-second shell command limit and + 700-second engine limit, and does not extend required PR critical-path + latency by more than two minutes. Full Backend fan-in, Agent Gates, global + 78-percent coverage, and protected 90-percent floors remain authoritative + and green. +- [ ] All required internal reviewers pass and external CI/review is resolved + before human merge. ## Verification commands -Refresh exact commands from merged 04M; at minimum run mutation-policy unit and -integration tests, strong/weak seeded behavior proof, Ruff, Agent Gates, -Markdown/stale checks, full hosted Backend, and exact blocking-workflow proof. +```bash +cd backend +.venv/bin/python -m pytest -q tests/test_mutation_policy.py +.venv/bin/python -m pytest -q tests/test_ci_test_lanes.py tests/test_coverage_contract.py +.venv/bin/ruff check scripts/mutation_policy.py tests/test_mutation_policy.py +cd .. +PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python3 -m pytest -q scripts/test_lightweight_agent_gates.py +python3 scripts/check_markdown_links.py +python3 scripts/check_stale_workstream_wording.py +python3 scripts/check_stale_authorization_docs.py +python3 scripts/check_stale_artifact_contracts.py +git diff --check +``` + +Focused policy tests must include docs/no-target `not_applicable`, applicable +changes without a claim, exact/additive claims, multiple/stale/symlinked claims, +omitted/unrelated/nested/async/decorator/module/deleted callable cases, every +engine status, missing/unknown outcomes, weak-calibration-only survival, and +allowed/rejected excluded mutants. Workflow invariant tests must prove +preflight occurs before dependency installation and ordinary verdicts use the +protected-base evaluator. Hosted PR evidence must include the blocking workflow +result and timing plus successful Backend lane/fan-in coverage and Agent Gates. ## Required reviewers -Senior engineering, QA/test, security/auth, product/ops, architecture, CI -integrity, docs, reuse/dedup, and test delta. +Senior engineering, QA/test, security, product/ops, architecture, CI integrity, +docs, reuse/dedup, and test delta. ## Human review focus -- Does the gate block weak behavior proof without blocking unrelated work? -- Can classifications or test selection be used as an escape hatch? -- Does the gate remain practical for external contributors? +- Does meaningful surviving behavior always block without a contributor escape + hatch? +- Is the exact weak calibration the only survivor that can pass? +- Can an ordinary contributor determine applicability and create the claim + without editing workflow internals or duplicating configuration? +- Does unrelated work avoid an irrelevant mutation job? ## Stop conditions -Stop on missing pilot evidence, unreviewed policy change, unacceptable hosted -latency/noise, coverage/Backend weakening, or need for broad exemptions. +Stop on an unclassifiable required survivor, unacceptable hosted latency/noise, +need for a broad exemption, weakening of Backend/coverage/test inventory, or a +new dependency, product, authorization, payment, migration, or data boundary. diff --git a/.agent-loop/initiatives/WS-QUAL-001-backend-coverage-floor/reviews/WS-QUAL-001-05M-external-review-response.md b/.agent-loop/initiatives/WS-QUAL-001-backend-coverage-floor/reviews/WS-QUAL-001-05M-external-review-response.md new file mode 100644 index 00000000..1e9c2a6f --- /dev/null +++ b/.agent-loop/initiatives/WS-QUAL-001-backend-coverage-floor/reviews/WS-QUAL-001-05M-external-review-response.md @@ -0,0 +1,26 @@ +# WS-QUAL-001-05M External Review Response + +## CodeRabbit review on `12bb1d96` + +Four substantive findings were accepted and fixed: + +1. Plain, undecorated class headers are always executable spans, so a class + rename cannot escape callable ownership. +2. Protected blocking capability detection uses the explicit + `workstream-mutation-capability:discover-v1` marker rather than argparse + source formatting. +3. Deleted eligible targets fail closed with `deleted_eligible_target`; absent + executable logic cannot be mutated and cannot be classified as unrelated. +4. Generated TOML reparse failures become + `invalid_generated_mutation_config` rather than escaping with a traceback. + +The valid test suggestions were also applied: preserved non-mutmut TOML, +nested-function ownership, independent module-level/deleted-callable guards, +and missing owner-target rejection now have regressions. + +The suggested owner-record refactor was not applied because it was explicitly +low value and would increase review surface without changing the corrected +behavior. The textual TOML rewrite remains fail closed; typed failure is the +required safety property for unsupported legacy shapes. + +Exact-head hosted CI and CodeRabbit rereview remain required after publication. diff --git a/.agent-loop/initiatives/WS-QUAL-001-backend-coverage-floor/reviews/WS-QUAL-001-05M-internal-review-evidence.md b/.agent-loop/initiatives/WS-QUAL-001-backend-coverage-floor/reviews/WS-QUAL-001-05M-internal-review-evidence.md new file mode 100644 index 00000000..2b8fd294 --- /dev/null +++ b/.agent-loop/initiatives/WS-QUAL-001-backend-coverage-floor/reviews/WS-QUAL-001-05M-internal-review-evidence.md @@ -0,0 +1,64 @@ +# WS-QUAL-001-05M Internal Review Evidence + +## Deterministic evidence gate + +Result: PASS with the approved L1 size exception. The policy, protected +workflow, schema/example, focused tests, dependency custody, and contributor +documentation form one blocking-gate review boundary. No product code, +migration, Backend lane, coverage floor, or product lifecycle changed. + +Commands and results: + +- Exact discovery against merged `04M`: `applicable`, one `05M` claim, 19 + changed policy callables, and exact ownership. +- Mutation-policy coverage: 58 tests passed; 91.97 percent. +- Mutation policy, CI-lane, and coverage-contract suites: 263 tests passed. +- Lightweight repository gate suite: 11 tests passed. +- Ruff, diff check, Markdown links, and stale Workstream wording: passed. +- Protected dependency replay: hash-locked `mutmut==3.7.0` and `uv==0.11.7` + installed; protected-lock FastAPI, SQLAlchemy, Pydantic settings, and + pytest-asyncio imports passed; mutation-policy suite passed in that + disposable environment. + +Hosted GitHub Actions and CodeRabbit remain external merge gates on the exact +published head. + +## Reviewer results + +| Track | Result | Material outcome | +|---|---|---| +| Plan | PASS | Exact authority, applicability, runtime, and proof requirements were explicit before implementation. | +| Architecture | PASS after fixes | Calibration is policy-owned and exact; changed-callable ownership cannot be narrowed by a claim. | +| Senior engineering | PASS with low risk | Protected locked backend dependencies make eligible app tests executable; stable legacy `pilot` job id is intentionally retained for check continuity. | +| QA | PASS after fixes | Merge-base mapping, AST existence, calibration impostors, and claim-only targets are covered. | +| Security | PASS after fix | Candidate tests receive a minimal environment without GitHub command files or credentials. | +| CI integrity | PASS after fix | Protected-base evaluator/dependencies, stable always-emitted job, bounded execution, and existing Backend gates remain intact. | +| Test delta | PASS with low risk | Full enforced-survivor failure and complete generated TOML configuration are regression-tested; no skips or weakened assertions. | +| Docs | PASS after correction | Local discovery and hosted selection/evidence interpretation are documented. | +| Product/ops | PASS with low risk | Engineering evidence remains separate from Workstream product review, compensation, and reputation. | +| Reuse/dedup | PASS with low risk | Existing Git-delta discovery is reused; line-range parsing can move to the shared helper in later maintenance. | + +## Findings resolved + +- Replaced contributor-coupled calibration selection and substring identity + with exact policy-owned strong and weak controls. +- Required exact equality between changed callables and claim ownership. +- Used merge-base hunks and rejected claimed callables absent from the target + AST. +- Preserved the stable required job id and future protected-main enforcement. +- Removed secrets and GitHub command-file paths from candidate execution. +- Added protected-base locked backend runtime/test dependencies so eligible + `backend/app/**` tests reach mutation rather than failing imports. +- Added end-to-end enforced-survivor and complete generated-config tests. +- Added copyable local discovery and evidence-repair guidance. +- Made plain class headers fail closed, replaced formatting-sensitive capability + detection, made deleted eligible targets fail closed, and typed + generated-TOML parse failure after CodeRabbit review. + +## Residual low risks + +- The stable job id remains `pilot` for branch-protection continuity although + the workflow display name is `Behavior Mutation Gate`. +- Workflow structure has string-invariant tests; the critical enforcement path + is exercised behaviorally in the backend suite. +- Local diff-line parsing may be centralized in `scripts/git_delta.py` later. diff --git a/.agent-loop/initiatives/WS-QUAL-001-backend-coverage-floor/reviews/WS-QUAL-001-05M-pr-trust-bundle.md b/.agent-loop/initiatives/WS-QUAL-001-backend-coverage-floor/reviews/WS-QUAL-001-05M-pr-trust-bundle.md new file mode 100644 index 00000000..e5d1dc16 --- /dev/null +++ b/.agent-loop/initiatives/WS-QUAL-001-backend-coverage-floor/reviews/WS-QUAL-001-05M-pr-trust-bundle.md @@ -0,0 +1,98 @@ +# WS-QUAL-001-05M PR Trust Bundle + +## Chunk + +`WS-QUAL-001-05M` — Blocking Changed-Scope Behavior Mutation Gate (L1). + +## Goal and human-approved intent + +Turn the accepted `04M` observational calibration into a bounded required gate: +eligible changed behavior must have exact ownership, owning tests, and no +meaningful surviving selected mutant. The user explicitly accepted calibration +and started this chunk. + +## What changed and why + +- Replaced mutable static mutation configuration with exact selection-derived + disposable configuration. +- Added exact claim discovery, merge-base callable mapping, AST existence, and + fail-closed status reconciliation. +- Made the stable `pilot` check always emit and block meaningful survivors. +- Kept evaluator, helper, mutation toolchain, and backend dependency authority + at protected base after the explicit one-time bootstrap. +- Added contributor claim/schema/example guidance, local discovery, evidence + interpretation, focused tests, and 90-percent subsystem coverage proof. + +Coverage alone does not prove assertions detect wrong behavior. This gate adds +that proof only for the changed, explicitly owned callable scope; it does not +introduce a global mutation score. + +## Design chosen and alternatives rejected + +The protected evaluator derives every eligible changed target and callable. +One exact claim supplies owning tests, observable outcomes, and real boundaries +but cannot narrow the derived scope. Mutmut runs in an archived disposable tree +with a hash-locked runner and runtime/test dependencies resolved from the +protected backend lock. Exact policy-owned strong/weak controls calibrate the +engine. All meaningful selected survivors and unsafe/incomplete statuses block. + +Rejected: full-repository mutation, score thresholds, PR-head evaluator or +dependencies, free-form survivor exemptions, workflow path filters, and +running candidate tests with repository/CI credentials. + +## Scope control and product behavior + +No application behavior, migration, Workstream product lifecycle, +authorization, payment, reputation, Backend lane, or coverage threshold +changed. Review decisions remain `accept`, `needs_revision`, and `reject`; +mutation findings are engineering evidence only. Existing global 78-percent +and protected 90-percent floors remain blocking. + +## Acceptance proof and test delta + +- Exact discovery: applicable `05M`, one claim, exact changed-callable ownership. +- Mutation policy: 58 passed; 91.97-percent focused coverage. +- Policy/CI/coverage contract suites: 263 passed. +- Repository workflow invariants: 11 passed. +- Locked disposable environment replay: backend imports and policy suite passed. +- Ruff, Markdown links, stale wording, and diff checks passed. +- No test was removed, skipped, xfailed, deselected, or weakened. +- Enforced selected-survivor failure and the complete generated mutmut table + now have behavioral regression tests. + +## CI integrity and reviewer results + +The workflow retains stable job id `pilot`, read-only permissions, pinned +Actions, no path filters, preflight before installation, protected-base +authority, 15-minute job/720-second command/700-second engine limits, and +exact-head evidence. Architecture, senior engineering, QA, security, +product/ops, CI integrity, docs, reuse/dedup, and test-delta tracks pass after +valid findings were fixed. Detailed results are in the internal review evidence. + +## External review + +CodeRabbit's first review produced four substantive findings. They were fixed: +plain class headers now fail closed, capability detection uses an explicit +marker, deleted eligible targets fail closed, and generated-TOML reparse +failures are typed. GitHub Actions and CodeRabbit must rerun on the exact final +head. The bootstrap PR is nonblocking for mutation outcomes because merged +`04M` lacks the new evaluator and locked `uv` runner; the first protected-main +run establishes `05M` authority for later PRs. + +## Remaining risks and follow-up + +Hosted dependency/runtime timing must remain within the accepted limits. The +stable internal job id `pilot` is retained to avoid breaking required-check +configuration. Shared diff-line parsing may be deduplicated later; neither is a +merge blocker. Do not start another QUAL chunk from this PR. + +## Human review focus and merge ownership + +- Confirm protected-base evaluator and dependency custody after bootstrap. +- Confirm exact callable ownership cannot omit changed behavior. +- Inspect hosted `selection.json`, `executed-selection.json`, and + `evidence.json`, especially calibration and `verdict.blockers`. +- Confirm Backend, Agent Gates, CodeRabbit, and this required gate pass on the + exact final head. + +Only the user may approve and merge this specific PR. diff --git a/.ci/behavior-claims/README.md b/.ci/behavior-claims/README.md index f08c4bf8..ce96a4c5 100644 --- a/.ci/behavior-claims/README.md +++ b/.ci/behavior-claims/README.md @@ -11,7 +11,38 @@ any essential real boundaries. Unknown fields, unsafe paths, missing files, duplicate entries, unowned changed targets, or stale chunk identifiers fail closed. -During the `WS-QUAL-001-04M` pilot, results are observational. Infrastructure, -custody, selection, baseline-test, or evidence failures remain blocking, but no -mutation percentage is a contributor requirement until a later human-approved -policy chunk. +The required behavior-mutation check discovers the one claim changed by the +pull request; labels, workflow inputs, environment variables, and PR prose +cannot select it. Copy `example.behavior-claim.json`, rename it to the bounded +chunk identifier, and replace every example target, callable, test, outcome, +and boundary. Eligible production changes without exactly one changed claim +fail closed. A test-only behavior claim is additive and cannot remove an +eligible changed target. + +The check has no mutation percentage. Killed mutants pass. A meaningful +survivor, timeout, suspicious result, engine error, malformed or stale evidence, +target escape, or excluded mutant inside the selected callable scope blocks. +The only surviving control allowed by policy is Workstream's exact deliberately +weak calibration callable; contributors cannot add survivor allowlists, +free-form exemptions, or source mutation pragmas. + +Changes with no eligible target and no claim produce typed `not_applicable` +evidence before the mutation toolchain is installed. Ordinary PR verdicts are +calculated by the evaluator and Git-delta helper archived from protected base, +not by PR-head policy code. + +Validate claim discovery locally from the repository root: + +```bash +backend/.venv/bin/python backend/scripts/mutation_policy.py \ + --repository-root . \ + --base-sha "$(git merge-base origin/main HEAD)" \ + --head-sha "$(git rev-parse HEAD)" \ + --discover \ + --selection-output /tmp/workstream-mutation-selection.json +``` + +An unrelated delta reports `applicability: not_applicable`. An applicable +delta must report the exact changed targets, callable ownership, and owning +tests expected by the contributor. Discovery errors are policy failures; fix +the claim or delta rather than editing generated evidence. diff --git a/.ci/behavior-claims/WS-QUAL-001-05M.json b/.ci/behavior-claims/WS-QUAL-001-05M.json new file mode 100644 index 00000000..ee6355c9 --- /dev/null +++ b/.ci/behavior-claims/WS-QUAL-001-05M.json @@ -0,0 +1,48 @@ +{ + "schema_version": 1, + "chunk_id": "WS-QUAL-001-05M", + "claims": [ + { + "target": "backend/scripts/mutation_policy.py", + "callables": [ + "scripts.mutation_policy.discover_claim_path", + "scripts.mutation_policy._diff_lines", + "scripts.mutation_policy._source_at", + "scripts.mutation_policy._safe_path", + "scripts.mutation_policy._minimal_runtime_environment", + "scripts.mutation_policy._callable_spans", + "scripts.mutation_policy._callable_spans.visit", + "scripts.mutation_policy._map_changed_lines", + "scripts.mutation_policy.changed_callables", + "scripts.mutation_policy._read_claim", + "scripts.mutation_policy.build_selection", + "scripts.mutation_policy.discover_selection", + "scripts.mutation_policy._write_mutmut_config", + "scripts.mutation_policy._mutant_filters", + "scripts.mutation_policy.classify_outcomes", + "scripts.mutation_policy.policy_self_test", + "scripts.mutation_policy._validate_calibration", + "scripts.mutation_policy.execute_pilot", + "scripts.mutation_policy._main" + ], + "tests": [ + "backend/tests/test_mutation_policy.py::TestMutationPolicy::test_changed_targets_are_mandatory_and_claims_are_additive", + "backend/tests/test_mutation_policy.py::TestMutationPolicy::test_claim_validation_fails_closed", + "backend/tests/test_mutation_policy.py::TestMutationPolicy::test_no_target_no_claim_is_typed_not_applicable", + "backend/tests/test_mutation_policy.py::TestMutationPolicy::test_applicable_delta_requires_one_changed_claim", + "backend/tests/test_mutation_policy.py::TestMutationPolicy::test_multiple_changed_claims_fail_closed", + "backend/tests/test_mutation_policy.py::TestMutationPolicy::test_changed_callable_mapping_covers_decorated_async_and_nested_methods", + "backend/tests/test_mutation_policy.py::TestMutationPolicy::test_module_level_and_deleted_callable_changes_fail_closed", + "backend/tests/test_mutation_policy.py::TestMutationPolicy::test_blocking_verdict_allows_only_weak_control_and_unselected_exclusions", + "backend/tests/test_mutation_policy.py::TestMutationPolicy::test_incomplete_and_unknown_outcomes_fail_closed", + "backend/tests/test_mutation_policy.py::TestMutationPolicy::test_strong_calibration_asserts_the_exact_boundary", + "backend/tests/test_mutation_policy.py::TestMutationPolicy::test_weak_calibration_deliberately_asserts_only_the_result_type" + ], + "outcomes": [ + "return", + "mapped_error" + ], + "boundaries": [] + } + ] +} diff --git a/.ci/behavior-claims/example.behavior-claim.json b/.ci/behavior-claims/example.behavior-claim.json new file mode 100644 index 00000000..d71d95e6 --- /dev/null +++ b/.ci/behavior-claims/example.behavior-claim.json @@ -0,0 +1,21 @@ +{ + "schema_version": 1, + "chunk_id": "WS-EXAMPLE-001-01", + "claims": [ + { + "target": "backend/app/modules/example/service.py", + "callables": [ + "app.modules.example.service.ExampleService.create" + ], + "tests": [ + "backend/tests/test_example_service.py::test_create_persists_the_result" + ], + "outcomes": [ + "persisted_state" + ], + "boundaries": [ + "postgresql" + ] + } + ] +} diff --git a/.github/workflows/mutation-pilot.yml b/.github/workflows/mutation-pilot.yml index 3036d205..0dc1b954 100644 --- a/.github/workflows/mutation-pilot.yml +++ b/.github/workflows/mutation-pilot.yml @@ -1,31 +1,13 @@ -name: Mutation Pilot +name: Behavior Mutation Gate on: pull_request: - paths: - - "backend/scripts/mutation_policy.py" - - "backend/tests/test_mutation_policy.py" - - "backend/pyproject.toml" - - "scripts/git_delta.py" - - "scripts/test_git_delta.py" - - "scripts/behavior-claim.schema.json" - - ".ci/behavior-claims/WS-QUAL-001-04M.json" - - ".github/workflows/mutation-pilot.yml" push: branches: - main - paths: - - "backend/scripts/mutation_policy.py" - - "backend/tests/test_mutation_policy.py" - - "backend/pyproject.toml" - - "scripts/git_delta.py" - - "scripts/test_git_delta.py" - - "scripts/behavior-claim.schema.json" - - ".ci/behavior-claims/WS-QUAL-001-04M.json" - - ".github/workflows/mutation-pilot.yml" concurrency: - group: mutation-pilot-${{ github.event.pull_request.number || github.ref }} + group: behavior-mutation-${{ github.event.pull_request.number || github.ref }} cancel-in-progress: true permissions: @@ -36,8 +18,8 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 15 env: - PILOT_HEAD_SHA: ${{ github.event.pull_request.head.sha || github.sha }} - PILOT_BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }} + MUTATION_HEAD_SHA: ${{ github.event.pull_request.head.sha || github.sha }} + MUTATION_BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }} steps: - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 @@ -51,65 +33,114 @@ jobs: python-version: "3.12" - id: custody - name: Bind exact tree and protected dependency authority + name: Bind exact tree and protected evaluator authority shell: bash run: | set -euo pipefail - pilot_dir="${RUNNER_TEMP}/workstream-mutation-pilot" - test "$(git rev-parse HEAD)" = "${PILOT_HEAD_SHA}" + gate_dir="${RUNNER_TEMP}/workstream-mutation-gate" + evaluator_dir="${gate_dir}/protected-evaluator" + test "$(git rev-parse HEAD)" = "${MUTATION_HEAD_SHA}" test -z "$(git status --porcelain)" head_tree="$(git rev-parse 'HEAD^{tree}')" - base_sha="$(git rev-parse "${PILOT_BASE_SHA}^{commit}")" - install -d -m 700 "${pilot_dir}" - git show "${base_sha}:scripts/mutation-requirements.txt" \ - > "${pilot_dir}/protected-requirements.txt" - manifest_sha256="$(sha256sum "${pilot_dir}/protected-requirements.txt" | cut -d ' ' -f 1)" + base_sha="$(git rev-parse "${MUTATION_BASE_SHA}^{commit}")" + install -d -m 700 "${evaluator_dir}/backend/scripts" "${evaluator_dir}/scripts" "${gate_dir}/protected-backend" + git show "${base_sha}:backend/scripts/mutation_policy.py" > "${evaluator_dir}/backend/scripts/mutation_policy.py" + git show "${base_sha}:scripts/git_delta.py" > "${evaluator_dir}/scripts/git_delta.py" + evaluator="${evaluator_dir}/backend/scripts/mutation_policy.py" + bootstrap=false + if ! grep -q -- 'workstream-mutation-capability:discover-v1' "${evaluator}"; then + bootstrap=true + evaluator="${GITHUB_WORKSPACE}/backend/scripts/mutation_policy.py" + fi + git show "${base_sha}:scripts/mutation-requirements.txt" > "${gate_dir}/protected-requirements.txt" + if [[ "${bootstrap}" == "true" ]]; then + cp "${GITHUB_WORKSPACE}/scripts/mutation-requirements.txt" "${gate_dir}/protected-requirements.txt" + fi + git show "${base_sha}:backend/pyproject.toml" > "${gate_dir}/protected-backend/pyproject.toml" + git show "${base_sha}:backend/uv.lock" > "${gate_dir}/protected-backend/uv.lock" + manifest_sha256="$(sha256sum "${gate_dir}/protected-requirements.txt" | cut -d ' ' -f 1)" echo "head_tree=${head_tree}" >> "${GITHUB_OUTPUT}" echo "base_sha=${base_sha}" >> "${GITHUB_OUTPUT}" + echo "evaluator=${evaluator}" >> "${GITHUB_OUTPUT}" + echo "bootstrap=${bootstrap}" >> "${GITHUB_OUTPUT}" echo "manifest_sha256=${manifest_sha256}" >> "${GITHUB_OUTPUT}" + - id: preflight + name: Discover exact mutation applicability before installation + shell: bash + run: | + set -euo pipefail + gate_dir="${RUNNER_TEMP}/workstream-mutation-gate" + python "${{ steps.custody.outputs.evaluator }}" \ + --repository-root . \ + --base-sha "${{ steps.custody.outputs.base_sha }}" \ + --head-sha "${MUTATION_HEAD_SHA}" \ + --discover \ + --selection-output "${gate_dir}/selection.json" + applicability="$(python -c 'import json,sys; print(json.load(open(sys.argv[1]))["applicability"])' "${gate_dir}/selection.json")" + echo "applicability=${applicability}" >> "${GITHUB_OUTPUT}" + + - name: Verify protected-main blocking evaluator + if: ${{ github.event_name == 'push' }} + shell: bash + run: python "${{ steps.custody.outputs.evaluator }}" --self-test + - name: Install protected hash-locked mutation toolchain + if: ${{ steps.preflight.outputs.applicability == 'applicable' }} shell: bash run: | set -euo pipefail - pilot_dir="${RUNNER_TEMP}/workstream-mutation-pilot" - python -m venv "${pilot_dir}/venv" - "${pilot_dir}/venv/bin/python" -m pip install \ + gate_dir="${RUNNER_TEMP}/workstream-mutation-gate" + python -m venv "${gate_dir}/venv" + "${gate_dir}/venv/bin/python" -m pip install \ --disable-pip-version-check \ --require-hashes \ - -r "${pilot_dir}/protected-requirements.txt" - test "$("${pilot_dir}/venv/bin/python" -c 'import importlib.metadata; print(importlib.metadata.version("mutmut"))')" = "3.7.0" + -r "${gate_dir}/protected-requirements.txt" + test "$("${gate_dir}/venv/bin/python" -c 'import importlib.metadata; print(importlib.metadata.version("mutmut"))')" = "3.7.0" + test "$("${gate_dir}/venv/bin/uv" --version)" = "uv 0.11.7 (x86_64-unknown-linux-gnu)" + UV_PROJECT_ENVIRONMENT="${gate_dir}/venv" \ + "${gate_dir}/venv/bin/uv" sync \ + --project "${gate_dir}/protected-backend" \ + --locked \ + --extra dev \ + --inexact \ + --no-install-project - - name: Run bounded changed-scope mutation pilot + - name: Run required bounded behavior mutation + if: ${{ steps.preflight.outputs.applicability == 'applicable' }} shell: bash run: | set -euo pipefail - pilot_dir="${RUNNER_TEMP}/workstream-mutation-pilot" + gate_dir="${RUNNER_TEMP}/workstream-mutation-gate" + enforcement=--enforce + if [[ "${{ steps.custody.outputs.bootstrap }}" == "true" ]]; then + enforcement="" + fi timeout --signal=TERM --kill-after=15s 720s \ - "${pilot_dir}/venv/bin/python" backend/scripts/mutation_policy.py \ + "${gate_dir}/venv/bin/python" "${{ steps.custody.outputs.evaluator }}" \ --repository-root . \ --base-sha "${{ steps.custody.outputs.base_sha }}" \ - --head-sha "${PILOT_HEAD_SHA}" \ - --chunk-id WS-QUAL-001-04M \ - --claim-file .ci/behavior-claims/WS-QUAL-001-04M.json \ - --selection-output "${pilot_dir}/selection.json" \ - --execute \ - --manifest "${pilot_dir}/protected-requirements.txt" \ + --head-sha "${MUTATION_HEAD_SHA}" \ + --discover \ + --selection-output "${gate_dir}/executed-selection.json" \ + --execute ${enforcement} \ + --manifest "${gate_dir}/protected-requirements.txt" \ --manifest-digest "${{ steps.custody.outputs.manifest_sha256 }}" \ - --mutmut-executable "${pilot_dir}/venv/bin/mutmut" \ - --evidence-output "${pilot_dir}/evidence.json" \ + --mutmut-executable "${gate_dir}/venv/bin/mutmut" \ + --evidence-output "${gate_dir}/evidence.json" \ --timeout-seconds 700 test -z "$(git status --porcelain --untracked-files=no)" test "$(git rev-parse 'HEAD^{tree}')" = "${{ steps.custody.outputs.head_tree }}" - - name: Upload exact-head pilot evidence + - name: Upload exact-head mutation evidence if: ${{ always() }} uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 with: - name: mutation-pilot-${{ env.PILOT_HEAD_SHA }} + name: behavior-mutation-${{ env.MUTATION_HEAD_SHA }} path: | - ${{ runner.temp }}/workstream-mutation-pilot/selection.json - ${{ runner.temp }}/workstream-mutation-pilot/evidence.json + ${{ runner.temp }}/workstream-mutation-gate/selection.json + ${{ runner.temp }}/workstream-mutation-gate/executed-selection.json + ${{ runner.temp }}/workstream-mutation-gate/evidence.json include-hidden-files: true if-no-files-found: error retention-days: 7 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f22b0bb5..d21304ae 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -70,6 +70,37 @@ Different initiatives may proceed concurrently in separate branches or worktrees. If another pull request changes the base, inspect the new delta and rerun affected checks; unchanged evidence does not need ceremonial repetition. +## Behavior Mutation Claims + +Eligible Python changes under `backend/app/` or `backend/scripts/` require one +schema-v1 claim at `.ci/behavior-claims/.json`. Start from +[the copyable example](.ci/behavior-claims/example.behavior-claim.json) and name +the exact changed callable, its owning pytest node, the observable outcome, and +any essential PostgreSQL, MinIO, HTTP, lock, trigger, or concurrency boundary. +The [claim guide](.ci/behavior-claims/README.md) contains the closed rules. + +Run `cd backend && .venv/bin/python -m pytest -q ` before +opening the PR. From the repository root, validate discovery against the PR +delta with: + +```bash +backend/.venv/bin/python backend/scripts/mutation_policy.py \ + --repository-root . \ + --base-sha "$(git merge-base origin/main HEAD)" \ + --head-sha "$(git rev-parse HEAD)" \ + --discover \ + --selection-output /tmp/workstream-mutation-selection.json +``` + +Inspect `applicability`, `changed_targets`, `changed_callables`, `tests`, and +`target_owners` in that output. The required mutation check derives targets from the exact git +delta and uses the claim only for bounded callable/test ownership. It does not +use a global score: meaningful survivors and incomplete or unsafe evidence +block. Repair a survivor by strengthening the owning behavior assertion or by +correcting the production behavior; do not add skips, exclusions, allowlists, +or mutation pragmas. Unrelated changes return `not_applicable` automatically +and do not install the mutation engine. + ## Durable Records Keep useful plans, contracts, review notes, and historical `.agent-loop/` diff --git a/backend/pyproject.toml b/backend/pyproject.toml index 04f6ce68..4ad57835 100644 --- a/backend/pyproject.toml +++ b/backend/pyproject.toml @@ -53,21 +53,3 @@ testpaths = ["tests"] [tool.ruff] line-length = 100 target-version = "py311" - -[tool.mutmut] -source_paths = ["scripts"] -only_mutate = ["scripts/mutation_policy.py", "scripts/run_test_lanes.py"] -pytest_add_cli_args_test_selection = [ - "tests/test_ci_test_lanes.py::test_committed_lanes_cover_recursive_inventory_exactly_once", - "tests/test_ci_test_lanes.py::test_inventory_fails_closed", - "tests/test_ci_test_lanes.py::test_measured_hotspots_have_explicit_semantic_owners", - "tests/test_mutation_policy.py::TestMutationPolicy::test_changed_targets_are_mandatory_and_claims_are_additive", - "tests/test_mutation_policy.py::TestMutationPolicy::test_claim_validation_fails_closed", - "tests/test_mutation_policy.py::TestMutationPolicy::test_strong_calibration_asserts_the_exact_boundary", - "tests/test_mutation_policy.py::TestMutationPolicy::test_weak_calibration_deliberately_asserts_only_the_result_type", -] -pytest_add_cli_args = ["-q", "--noconftest"] -use_git_change_detection = false -debug = true -timeout_multiplier = 4.0 -timeout_constant = 2.0 diff --git a/backend/scripts/mutation_policy.py b/backend/scripts/mutation_policy.py index 24a3f62c..cdd64b66 100644 --- a/backend/scripts/mutation_policy.py +++ b/backend/scripts/mutation_policy.py @@ -4,7 +4,9 @@ from __future__ import annotations import argparse +import ast from collections import Counter +from fnmatch import fnmatchcase import hashlib import json import os @@ -18,6 +20,7 @@ import tomllib from typing import Any + def _repository_root() -> Path: """Locate the archive root from either original or mutmut-copied code.""" for candidate in Path(__file__).resolve().parents: @@ -39,6 +42,7 @@ def _repository_root() -> Path: CHUNK_RE = re.compile(r"^WS-[A-Z]+-[0-9]{3}-[A-Z0-9]+$") TEST_NODE_RE = re.compile(r"^backend/tests/test_[A-Za-z0-9_/]+\.py::[^\s]+$") CALLABLE_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_.]+$") +HUNK_RE = re.compile(r"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@") ELIGIBLE_PREFIXES = ("backend/app/", "backend/scripts/") OBSERVABLE_OUTCOMES = { "return", @@ -50,6 +54,31 @@ def _repository_root() -> Path: "recovery_outcome", } REAL_BOUNDARIES = {"postgresql", "minio", "http", "lock", "trigger", "concurrency"} +CALIBRATION_TARGET = "backend/scripts/mutation_policy.py" +CALIBRATION_CALLABLES = ( + "scripts.mutation_policy._strong_calibration", + "scripts.mutation_policy._weak_calibration", +) +CALIBRATION_TESTS = ( + "backend/tests/test_mutation_policy.py::TestMutationPolicy::test_strong_calibration_asserts_the_exact_boundary", + "backend/tests/test_mutation_policy.py::TestMutationPolicy::test_weak_calibration_deliberately_asserts_only_the_result_type", +) +WEAK_CALIBRATION_FILTER = "scripts.mutation_policy.x__weak_calibration__mutmut_*" +STRONG_CALIBRATION_FILTER = "scripts.mutation_policy.x__strong_calibration__mutmut_*" +# workstream-mutation-capability:discover-v1 +POLICY_CAPABILITY_MARKER = "workstream-mutation-capability:discover-v1" +RUNTIME_ENV_ALLOWLIST = { + "HOME", + "LANG", + "LC_ALL", + "PATH", + "SSL_CERT_FILE", + "TEMP", + "TMP", + "TMPDIR", + "TZ", + "VIRTUAL_ENV", +} OUTCOMES = ( "generated", "killed", @@ -102,6 +131,15 @@ def _sha256(data: bytes) -> str: return hashlib.sha256(data).hexdigest() +def _minimal_runtime_environment(source: dict[str, str] | None = None) -> dict[str, str]: + """Return only non-authority runtime values for untrusted candidate code.""" + values = os.environ if source is None else source + environment = {key: values[key] for key in RUNTIME_ENV_ALLOWLIST if key in values} + environment["PYTEST_DISABLE_PLUGIN_AUTOLOAD"] = "1" + environment["PYTHONHASHSEED"] = "0" + return environment + + def _git(root: Path, *arguments: str) -> str: result = subprocess.run( ["git", *arguments], @@ -119,12 +157,7 @@ def _git(root: Path, *arguments: str) -> str: def _safe_path(value: str) -> PurePosixPath: path = PurePosixPath(value) - if ( - not value - or path.is_absolute() - or ".." in path.parts - or value != path.as_posix() - ): + if not value or path.is_absolute() or ".." in path.parts or value != path.as_posix(): raise MutationPolicyError("unsafe_path") return path @@ -152,6 +185,115 @@ def _regular_repository_file(root: Path, value: str) -> bool: return False +def discover_claim_path(root: Path, base_sha: str, head_sha: str) -> Path | None: + """Discover one exact changed behavior claim without workflow input.""" + candidates = sorted( + path + for path in changed_files(base_sha, head_sha, repository_root=root, include_local=False) + if path.startswith(".ci/behavior-claims/") + and path.endswith(".json") + and path != ".ci/behavior-claims/example.behavior-claim.json" + ) + if len(candidates) > 1: + raise MutationPolicyError("multiple_behavior_claims") + if not candidates: + return None + candidate = root / candidates[0] + if not _regular_repository_file(root, candidates[0]): + raise MutationPolicyError("invalid_behavior_claim_path") + return candidate + + +def _diff_lines(root: Path, base_sha: str, head_sha: str, path: str) -> tuple[set[int], set[int]]: + """Return exact old/new line numbers touched by a zero-context diff.""" + output = _git(root, "diff", "--unified=0", "--no-ext-diff", base_sha, head_sha, "--", path) + old_lines: set[int] = set() + new_lines: set[int] = set() + for line in output.splitlines(): + match = HUNK_RE.match(line) + if match is None: + continue + old_start, old_count, new_start, new_count = match.groups() + old_size = int(old_count or "1") + new_size = int(new_count or "1") + old_lines.update(range(int(old_start), int(old_start) + old_size)) + new_lines.update(range(int(new_start), int(new_start) + new_size)) + return old_lines, new_lines + + +def _source_at(root: Path, revision: str, path: str) -> str: + return _git(root, "show", f"{revision}:{path}") + + +def _callable_spans( + source: str, module: str +) -> tuple[list[tuple[int, int, str]], list[tuple[int, int]]]: + """Return qualified callable spans and module/class executable spans.""" + try: + tree = ast.parse(source) + except SyntaxError as exc: + raise MutationPolicyError("invalid_target_syntax") from exc + callables: list[tuple[int, int, str]] = [] + executable: list[tuple[int, int]] = [] + + def visit(nodes: list[ast.stmt], parents: tuple[str, ...] = ()) -> None: + for node in nodes: + end = getattr(node, "end_lineno", node.lineno) + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)): + start = min([node.lineno, *[item.lineno for item in node.decorator_list]]) + qualified = ".".join((module, *parents, node.name)) + callables.append((start, end, qualified)) + visit(node.body, (*parents, node.name)) + elif isinstance(node, ast.ClassDef): + start = min([node.lineno, *[item.lineno for item in node.decorator_list]]) + executable.append((start, node.lineno)) + visit(node.body, (*parents, node.name)) + else: + executable.append((node.lineno, end)) + + visit(tree.body) + return callables, executable + + +def _map_changed_lines(source: str, module: str, lines: set[int]) -> tuple[set[str], bool]: + callables, executable = _callable_spans(source, module) + owners: set[str] = set() + unmapped = False + for line in lines: + matches = [item for item in callables if item[0] <= line <= item[1]] + if matches: + owners.add(min(matches, key=lambda item: item[1] - item[0])[2]) + elif any(start <= line <= end for start, end in executable): + unmapped = True + return owners, unmapped + + +def changed_callables( + root: Path, base_sha: str, head_sha: str, target: str, *, allow_unmapped: bool = False +) -> list[str]: + """Derive complete current callable ownership for executable target hunks.""" + module = target.removeprefix("backend/").removesuffix(".py").replace("/", ".") + delta_base = _git(root, "merge-base", base_sha, head_sha) + old_lines, new_lines = _diff_lines(root, delta_base, head_sha, target) + current_source = _source_at(root, head_sha, target) + current, current_unmapped = _map_changed_lines(current_source, module, new_lines) + try: + base_source = _source_at(root, delta_base, target) + except MutationPolicyError: + base_source = "" + previous: set[str] = set() + previous_unmapped = False + if base_source: + previous, previous_unmapped = _map_changed_lines(base_source, module, old_lines) + removed = previous - {item[2] for item in _callable_spans(current_source, module)[0]} + if (current_unmapped or previous_unmapped or removed) and not allow_unmapped: + raise MutationPolicyError("unmappable_changed_logic") + derived = sorted(current | (previous - removed if allow_unmapped else previous)) + if not derived: + raise MutationPolicyError("zero_changed_callables") + return derived + + def _read_claim(path: Path | None, root: Path, expected_chunk: str) -> list[dict[str, Any]]: if path is None: return [] @@ -194,8 +336,11 @@ def _read_claim(path: Path | None, root: Path, expected_chunk: str) -> list[dict if ( not isinstance(callables, list) or not callables - or len(callables) > 12 - or any(not isinstance(item, str) or CALLABLE_RE.fullmatch(item) is None for item in callables) + or len(callables) > 24 + or any( + not isinstance(item, str) or CALLABLE_RE.fullmatch(item) is None + for item in callables + ) or len(set(callables)) != len(callables) ): raise MutationPolicyError("invalid_claim_callables") @@ -217,7 +362,9 @@ def _read_claim(path: Path | None, root: Path, expected_chunk: str) -> list[dict if ( not isinstance(outcomes, list) or not outcomes - or any(not isinstance(item, str) or item not in OBSERVABLE_OUTCOMES for item in outcomes) + or any( + not isinstance(item, str) or item not in OBSERVABLE_OUTCOMES for item in outcomes + ) or len(set(outcomes)) != len(outcomes) ): raise MutationPolicyError("invalid_claim_outcomes") @@ -260,7 +407,21 @@ def build_selection( repository_root=root, include_local=False, ) - changed_targets = sorted(path for path in changed if _eligible_target(path)) + delta_base = _git(root, "merge-base", base_sha, head_sha) + deleted = set( + filter( + None, + _git( + root, "diff", "--diff-filter=D", "--name-only", delta_base, head_sha, "--" + ).splitlines(), + ) + ) + deleted_targets = sorted(path for path in changed if _eligible_target(path) and path in deleted) + if deleted_targets: + raise MutationPolicyError("deleted_eligible_target") + changed_targets = sorted( + path for path in changed if _eligible_target(path) and path not in deleted + ) if claim_path is not None: expected_claim = root / ".ci" / "behavior-claims" / f"{chunk_id}.json" try: @@ -274,8 +435,36 @@ def build_selection( unowned_targets = sorted(set(changed_targets) - set(claims_by_target)) if unowned_targets: raise MutationPolicyError("changed_target_without_behavior_claim") + for target, claim in claims_by_target.items(): + module = target.removeprefix("backend/").removesuffix(".py").replace("/", ".") + available = { + item[2] for item in _callable_spans(_source_at(root, head_sha, target), module)[0] + } + if not set(claim["callables"]).issubset(available): + raise MutationPolicyError("missing_claim_callable") targets = sorted(set(changed_targets) | set(claims_by_target)) - tests = sorted({node for claim in claims for node in claim["tests"]}) + bootstrap = False + blocking_policy = chunk_id == "WS-QUAL-001-05M" + try: + base_policy = _source_at(root, base_sha, "backend/scripts/mutation_policy.py") + except MutationPolicyError: + base_policy = "" + if chunk_id == "WS-QUAL-001-05M": + bootstrap = POLICY_CAPABILITY_MARKER not in base_policy + blocking_policy = blocking_policy or POLICY_CAPABILITY_MARKER in base_policy + derived_callables = { + target: changed_callables(root, base_sha, head_sha, target, allow_unmapped=bootstrap) + for target in changed_targets + } + for target, required in derived_callables.items(): + if set(required) != set(claims_by_target[target]["callables"]): + raise MutationPolicyError("unowned_changed_callable") + if blocking_policy: + targets = sorted(set(targets) | {CALIBRATION_TARGET}) + tests = {node for claim in claims for node in claim["tests"]} + if blocking_policy: + tests.update(CALIBRATION_TESTS) + tests = sorted(tests) if not targets: raise MutationPolicyError("zero_mutation_targets") if not tests: @@ -288,11 +477,47 @@ def build_selection( "head_tree": _git(root, "rev-parse", f"{head_sha}^{{tree}}"), "changed_paths": changed, "changed_targets": changed_targets, + "changed_callables": derived_callables, "claims": claims, "target_owners": [ { - **claims_by_target[target], - "selection_reason": "changed" if target in changed_targets else "claim", + **( + { + "target": target, + "callables": sorted( + set(claims_by_target.get(target, {}).get("callables", [])) + | ( + set(CALIBRATION_CALLABLES) + if blocking_policy and target == CALIBRATION_TARGET + else set() + ) + ), + "tests": sorted( + set(claims_by_target.get(target, {}).get("tests", [])) + | ( + set(CALIBRATION_TESTS) + if blocking_policy and target == CALIBRATION_TARGET + else set() + ) + ), + "outcomes": sorted( + set(claims_by_target.get(target, {}).get("outcomes", [])) + | ( + {"return"} + if blocking_policy and target == CALIBRATION_TARGET + else set() + ) + ), + "boundaries": claims_by_target.get(target, {}).get("boundaries", []), + } + ), + "selection_reason": ( + "changed" + if target in changed_targets + else "claim" + if target in claims_by_target + else "calibration" + ), } for target in targets ], @@ -301,27 +526,87 @@ def build_selection( } -def _verify_mutmut_config(backend: Path, selection: dict[str, Any]) -> None: +def discover_selection(root: Path, base_sha: str, head_sha: str) -> dict[str, Any]: + """Discover applicability and the only canonical claim from the git delta.""" + changed = changed_files(base_sha, head_sha, repository_root=root, include_local=False) + delta_base = _git(root, "merge-base", base_sha, head_sha) + deleted = set( + filter( + None, + _git( + root, "diff", "--diff-filter=D", "--name-only", delta_base, head_sha, "--" + ).splitlines(), + ) + ) + deleted_targets = sorted(path for path in changed if _eligible_target(path) and path in deleted) + if deleted_targets: + raise MutationPolicyError("deleted_eligible_target") + changed_targets = sorted( + path for path in changed if _eligible_target(path) and path not in deleted + ) + claim_path = discover_claim_path(root, base_sha, head_sha) + if not changed_targets and claim_path is None: + return { + "schema_version": SCHEMA_VERSION, + "applicability": "not_applicable", + "base_sha": base_sha, + "head_sha": head_sha, + "changed_paths": changed, + } + if claim_path is None: + raise MutationPolicyError("missing_behavior_claim") + try: + raw = json.loads(claim_path.read_text(encoding="utf-8")) + chunk_id = raw["chunk_id"] + except (OSError, UnicodeDecodeError, json.JSONDecodeError, KeyError, TypeError) as exc: + raise MutationPolicyError("invalid_behavior_claim_json") from exc + selection = build_selection(root, base_sha, head_sha, chunk_id, claim_path) + selection["applicability"] = "applicable" + selection["claim_path"] = claim_path.relative_to(root).as_posix() + return selection + + +def _write_mutmut_config(backend: Path, selection: dict[str, Any]) -> str: + """Replace any static mutmut section with exact policy-derived config.""" pyproject = backend / "pyproject.toml" try: - config = tomllib.loads(pyproject.read_text(encoding="utf-8"))["tool"]["mutmut"] - except (OSError, UnicodeDecodeError, tomllib.TOMLDecodeError, KeyError) as exc: + original = pyproject.read_text(encoding="utf-8") + tomllib.loads(original) + except (OSError, UnicodeDecodeError, tomllib.TOMLDecodeError) as exc: raise MutationPolicyError("invalid_mutation_config") from exc relative_targets = [target.removeprefix("backend/") for target in selection["targets"]] source_paths = sorted({target.split("/", 1)[0] for target in relative_targets}) test_nodes = [node.removeprefix("backend/") for node in selection["tests"]] - expected = { - "source_paths": source_paths, - "only_mutate": relative_targets, - "pytest_add_cli_args_test_selection": test_nodes, - "pytest_add_cli_args": ["-q", "--noconftest"], - "use_git_change_detection": False, - "debug": True, - "timeout_multiplier": 4.0, - "timeout_constant": 2.0, - } - if config != expected: - raise MutationPolicyError("mutation_config_selection_mismatch") + lines = original.splitlines() + retained: list[str] = [] + skipping = False + for line in lines: + if line.strip() == "[tool.mutmut]": + skipping = True + continue + if skipping and line.startswith("["): + skipping = False + if not skipping: + retained.append(line) + config_lines = [ + "[tool.mutmut]", + f"source_paths = {json.dumps(source_paths)}", + f"only_mutate = {json.dumps(relative_targets)}", + f"pytest_add_cli_args_test_selection = {json.dumps(test_nodes)}", + 'pytest_add_cli_args = ["-q", "--noconftest"]', + "use_git_change_detection = false", + "debug = true", + "timeout_multiplier = 4.0", + "timeout_constant = 2.0", + ] + rendered = "\n".join([*retained, "", *config_lines, ""]) + pyproject.write_text(rendered, encoding="utf-8") + try: + config = tomllib.loads(rendered)["tool"]["mutmut"] + except (tomllib.TOMLDecodeError, KeyError, TypeError) as exc: + raise MutationPolicyError("invalid_generated_mutation_config") from exc + digest = _sha256(_json_bytes(config)) + return digest def _reject_disposable_special_files(disposable: Path) -> None: @@ -339,12 +624,80 @@ def _mutant_filters(selection: dict[str, Any]) -> list[str]: """Translate reviewed qualified callables into exact mutmut name globs.""" filters: list[str] = [] for owner in selection["target_owners"]: + target = owner.get("target") + if not isinstance(target, str) or not target: + raise MutationPolicyError("missing_owner_target") + target_module = target.removeprefix("backend/").removesuffix(".py").replace("/", ".") for callable_name in owner["callables"]: - module, function = callable_name.rsplit(".", 1) - filters.append(f"{module}.x_{function}__mutmut_*") + relative = callable_name.removeprefix(f"{target_module}.") + filters.append(f"{target_module}.x_{relative.replace('.', '__')}__mutmut_*") return sorted(set(filters)) +def classify_outcomes( + counts: dict[str, int], mutants: list[dict[str, str]], filters: list[str] +) -> dict[str, Any]: + """Produce the closed blocking verdict for complete mutation outcomes.""" + if set(counts) != set(OUTCOMES) or counts["generated"] != len(mutants): + raise MutationPolicyError("incomplete_mutation_outcomes") + if any(not isinstance(value, int) or value < 0 for value in counts.values()): + raise MutationPolicyError("invalid_mutation_outcomes") + controls: list[str] = [] + blockers: list[dict[str, str]] = [] + for mutant in mutants: + name = mutant.get("name") + outcome = mutant.get("outcome") + if not isinstance(name, str) or outcome not in OUTCOMES[1:]: + raise MutationPolicyError("unknown_mutation_outcome") + selected = any(fnmatchcase(name, pattern) for pattern in filters) + weak_control = fnmatchcase(name, WEAK_CALIBRATION_FILTER) + if outcome == "killed": + continue + if outcome == "survived" and weak_control and selected: + controls.append(name) + continue + if outcome == "excluded" and not selected: + continue + blockers.append({"name": name, "outcome": outcome}) + return { + "status": "pass" if not blockers else "block", + "classification": "calibrated" if controls else "strict", + "calibration_controls": sorted(controls), + "blockers": sorted(blockers, key=lambda item: (item["outcome"], item["name"])), + } + + +def policy_self_test() -> None: + """Prove the protected evaluator blocks and permits only closed outcomes.""" + filters = [WEAK_CALIBRATION_FILTER] + outcomes = {name: 0 for name in OUTCOMES} + outcomes.update({"generated": 1, "survived": 1}) + control = [ + {"name": "scripts.mutation_policy.x__weak_calibration__mutmut_1", "outcome": "survived"} + ] + if classify_outcomes(outcomes, control, filters)["status"] != "pass": + raise MutationPolicyError("self_test_control_failed") + blocker = [{"name": "scripts.control.x_changed__mutmut_1", "outcome": "survived"}] + if classify_outcomes(outcomes, blocker, filters)["status"] != "block": + raise MutationPolicyError("self_test_blocker_failed") + + +def _validate_calibration(mutants: list[dict[str, str]]) -> dict[str, dict[str, int]]: + """Validate only the exact repository-owned strong and weak controls.""" + strong = [ + mutant for mutant in mutants if fnmatchcase(mutant["name"], STRONG_CALIBRATION_FILTER) + ] + weak = [mutant for mutant in mutants if fnmatchcase(mutant["name"], WEAK_CALIBRATION_FILTER)] + if not any(mutant["outcome"] == "killed" for mutant in strong): + raise MutationPolicyError("strong_calibration_not_killed") + if not any(mutant["outcome"] == "survived" for mutant in weak): + raise MutationPolicyError("weak_calibration_not_survived") + return { + "strong": dict(Counter(mutant["outcome"] for mutant in strong)), + "weak": dict(Counter(mutant["outcome"] for mutant in weak)), + } + + def _parse_outcomes(backend: Path) -> tuple[dict[str, int], list[dict[str, str]]]: counts = Counter({outcome: 0 for outcome in OUTCOMES}) mutants: list[dict[str, str]] = [] @@ -376,6 +729,8 @@ def execute_pilot( mutmut_executable: Path, output: Path, timeout_seconds: int, + *, + enforce: bool = False, ) -> None: """Run mutation testing in an archived disposable tree and emit evidence.""" if timeout_seconds < 1 or timeout_seconds > 720: @@ -410,11 +765,8 @@ def execute_pilot( raise MutationPolicyError("archive_extract_failed") _reject_disposable_special_files(disposable) backend = disposable / "backend" - _verify_mutmut_config(backend, selection) - environment = os.environ.copy() - environment.pop("GITHUB_TOKEN", None) - environment.pop("GH_TOKEN", None) - environment["PYTEST_DISABLE_PLUGIN_AUTOLOAD"] = "1" + generated_config_sha256 = _write_mutmut_config(backend, selection) + environment = _minimal_runtime_environment() baseline = subprocess.run( [ sys.executable, @@ -456,24 +808,8 @@ def execute_pilot( if result.returncode != 0: raise MutationPolicyError("mutation_engine_error") counts, mutants = _parse_outcomes(backend) - strong = [ - mutant - for mutant in mutants - if ".x__strong_calibration__mutmut_" in mutant["name"] - ] - weak = [ - mutant - for mutant in mutants - if ".x__weak_calibration__mutmut_" in mutant["name"] - ] - if not any(mutant["outcome"] == "killed" for mutant in strong): - raise MutationPolicyError("strong_calibration_not_killed") - if not any(mutant["outcome"] == "survived" for mutant in weak): - raise MutationPolicyError("weak_calibration_not_survived") - calibration = { - "strong": dict(Counter(mutant["outcome"] for mutant in strong)), - "weak": dict(Counter(mutant["outcome"] for mutant in weak)), - } + verdict = classify_outcomes(counts, mutants, _mutant_filters(selection)) + calibration = _validate_calibration(mutants) elapsed = round(time.monotonic() - started, 3) if ( _git(root, "status", "--porcelain", "--untracked-files=no") @@ -495,24 +831,31 @@ def execute_pilot( "target_owners": selection["target_owners"], }, "selection_sha256": _sha256(_json_bytes(selection)), + "generated_config_sha256": generated_config_sha256, "elapsed_seconds": elapsed, "outcomes": counts, "calibration": calibration, "mutants": mutants, + "verdict": verdict, } output.parent.mkdir(parents=True, exist_ok=True) descriptor = os.open(output, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) with os.fdopen(descriptor, "wb") as stream: stream.write(_json_bytes(evidence)) + if enforce and verdict["status"] != "pass": + raise MutationPolicyError("blocking_mutation_outcome") def _main() -> int: parser = argparse.ArgumentParser() parser.add_argument("--repository-root", type=Path, default=ROOT) - parser.add_argument("--base-sha", required=True) - parser.add_argument("--head-sha", required=True) - parser.add_argument("--chunk-id", required=True) + parser.add_argument("--base-sha") + parser.add_argument("--head-sha") + parser.add_argument("--chunk-id") parser.add_argument("--claim-file", type=Path) + parser.add_argument("--discover", action="store_true") + parser.add_argument("--enforce", action="store_true") + parser.add_argument("--self-test", action="store_true") parser.add_argument("--selection-output", type=Path) parser.add_argument("--execute", action="store_true") parser.add_argument("--manifest", type=Path) @@ -522,18 +865,34 @@ def _main() -> int: parser.add_argument("--timeout-seconds", type=int, default=720) args = parser.parse_args() try: + if args.self_test: + policy_self_test() + return 0 + if not args.base_sha or not args.head_sha: + raise MutationPolicyError("missing_revision") root = args.repository_root.resolve(strict=True) - selection = build_selection( - root, - args.base_sha, - args.head_sha, - args.chunk_id, - args.claim_file, - ) + if args.discover: + selection = discover_selection(root, args.base_sha, args.head_sha) + else: + if args.chunk_id is None: + raise MutationPolicyError("missing_chunk_id") + selection = build_selection( + root, + args.base_sha, + args.head_sha, + args.chunk_id, + args.claim_file, + ) if args.selection_output: args.selection_output.write_bytes(_json_bytes(selection)) + if selection.get("applicability") == "not_applicable": + if args.execute: + raise MutationPolicyError("inapplicable_execution") + return 0 if args.execute: - if not all((args.manifest, args.manifest_digest, args.mutmut_executable, args.evidence_output)): + if not all( + (args.manifest, args.manifest_digest, args.mutmut_executable, args.evidence_output) + ): raise MutationPolicyError("missing_execution_argument") execute_pilot( root, @@ -543,6 +902,7 @@ def _main() -> int: args.mutmut_executable, args.evidence_output, args.timeout_seconds, + enforce=args.enforce, ) except (MutationPolicyError, OSError, subprocess.TimeoutExpired) as exc: print(f"mutation_policy_error:{exc}", file=sys.stderr) diff --git a/backend/tests/test_mutation_policy.py b/backend/tests/test_mutation_policy.py index 61818642..a078b187 100644 --- a/backend/tests/test_mutation_policy.py +++ b/backend/tests/test_mutation_policy.py @@ -8,12 +8,14 @@ import subprocess import sys import tempfile +import tomllib import pytest from scripts.mutation_policy import MutationPolicyError from scripts.mutation_policy import _eligible_target from scripts.mutation_policy import _main +from scripts.mutation_policy import _minimal_runtime_environment from scripts.mutation_policy import _mutant_filters from scripts.mutation_policy import _parse_outcomes from scripts.mutation_policy import _reject_disposable_special_files @@ -22,9 +24,15 @@ from scripts.mutation_policy import _safe_path from scripts.mutation_policy import _strong_calibration from scripts.mutation_policy import _weak_calibration -from scripts.mutation_policy import _verify_mutmut_config +from scripts.mutation_policy import _validate_calibration +from scripts.mutation_policy import _write_mutmut_config from scripts.mutation_policy import build_selection +from scripts.mutation_policy import changed_callables +from scripts.mutation_policy import classify_outcomes +from scripts.mutation_policy import discover_claim_path +from scripts.mutation_policy import discover_selection from scripts.mutation_policy import execute_pilot +from scripts.mutation_policy import policy_self_test class TestMutationPolicy: @@ -50,39 +58,31 @@ def test_changed_targets_are_mandatory_and_claims_are_additive(self) -> None: { "target": "backend/scripts/changed.py", "callables": ["scripts.changed.changed"], - "tests": [ - "backend/tests/test_claimed.py::test_claimed" - ], + "tests": ["backend/tests/test_claimed.py::test_claimed"], "outcomes": ["return"], "boundaries": [], }, { "target": "backend/scripts/claimed.py", "callables": ["scripts.claimed.claimed"], - "tests": [ - "backend/tests/test_claimed.py::test_claimed" - ], + "tests": ["backend/tests/test_claimed.py::test_claimed"], "outcomes": ["return"], "boundaries": [], - } + }, ], } ), encoding="utf-8", ) - selection = build_selection( - root, self.base, head, "WS-QUAL-001-04M", claim - ) + selection = build_selection(root, self.base, head, "WS-QUAL-001-04M", claim) assert selection["changed_targets"] == ["backend/scripts/changed.py"] assert selection["targets"] == [ "backend/scripts/changed.py", "backend/scripts/claimed.py", ] - assert selection["tests"] == [ - "backend/tests/test_claimed.py::test_claimed" - ] + assert selection["tests"] == ["backend/tests/test_claimed.py::test_claimed"] def test_claim_validation_fails_closed(self) -> None: with tempfile.TemporaryDirectory() as temporary: @@ -176,15 +176,35 @@ def test_unsafe_paths_fail_closed(self, path: str) -> None: @pytest.mark.parametrize( ("claim", "error"), [ - ({"schema_version": 1, "chunk_id": "WS-QUAL-001-04M", "claims": [], "extra": True}, "invalid_behavior_claim_shape"), - ({"schema_version": 1, "chunk_id": "WS-QUAL-001-04M", "claims": [42]}, "invalid_behavior_claim"), - ({"schema_version": 2, "chunk_id": "WS-QUAL-001-04M", "claims": []}, "unsupported_behavior_claim_schema"), - ({"schema_version": 1, "chunk_id": "WS-QUAL-001-04M", "claims": "bad"}, "invalid_behavior_claim_count"), + ( + {"schema_version": 1, "chunk_id": "WS-QUAL-001-04M", "claims": [], "extra": True}, + "invalid_behavior_claim_shape", + ), + ( + {"schema_version": 1, "chunk_id": "WS-QUAL-001-04M", "claims": [42]}, + "invalid_behavior_claim", + ), + ( + {"schema_version": 2, "chunk_id": "WS-QUAL-001-04M", "claims": []}, + "unsupported_behavior_claim_schema", + ), + ( + {"schema_version": 1, "chunk_id": "WS-QUAL-001-04M", "claims": "bad"}, + "invalid_behavior_claim_count", + ), ( { "schema_version": 1, "chunk_id": "WS-QUAL-001-04M", - "claims": [{"target": "docs/no.py", "callables": ["docs.no.no"], "tests": ["backend/tests/test_claimed.py::test_claimed"], "outcomes": ["return"], "boundaries": []}], + "claims": [ + { + "target": "docs/no.py", + "callables": ["docs.no.no"], + "tests": ["backend/tests/test_claimed.py::test_claimed"], + "outcomes": ["return"], + "boundaries": [], + } + ], }, "ineligible_claim_target", ), @@ -192,7 +212,15 @@ def test_unsafe_paths_fail_closed(self, path: str) -> None: { "schema_version": 1, "chunk_id": "WS-QUAL-001-04M", - "claims": [{"target": "backend/scripts/missing.py", "callables": ["scripts.missing.missing"], "tests": ["backend/tests/test_claimed.py::test_claimed"], "outcomes": ["return"], "boundaries": []}], + "claims": [ + { + "target": "backend/scripts/missing.py", + "callables": ["scripts.missing.missing"], + "tests": ["backend/tests/test_claimed.py::test_claimed"], + "outcomes": ["return"], + "boundaries": [], + } + ], }, "missing_claim_target", ), @@ -200,7 +228,15 @@ def test_unsafe_paths_fail_closed(self, path: str) -> None: { "schema_version": 1, "chunk_id": "WS-QUAL-001-04M", - "claims": [{"target": "backend/scripts/claimed.py", "callables": ["scripts.claimed.claimed"], "tests": [], "outcomes": ["return"], "boundaries": []}], + "claims": [ + { + "target": "backend/scripts/claimed.py", + "callables": ["scripts.claimed.claimed"], + "tests": [], + "outcomes": ["return"], + "boundaries": [], + } + ], }, "invalid_claim_tests", ), @@ -208,7 +244,15 @@ def test_unsafe_paths_fail_closed(self, path: str) -> None: { "schema_version": 1, "chunk_id": "WS-QUAL-001-04M", - "claims": [{"target": "backend/scripts/claimed.py", "callables": ["scripts.claimed.claimed"], "tests": ["not-a-node"], "outcomes": ["return"], "boundaries": []}], + "claims": [ + { + "target": "backend/scripts/claimed.py", + "callables": ["scripts.claimed.claimed"], + "tests": ["not-a-node"], + "outcomes": ["return"], + "boundaries": [], + } + ], }, "invalid_claim_test_node", ), @@ -216,7 +260,15 @@ def test_unsafe_paths_fail_closed(self, path: str) -> None: { "schema_version": 1, "chunk_id": "WS-QUAL-001-04M", - "claims": [{"target": "backend/scripts/claimed.py", "callables": ["scripts.claimed.claimed"], "tests": ["backend/tests/test_missing.py::test_missing"], "outcomes": ["return"], "boundaries": []}], + "claims": [ + { + "target": "backend/scripts/claimed.py", + "callables": ["scripts.claimed.claimed"], + "tests": ["backend/tests/test_missing.py::test_missing"], + "outcomes": ["return"], + "boundaries": [], + } + ], }, "missing_claim_test_module", ), @@ -250,7 +302,10 @@ def test_claim_path_must_match_the_chunk_contract(self) -> None: ("overrides", "error"), [ ({"callables": []}, "invalid_claim_callables"), - ({"tests": ["backend/tests/test_claimed.py::test_claimed"] * 2}, "duplicate_claim_test_node"), + ( + {"tests": ["backend/tests/test_claimed.py::test_claimed"] * 2}, + "duplicate_claim_test_node", + ), ({"outcomes": ["unknown"]}, "invalid_claim_outcomes"), ({"boundaries": ["unknown"]}, "invalid_claim_boundaries"), ], @@ -330,7 +385,7 @@ def test_invalid_mutmut_metadata_and_zero_mutants_fail_closed(self) -> None: with pytest.raises(MutationPolicyError, match="zero_generated_mutants"): _parse_outcomes(backend) - def test_mutmut_configuration_parse_and_selection_drift_fail_closed(self) -> None: + def test_mutmut_configuration_is_generated_from_selection(self) -> None: with tempfile.TemporaryDirectory() as temporary: backend = Path(temporary) pyproject = backend / "pyproject.toml" @@ -340,13 +395,52 @@ def test_mutmut_configuration_parse_and_selection_drift_fail_closed(self) -> Non } pyproject.write_text("not = [valid", encoding="utf-8") with pytest.raises(MutationPolicyError, match="invalid_mutation_config"): - _verify_mutmut_config(backend, selection) + _write_mutmut_config(backend, selection) pyproject.write_text( + "[project]\nname = 'example'\nversion = '0.1.0'\n" "[tool.mutmut]\nsource_paths = ['wrong']\n", encoding="utf-8", ) - with pytest.raises(MutationPolicyError, match="mutation_config_selection_mismatch"): - _verify_mutmut_config(backend, selection) + digest = _write_mutmut_config(backend, selection) + document = tomllib.loads(pyproject.read_text(encoding="utf-8")) + assert document["project"] == {"name": "example", "version": "0.1.0"} + rendered = document["tool"]["mutmut"] + assert rendered == { + "source_paths": ["scripts"], + "only_mutate": ["scripts/example.py"], + "pytest_add_cli_args_test_selection": ["tests/test_example.py::test_example"], + "pytest_add_cli_args": ["-q", "--noconftest"], + "use_git_change_detection": False, + "debug": True, + "timeout_multiplier": 4.0, + "timeout_constant": 2.0, + } + assert len(digest) == 64 + + def test_generated_mutmut_parse_failure_is_typed(self, monkeypatch: pytest.MonkeyPatch) -> None: + with tempfile.TemporaryDirectory() as temporary: + backend = Path(temporary) + (backend / "pyproject.toml").write_text( + "[project]\nname = 'example'\nversion = '0.1.0'\n", + encoding="utf-8", + ) + selection = { + "targets": ["backend/scripts/example.py"], + "tests": ["backend/tests/test_example.py::test_example"], + } + real_loads = tomllib.loads + calls = 0 + + def fail_second_parse(value: str) -> dict[str, object]: + nonlocal calls + calls += 1 + if calls == 2: + raise tomllib.TOMLDecodeError("generated failure", value, 0) + return real_loads(value) + + monkeypatch.setattr(tomllib, "loads", fail_second_parse) + with pytest.raises(MutationPolicyError, match="invalid_generated_mutation_config"): + _write_mutmut_config(backend, selection) def test_disposable_symlinks_and_invalid_result_values_fail_closed(self) -> None: with tempfile.TemporaryDirectory() as temporary: @@ -384,11 +478,12 @@ def test_callable_filters_are_exact_and_deterministic(self) -> None: selection = { "target_owners": [ { + "target": "backend/scripts/example.py", "callables": [ "scripts.example.public", "scripts.example._private", "scripts.example.public", - ] + ], } ] } @@ -396,6 +491,373 @@ def test_callable_filters_are_exact_and_deterministic(self) -> None: "scripts.example.x__private__mutmut_*", "scripts.example.x_public__mutmut_*", ] + with pytest.raises(MutationPolicyError, match="missing_owner_target"): + _mutant_filters({"target_owners": [{"callables": ["scripts.example.public"]}]}) + + def test_no_target_no_claim_is_typed_not_applicable(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + self._initialize(root) + (root / "README.md").write_text("docs\n", encoding="utf-8") + self._git(root, "add", ".") + self._git(root, "commit", "-m", "docs") + head = self._git(root, "rev-parse", "HEAD") + selection = discover_selection(root, self.base, head) + assert selection["applicability"] == "not_applicable" + assert "targets" not in selection + + def test_deleted_target_fails_closed(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + self._initialize(root) + (root / "backend/scripts/claimed.py").unlink() + self._git(root, "add", ".") + self._git(root, "commit", "-m", "delete target") + head = self._git(root, "rev-parse", "HEAD") + + with pytest.raises(MutationPolicyError, match="deleted_eligible_target"): + discover_selection(root, self.base, head) + + def test_applicable_delta_requires_one_changed_claim(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + self._initialize(root) + changed = root / "backend/scripts/claimed.py" + changed.write_text("def claimed():\n return False\n", encoding="utf-8") + self._git(root, "add", ".") + self._git(root, "commit", "-m", "behavior") + head = self._git(root, "rev-parse", "HEAD") + with pytest.raises(MutationPolicyError, match="missing_behavior_claim"): + discover_selection(root, self.base, head) + + def test_multiple_changed_claims_fail_closed(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + self._initialize(root) + claims = root / ".ci/behavior-claims" + claims.mkdir(parents=True) + for name in ("WS-QUAL-001-A.json", "WS-QUAL-001-B.json"): + (claims / name).write_text("{}", encoding="utf-8") + self._git(root, "add", ".") + self._git(root, "commit", "-m", "claims") + head = self._git(root, "rev-parse", "HEAD") + with pytest.raises(MutationPolicyError, match="multiple_behavior_claims"): + discover_claim_path(root, self.base, head) + + def test_changed_callable_mapping_covers_decorated_async_and_nested_methods(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + self._initialize(root) + target = root / "backend/scripts/claimed.py" + target.write_text( + "class Service:\n" + " @staticmethod\n" + " async def claimed():\n" + " return True\n", + encoding="utf-8", + ) + self._git(root, "add", ".") + self._git(root, "commit", "-m", "nested base") + self.base = self._git(root, "rev-parse", "HEAD") + target.write_text( + "class Service:\n" + " @staticmethod\n" + " async def claimed():\n" + " return False\n", + encoding="utf-8", + ) + self._git(root, "add", ".") + self._git(root, "commit", "-m", "nested") + head = self._git(root, "rev-parse", "HEAD") + assert changed_callables(root, self.base, head, "backend/scripts/claimed.py") == [ + "scripts.claimed.Service.claimed" + ] + + def test_plain_class_header_change_fails_closed(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + self._initialize(root) + target = root / "backend/scripts/claimed.py" + target.write_text( + "class Service:\n def claimed(self):\n return True\n", + encoding="utf-8", + ) + self._git(root, "add", ".") + self._git(root, "commit", "-m", "class base") + self.base = self._git(root, "rev-parse", "HEAD") + target.write_text( + "class RenamedService:\n def claimed(self):\n return True\n", + encoding="utf-8", + ) + self._git(root, "add", ".") + self._git(root, "commit", "-m", "rename class") + head = self._git(root, "rev-parse", "HEAD") + with pytest.raises(MutationPolicyError, match="unmappable_changed_logic"): + changed_callables(root, self.base, head, "backend/scripts/claimed.py") + + def test_function_nested_in_function_maps_to_inner_owner(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + self._initialize(root) + target = root / "backend/scripts/claimed.py" + body = "def outer():\n def inner():\n return {value}\n return inner\n" + target.write_text(body.format(value="True"), encoding="utf-8") + self._git(root, "add", ".") + self._git(root, "commit", "-m", "nested function base") + self.base = self._git(root, "rev-parse", "HEAD") + target.write_text(body.format(value="False"), encoding="utf-8") + self._git(root, "add", ".") + self._git(root, "commit", "-m", "nested function change") + head = self._git(root, "rev-parse", "HEAD") + assert changed_callables(root, self.base, head, "backend/scripts/claimed.py") == [ + "scripts.claimed.outer.inner" + ] + + def test_module_level_and_deleted_callable_changes_fail_closed(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + self._initialize(root) + target = root / "backend/scripts/claimed.py" + target.write_text( + "def claimed():\n return True\n\nSETTING = True\n", encoding="utf-8" + ) + self._git(root, "add", ".") + self._git(root, "commit", "-m", "module") + head = self._git(root, "rev-parse", "HEAD") + with pytest.raises(MutationPolicyError, match="unmappable_changed_logic"): + changed_callables(root, self.base, head, "backend/scripts/claimed.py") + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + self._initialize(root) + target = root / "backend/scripts/claimed.py" + target.write_text("", encoding="utf-8") + self._git(root, "add", ".") + self._git(root, "commit", "-m", "remove callable") + head = self._git(root, "rev-parse", "HEAD") + with pytest.raises(MutationPolicyError, match="unmappable_changed_logic"): + changed_callables(root, self.base, head, "backend/scripts/claimed.py") + + def test_callable_mapping_uses_merge_base_not_advanced_main(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + self._initialize(root) + self._git(root, "branch", "feature") + target = root / "backend/scripts/claimed.py" + target.write_text( + "def claimed():\n return True\n\ndef main_only():\n return True\n", + encoding="utf-8", + ) + self._git(root, "add", ".") + self._git(root, "commit", "-m", "main moved") + advanced_main = self._git(root, "rev-parse", "HEAD") + self._git(root, "switch", "feature") + target.write_text("def claimed():\n return False\n", encoding="utf-8") + self._git(root, "add", ".") + self._git(root, "commit", "-m", "feature behavior") + feature = self._git(root, "rev-parse", "HEAD") + assert changed_callables( + root, advanced_main, feature, "backend/scripts/claimed.py" + ) == ["scripts.claimed.claimed"] + + def test_claim_only_callable_must_exist_in_target_ast(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + self._initialize(root) + head = self._git(root, "rev-parse", "HEAD") + claim = root / ".ci/behavior-claims/WS-TEST-001-01.json" + claim.parent.mkdir(parents=True) + claim.write_text( + json.dumps( + { + "schema_version": 1, + "chunk_id": "WS-TEST-001-01", + "claims": [ + { + "target": "backend/scripts/claimed.py", + "callables": ["scripts.claimed.missing"], + "tests": ["backend/tests/test_claimed.py::test_claimed"], + "outcomes": ["return"], + "boundaries": [], + } + ], + } + ), + encoding="utf-8", + ) + with pytest.raises(MutationPolicyError, match="missing_claim_callable"): + build_selection(root, self.base, head, "WS-TEST-001-01", claim) + + def test_blocking_verdict_allows_only_weak_control_and_unselected_exclusions(self) -> None: + filters = ["scripts.example.x_public__mutmut_*"] + pass_mutants = [ + {"name": "scripts.example.x_public__mutmut_1", "outcome": "killed"}, + { + "name": "scripts.mutation_policy.x__weak_calibration__mutmut_1", + "outcome": "survived", + }, + {"name": "scripts.example.x_other__mutmut_1", "outcome": "excluded"}, + ] + pass_filters = [*filters, "scripts.mutation_policy.x__weak_calibration__mutmut_*"] + counts = { + name: 0 + for name in ( + "generated", + "killed", + "survived", + "timeout", + "suspicious", + "excluded", + "error", + ) + } + counts.update({"generated": 3, "killed": 1, "survived": 1, "excluded": 1}) + assert classify_outcomes(counts, pass_mutants, pass_filters)["status"] == "pass" + + impostor = [{"name": "app.example.x__weak_calibration__mutmut_1", "outcome": "survived"}] + impostor_filters = ["app.example.x__weak_calibration__mutmut_*"] + impostor_counts = {name: 0 for name in counts} + impostor_counts.update({"generated": 1, "survived": 1}) + assert classify_outcomes(impostor_counts, impostor, impostor_filters)["status"] == "block" + + for outcome in ("survived", "timeout", "suspicious", "error", "excluded"): + mutants = [{"name": "scripts.example.x_public__mutmut_1", "outcome": outcome}] + blocked = {name: 0 for name in counts} + blocked.update({"generated": 1, outcome: 1}) + assert classify_outcomes(blocked, mutants, filters)["status"] == "block" + + def test_incomplete_and_unknown_outcomes_fail_closed(self) -> None: + with pytest.raises(MutationPolicyError, match="incomplete_mutation_outcomes"): + classify_outcomes({"generated": 0}, [], []) + counts = { + name: 0 + for name in ( + "generated", + "killed", + "survived", + "timeout", + "suspicious", + "excluded", + "error", + ) + } + counts["generated"] = 1 + with pytest.raises(MutationPolicyError, match="unknown_mutation_outcome"): + classify_outcomes(counts, [{"name": "mutant", "outcome": "unknown"}], []) + counts["generated"] = 0 + counts["killed"] = -1 + with pytest.raises(MutationPolicyError, match="invalid_mutation_outcomes"): + classify_outcomes(counts, [], []) + + def test_policy_self_test_proves_control_and_blocker(self) -> None: + policy_self_test() + + def test_calibration_rejects_contributor_named_impostors(self) -> None: + impostors = [ + {"name": "app.example.x__strong_calibration__mutmut_1", "outcome": "killed"}, + {"name": "app.example.x__weak_calibration__mutmut_1", "outcome": "survived"}, + ] + with pytest.raises(MutationPolicyError, match="strong_calibration_not_killed"): + _validate_calibration(impostors) + assert _validate_calibration( + [ + { + "name": "scripts.mutation_policy.x__strong_calibration__mutmut_1", + "outcome": "killed", + }, + { + "name": "scripts.mutation_policy.x__weak_calibration__mutmut_1", + "outcome": "survived", + }, + ] + ) == {"strong": {"killed": 1}, "weak": {"survived": 1}} + + def test_candidate_runtime_environment_excludes_ci_authority(self) -> None: + environment = _minimal_runtime_environment( + { + "PATH": "/bin", + "HOME": "/tmp/home", + "GITHUB_ENV": "/tmp/commands", + "GITHUB_TOKEN": "secret", + "SERVICE_PASSWORD": "secret", + "SIGNING_KEY": "secret", + } + ) + assert environment == { + "PATH": "/bin", + "HOME": "/tmp/home", + "PYTEST_DISABLE_PLUGIN_AUTOLOAD": "1", + "PYTHONHASHSEED": "0", + } + + def test_discovery_builds_applicable_exact_claim(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + self._initialize(root) + target = root / "backend/scripts/claimed.py" + target.write_text("def claimed():\n return False\n", encoding="utf-8") + claim = root / ".ci/behavior-claims/WS-TEST-001-01.json" + claim.parent.mkdir(parents=True) + claim.write_text( + json.dumps( + { + "schema_version": 1, + "chunk_id": "WS-TEST-001-01", + "claims": [ + { + "target": "backend/scripts/claimed.py", + "callables": ["scripts.claimed.claimed"], + "tests": ["backend/tests/test_claimed.py::test_claimed"], + "outcomes": ["return"], + "boundaries": [], + } + ], + } + ), + encoding="utf-8", + ) + self._git(root, "add", ".") + self._git(root, "commit", "-m", "claimed behavior") + head = self._git(root, "rev-parse", "HEAD") + selection = discover_selection(root, self.base, head) + assert selection["applicability"] == "applicable" + assert selection["claim_path"] == ".ci/behavior-claims/WS-TEST-001-01.json" + assert selection["changed_callables"] == { + "backend/scripts/claimed.py": ["scripts.claimed.claimed"] + } + + def test_main_supports_self_test_and_not_applicable_discovery( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.setattr(sys, "argv", ["mutation_policy.py", "--self-test"]) + assert _main() == 0 + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + self._initialize(root) + (root / "README.md").write_text("docs\n", encoding="utf-8") + self._git(root, "add", ".") + self._git(root, "commit", "-m", "docs") + head = self._git(root, "rev-parse", "HEAD") + output = root / "selection.json" + monkeypatch.setattr( + sys, + "argv", + [ + "mutation_policy.py", + "--repository-root", + str(root), + "--base-sha", + self.base, + "--head-sha", + head, + "--discover", + "--selection-output", + str(output), + ], + ) + assert _main() == 0 + assert ( + json.loads(output.read_text(encoding="utf-8"))["applicability"] == "not_applicable" + ) def test_zero_targets_and_invalid_revisions_fail_closed(self) -> None: with tempfile.TemporaryDirectory() as temporary: @@ -470,6 +932,53 @@ def test_execute_pilot_uses_disposable_tree_and_writes_reconciled_evidence(self) } assert self._git(root, "status", "--porcelain", "--untracked-files=no") == "" + def test_execute_pilot_enforcement_blocks_selected_survivor(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) / "repository" + root.mkdir() + self._initialize_execution_repository(root) + head = self._git(root, "rev-parse", "HEAD") + selection = build_selection( + root, + self.base, + head, + "WS-QUAL-001-04M", + root / ".ci/behavior-claims/WS-QUAL-001-04M.json", + ) + manifest = root / "trusted.txt" + manifest.write_text("trusted\n", encoding="utf-8") + executable = root / "fake-mutmut" + executable.write_text( + """#!/usr/bin/env python3 +import json +from pathlib import Path +path = Path('mutants/scripts/mutation_policy.py.meta') +path.parent.mkdir(parents=True) +path.write_text(json.dumps({'exit_code_by_key': { + 'scripts.mutation_policy.x__strong_calibration__mutmut_1': 1, + 'scripts.mutation_policy.x__weak_calibration__mutmut_1': 0, + 'scripts.mutation_policy.x_policy__mutmut_1': 0, +}})) +""", + encoding="utf-8", + ) + executable.chmod(0o700) + output = root / "evidence.json" + + with pytest.raises(MutationPolicyError, match="blocking_mutation_outcome"): + execute_pilot( + root, + selection, + manifest, + hashlib.sha256(manifest.read_bytes()).hexdigest(), + executable, + output, + 30, + enforce=True, + ) + + assert json.loads(output.read_text(encoding="utf-8"))["verdict"]["status"] == "block" + def test_execution_rejects_bad_digest_timeout_and_dirty_tree(self) -> None: with tempfile.TemporaryDirectory() as temporary: root = Path(temporary) / "repository" @@ -496,7 +1005,9 @@ def test_execution_rejects_bad_digest_timeout_and_dirty_tree(self) -> None: with pytest.raises(MutationPolicyError, match="dirty_source_tree"): execute_pilot(root, selection, manifest, digest, executable, output, 30) - def test_main_reports_policy_errors(self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]) -> None: + def test_main_reports_policy_errors( + self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] + ) -> None: monkeypatch.setattr( sys, "argv", diff --git a/docs/operations_backend_testing.md b/docs/operations_backend_testing.md index 453cdf08..84ad0243 100644 --- a/docs/operations_backend_testing.md +++ b/docs/operations_backend_testing.md @@ -171,41 +171,56 @@ does not override otherwise passing correctness, custody, service-contract, API, and coverage gates. Never lower coverage, skip nodes, or add a silent fallback to meet the target. -## Changed-scope mutation pilot - -`Mutation Pilot` is an independent, observational WS-QUAL-001-04M workflow. It -does not join the required Backend fan-in, alter the 78 percent global floor, or -alter any protected 90 percent subsystem floor. It runs only when the pilot's -policy, tests, claim, schema, Git-delta helper, or workflow changes. - -The pilot always selects eligible changed Python targets under `backend/app/` -or `backend/scripts/`. Schema-v1 files under `.ci/behavior-claims/` may add a -bounded target with qualified callables, exact owning pytest nodes, typed -observable outcomes, and any essential real boundaries. Every eligible changed -target requires this ownership mapping; claims cannot remove or replace one. -Malformed claims, stale chunk identifiers, unsafe paths, symlinks, missing -targets or tests, empty selection, and configuration drift fail closed. - -For pull requests, the hash-locked mutation toolchain is read only from -`scripts/mutation-requirements.txt` at the protected base revision and installed -with `pip --require-hashes`. It does not run PR-controlled packaging hooks or -install the backend package; the focused tests run with conftest loading -disabled because they use only the protected mutation-test toolchain. The -workflow has read-only permissions, disables persisted checkout credentials, -and receives no secrets. The policy wrapper runs from the exact PR-head -checkout; it archives that same head and runs baseline tests and mutmut only in -the disposable extraction with token environment variables removed. Symlinks -and special archive entries are rejected before test or mutation execution. -Evidence and the virtual environment live in runner-temporary storage outside -the PR checkout. The checked-out source tree must stay clean and retain the -same exact tree hash. - -The independent job is limited to 15 minutes, with mutation execution bounded -below 12 minutes. Its seven-day artifact binds base SHA, head SHA and tree, -manifest digest, exact configuration, selected targets and tests, elapsed time, -and generated, killed, survived, timeout, suspicious, excluded, and error -outcomes. Mutation score is not a contributor gate during this pilot. Baseline -test failure, dependency-custody failure, target escape, engine error, timeout, -or malformed evidence still fails the workflow. Only separately instructed -`WS-QUAL-001-05M`, after accepted hosted evidence and a human calibration -checkpoint, may propose a blocking survivor policy. +## Required changed-scope behavior mutation + +`Behavior Mutation Gate` is an independent required check. It does not join the +Backend fan-in, alter the 78 percent global floor, or alter any protected 90 +percent subsystem floor. It always emits a stable pull-request result. Internal +preflight returns typed `not_applicable` before dependency installation when the +exact delta contains neither an eligible target nor a behavior claim. + +The gate selects eligible changed Python targets under `backend/app/` or +`backend/scripts/`. One changed schema-v1 file under `.ci/behavior-claims/` +provides qualified callable ownership, exact pytest nodes, typed observable +outcomes, and essential real boundaries. Exact executable diff hunks must map +to claimed callables. Missing, multiple, stale, unsafe, symlinked, narrowed, or +unmappable claims fail closed. Mutmut configuration is generated only inside +the disposable archive from the validated selection. + +The hash-locked toolchain is read only from +`scripts/mutation-requirements.txt` at protected base and installed with +`pip --require-hashes`. The same protected base supplies `backend/uv.lock` and +`backend/pyproject.toml`; `uv sync --locked` installs the runtime and test +dependencies needed by owning backend tests without trusting dependency edits +from the pull-request head. Ordinary PR selection, classification, and verdict use +the evaluator and Git-delta helper archived from protected base; PR-head policy +code is not its own authority. Execution receives no secrets, uses read-only +permissions and no persisted checkout credentials, removes token environment +variables, and mutates only an exact-head disposable archive. Special entries, +source-tree drift, custody drift, and baseline failure block. + +The independent job has a 15-minute cap, 720-second shell limit, and 700-second +engine limit. Seven-day evidence binds the exact revisions/tree, protected +manifest, generated configuration, selection, targets, tests, elapsed time, +every mutant outcome, and the closed verdict. There is no score threshold. +Killed mutants pass. Meaningful survivors, timeout, suspicious, error, unknown, +or incomplete outcomes block. Excluded mutants pass only outside the selected +callable filters. The repository's exact weak calibration survivor is the sole +allowed control; contributors cannot add classifications, allowlists, free-form +exemptions, or source mutation pragmas. + +For local discovery, run the command documented in +`.ci/behavior-claims/README.md` and inspect the generated selection before +publishing. The hosted artifact contains: + +- `selection.json`: pre-install applicability and exact claim/target selection; +- `executed-selection.json`: the selection regenerated immediately before + execution; +- `evidence.json`: exact-head custody, configuration digests, elapsed time, + calibration, complete mutant outcomes, and the closed verdict. + +For `not_applicable`, only `selection.json` is expected. For an applicable +failure, first compare both selections, then inspect `verdict.status` and +`verdict.blockers` in `evidence.json`. A selected survivor must be repaired in +the owning assertion or production behavior. Missing evidence means the named +earlier step failed; use its job log rather than manufacturing an artifact. diff --git a/scripts/behavior-claim.schema.json b/scripts/behavior-claim.schema.json index b0974476..6083a079 100644 --- a/scripts/behavior-claim.schema.json +++ b/scripts/behavior-claim.schema.json @@ -20,7 +20,7 @@ "callables": { "type": "array", "minItems": 1, - "maxItems": 12, + "maxItems": 24, "uniqueItems": true, "items": {"type": "string", "pattern": "^[A-Za-z_][A-Za-z0-9_.]+$"} }, diff --git a/scripts/mutation-requirements.in b/scripts/mutation-requirements.in index 2da5ce7c..7a5bd642 100644 --- a/scripts/mutation-requirements.in +++ b/scripts/mutation-requirements.in @@ -5,3 +5,5 @@ mutmut==3.7.0 coverage==7.15.2 packaging==26.2 pytest==8.4.2 +# Install the protected backend lock into the disposable mutation environment. +uv==0.11.7 diff --git a/scripts/mutation-requirements.txt b/scripts/mutation-requirements.txt index 755fa1ef..146d833f 100644 --- a/scripts/mutation-requirements.txt +++ b/scripts/mutation-requirements.txt @@ -2,7 +2,7 @@ # This file is autogenerated by pip-compile with Python 3.12 # by the following command: # -# pip-compile --allow-unsafe --generate-hashes --output-file=scripts/mutation-requirements.txt --strip-extras scripts/mutation-requirements.in +# pip-compile --allow-unsafe --cert=None --client-cert=None --generate-hashes --index-url=None --output-file=scripts/mutation-requirements.txt --pip-args=None --strip-extras scripts/mutation-requirements.in # click==8.4.2 \ --hash=sha256:9a6cea6e60b17ebe0a44c5cc636d94f09bd66142c1cd7d8b4cd731c4917a15f6 \ @@ -399,3 +399,24 @@ uc-micro-py==2.0.0 \ --hash=sha256:3603a3859af53e5a39bc7677713c78ea6589ff188d70f4fee165db88e22b242c \ --hash=sha256:c53691e495c8db60e16ffc4861a35469b0ba0821fe409a8a7a0a71864d33a811 # via linkify-it-py +uv==0.11.7 \ + --hash=sha256:0df59ab0c6a4b14a763e8445e1c303af9abeb53cdfa4428daf9ff9642c0a3cce \ + --hash=sha256:162fa961a9a081dcea6e889c79f738a5ae56507047e4672964972e33c301bea9 \ + --hash=sha256:23d457d6731ebdb83f1bffebe4894edab2ef43c1ec5488433c74300db4958924 \ + --hash=sha256:46d971489b00bdb27e0aa715e4a5cd4ef2c28ea5b6ef78f2b67bf861eb44b405 \ + --hash=sha256:4e4d5e31bea86e1b6e0f5a0f95e14e80018e6f6c0129256d2915a4b3d793644d \ + --hash=sha256:553e67cc766d013ce24353fecd4ea5533d2aedcfd35f9fac430e07b1d1f23ed4 \ + --hash=sha256:5674dfb5944513f4b3735b05c2deba6b1b01151f46729d533d413a9a905f8c5d \ + --hash=sha256:5985a15a92bd9a170fc1947abb1fbc3e9828c5a430ad85b5bed8356c20b67a71 \ + --hash=sha256:6158b7e39464f1aa1e040daa0186cae4749a78b5cd80ac769f32ca711b8976b1 \ + --hash=sha256:750ee5b96959b807cf442b73dd8b55111862d63f258f896787ea5f06b68aaca9 \ + --hash=sha256:7d6a17507b8139b8803f445a03fd097f732ce8356b1b7b13cdb4dd8ef7f4b2e0 \ + --hash=sha256:8b2fe1ec6775dad10183e3fdce430a5b37b7857d49763c884f3a67eaa8ca6f8a \ + --hash=sha256:ceae53b202ea92bc954759bc7c7570cdcd5c3512fce15701198c19fd2dfb8605 \ + --hash=sha256:dd48823ca4b505124389f49ae50626ba9f57212b9047738efc95126ed5f3844d \ + --hash=sha256:eb91f52ee67e10d5290f2c2897e2171357f1a10966de38d83eefa93d96843b0c \ + --hash=sha256:f394331f0507e80ee732cb3df737589de53bed999dd02a6d24682f08c2f8ac4f \ + --hash=sha256:f422d39530516b1dfb28bb6e90c32bb7dacd50f6a383cd6e40c1a859419fbc8c \ + --hash=sha256:f97e9f4e4d44fb5c4dfaa05e858ef3414a96416a2e4af270ecd88a3e5fb049a9 \ + --hash=sha256:fab0bb43fbbc0ee5b5fee212078d2300c371b725faff7cf72eeaafa0bff0606b + # via -r scripts/mutation-requirements.in diff --git a/scripts/test_lightweight_agent_gates.py b/scripts/test_lightweight_agent_gates.py index 84b5b979..08c54754 100644 --- a/scripts/test_lightweight_agent_gates.py +++ b/scripts/test_lightweight_agent_gates.py @@ -124,28 +124,50 @@ def test_backend_uses_distributed_semantic_lanes_and_stable_fan_in(self) -> None self.assertIn("pull_request_review:", agent_gates) self.assertIn("--require-pr-approval", agent_gates) - def test_mutation_pilot_is_bounded_read_only_and_independent(self) -> None: + def test_behavior_mutation_gate_is_bounded_protected_and_independent(self) -> None: workflow = Path(".github/workflows/mutation-pilot.yml").read_text(encoding="utf-8") backend = Path(".github/workflows/backend.yml").read_text(encoding="utf-8") self.assertIn(" pull_request:\n", workflow) self.assertIn(" push:\n", workflow) self.assertNotIn("pull_request_target", workflow) + self.assertNotIn(" paths:", workflow) self.assertIn("permissions:\n contents: read", workflow) self.assertNotIn("contents: write", workflow) self.assertNotIn("continue-on-error", workflow) self.assertIn("timeout-minutes: 15", workflow) - self.assertEqual(workflow.count(' - "backend/pyproject.toml"'), 2) self.assertIn("timeout --signal=TERM --kill-after=15s 720s", workflow) self.assertIn("persist-credentials: false", workflow) self.assertIn("--require-hashes", workflow) self.assertIn( 'git show "${base_sha}:scripts/mutation-requirements.txt"', workflow ) + self.assertIn( + 'git show "${base_sha}:backend/pyproject.toml"', workflow + ) + self.assertIn('git show "${base_sha}:backend/uv.lock"', workflow) + self.assertIn('if [[ "${bootstrap}" == "true" ]]', workflow) + self.assertIn('"${gate_dir}/venv/bin/uv" sync', workflow) + self.assertIn("--locked", workflow) + self.assertIn("--inexact", workflow) + self.assertIn( + 'git show "${base_sha}:backend/scripts/mutation_policy.py"', workflow + ) + self.assertIn('git show "${base_sha}:scripts/git_delta.py"', workflow) + self.assertLess( + workflow.index("Discover exact mutation applicability before installation"), + workflow.index("Install protected hash-locked mutation toolchain"), + ) + self.assertIn("steps.preflight.outputs.applicability == 'applicable'", workflow) + self.assertIn("--discover", workflow) + self.assertIn("workstream-mutation-capability:discover-v1", workflow) + self.assertIn("--enforce", workflow) + self.assertIn("--self-test", workflow) self.assertIn("--timeout-seconds 700", workflow) self.assertIn("retention-days: 7", workflow) self.assertIn("include-hidden-files: true", workflow) self.assertNotIn('pip install -e "backend[dev]"', workflow) + self.assertNotIn('pip install -e ".[dev]"', workflow) self.assertNotIn("mutation-pilot", backend) if __name__ == "__main__":