From 41f59eb99f34b6d4b9cb7849046704d5af5b2441 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Tue, 11 Aug 2026 15:09:02 -0500 Subject: [PATCH 1/5] docs: propose mutation falsification service Assisted-by: AI Signed-off-by: Tim Nunamaker --- .../.openspec.yaml | 2 + .../design.md | 98 +++++++++++++++++ .../proposal.md | 29 +++++ .../specs/mutation-falsification/spec.md | 101 ++++++++++++++++++ .../tasks.md | 43 ++++++++ 5 files changed, 273 insertions(+) create mode 100644 openspec/changes/add-mutation-falsification-service/.openspec.yaml create mode 100644 openspec/changes/add-mutation-falsification-service/design.md create mode 100644 openspec/changes/add-mutation-falsification-service/proposal.md create mode 100644 openspec/changes/add-mutation-falsification-service/specs/mutation-falsification/spec.md create mode 100644 openspec/changes/add-mutation-falsification-service/tasks.md diff --git a/openspec/changes/add-mutation-falsification-service/.openspec.yaml b/openspec/changes/add-mutation-falsification-service/.openspec.yaml new file mode 100644 index 000000000..a8821c74d --- /dev/null +++ b/openspec/changes/add-mutation-falsification-service/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-08-11 diff --git a/openspec/changes/add-mutation-falsification-service/design.md b/openspec/changes/add-mutation-falsification-service/design.md new file mode 100644 index 000000000..968962e7e --- /dev/null +++ b/openspec/changes/add-mutation-falsification-service/design.md @@ -0,0 +1,98 @@ +## Context + +See `proposal.md` for motivation. PDPP uses Node's built-in test runner with `tsx`, a fail-closed test-accounting manifest, and several bespoke falsifiability tools. Two current examples are `scripts/test-migration/mutation-oracle.ts`, which applies named defects and proves rollback, and `packages/polyfill-connectors/scripts/mock-mutation-check.ts`, which mutates fake-server path facts. These tools produce useful evidence but have different inputs, classifications, and receipts. + +The repository is large enough that exhaustive mutation is an unattractive default. Tests also span subprocesses, databases, browsers, fixtures, and dynamic inputs, so a library's import graph cannot be the sole selection authority. Local execution must respect constrained-host concurrency controls. + +## Goals / Non-Goals + +**Goals:** + +- Make deliberate-fault evidence deterministic, reviewable, replayable, and cheap enough for routine agent use. +- Reuse existing test accounting, local resource controls, and falsifiability oracles. +- Optimize useful risk findings per reviewer-minute and compute-minute. +- Create an adapter boundary that permits domain operators, third-party engines, and later agent-generated mutants. + +**Non-Goals:** + +- A repository-wide mutation score or a 100% mutation target. +- Exhaustive mutation of the monorepo. +- Automatic test generation, deletion, or quality grading. +- Mutating live services, personal data, or credentialed browser sessions. +- Replacing coverage, integration tests, or existing conformance harnesses. + +## Decisions + +### 1. The stable abstraction is a risk-falsification packet, not a mutation engine + +A packet carries the risk source, exact mutation, expected affected behavior, selected tests, backstop, and budget. A receipt carries what ran and what happened. Generators and runners sit behind adapters. + +This keeps StrykerJS, bespoke TypeScript transforms, historical bug patches, and future agent-generated mutations interchangeable. It also makes the judgment boundary explicit: generators propose faults; deterministic tests and receipt validation judge them. + +Alternative: make StrykerJS configuration the architecture. Rejected because PDPP primarily uses `node:test` through `tsx`; StrykerJS has no native Node test-runner integration, and its command runner loses per-test intelligence. It remains a useful feasibility adapter for pure, precompiled TypeScript islands. + +### 2. Start by wrapping the test-migration oracle + +The first adapter will express the named cases from `scripts/test-migration/mutation-oracle.ts` as packets and preserve its byte-identical rollback proof. This surface is small, deterministic, already falsifiable, and has no live-service dependency. It proves the substrate without first inventing a source transformer. + +After the substrate works, add two or three domain mutations on one small high-risk pure surface. Good candidates include authorization decisions, projection/filter boundaries, or connector cursor/frontier rules. Select the final target by runtime, isolation, and the availability of an independent oracle. + +Alternative: begin with all connector path mutations. Rejected for the first slice because the current connector tool intentionally reports many unknown surfaces and has broader runtime variability. + +### 3. Use explicit selection first, then measure smarter routing + +Version one packets name focused tests and a relevant accounted backstop. The runner does not infer completeness from static imports. It records both results so later analysis can measure whether focused selection missed a kill found by the backstop. + +Later routing may combine changed-code coverage, import and literal-input graphs, historical failures, and test accounting. It may become a fast lane only after shadow comparisons establish an acceptable miss rate. + +Alternative: immediately implement whole-repository per-test coverage selection. Rejected because it adds substantial instrumentation cost before the packet and receipt model is proven. + +### 4. Execute in disposable workspaces with baseline-first judgment + +The executor uses a clean, disposable worktree or fixture repository. For each packet it: + +1. validates revision and packet bindings; +2. runs the focused clean baseline; +3. applies one mutation; +4. runs the focused tests; +5. optionally runs the declared backstop according to packet policy; +6. restores and verifies the exact tree; +7. emits and validates a receipt. + +Killed requires a passing baseline plus a mutation-present failure attributable in the receipt. A green mutant is survived unless execution evidence supports not-exercised, or a reviewer records equivalent-suspect or uninteresting with independent justification. + +Alternative: edit the developer's working tree and revert files in place. Rejected because interruption, concurrent agents, and unrelated dirty changes make cleanup evidence weaker. + +### 5. Bind receipts to content and effective execution + +Canonical JSON packets and receipts use a versioned schema. Content hashes bind the packet, base tree, mutated target before and after, mutation patch/operator parameters, exact effective argv, selected profile, captured results, and restored tree. The runner writes receipts atomically. Validation recomputes bindings and rejects partial or forged records. + +The schema records generator kind and provenance without assigning it authority. A later agent-generated mutant therefore remains a proposal judged by deterministic execution. + +Alternative: human-readable logs only. Rejected because agents cannot reliably compare, replay, or audit unbound prose. + +### 6. Calibrate with outcome distributions, not an aggregate score + +The pilot records killed, survived, not exercised, timeout, execution error, equivalent-suspect, and uninteresting counts by operator and risk source. Primary operating measures are productive-mutant rate, actionable-survivor rate, reviewer time, compute time, focused-to-backstop miss rate, cleanup failures, and flaky baseline rate. + +No aggregate mutation percentage gates a pull request. A future gate must be scoped to a stable risk/operator class and justified by observed signal and cost. + +## Risks / Trade-offs + +- **Mutants are easy to generate but expensive to judge** → cap packets, suppress obviously arid targets, and measure actionable findings per reviewer-minute. +- **Equivalent or uninteresting mutants create false urgency** → preserve distinct classifications and require recorded triage rather than forcing tests to kill everything. +- **Focused selection misses dynamic dependencies** → require an explicit backstop and measure focused-to-backstop misses before using selection as a gate. +- **Mutation trials overload local hosts** → use finite packet budgets, low default concurrency, and the repository's cross-process local test guard. +- **An interrupted mutation contaminates later work** → use disposable workspaces, atomic receipts, bounded cleanup, and exact tree-identity verification. +- **The same agent authors code, mutant, and test** → treat generator output as untrusted; require deterministic oracles and preserve generator/reviewer provenance. +- **A generic engine becomes a maintenance burden** → keep it behind the adapter contract and continue only if a measured pilot outperforms focused domain operators. + +## Migration Plan + +1. Add schemas, validators, receipt verification, and adversarial self-tests. +2. Wrap the existing test-migration oracle and run it locally in advisory mode. +3. Add one bounded domain pilot and compare focused tests with its accounted backstop. +4. Run a separate StrykerJS feasibility experiment on precompiled pure TypeScript; retain it only if setup, routing, and signal costs are competitive. +5. Review pilot evidence before adding CI scheduling, additional operators, or any narrow blocking rule. + +Rollback removes the advisory entry point and generated receipts. Existing mutation oracles remain independently runnable until their adapter demonstrates equivalent evidence. diff --git a/openspec/changes/add-mutation-falsification-service/proposal.md b/openspec/changes/add-mutation-falsification-service/proposal.md new file mode 100644 index 000000000..ff6592c81 --- /dev/null +++ b/openspec/changes/add-mutation-falsification-service/proposal.md @@ -0,0 +1,29 @@ +## Why + +PDPP has strong bespoke falsifiability tests, but no common way to define, run, and audit deliberate faults across its AI-authored implementation and tests. A mutation score would hide the important question: whether a relevant test detects a plausible failure at an acceptable evidence and compute cost. + +## What Changes + +- Add a risk-falsification service that accepts machine-readable mutation packets and emits replayable, revision-bound receipts. +- Require baseline-first, isolated execution, exact cleanup, explicit resource budgets, and honest outcome classification. +- Route mutation checks through the existing test-accounting authority and record both focused tests and a relevant backstop. +- Add an adapter for an existing PDPP falsifiability oracle as the first end-to-end implementation. +- Keep mutation runs advisory during calibration. Do not introduce a repository-wide mutation score, coverage quota, or automatic test-deletion rule. +- Leave room for later domain-mutator, StrykerJS, and agent-generated-mutant adapters without making any one generator the architecture. + +## Capabilities + +### New Capabilities + +- `mutation-falsification`: Defines mutation packets, execution safety, result classification, evidence receipts, test-authority integration, and calibrated rollout. + +### Modified Capabilities + +None. + +## Impact + +- Adds repository tooling, schemas, tests, and documentation for mutation packets and receipts. +- Integrates with `test-accounting.manifest.json` and its runners without changing suite ownership. +- Initially wraps one existing mutation or falsifiability oracle; production behavior and public APIs do not change. +- Mutation work remains bounded and non-blocking until observed evidence supports a narrower gate. diff --git a/openspec/changes/add-mutation-falsification-service/specs/mutation-falsification/spec.md b/openspec/changes/add-mutation-falsification-service/specs/mutation-falsification/spec.md new file mode 100644 index 000000000..18b65ae2f --- /dev/null +++ b/openspec/changes/add-mutation-falsification-service/specs/mutation-falsification/spec.md @@ -0,0 +1,101 @@ +## Purpose + +Provide bounded, replayable evidence that PDPP tests detect plausible faults without turning mutation counts or scores into quality gates. + +## ADDED Requirements + +### Requirement: Mutation work SHALL be declared as a validated packet + +Each run SHALL accept a machine-readable packet that identifies the repository revision, risk being tested, mutation target and change, generator provenance, selected test authority, relevant backstop, resource budget, and forbidden execution profiles. The service SHALL reject incomplete, malformed, or revision-mismatched packets before changing source files. + +#### Scenario: Valid packet is accepted +- **WHEN** a packet identifies a clean matching revision, a concrete mutation, accounted tests, a backstop, and finite budgets +- **THEN** the service SHALL accept it for execution + +#### Scenario: Packet does not match the checkout +- **WHEN** the packet's revision or target content does not match the checkout +- **THEN** the service SHALL reject it without applying the mutation + +### Requirement: Mutation execution SHALL establish a passing baseline + +The service SHALL run the packet's selected tests against the unmodified revision before applying the mutation. It SHALL NOT classify a mutant as killed or survived unless the baseline passes under the same declared execution conditions. + +#### Scenario: Baseline fails +- **WHEN** any selected baseline test fails, times out, or cannot run +- **THEN** the service SHALL stop that trial and classify it as an execution error rather than a mutation result + +### Requirement: Mutation execution SHALL be isolated and reversible + +The service SHALL apply each mutation in isolation, SHALL prevent mutation trials from overlapping beyond the declared resource budget, and SHALL restore the checkout to its exact pre-trial state before another trial or successful exit. A cleanup failure SHALL be visible and SHALL prevent subsequent trials in that checkout. + +#### Scenario: Mutant is exercised +- **WHEN** the service applies a mutation and runs its selected tests +- **THEN** no other packet's source mutation SHALL be present in that trial + +#### Scenario: Trial is interrupted +- **WHEN** a trial exits by failure, timeout, or handled termination +- **THEN** the service SHALL attempt restoration and SHALL verify the pre-trial tree identity before permitting more work + +### Requirement: Outcomes SHALL preserve uncertainty + +The service SHALL distinguish at least killed, survived, not exercised, timeout, execution error, equivalent-suspect, and uninteresting outcomes. Equivalent-suspect and uninteresting SHALL require recorded triage and SHALL NOT be inferred only from a passing test run. + +#### Scenario: Selected test detects the mutation +- **WHEN** the clean baseline passes and an otherwise valid selected test fails because the mutation is present +- **THEN** the service SHALL classify the mutant as killed and record the detecting test evidence + +#### Scenario: Selected tests stay green +- **WHEN** the clean baseline passes and all selected tests pass with the mutation present +- **THEN** the service SHALL classify the mutant as survived unless independent evidence supports a more specific classification + +#### Scenario: Mutation target is not exercised +- **WHEN** available execution evidence shows that no selected test reaches the mutated behavior +- **THEN** the service SHALL classify the mutant as not exercised rather than survived + +### Requirement: Every trial SHALL emit a verifiable receipt + +The service SHALL emit a machine-readable receipt bound to the packet, repository revision, pre- and post-restoration tree identity, exact effective commands, environment profile, mutant identity, observed outcomes, timestamps, duration, exit status, and captured diagnostic output. Receipt validation SHALL reject missing or inconsistent evidence. + +#### Scenario: Receipt is replayed +- **WHEN** a reviewer validates a receipt against its packet and repository revision +- **THEN** the reviewer SHALL be able to identify the exact mutation, tests, budgets, result, and cleanup proof without trusting prose + +#### Scenario: Receipt is altered +- **WHEN** a bound packet field, command, result, mutant identity, or tree identity is changed after issue +- **THEN** receipt validation SHALL fail + +### Requirement: Test selection SHALL respect executable accounting + +Selected tests and backstops SHALL resolve through the repository's executable test-accounting authority or an explicitly declared, validated mutation-oracle command. The service SHALL record why each selected test is relevant and SHALL NOT treat static imports alone as complete dependency evidence. + +#### Scenario: Accounted focused tests and backstop are declared +- **WHEN** a packet selects a focused test set for fast feedback +- **THEN** it SHALL also identify the relevant accounted package, suite, or control-lane backstop used to measure selection misses + +#### Scenario: Unaccounted side lane is requested +- **WHEN** a packet names an executable test that is neither accounted nor an approved mutation oracle +- **THEN** the service SHALL reject the packet + +### Requirement: Mutation runs SHALL obey explicit safety and resource bounds + +Each packet SHALL declare finite wall-time, trial-count, process-concurrency, and cleanup bounds. Local execution SHALL cooperate with the repository's local test-resource controls. Live third-party services, personal data, production credentials, and stateful browser profiles SHALL be forbidden unless a later capability defines an isolated mutation profile for them. + +#### Scenario: Budget is exhausted +- **WHEN** a packet reaches a declared time, trial, or process limit +- **THEN** the service SHALL stop new work, restore the checkout, and report a bounded outcome + +#### Scenario: Packet requests a forbidden profile +- **WHEN** a packet requires live credentials, personal data, or an undeclared stateful external profile +- **THEN** the service SHALL reject it before mutation execution + +### Requirement: Mutation evidence SHALL remain advisory until calibrated + +Initial mutation execution SHALL report evidence without enforcing a repository-wide score or adequacy threshold. A later blocking rule SHALL identify a narrow risk class, a stable operator set, an accountable backstop, observed false-positive and miss evidence, and an explicit rollback path. + +#### Scenario: A survived mutant is reported during calibration +- **WHEN** an advisory run produces a survived mutant +- **THEN** the service SHALL nominate it for triage and SHALL NOT automatically fail unrelated changes or require a new test + +#### Scenario: A test is proposed for deletion +- **WHEN** mutation evidence shows that another test kills the same sampled mutants +- **THEN** that evidence alone SHALL NOT authorize deletion because the test may protect a different behavior or fault class diff --git a/openspec/changes/add-mutation-falsification-service/tasks.md b/openspec/changes/add-mutation-falsification-service/tasks.md new file mode 100644 index 000000000..d4b696286 --- /dev/null +++ b/openspec/changes/add-mutation-falsification-service/tasks.md @@ -0,0 +1,43 @@ +## 1. Packet and Receipt Contracts + +- [ ] 1.1 Define versioned mutation-packet and mutation-receipt schemas with revision, risk, mutation, provenance, test selection, backstop, budget, outcome, command, diagnostic, and tree-identity fields. +- [ ] 1.2 Implement strict packet and receipt parsers that reject unknown schema versions, missing bounds, revision mismatches, forbidden profiles, and inconsistent outcome evidence. +- [ ] 1.3 Add canonical serialization and content binding for packets, receipts, effective commands, mutant content, and pre/post tree identities. +- [ ] 1.4 Add adversarial tests that alter each bound receipt class and prove validation fails. + +## 2. Safe Mutation Executor + +- [ ] 2.1 Implement a baseline-first executor in a disposable clean workspace with one mutation per trial. +- [ ] 2.2 Integrate finite wall-time, trial-count, process-concurrency, and cleanup bounds with the repository's local test-resource controls. +- [ ] 2.3 Implement killed, survived, not-exercised, timeout, execution-error, equivalent-suspect, and uninteresting result handling without collapsing uncertainty. +- [ ] 2.4 Emit receipts atomically with exact effective argv, captured diagnostics, durations, exit status, and cleanup evidence. +- [ ] 2.5 Add interruption and failure-path tests that prove bounded termination, child cleanup, exact restoration, and refusal to continue after cleanup failure. + +## 3. Test Authority and Selection + +- [ ] 3.1 Resolve declared focused tests and backstops against `test-accounting.manifest.json` or an explicit approved mutation-oracle entry. +- [ ] 3.2 Reject unaccounted executable side lanes and static-import-only completeness claims. +- [ ] 3.3 Record focused and backstop results separately so selection misses can be measured. +- [ ] 3.4 Add tests for dynamic/literal inputs, profile mismatch, missing backstop, and focused-pass/backstop-kill cases. + +## 4. Existing-Oracle Adapter + +- [ ] 4.1 Express each named case in `scripts/test-migration/mutation-oracle.ts` as a mutation packet without weakening its current checks. +- [ ] 4.2 Run the existing oracle through the common executor and preserve its byte-identical rollback proof in receipts. +- [ ] 4.3 Add differential tests proving dropped tests, silent skips, assertion loss, import breakage, and stale literal paths remain detectable. +- [ ] 4.4 Keep the legacy entry point available until receipt comparison proves the adapter has equal or stronger evidence. + +## 5. Domain Pilot + +- [ ] 5.1 Select one small, high-risk, hermetic domain surface using measured runtime, independent-oracle strength, and expected operator value. +- [ ] 5.2 Implement two or three domain-specific mutation operators with explicit risk provenance and compile-valid output checks. +- [ ] 5.3 Run the focused selection and relevant accounted backstop in advisory mode; capture outcome distribution, misses, runtime, and reviewer effort. +- [ ] 5.4 Add regression tests for every actionable survivor repaired during the pilot and prove each test fails before the repair or kills its nominated mutant. + +## 6. Feasibility and Rollout + +- [ ] 6.1 Run a time-boxed StrykerJS command-runner experiment on one precompiled pure-TypeScript island, including setup cost, test routing quality, mutant quality, and compute cost. +- [ ] 6.2 Record an explicit continue, adapt, or stop decision for the StrykerJS adapter based on comparison with domain operators. +- [ ] 6.3 Document packet authoring, local advisory execution, receipt triage, equivalent-suspect review, and forbidden live-data profiles. +- [ ] 6.4 Add a non-blocking control lane only after local reliability is proven, with explicit concurrency and time budgets. +- [ ] 6.5 Publish a pilot report covering productive-mutant rate, actionable survivors, focused-to-backstop misses, flaky baselines, cleanup failures, reviewer time, and compute time; propose no blocking gate without calibrated evidence. From 80d158ac0216e0449d18cdbd010143963d10c0db Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Tue, 11 Aug 2026 15:35:57 -0500 Subject: [PATCH 2/5] docs: narrow mutation falsification proposal Assisted-by: AI Signed-off-by: Tim Nunamaker --- .../design.md | 116 ++++++++-------- .../proposal.md | 24 ++-- .../specs/mutation-falsification/spec.md | 130 +++++++++--------- .../tasks.md | 68 ++++----- 4 files changed, 164 insertions(+), 174 deletions(-) diff --git a/openspec/changes/add-mutation-falsification-service/design.md b/openspec/changes/add-mutation-falsification-service/design.md index 968962e7e..4b59a599c 100644 --- a/openspec/changes/add-mutation-falsification-service/design.md +++ b/openspec/changes/add-mutation-falsification-service/design.md @@ -1,98 +1,106 @@ ## Context -See `proposal.md` for motivation. PDPP uses Node's built-in test runner with `tsx`, a fail-closed test-accounting manifest, and several bespoke falsifiability tools. Two current examples are `scripts/test-migration/mutation-oracle.ts`, which applies named defects and proves rollback, and `packages/polyfill-connectors/scripts/mock-mutation-check.ts`, which mutates fake-server path facts. These tools produce useful evidence but have different inputs, classifications, and receipts. +See `proposal.md` for motivation. PDPP's current test-migration oracle already owns a strong, self-contained lifecycle: it creates fixture repositories, applies named faults, invokes mutation-specific judges, runs a positive control, proves rollback, and disposes its fixtures. The GroupMe connector also has real mutation-killing tests for historical pagination and cursor-progress faults. -The repository is large enough that exhaustive mutation is an unattractive default. Tests also span subprocesses, databases, browsers, fixtures, and dynamic inputs, so a library's import graph cannot be the sole selection authority. Local execution must respect constrained-host concurrency controls. +The current test-accounting authority issues receipts only for manifest-owned complete plans on clean trees. It does not authorize arbitrary focused subsets or dirty mutants. A Git worktree isolates tracked source state but does not constrain filesystem, environment, network, credentials, processes, caches, databases, or Docker access. Local hashes bind content but do not authenticate an issuer. ## Goals / Non-Goals **Goals:** -- Make deliberate-fault evidence deterministic, reviewable, replayable, and cheap enough for routine agent use. -- Reuse existing test accounting, local resource controls, and falsifiability oracles. -- Optimize useful risk findings per reviewer-minute and compute-minute. -- Create an adapter boundary that permits domain operators, third-party engines, and later agent-generated mutants. +- Make one existing oracle's evidence structured without weakening its lifecycle. +- Run one trusted real-domain pilot against the current accounting authority's actual contract. +- Separate requested intent, machine observations, derived projections, and independent triage. +- Measure whether any shared infrastructure would reduce real repeated reasoning and audit cost. **Non-Goals:** -- A repository-wide mutation score or a 100% mutation target. -- Exhaustive mutation of the monorepo. -- Automatic test generation, deletion, or quality grading. -- Mutating live services, personal data, or credentialed browser sessions. -- Replacing coverage, integration tests, or existing conformance harnesses. +- A generic source-mutating executor or untrusted-code sandbox. +- Arbitrary, packet-authored, or agent-generated patches and commands. +- A new test-accounting subset authority. +- StrykerJS, CI scheduling, blocking gates, mutation scores, or test-deletion authority. +- Hard CPU or memory containment where the host cannot enforce it. ## Decisions -### 1. The stable abstraction is a risk-falsification packet, not a mutation engine +### 1. Version one is a trusted local evidence program, not a service -A packet carries the risk source, exact mutation, expected affected behavior, selected tests, backstop, and budget. A receipt carries what ran and what happened. Generators and runners sit behind adapters. +There is no daemon, queue, server, network API, or remotely supplied executable input. A repository-owned registry names the only permitted adapters and operators. Intent may request a registered risk and a stricter budget; versioned repository policy derives the effective command, working directory, environment allowlist, immutable judge closure, focused evidence, complete backstop, and host limits. -This keeps StrykerJS, bespoke TypeScript transforms, historical bug patches, and future agent-generated mutations interchangeable. It also makes the judgment boundary explicit: generators propose faults; deterministic tests and receipt validation judge them. +This prevents a generator from choosing its own judge or safety policy. Arbitrary and agent-generated executable mutations require a separate sandbox design that proves filesystem, network, environment, process-tree, CPU, memory, disk, and output containment. -Alternative: make StrykerJS configuration the architecture. Rejected because PDPP primarily uses `node:test` through `tsx`; StrykerJS has no native Node test-runner integration, and its command runner loses per-test intelligence. It remains a useful feasibility adapter for pure, precompiled TypeScript islands. +### 2. Adapters own mutation mechanics until common structure is earned -### 2. Start by wrapping the test-migration oracle +The migration oracle remains self-contained. It gains structured output but keeps its current named cases, fixture repositories, mutation-specific judges, positive control, and rollback proof. It demonstrates evidence shape and crash honesty, not a generic executor. -The first adapter will express the named cases from `scripts/test-migration/mutation-oracle.ts` as packets and preserve its byte-identical rollback proof. This surface is small, deterministic, already falsifiable, and has no live-service dependency. It proves the substrate without first inventing a source transformer. +The second adapter is a GroupMe cursor/frontier pilot. It uses two or three checked-in declarative operators over `packages/polyfill-connectors/connectors/groupme/index.ts`, such as reintroducing the historical page ceiling or weakening non-progress detection. The operators have exact preimages and permitted postimages; they cannot alter tests, runners, policy, or manifests. -After the substrate works, add two or three domain mutations on one small high-risk pure surface. Good candidates include authorization decisions, projection/filter boundaries, or connector cursor/frontier rules. Select the final target by runtime, isolation, and the availability of an independent oracle. +Only after both adapters run will a decision memo identify whether they share a deep stable boundary. Until then, duplication is preferable to a shallow common executor with adapter-specific escape hatches. -Alternative: begin with all connector path mutations. Rejected for the first slice because the current connector tool intentionally reports many unknown surfaces and has broader runtime variability. +### 3. Evidence uses three immutable artifact types -### 3. Use explicit selection first, then measure smarter routing +An **intent packet** records requested risk, base identity, adapter/operator descriptor, and requested bounds. Its canonical digest is its identifier; callers do not supply the identifier. -Version one packets name focused tests and a relevant accounted backstop. The runner does not infer completeness from static imports. It records both results so later analysis can measure whether focused selection missed a kill found by the backstop. +An **attempt receipt** records one execution's raw observations: issued random attempt ID, deterministic trial key, resolved policy, exact effective plan, environment profile, base/mutant/judge identities, bounded artifact digests and sizes, baseline/materialization/focused/backstop/reachability/cleanup axes, duration, exit or signal, and any referenced accounting receipt digests. -Later routing may combine changed-code coverage, import and literal-input graphs, historical failures, and test accounting. It may become a fast lane only after shadow comparisons establish an acceptable miss rate. +A **triage receipt** is append-only and binds one attempt digest. It records an independent reviewer's claimed identity, disposition, evidence, reason, and timestamp. Version one does not authenticate that identity. A different reviewer from the operator/test author is required before likely-equivalent or uninteresting evidence is excluded from reported actionable results. -Alternative: immediately implement whole-repository per-test coverage selection. Rejected because it adds substantial instrumentation cost before the packet and receipt model is proven. +`killed`, `survived`, and `inconclusive` are computed projections, never caller fields. `not_exercised` requires adapter-supplied validated reachability evidence; otherwise reachability is `unknown`. Timeout remains a timeout unless a later predeclared repeat policy supports a stronger interpretation. -### 4. Execute in disposable workspaces with baseline-first judgment +### 4. Digests provide integrity binding, not authenticity -The executor uses a clean, disposable worktree or fixture repository. For each packet it: +Canonical JSON, schema version, canonicalization version, and hash algorithm are explicit. A trial key binds the intent digest, repository tree, adapter version, policy version, and mutation identity. Each run gets a random attempt ID, so replay has stable identity but different observations. -1. validates revision and packet bindings; -2. runs the focused clean baseline; -3. applies one mutation; -4. runs the focused tests; -5. optionally runs the declared backstop according to packet policy; -6. restores and verifies the exact tree; -7. emits and validates a receipt. +Before spawn, the adapter-specific runner writes an issued attempt marker in a verifier-owned run directory. Complete receipts publish atomically only after structured output validation and cleanup evidence. Interrupted markers remain incomplete and are discovered at next start. Referenced transcripts are bounded artifacts whose digests and sizes appear in the receipt. -Killed requires a passing baseline plus a mutation-present failure attributable in the receipt. A green mutant is survived unless execution evidence supports not-exercised, or a reviewer records equivalent-suspect or uninteresting with independent justification. +Anyone controlling the host can rewrite records and recompute an unkeyed digest. Version one therefore claims internal consistency and tamper evidence relative to a separately retained digest, not issuer authenticity. Authenticated provenance would require a later CI signature or platform attestation. -Alternative: edit the developer's working tree and revert files in place. Rejected because interruption, concurrent agents, and unrelated dirty changes make cleanup evidence weaker. +### 5. The migration oracle is the evidence pilot -### 5. Bind receipts to content and effective execution +The first slice adds a structured JSON mode to `scripts/test-migration/mutation-oracle.ts`. Legacy and structured modes must agree on every named case, catching check, hole, positive control, and rollback result. The source checkout must remain unchanged. -Canonical JSON packets and receipts use a versioned schema. Content hashes bind the packet, base tree, mutated target before and after, mutation patch/operator parameters, exact effective argv, selected profile, captured results, and restored tree. The runner writes receipts atomically. Validation recomputes bindings and rejects partial or forged records. +The adapter-specific runner accepts only `test-migration-oracle/v1`; it derives the command and allowlisted environment. It imposes hard wall-time and output-byte limits, records partial/crash states honestly, and never interprets missing output as success. It does not claim focused selection, test-accounting authority, or domain value. -The schema records generator kind and provenance without assigning it authority. A later agent-generated mutant therefore remains a proposal judged by deterministic execution. +### 6. GroupMe is the real-domain calibration pilot -Alternative: human-readable logs only. Rejected because agents cannot reliably compare, replay, or audit unbound prose. +The pilot uses the existing hermetic GroupMe cursor/frontier tests as focused adapter evidence. It creates a fresh one-commit mutant descendant for each trusted operator. The clean and mutant complete `polyfill-connectors` suite run through the unchanged test-accounting authority; their verified receipt digests are referenced rather than duplicated. -### 6. Calibrate with outcome distributions, not an aggregate score +The focused clean baseline and complete clean backstop must pass before interpreting mutant evidence. Every focused survivor receives the complete mutant backstop. A focused pass plus backstop failure is a selector miss and blocks selector promotion. If a required backstop cannot complete, the attempt is inconclusive. -The pilot records killed, survived, not exercised, timeout, execution error, equivalent-suspect, and uninteresting counts by operator and risk source. Primary operating measures are productive-mutant rate, actionable-survivor rate, reviewer time, compute time, focused-to-backstop miss rate, cleanup failures, and flaky baseline rate. +A clean baseline may be reused across operators only when the repository tree, judge and adapter closure, effective command, allowlisted environment, dependency identity, and budget remain digest-identical. Otherwise it must run again. -No aggregate mutation percentage gates a pull request. A future gate must be scoped to a stable risk/operator class and justified by observed signal and cost. +The source and judge closures are separate. Operators can change only predeclared production ranges in GroupMe's implementation. Tests, policy, runner, manifest, lockfile, and receipt validator remain immutable and are digest-bound. + +### 7. Workspaces are one-shot source isolation, not sandboxes + +Domain attempts use a configured disk-backed root with free-space preflight. Each attempt gets a fresh workspace and clean committed mutant descendant. Dependencies are materialized with the repository-pinned `pnpm@10.33.0`, offline and frozen from the lockfile, with lifecycle scripts disabled for this pure TypeScript pilot. If that exact materialization cannot satisfy the pilot, execution stops and the design must be reviewed rather than silently sharing mutable build output. + +Successful workspaces are deleted before the completed attempt receipt is published. Interrupted or cleanup-failed workspaces are quarantined and never reused. Next-start recovery reports owned incomplete markers and quarantines before new work. This handles hard termination honestly; it does not pretend `finally` runs after power loss or `SIGKILL`. + +Initial host policy runs one adapter at a time, at most 20 domain mutants and 10 wall-clock minutes, with hard limits for adapter wall time, captured output, workspace size, and cleanup time. CPU use targets no more than 50% of available processors and memory is observed, but neither is described as hard containment without a host mechanism. Live credentials, personal data, third-party network, stateful browsers, and shared production-like databases are prohibited. Required unsupported controls cause refusal. + +### 8. The decision gate precedes shared infrastructure + +The pilot predefines valid-trial denominators and reports raw counts for execution axes, projections, selector misses, triage dispositions, runtime, output/workspace size, cleanup, and reviewer minutes. Setup time is reported separately. + +Stop or narrow immediately on a cleanup or containment failure, an unexplained selector miss, authority/receipt mismatch, or evidence corruption. Stop generalization if setup consumes most runtime, review exceeds five minutes per disputed attempt, most operators are invalid/trivial, no useful risk evidence appears within 20 mutants/10 minutes, or the adapters do not expose repeated policy/evidence logic. + +Continue only if evidence is interpretable, costs are acceptable, there are no unexplained selector or cleanup failures, and a proposed shared module would hide substantial repeated invariants across both adapters. Any coordinator, generic executor, Stryker experiment, CI lane, or sandbox is a new reviewed OpenSpec change. ## Risks / Trade-offs -- **Mutants are easy to generate but expensive to judge** → cap packets, suppress obviously arid targets, and measure actionable findings per reviewer-minute. -- **Equivalent or uninteresting mutants create false urgency** → preserve distinct classifications and require recorded triage rather than forcing tests to kill everything. -- **Focused selection misses dynamic dependencies** → require an explicit backstop and measure focused-to-backstop misses before using selection as a gate. -- **Mutation trials overload local hosts** → use finite packet budgets, low default concurrency, and the repository's cross-process local test guard. -- **An interrupted mutation contaminates later work** → use disposable workspaces, atomic receipts, bounded cleanup, and exact tree-identity verification. -- **The same agent authors code, mutant, and test** → treat generator output as untrusted; require deterministic oracles and preserve generator/reviewer provenance. -- **A generic engine becomes a maintenance burden** → keep it behind the adapter contract and continue only if a measured pilot outperforms focused domain operators. +- **The narrow slices may not justify a framework** → treat stopping with two useful purpose-fit adapters as success. +- **Local evidence lacks authenticated issuer identity** → state the trust boundary and preserve external attestation as a separate future capability. +- **Full connector backstops may dominate runtime** → measure setup and execution separately; stop rather than weaken the mandatory calibration backstop. +- **A trusted mutant can still affect ambient host state accidentally** → constrain paths/operators/environment, use hermetic tests, run sequentially, and never call a worktree a sandbox. +- **Offline dependency materialization may fail** → refuse the attempt and review the dependency strategy; do not share mutable outputs silently. +- **Small samples cannot prove selector completeness** → report raw counts and treat any observed miss as disqualifying, without claiming zero misses proves completeness. ## Migration Plan -1. Add schemas, validators, receipt verification, and adversarial self-tests. -2. Wrap the existing test-migration oracle and run it locally in advisory mode. -3. Add one bounded domain pilot and compare focused tests with its accounted backstop. -4. Run a separate StrykerJS feasibility experiment on precompiled pure TypeScript; retain it only if setup, routing, and signal costs are competitive. -5. Review pilot evidence before adding CI scheduling, additional operators, or any narrow blocking rule. +1. Land this revised design only after a second independent architecture review returns LAND. +2. Add structured evidence to the existing migration oracle and validate it without generic execution infrastructure. +3. Run the trusted GroupMe cursor/frontier pilot under mandatory clean and mutant backstops. +4. Publish a continue, narrow, or stop decision memo. Do not implement shared infrastructure from this change. -Rollback removes the advisory entry point and generated receipts. Existing mutation oracles remain independently runnable until their adapter demonstrates equivalent evidence. +Rollback removes structured output and pilot artifacts while preserving both existing test suites and their legacy entry points. diff --git a/openspec/changes/add-mutation-falsification-service/proposal.md b/openspec/changes/add-mutation-falsification-service/proposal.md index ff6592c81..7c39d2663 100644 --- a/openspec/changes/add-mutation-falsification-service/proposal.md +++ b/openspec/changes/add-mutation-falsification-service/proposal.md @@ -1,21 +1,21 @@ ## Why -PDPP has strong bespoke falsifiability tests, but no common way to define, run, and audit deliberate faults across its AI-authored implementation and tests. A mutation score would hide the important question: whether a relevant test detects a plausible failure at an acceptable evidence and compute cost. +PDPP has valuable bespoke falsifiability tests, but their evidence is not consistently machine-readable or comparable. Before adding shared mutation infrastructure, PDPP needs to prove that structured evidence from one existing oracle and one trusted real-domain pilot produces useful signal at acceptable cost. ## What Changes -- Add a risk-falsification service that accepts machine-readable mutation packets and emits replayable, revision-bound receipts. -- Require baseline-first, isolated execution, exact cleanup, explicit resource budgets, and honest outcome classification. -- Route mutation checks through the existing test-accounting authority and record both focused tests and a relevant backstop. -- Add an adapter for an existing PDPP falsifiability oracle as the first end-to-end implementation. -- Keep mutation runs advisory during calibration. Do not introduce a repository-wide mutation score, coverage quota, or automatic test-deletion rule. -- Leave room for later domain-mutator, StrykerJS, and agent-generated-mutant adapters without making any one generator the architecture. +- Add structured evidence to the existing test-migration oracle without changing its mutation lifecycle, judges, positive control, or rollback proof. +- Define separate intent, execution-attempt, and reviewer-triage artifacts with honest integrity and provenance claims. +- Run one trusted, repository-owned GroupMe cursor/frontier pilot with checked-in fault operators, focused adapter evidence, and mandatory complete test-accounting backstops. +- Pre-register resource budgets, selector-miss rules, evidence metrics, and continue, narrow, or stop criteria. +- Keep all mutation work local, sequential, advisory, and limited to reviewed adapters. +- Defer a shared coordinator, generic source-mutating executor, arbitrary or agent-generated patches, StrykerJS, CI scheduling, blocking gates, and automatic test deletion until the pilot evidence justifies a separate proposal. ## Capabilities ### New Capabilities -- `mutation-falsification`: Defines mutation packets, execution safety, result classification, evidence receipts, test-authority integration, and calibrated rollout. +- `mutation-falsification`: Defines trusted local mutation evidence, independent triage, one bounded domain calibration pilot, and the decision gate for any later shared infrastructure. ### Modified Capabilities @@ -23,7 +23,7 @@ None. ## Impact -- Adds repository tooling, schemas, tests, and documentation for mutation packets and receipts. -- Integrates with `test-accounting.manifest.json` and its runners without changing suite ownership. -- Initially wraps one existing mutation or falsifiability oracle; production behavior and public APIs do not change. -- Mutation work remains bounded and non-blocking until observed evidence supports a narrower gate. +- Adds structured output and evidence validation around an existing repository oracle. +- Adds a bounded GroupMe pilot that uses existing tests and the unchanged test-accounting authority. +- Does not change product behavior, public APIs, test-accounting authority, or CI requirements. +- Produces a decision memo before any reusable mutation execution framework is proposed. diff --git a/openspec/changes/add-mutation-falsification-service/specs/mutation-falsification/spec.md b/openspec/changes/add-mutation-falsification-service/specs/mutation-falsification/spec.md index 18b65ae2f..f7eb78b2f 100644 --- a/openspec/changes/add-mutation-falsification-service/specs/mutation-falsification/spec.md +++ b/openspec/changes/add-mutation-falsification-service/specs/mutation-falsification/spec.md @@ -1,101 +1,101 @@ ## Purpose -Provide bounded, replayable evidence that PDPP tests detect plausible faults without turning mutation counts or scores into quality gates. +Provide bounded, reviewable evidence that selected PDPP tests detect plausible trusted faults before the repository invests in shared mutation infrastructure. ## ADDED Requirements -### Requirement: Mutation work SHALL be declared as a validated packet +### Requirement: Version one SHALL execute only trusted repository adapters -Each run SHALL accept a machine-readable packet that identifies the repository revision, risk being tested, mutation target and change, generator provenance, selected test authority, relevant backstop, resource budget, and forbidden execution profiles. The service SHALL reject incomplete, malformed, or revision-mismatched packets before changing source files. +Version one SHALL be a local developer harness with no daemon, queue, server, or network API. It SHALL execute only reviewed, repository-owned adapter and operator identifiers with validated parameters and exact target preimages. It SHALL NOT execute caller-supplied commands, arbitrary patches, or agent-generated source. -#### Scenario: Valid packet is accepted -- **WHEN** a packet identifies a clean matching revision, a concrete mutation, accounted tests, a backstop, and finite budgets -- **THEN** the service SHALL accept it for execution +#### Scenario: Trusted adapter is requested +- **WHEN** an intent names a registered adapter and valid checked-in operator +- **THEN** repository policy SHALL derive the effective command, environment, judge, and limits -#### Scenario: Packet does not match the checkout -- **WHEN** the packet's revision or target content does not match the checkout -- **THEN** the service SHALL reject it without applying the mutation +#### Scenario: Arbitrary executable input is requested +- **WHEN** an intent supplies executable source, command arguments, working directory, environment policy, or an unregistered operator +- **THEN** the harness SHALL reject it before execution -### Requirement: Mutation execution SHALL establish a passing baseline +### Requirement: Intent, attempt, and triage evidence SHALL remain separate -The service SHALL run the packet's selected tests against the unmodified revision before applying the mutation. It SHALL NOT classify a mutant as killed or survived unless the baseline passes under the same declared execution conditions. +An intent packet SHALL record the requested risk, trusted adapter and operator descriptors, base identity, and requested budget. An attempt receipt SHALL record immutable machine observations from one execution. A triage receipt SHALL separately record a later reviewer judgment bound to one attempt. The harness SHALL derive conventional killed, survived, or inconclusive projections from valid attempt evidence and SHALL NOT accept those projections or triage dispositions from the intent. -#### Scenario: Baseline fails -- **WHEN** any selected baseline test fails, times out, or cannot run -- **THEN** the service SHALL stop that trial and classify it as an execution error rather than a mutation result +#### Scenario: Execution completes +- **WHEN** a trusted adapter produces valid execution evidence +- **THEN** the attempt receipt SHALL preserve baseline, materialization, focused, backstop, reachability, cleanup, timeout, and error observations without adding a human judgment -### Requirement: Mutation execution SHALL be isolated and reversible +#### Scenario: Reviewer classifies evidence +- **WHEN** an independent reviewer judges an attempt actionable, likely equivalent, uninteresting, deferred, or invalid +- **THEN** that judgment SHALL be appended as a separate triage receipt with reviewer claim, evidence reference, reason, and timestamp -The service SHALL apply each mutation in isolation, SHALL prevent mutation trials from overlapping beyond the declared resource budget, and SHALL restore the checkout to its exact pre-trial state before another trial or successful exit. A cleanup failure SHALL be visible and SHALL prevent subsequent trials in that checkout. +### Requirement: Evidence identity and integrity claims SHALL be precise -#### Scenario: Mutant is exercised -- **WHEN** the service applies a mutation and runs its selected tests -- **THEN** no other packet's source mutation SHALL be present in that trial +Canonicalization and hash algorithms SHALL be versioned. The deterministic trial key SHALL bind the intent digest, repository tree, adapter version, policy version, and mutation identity. Each execution SHALL receive a distinct random attempt identifier. Content digests SHALL establish internal consistency or tamper evidence relative to an independently retained digest; they SHALL NOT be represented as issuer authentication. -#### Scenario: Trial is interrupted -- **WHEN** a trial exits by failure, timeout, or handled termination -- **THEN** the service SHALL attempt restoration and SHALL verify the pre-trial tree identity before permitting more work +#### Scenario: Same trial is repeated +- **WHEN** the same bound trial executes more than once +- **THEN** the attempts SHALL share a trial key and SHALL have different attempt identifiers and runtime observations -### Requirement: Outcomes SHALL preserve uncertainty +#### Scenario: Authenticated provenance is unavailable +- **WHEN** evidence is produced only on a developer-controlled host without a trusted signer or platform attestation +- **THEN** provenance SHALL be recorded as a claim and SHALL NOT be described as authenticated identity -The service SHALL distinguish at least killed, survived, not exercised, timeout, execution error, equivalent-suspect, and uninteresting outcomes. Equivalent-suspect and uninteresting SHALL require recorded triage and SHALL NOT be inferred only from a passing test run. +### Requirement: The existing migration oracle SHALL retain its purpose-fit lifecycle -#### Scenario: Selected test detects the mutation -- **WHEN** the clean baseline passes and an otherwise valid selected test fails because the mutation is present -- **THEN** the service SHALL classify the mutant as killed and record the detecting test evidence +The test-migration oracle SHALL retain its existing named mutations, mutation-specific judges, positive control, fixture repositories, and rollback proof. Its structured mode SHALL report the same cases and decisions as its human-readable mode. This adapter SHALL validate structured evidence and crash honesty only; it SHALL NOT be presented as validation of a generic source-mutating executor. -#### Scenario: Selected tests stay green -- **WHEN** the clean baseline passes and all selected tests pass with the mutation present -- **THEN** the service SHALL classify the mutant as survived unless independent evidence supports a more specific classification +#### Scenario: Structured mode runs successfully +- **WHEN** the existing oracle is run in structured mode +- **THEN** its named cases, catching checks, holes, positive control, and rollback decision SHALL agree with the legacy result -#### Scenario: Mutation target is not exercised -- **WHEN** available execution evidence shows that no selected test reaches the mutated behavior -- **THEN** the service SHALL classify the mutant as not exercised rather than survived +#### Scenario: Structured evidence is incomplete +- **WHEN** output is partial, malformed, omits a case or control, exceeds its bound, or the process is interrupted +- **THEN** the attempt SHALL remain incomplete or invalid and SHALL NOT be projected as killed or survived -### Requirement: Every trial SHALL emit a verifiable receipt +### Requirement: Test-accounting authority SHALL NOT be overstated -The service SHALL emit a machine-readable receipt bound to the packet, repository revision, pre- and post-restoration tree identity, exact effective commands, environment profile, mutant identity, observed outcomes, timestamps, duration, exit status, and captured diagnostic output. Receipt validation SHALL reject missing or inconsistent evidence. +Focused checks in the domain pilot SHALL be labeled adapter evidence, not test-accounting authority receipts. Repository-owned policy SHALL select the focused check and mandatory complete owning-suite backstop. Clean and mutant complete backstops SHALL run through the unchanged test-accounting authority on clean committed trees, and mutation evidence SHALL reference its verified receipt digests. -#### Scenario: Receipt is replayed -- **WHEN** a reviewer validates a receipt against its packet and repository revision -- **THEN** the reviewer SHALL be able to identify the exact mutation, tests, budgets, result, and cleanup proof without trusting prose +#### Scenario: Focused mutant check passes +- **WHEN** the focused check passes with a trusted mutant present +- **THEN** the mandatory mutant backstop SHALL run; a focused pass followed by a backstop failure SHALL be recorded as a selector miss -#### Scenario: Receipt is altered -- **WHEN** a bound packet field, command, result, mutant identity, or tree identity is changed after issue -- **THEN** receipt validation SHALL fail +#### Scenario: Required backstop cannot complete +- **WHEN** a clean or mutant backstop is missing, fails its baseline, exceeds budget, or cannot issue a valid authority receipt +- **THEN** the attempt SHALL be inconclusive and SHALL NOT count as survived -### Requirement: Test selection SHALL respect executable accounting +### Requirement: Domain mutation attempts SHALL be one-shot and bounded -Selected tests and backstops SHALL resolve through the repository's executable test-accounting authority or an explicitly declared, validated mutation-oracle command. The service SHALL record why each selected test is relevant and SHALL NOT treat static imports alone as complete dependency evidence. +The GroupMe cursor/frontier pilot SHALL use only checked-in declarative fault operators over permitted production paths while the judge and runner closure remain immutable. Each source-mutating attempt SHALL use a fresh clean committed descendant in a disk-backed workspace. A successful workspace SHALL be deleted; an interrupted or cleanup-failed workspace SHALL be quarantined and detected on the next start. -#### Scenario: Accounted focused tests and backstop are declared -- **WHEN** a packet selects a focused test set for fast feedback -- **THEN** it SHALL also identify the relevant accounted package, suite, or control-lane backstop used to measure selection misses +#### Scenario: Trusted domain operator runs +- **WHEN** a registered GroupMe operator matches its exact target preimage +- **THEN** the attempt SHALL record base and mutant trees, permitted changed paths, focused and backstop evidence, environment profile, artifacts, resource observations, and cleanup state -#### Scenario: Unaccounted side lane is requested -- **WHEN** a packet names an executable test that is neither accounted nor an approved mutation oracle -- **THEN** the service SHALL reject the packet +#### Scenario: Attempt is abandoned +- **WHEN** execution ends before a complete receipt and verified cleanup +- **THEN** the issued attempt marker SHALL remain incomplete, its workspace SHALL NOT be reused, and a later run SHALL report or quarantine it -### Requirement: Mutation runs SHALL obey explicit safety and resource bounds +### Requirement: Local execution SHALL use enforceable bounds and honest limitations -Each packet SHALL declare finite wall-time, trial-count, process-concurrency, and cleanup bounds. Local execution SHALL cooperate with the repository's local test-resource controls. Live third-party services, personal data, production credentials, and stateful browser profiles SHALL be forbidden unless a later capability defines an isolated mutation profile for them. +Repository policy SHALL run one adapter at a time and SHALL set finite limits for attempts, wall time, captured output, workspace bytes, and cleanup time. It SHALL use an environment allowlist and SHALL forbid live credentials, personal data, live third-party services, stateful browsers, and shared production-like databases. CPU and memory limits SHALL be identified as hard only when the host mechanism enforces them; otherwise they SHALL be recorded as advisory observations. Unsupported required limits SHALL cause refusal rather than a portability claim. -#### Scenario: Budget is exhausted -- **WHEN** a packet reaches a declared time, trial, or process limit -- **THEN** the service SHALL stop new work, restore the checkout, and report a bounded outcome +#### Scenario: Hard limit is exceeded +- **WHEN** wall time, output, workspace, or another enforced limit is exceeded +- **THEN** execution SHALL stop or be marked interrupted, bounded evidence SHALL be retained, and the attempt SHALL be inconclusive -#### Scenario: Packet requests a forbidden profile -- **WHEN** a packet requires live credentials, personal data, or an undeclared stateful external profile -- **THEN** the service SHALL reject it before mutation execution +#### Scenario: Host cannot enforce required safety +- **WHEN** the selected adapter requires a hard control that the host does not provide +- **THEN** the harness SHALL refuse the attempt instead of silently weakening the policy -### Requirement: Mutation evidence SHALL remain advisory until calibrated +### Requirement: Calibration SHALL end with a pre-registered decision -Initial mutation execution SHALL report evidence without enforcing a repository-wide score or adequacy threshold. A later blocking rule SHALL identify a narrow risk class, a stable operator set, an accountable backstop, observed false-positive and miss evidence, and an explicit rollback path. +The pilot SHALL remain advisory and SHALL publish raw counts and defined denominators for valid trials, focused-to-backstop misses, invalid faults, execution failures, cleanup failures, runtime, artifact sizes, and reviewer time. It SHALL stop or narrow on any containment or cleanup failure, unexplained selector miss, dominant setup cost, predominantly invalid or trivial faults, or lack of useful evidence within its declared budget. Shared infrastructure SHALL require a later proposal supported by repeated invariants across both adapters and measured reduction in audit cost. -#### Scenario: A survived mutant is reported during calibration -- **WHEN** an advisory run produces a survived mutant -- **THEN** the service SHALL nominate it for triage and SHALL NOT automatically fail unrelated changes or require a new test +#### Scenario: Pilot stays within bounds and produces useful evidence +- **WHEN** the two adapters produce interpretable evidence without cleanup failures or unexplained selector misses at acceptable compute and review cost +- **THEN** the decision memo MAY recommend a narrowly scoped shared evidence module and SHALL identify the repeated invariants it would hide -#### Scenario: A test is proposed for deletion -- **WHEN** mutation evidence shows that another test kills the same sampled mutants -- **THEN** that evidence alone SHALL NOT authorize deletion because the test may protect a different behavior or fault class +#### Scenario: Pilot does not justify generalization +- **WHEN** signal is weak, costs dominate, safety fails, or the adapters do not share a deep stable boundary +- **THEN** the decision memo SHALL stop or narrow the initiative and retain purpose-fit adapters rather than manufacturing a framework diff --git a/openspec/changes/add-mutation-falsification-service/tasks.md b/openspec/changes/add-mutation-falsification-service/tasks.md index d4b696286..d9872ad70 100644 --- a/openspec/changes/add-mutation-falsification-service/tasks.md +++ b/openspec/changes/add-mutation-falsification-service/tasks.md @@ -1,43 +1,25 @@ -## 1. Packet and Receipt Contracts - -- [ ] 1.1 Define versioned mutation-packet and mutation-receipt schemas with revision, risk, mutation, provenance, test selection, backstop, budget, outcome, command, diagnostic, and tree-identity fields. -- [ ] 1.2 Implement strict packet and receipt parsers that reject unknown schema versions, missing bounds, revision mismatches, forbidden profiles, and inconsistent outcome evidence. -- [ ] 1.3 Add canonical serialization and content binding for packets, receipts, effective commands, mutant content, and pre/post tree identities. -- [ ] 1.4 Add adversarial tests that alter each bound receipt class and prove validation fails. - -## 2. Safe Mutation Executor - -- [ ] 2.1 Implement a baseline-first executor in a disposable clean workspace with one mutation per trial. -- [ ] 2.2 Integrate finite wall-time, trial-count, process-concurrency, and cleanup bounds with the repository's local test-resource controls. -- [ ] 2.3 Implement killed, survived, not-exercised, timeout, execution-error, equivalent-suspect, and uninteresting result handling without collapsing uncertainty. -- [ ] 2.4 Emit receipts atomically with exact effective argv, captured diagnostics, durations, exit status, and cleanup evidence. -- [ ] 2.5 Add interruption and failure-path tests that prove bounded termination, child cleanup, exact restoration, and refusal to continue after cleanup failure. - -## 3. Test Authority and Selection - -- [ ] 3.1 Resolve declared focused tests and backstops against `test-accounting.manifest.json` or an explicit approved mutation-oracle entry. -- [ ] 3.2 Reject unaccounted executable side lanes and static-import-only completeness claims. -- [ ] 3.3 Record focused and backstop results separately so selection misses can be measured. -- [ ] 3.4 Add tests for dynamic/literal inputs, profile mismatch, missing backstop, and focused-pass/backstop-kill cases. - -## 4. Existing-Oracle Adapter - -- [ ] 4.1 Express each named case in `scripts/test-migration/mutation-oracle.ts` as a mutation packet without weakening its current checks. -- [ ] 4.2 Run the existing oracle through the common executor and preserve its byte-identical rollback proof in receipts. -- [ ] 4.3 Add differential tests proving dropped tests, silent skips, assertion loss, import breakage, and stale literal paths remain detectable. -- [ ] 4.4 Keep the legacy entry point available until receipt comparison proves the adapter has equal or stronger evidence. - -## 5. Domain Pilot - -- [ ] 5.1 Select one small, high-risk, hermetic domain surface using measured runtime, independent-oracle strength, and expected operator value. -- [ ] 5.2 Implement two or three domain-specific mutation operators with explicit risk provenance and compile-valid output checks. -- [ ] 5.3 Run the focused selection and relevant accounted backstop in advisory mode; capture outcome distribution, misses, runtime, and reviewer effort. -- [ ] 5.4 Add regression tests for every actionable survivor repaired during the pilot and prove each test fails before the repair or kills its nominated mutant. - -## 6. Feasibility and Rollout - -- [ ] 6.1 Run a time-boxed StrykerJS command-runner experiment on one precompiled pure-TypeScript island, including setup cost, test routing quality, mutant quality, and compute cost. -- [ ] 6.2 Record an explicit continue, adapt, or stop decision for the StrykerJS adapter based on comparison with domain operators. -- [ ] 6.3 Document packet authoring, local advisory execution, receipt triage, equivalent-suspect review, and forbidden live-data profiles. -- [ ] 6.4 Add a non-blocking control lane only after local reliability is proven, with explicit concurrency and time budgets. -- [ ] 6.5 Publish a pilot report covering productive-mutant rate, actionable survivors, focused-to-backstop misses, flaky baselines, cleanup failures, reviewer time, and compute time; propose no blocking gate without calibrated evidence. +## 1. Existing-Oracle Structured Evidence + +- [ ] 1.1 Define versioned adapter-specific intent, attempt, and triage schemas for `test-migration-oracle/v1`, including canonicalization golden vectors and explicit integrity-versus-authenticity language. +- [ ] 1.2 Add structured JSON output to the existing test-migration oracle without changing its named mutations, judges, positive control, fixture lifecycle, human output, or rollback proof. +- [ ] 1.3 Add differential tests requiring legacy and structured modes to report identical cases, catching checks, holes, positive-control result, and rollback result. +- [ ] 1.4 Add bounded adapter-specific execution evidence for wall time and output, with issued/incomplete/completed attempt states and exact source-checkout unchanged proof. +- [ ] 1.5 Add corruption, unknown-version, partial-output, crash, timeout, output-cap, missing-case, missing-control, and cleanup-evidence tests. +- [ ] 1.6 Run the legacy and structured oracle twice on clean revisions, record runtime and artifact costs, and obtain independent review before starting the domain pilot. + +## 2. Trusted GroupMe Domain Pilot + +- [ ] 2.1 Register two or three reviewed declarative operators for GroupMe page-ceiling and cursor-progress risks, with exact implementation preimages, permitted postimages, and immutable judge closure. +- [ ] 2.2 Implement one-shot disk-backed mutant workspaces with free-space preflight, pinned offline frozen dependency materialization, issued markers, successful deletion, interruption quarantine, and next-start recovery. +- [ ] 2.3 Record focused clean and mutant GroupMe checks as adapter evidence; do not label them test-accounting authority receipts. +- [ ] 2.4 Run clean and mutant complete `polyfill-connectors` backstops through the unchanged test-accounting authority, including the mutant backstop for every focused survivor, and bind verified receipt digests. +- [ ] 2.5 Add fault-injection tests for preimage mismatch, forbidden path changes, immutable-judge changes, backstop absence, selector miss, timeout, output/workspace limit, abandoned attempt, cleanup failure, and altered evidence. +- [ ] 2.6 Run at most 20 trials within 10 wall-clock minutes and sequential local resource policy; capture raw execution axes, projections, selector misses, runtime, setup time, artifact sizes, cleanup, and reviewer minutes. +- [ ] 2.7 Obtain independent triage for every survivor, likely-equivalent, uninteresting, or invalid-fault disposition. + +## 3. Decision Gate + +- [ ] 3.1 Publish metric definitions and raw pilot results, including invalid trials and excluded denominators rather than only percentages. +- [ ] 3.2 Stop or narrow on any cleanup/containment failure, unexplained selector miss, authority mismatch, dominant setup cost, predominantly invalid or trivial operators, or insufficient useful evidence within budget. +- [ ] 3.3 Compare the migration and GroupMe adapters and identify any substantial repeated policy/evidence invariants; do not generalize superficial lifecycle differences. +- [ ] 3.4 Publish a continue, narrow, or stop memo. Require a new independently reviewed OpenSpec proposal before any shared coordinator, generic executor, StrykerJS experiment, CI scheduling, blocking gate, agent-generated mutant, or test-deletion automation. From f1352e25eeb2f55c36de4562841874b5fd0aa46f Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Tue, 11 Aug 2026 15:43:13 -0500 Subject: [PATCH 3/5] docs: specify mutation process containment Assisted-by: AI Signed-off-by: Tim Nunamaker --- .../add-mutation-falsification-service/design.md | 9 +++++++-- .../specs/mutation-falsification/spec.md | 12 ++++++++---- .../add-mutation-falsification-service/tasks.md | 10 ++++++---- 3 files changed, 21 insertions(+), 10 deletions(-) diff --git a/openspec/changes/add-mutation-falsification-service/design.md b/openspec/changes/add-mutation-falsification-service/design.md index 4b59a599c..60c646a57 100644 --- a/openspec/changes/add-mutation-falsification-service/design.md +++ b/openspec/changes/add-mutation-falsification-service/design.md @@ -63,7 +63,7 @@ The adapter-specific runner accepts only `test-migration-oracle/v1`; it derives ### 6. GroupMe is the real-domain calibration pilot -The pilot uses the existing hermetic GroupMe cursor/frontier tests as focused adapter evidence. It creates a fresh one-commit mutant descendant for each trusted operator. The clean and mutant complete `polyfill-connectors` suite run through the unchanged test-accounting authority; their verified receipt digests are referenced rather than duplicated. +The pilot uses the existing hermetic GroupMe cursor/frontier tests as focused adapter evidence. It creates a fresh one-commit mutant descendant for each trusted operator. The clean complete `polyfill-connectors` suite always runs through the unchanged test-accounting authority. The mutant complete suite runs for every focused survivor; after a focused kill, policy may omit it only by recording `not_run_focused_kill`. Verified accounting receipt digests are referenced rather than duplicated. The focused clean baseline and complete clean backstop must pass before interpreting mutant evidence. Every focused survivor receives the complete mutant backstop. A focused pass plus backstop failure is a selector miss and blocks selector promotion. If a required backstop cannot complete, the attempt is inconclusive. @@ -77,7 +77,11 @@ Domain attempts use a configured disk-backed root with free-space preflight. Eac Successful workspaces are deleted before the completed attempt receipt is published. Interrupted or cleanup-failed workspaces are quarantined and never reused. Next-start recovery reports owned incomplete markers and quarantines before new work. This handles hard termination honestly; it does not pretend `finally` runs after power loss or `SIGKILL`. -Initial host policy runs one adapter at a time, at most 20 domain mutants and 10 wall-clock minutes, with hard limits for adapter wall time, captured output, workspace size, and cleanup time. CPU use targets no more than 50% of available processors and memory is observed, but neither is described as hard containment without a host mechanism. Live credentials, personal data, third-party network, stateful browsers, and shared production-like databases are prohibited. Required unsupported controls cause refusal. +Version one supports only Linux hosts with cgroup v2 and a usable systemd user manager. Each adapter, focused check, and authority backstop runs as a uniquely named transient systemd user service with `KillMode=control-group`, a finite `RuntimeMaxSec`, `TasksMax=64`, and policy-selected cgroup CPU and memory limits. The service-unit name and cgroup identity are recorded before launch and bound into effective policy and the attempt receipt. macOS, Windows, non-cgroup-v2 Linux, and Linux without the required user-service controls fail closed in version one. + +An atomic repository-scoped issued marker permits only one active mutation attempt. The unique unit name is recorded before the systemd start transaction. If the wrapper dies during or after launch, next-start recovery submits a stop for that prepublished unit name; systemd serializes the start and stop jobs, and recovery waits for the stop job to finish before verifying that the cgroup is empty and the unit inactive. Only then may it retire the marker or allow another attempt. It never relies on a recorded child PID, so PID reuse cannot establish cleanup. A missing, mismatched, or unverifiable unit identity quarantines the attempt and blocks further mutation work until reviewed. + +Initial policy permits one adapter or backstop service at a time, at most 20 domain mutants and 10 wall-clock minutes. It hard-bounds direct captured output and adapter workspace growth and sets a 60-second cleanup deadline. The cgroup caps the complete descendant tree at 50% of available CPU, a policy-selected memory maximum, and 64 tasks, including full-suite children and grandchildren. Live credentials, personal data, third-party network, stateful browsers, and shared production-like databases are prohibited. Required unsupported controls cause refusal. ### 8. The decision gate precedes shared infrastructure @@ -94,6 +98,7 @@ Continue only if evidence is interpretable, costs are acceptable, there are no u - **Full connector backstops may dominate runtime** → measure setup and execution separately; stop rather than weaken the mandatory calibration backstop. - **A trusted mutant can still affect ambient host state accidentally** → constrain paths/operators/environment, use hermetic tests, run sequentially, and never call a worktree a sandbox. - **Offline dependency materialization may fail** → refuse the attempt and review the dependency strategy; do not share mutable outputs silently. +- **Version-one containment is Linux-specific** → fail closed elsewhere and require separate reviewed host adapters before claiming macOS or Windows support. - **Small samples cannot prove selector completeness** → report raw counts and treat any observed miss as disqualifying, without claiming zero misses proves completeness. ## Migration Plan diff --git a/openspec/changes/add-mutation-falsification-service/specs/mutation-falsification/spec.md b/openspec/changes/add-mutation-falsification-service/specs/mutation-falsification/spec.md index f7eb78b2f..e54f4321d 100644 --- a/openspec/changes/add-mutation-falsification-service/specs/mutation-falsification/spec.md +++ b/openspec/changes/add-mutation-falsification-service/specs/mutation-falsification/spec.md @@ -54,7 +54,7 @@ The test-migration oracle SHALL retain its existing named mutations, mutation-sp ### Requirement: Test-accounting authority SHALL NOT be overstated -Focused checks in the domain pilot SHALL be labeled adapter evidence, not test-accounting authority receipts. Repository-owned policy SHALL select the focused check and mandatory complete owning-suite backstop. Clean and mutant complete backstops SHALL run through the unchanged test-accounting authority on clean committed trees, and mutation evidence SHALL reference its verified receipt digests. +Focused checks in the domain pilot SHALL be labeled adapter evidence, not test-accounting authority receipts. Repository-owned policy SHALL select the focused check and complete owning-suite backstop. The complete clean backstop SHALL always run. The complete mutant backstop SHALL run for every focused survivor. After a focused kill, policy SHALL either run it or record an explicit `not_run_focused_kill` observation. Every backstop that runs SHALL use the unchanged test-accounting authority on a clean committed tree, and mutation evidence SHALL reference its verified receipt digest. #### Scenario: Focused mutant check passes - **WHEN** the focused check passes with a trusted mutant present @@ -78,14 +78,18 @@ The GroupMe cursor/frontier pilot SHALL use only checked-in declarative fault op ### Requirement: Local execution SHALL use enforceable bounds and honest limitations -Repository policy SHALL run one adapter at a time and SHALL set finite limits for attempts, wall time, captured output, workspace bytes, and cleanup time. It SHALL use an environment allowlist and SHALL forbid live credentials, personal data, live third-party services, stateful browsers, and shared production-like databases. CPU and memory limits SHALL be identified as hard only when the host mechanism enforces them; otherwise they SHALL be recorded as advisory observations. Unsupported required limits SHALL cause refusal rather than a portability claim. +Repository policy SHALL run one adapter at a time and SHALL set finite limits for attempts, wall time, captured output, workspace bytes, descendant processes, and cleanup time. Every adapter, focused check, and complete backstop SHALL run inside an owned host containment unit that applies limits to the entire descendant process tree and can terminate and verify that tree independently of the wrapper process. The containment mechanism and version SHALL be bound into effective policy and the attempt receipt. Version one SHALL support only Linux hosts with the specified cgroup-v2 systemd containment; all other hosts SHALL refuse execution. It SHALL use an environment allowlist and SHALL forbid live credentials, personal data, live third-party services, stateful browsers, and shared production-like databases. #### Scenario: Hard limit is exceeded - **WHEN** wall time, output, workspace, or another enforced limit is exceeded -- **THEN** execution SHALL stop or be marked interrupted, bounded evidence SHALL be retained, and the attempt SHALL be inconclusive +- **THEN** the owned containment unit SHALL terminate the descendant tree, verify that no descendant remains, retain bounded evidence, and mark the attempt inconclusive + +#### Scenario: Wrapper dies after child start +- **WHEN** the wrapper exits after the contained process tree starts but before completion is recorded +- **THEN** the issued marker and containment-unit identity SHALL let next-start recovery terminate and verify the entire tree before another attempt can begin #### Scenario: Host cannot enforce required safety -- **WHEN** the selected adapter requires a hard control that the host does not provide +- **WHEN** the host is not a supported Linux cgroup-v2 systemd host or cannot apply the required descendant, task-count, wall-time, output, and cleanup controls - **THEN** the harness SHALL refuse the attempt instead of silently weakening the policy ### Requirement: Calibration SHALL end with a pre-registered decision diff --git a/openspec/changes/add-mutation-falsification-service/tasks.md b/openspec/changes/add-mutation-falsification-service/tasks.md index d9872ad70..aca3345e1 100644 --- a/openspec/changes/add-mutation-falsification-service/tasks.md +++ b/openspec/changes/add-mutation-falsification-service/tasks.md @@ -3,16 +3,18 @@ - [ ] 1.1 Define versioned adapter-specific intent, attempt, and triage schemas for `test-migration-oracle/v1`, including canonicalization golden vectors and explicit integrity-versus-authenticity language. - [ ] 1.2 Add structured JSON output to the existing test-migration oracle without changing its named mutations, judges, positive control, fixture lifecycle, human output, or rollback proof. - [ ] 1.3 Add differential tests requiring legacy and structured modes to report identical cases, catching checks, holes, positive-control result, and rollback result. -- [ ] 1.4 Add bounded adapter-specific execution evidence for wall time and output, with issued/incomplete/completed attempt states and exact source-checkout unchanged proof. -- [ ] 1.5 Add corruption, unknown-version, partial-output, crash, timeout, output-cap, missing-case, missing-control, and cleanup-evidence tests. -- [ ] 1.6 Run the legacy and structured oracle twice on clean revisions, record runtime and artifact costs, and obtain independent review before starting the domain pilot. +- [ ] 1.4 Add Linux cgroup-v2/systemd capability probing and fail-closed refusal for unsupported hosts; bind the containment mechanism, version, unit, cgroup, task, CPU, memory, wall-time, output, workspace, and cleanup policy into attempt evidence. +- [ ] 1.5 Add repository-scoped issued markers and next-start recovery that terminate and verify the recorded control group before another attempt can run, without relying on PID-only liveness. +- [ ] 1.6 Add bounded adapter-specific execution evidence with issued, incomplete, and completed attempt states and exact source-checkout unchanged proof. +- [ ] 1.7 Add corruption, unknown-version, partial-output, child-and-grandchild timeout, wrapper-death-after-spawn, PID-reuse, output-flood, task-limit, missing-case, missing-control, no-surviving-descendant, and cleanup-evidence tests. +- [ ] 1.8 Run the legacy and structured oracle twice on clean revisions, record runtime and artifact costs, and obtain independent review before starting the domain pilot. ## 2. Trusted GroupMe Domain Pilot - [ ] 2.1 Register two or three reviewed declarative operators for GroupMe page-ceiling and cursor-progress risks, with exact implementation preimages, permitted postimages, and immutable judge closure. - [ ] 2.2 Implement one-shot disk-backed mutant workspaces with free-space preflight, pinned offline frozen dependency materialization, issued markers, successful deletion, interruption quarantine, and next-start recovery. - [ ] 2.3 Record focused clean and mutant GroupMe checks as adapter evidence; do not label them test-accounting authority receipts. -- [ ] 2.4 Run clean and mutant complete `polyfill-connectors` backstops through the unchanged test-accounting authority, including the mutant backstop for every focused survivor, and bind verified receipt digests. +- [ ] 2.4 Always run the clean complete `polyfill-connectors` backstop through the unchanged test-accounting authority; run the mutant backstop for every focused survivor, record `not_run_focused_kill` when policy omits it after a focused kill, and bind every verified receipt digest. - [ ] 2.5 Add fault-injection tests for preimage mismatch, forbidden path changes, immutable-judge changes, backstop absence, selector miss, timeout, output/workspace limit, abandoned attempt, cleanup failure, and altered evidence. - [ ] 2.6 Run at most 20 trials within 10 wall-clock minutes and sequential local resource policy; capture raw execution axes, projections, selector misses, runtime, setup time, artifact sizes, cleanup, and reviewer minutes. - [ ] 2.7 Obtain independent triage for every survivor, likely-equivalent, uninteresting, or invalid-fault disposition. From fe3d965d8f547b4c41d4b292d914326df8dbdd0a Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Tue, 11 Aug 2026 15:47:35 -0500 Subject: [PATCH 4/5] docs: close mutation containment races Assisted-by: AI Signed-off-by: Tim Nunamaker --- .../add-mutation-falsification-service/design.md | 8 +++++--- .../specs/mutation-falsification/spec.md | 14 +++++++++----- .../add-mutation-falsification-service/tasks.md | 12 +++++++----- 3 files changed, 21 insertions(+), 13 deletions(-) diff --git a/openspec/changes/add-mutation-falsification-service/design.md b/openspec/changes/add-mutation-falsification-service/design.md index 60c646a57..dbc275a8b 100644 --- a/openspec/changes/add-mutation-falsification-service/design.md +++ b/openspec/changes/add-mutation-falsification-service/design.md @@ -77,11 +77,13 @@ Domain attempts use a configured disk-backed root with free-space preflight. Eac Successful workspaces are deleted before the completed attempt receipt is published. Interrupted or cleanup-failed workspaces are quarantined and never reused. Next-start recovery reports owned incomplete markers and quarantines before new work. This handles hard termination honestly; it does not pretend `finally` runs after power loss or `SIGKILL`. -Version one supports only Linux hosts with cgroup v2 and a usable systemd user manager. Each adapter, focused check, and authority backstop runs as a uniquely named transient systemd user service with `KillMode=control-group`, a finite `RuntimeMaxSec`, `TasksMax=64`, and policy-selected cgroup CPU and memory limits. The service-unit name and cgroup identity are recorded before launch and bound into effective policy and the attempt receipt. macOS, Windows, non-cgroup-v2 Linux, and Linux without the required user-service controls fail closed in version one. +Version one supports only Linux hosts with cgroup v2 and a usable systemd user manager. Each adapter, focused check, and authority backstop runs as a uniquely named transient systemd user service with `KillMode=control-group`, a finite `RuntimeMaxSec`, and policy-selected cgroup task, CPU, and memory limits. The initial `TasksMax` is 512, not an assumed Node file count; the clean complete backstop records peak tasks so later policy can be calibrated without invalidating the immutable command. macOS, Windows, non-cgroup-v2 Linux, and Linux without the required user-service controls fail closed in version one. -An atomic repository-scoped issued marker permits only one active mutation attempt. The unique unit name is recorded before the systemd start transaction. If the wrapper dies during or after launch, next-start recovery submits a stop for that prepublished unit name; systemd serializes the start and stop jobs, and recovery waits for the stop job to finish before verifying that the cgroup is empty and the unit inactive. Only then may it retire the marker or allow another attempt. It never relies on a recorded child PID, so PID reuse cannot establish cleanup. A missing, mismatched, or unverifiable unit identity quarantines the attempt and blocks further mutation work until reviewed. +The launcher only requests a uniquely named transient service; it never starts adapter work itself. The top-level attempt wrapper begins inside the manager-owned service with no adapter descendants. It writes and fsyncs a complete claim record containing the unit name to a unique temporary file, then atomically hard-links that record to the repository-scoped active-marker path. If the link loses to an existing marker, the service exits before spawning adapter work. If it wins, the wrapper queries systemd for its actual unit, `ControlGroup`, and `InvocationID`, atomically publishes the bound record, verifies that its own process is in that control group, and only then starts the adapter. -Initial policy permits one adapter or backstop service at a time, at most 20 domain mutants and 10 wall-clock minutes. It hard-bounds direct captured output and adapter workspace growth and sets a 60-second cleanup deadline. The cgroup caps the complete descendant tree at 50% of available CPU, a policy-selected memory maximum, and 64 tasks, including full-suite children and grandchildren. Live credentials, personal data, third-party network, stateful browsers, and shared production-like databases are prohibited. Required unsupported controls cause refusal. +This ordering removes the late-start race: a delayed start request can create only a service whose wrapper must win the same marker before doing work. Launcher death is harmless because systemd owns any accepted service. Recovery stops the marker-bound unit and invocation, waits for the stop job, verifies the cgroup absent or unpopulated and the unit inactive, and only then retires the marker. It never relies on a recorded child PID, so PID reuse cannot establish cleanup. An incomplete, mismatched, or unverifiable claim quarantines the attempt and blocks further mutation work until reviewed. + +Initial policy permits one adapter or backstop service at a time, at most 20 domain mutants and 10 wall-clock minutes. It hard-bounds captured output and sets a 60-second cleanup deadline. The cgroup caps the complete descendant tree at 50% of available CPU, a policy-selected memory maximum, and 512 tasks, including full-suite children, threads, and grandchildren. Workspace size uses a disk-free preflight, reserved host headroom, and periodic observation. Crossing the threshold stops and quarantines the attempt, but it is explicitly not a filesystem quota and cannot promise zero overshoot or host-disk impact. Live credentials, personal data, third-party network, stateful browsers, and shared production-like databases are prohibited. Required unsupported controls cause refusal. ### 8. The decision gate precedes shared infrastructure diff --git a/openspec/changes/add-mutation-falsification-service/specs/mutation-falsification/spec.md b/openspec/changes/add-mutation-falsification-service/specs/mutation-falsification/spec.md index e54f4321d..42c7d44f7 100644 --- a/openspec/changes/add-mutation-falsification-service/specs/mutation-falsification/spec.md +++ b/openspec/changes/add-mutation-falsification-service/specs/mutation-falsification/spec.md @@ -78,18 +78,22 @@ The GroupMe cursor/frontier pilot SHALL use only checked-in declarative fault op ### Requirement: Local execution SHALL use enforceable bounds and honest limitations -Repository policy SHALL run one adapter at a time and SHALL set finite limits for attempts, wall time, captured output, workspace bytes, descendant processes, and cleanup time. Every adapter, focused check, and complete backstop SHALL run inside an owned host containment unit that applies limits to the entire descendant process tree and can terminate and verify that tree independently of the wrapper process. The containment mechanism and version SHALL be bound into effective policy and the attempt receipt. Version one SHALL support only Linux hosts with the specified cgroup-v2 systemd containment; all other hosts SHALL refuse execution. It SHALL use an environment allowlist and SHALL forbid live credentials, personal data, live third-party services, stateful browsers, and shared production-like databases. +Repository policy SHALL run one adapter at a time and SHALL set finite hard limits for attempts, wall time, captured output, descendant tasks, CPU, memory, and cleanup time. Workspace bytes SHALL have a preflight reserve and an observed stop threshold, but SHALL NOT be described as a hard quota. Every adapter, focused check, and complete backstop SHALL run inside an owned host containment unit that applies hard limits to the entire descendant process tree and can terminate and verify that tree independently of the launcher process. The containment mechanism and version SHALL be bound into effective policy and the attempt receipt. Version one SHALL support only Linux hosts with the specified cgroup-v2 systemd containment; all other hosts SHALL refuse execution. It SHALL use an environment allowlist and SHALL forbid live credentials, personal data, live third-party services, stateful browsers, and shared production-like databases. #### Scenario: Hard limit is exceeded -- **WHEN** wall time, output, workspace, or another enforced limit is exceeded +- **WHEN** wall time, output, task count, CPU, memory, or another enforced limit is exceeded - **THEN** the owned containment unit SHALL terminate the descendant tree, verify that no descendant remains, retain bounded evidence, and mark the attempt inconclusive +#### Scenario: Workspace threshold is observed +- **WHEN** workspace observation detects that the advisory byte threshold has been crossed +- **THEN** the harness SHALL stop the contained tree, quarantine the workspace, record the observed overshoot, and SHALL NOT claim that host-disk impact was hard-contained + #### Scenario: Wrapper dies after child start -- **WHEN** the wrapper exits after the contained process tree starts but before completion is recorded -- **THEN** the issued marker and containment-unit identity SHALL let next-start recovery terminate and verify the entire tree before another attempt can begin +- **WHEN** the manager-owned attempt wrapper exits after claiming the repository marker and starting adapter descendants but before completion is recorded +- **THEN** the bound unit, invocation, and cgroup identity SHALL let next-start recovery terminate and verify the entire tree before another attempt can begin #### Scenario: Host cannot enforce required safety -- **WHEN** the host is not a supported Linux cgroup-v2 systemd host or cannot apply the required descendant, task-count, wall-time, output, and cleanup controls +- **WHEN** the host is not a supported Linux cgroup-v2 systemd host or cannot apply the required descendant, task-count, CPU, memory, wall-time, output, and cleanup controls - **THEN** the harness SHALL refuse the attempt instead of silently weakening the policy ### Requirement: Calibration SHALL end with a pre-registered decision diff --git a/openspec/changes/add-mutation-falsification-service/tasks.md b/openspec/changes/add-mutation-falsification-service/tasks.md index aca3345e1..93af9dc83 100644 --- a/openspec/changes/add-mutation-falsification-service/tasks.md +++ b/openspec/changes/add-mutation-falsification-service/tasks.md @@ -3,11 +3,13 @@ - [ ] 1.1 Define versioned adapter-specific intent, attempt, and triage schemas for `test-migration-oracle/v1`, including canonicalization golden vectors and explicit integrity-versus-authenticity language. - [ ] 1.2 Add structured JSON output to the existing test-migration oracle without changing its named mutations, judges, positive control, fixture lifecycle, human output, or rollback proof. - [ ] 1.3 Add differential tests requiring legacy and structured modes to report identical cases, catching checks, holes, positive-control result, and rollback result. -- [ ] 1.4 Add Linux cgroup-v2/systemd capability probing and fail-closed refusal for unsupported hosts; bind the containment mechanism, version, unit, cgroup, task, CPU, memory, wall-time, output, workspace, and cleanup policy into attempt evidence. -- [ ] 1.5 Add repository-scoped issued markers and next-start recovery that terminate and verify the recorded control group before another attempt can run, without relying on PID-only liveness. +- [ ] 1.4 Add Linux cgroup-v2/systemd capability probing and fail-closed refusal for unsupported hosts; bind the containment mechanism, version, unit, invocation, cgroup, task, CPU, memory, wall-time, output, workspace-threshold, and cleanup policy into attempt evidence. +- [ ] 1.5 Start the top-level wrapper inside the transient service; before any adapter work, atomically claim the repository marker from a complete record, bind and verify its actual unit/invocation/cgroup identity, and exit on claim loss. - [ ] 1.6 Add bounded adapter-specific execution evidence with issued, incomplete, and completed attempt states and exact source-checkout unchanged proof. -- [ ] 1.7 Add corruption, unknown-version, partial-output, child-and-grandchild timeout, wrapper-death-after-spawn, PID-reuse, output-flood, task-limit, missing-case, missing-control, no-surviving-descendant, and cleanup-evidence tests. -- [ ] 1.8 Run the legacy and structured oracle twice on clean revisions, record runtime and artifact costs, and obtain independent review before starting the domain pilot. +- [ ] 1.7 Add next-start recovery that stops the marker-bound unit and invocation, waits for the manager job, verifies the cgroup empty and unit inactive, and blocks on unverifiable claims without relying on PID-only liveness. +- [ ] 1.8 Add corruption, unknown-version, partial-output, delayed-start-submission, launcher-death, claim-race, child-and-grandchild timeout, wrapper-death-after-adapter-spawn, PID-reuse, output-flood, task-limit, missing-case, missing-control, no-surviving-descendant, and cleanup-evidence tests. +- [ ] 1.9 Add workspace fill-to-threshold and overshoot tests that prove stop-and-quarantine behavior and explicitly do not claim a hard disk quota. +- [ ] 1.10 Run the legacy and structured oracle twice on clean revisions, record runtime and artifact costs, and obtain independent review before starting the domain pilot. ## 2. Trusted GroupMe Domain Pilot @@ -15,7 +17,7 @@ - [ ] 2.2 Implement one-shot disk-backed mutant workspaces with free-space preflight, pinned offline frozen dependency materialization, issued markers, successful deletion, interruption quarantine, and next-start recovery. - [ ] 2.3 Record focused clean and mutant GroupMe checks as adapter evidence; do not label them test-accounting authority receipts. - [ ] 2.4 Always run the clean complete `polyfill-connectors` backstop through the unchanged test-accounting authority; run the mutant backstop for every focused survivor, record `not_run_focused_kill` when policy omits it after a focused kill, and bind every verified receipt digest. -- [ ] 2.5 Add fault-injection tests for preimage mismatch, forbidden path changes, immutable-judge changes, backstop absence, selector miss, timeout, output/workspace limit, abandoned attempt, cleanup failure, and altered evidence. +- [ ] 2.5 Add fault-injection tests for preimage mismatch, forbidden path changes, immutable-judge changes, backstop absence, selector miss, timeout, hard output limit, workspace soft-threshold overshoot, abandoned attempt, cleanup failure, and altered evidence. - [ ] 2.6 Run at most 20 trials within 10 wall-clock minutes and sequential local resource policy; capture raw execution axes, projections, selector misses, runtime, setup time, artifact sizes, cleanup, and reviewer minutes. - [ ] 2.7 Obtain independent triage for every survivor, likely-equivalent, uninteresting, or invalid-fault disposition. From 65a4ff0f7d3462e79b8c74d2b66526128a564853 Mon Sep 17 00:00:00 2001 From: Tim Nunamaker Date: Tue, 11 Aug 2026 16:10:09 -0500 Subject: [PATCH 5/5] docs: narrow mutation falsification evidence Assisted-by: AI Signed-off-by: Tim Nunamaker --- .../.openspec.yaml | 0 .../design.md | 139 ++++++++++++++++++ .../proposal.md | 5 +- .../specs/mutation-falsification/spec.md | 42 ++++-- .../tasks.md | 25 ++++ .../design.md | 113 -------------- .../tasks.md | 29 ---- 7 files changed, 195 insertions(+), 158 deletions(-) rename openspec/changes/{add-mutation-falsification-service => add-mutation-falsification-evidence}/.openspec.yaml (100%) create mode 100644 openspec/changes/add-mutation-falsification-evidence/design.md rename openspec/changes/{add-mutation-falsification-service => add-mutation-falsification-evidence}/proposal.md (82%) rename openspec/changes/{add-mutation-falsification-service => add-mutation-falsification-evidence}/specs/mutation-falsification/spec.md (54%) create mode 100644 openspec/changes/add-mutation-falsification-evidence/tasks.md delete mode 100644 openspec/changes/add-mutation-falsification-service/design.md delete mode 100644 openspec/changes/add-mutation-falsification-service/tasks.md diff --git a/openspec/changes/add-mutation-falsification-service/.openspec.yaml b/openspec/changes/add-mutation-falsification-evidence/.openspec.yaml similarity index 100% rename from openspec/changes/add-mutation-falsification-service/.openspec.yaml rename to openspec/changes/add-mutation-falsification-evidence/.openspec.yaml diff --git a/openspec/changes/add-mutation-falsification-evidence/design.md b/openspec/changes/add-mutation-falsification-evidence/design.md new file mode 100644 index 000000000..cfb87b3fd --- /dev/null +++ b/openspec/changes/add-mutation-falsification-evidence/design.md @@ -0,0 +1,139 @@ +## Context + +See `proposal.md` for motivation. PDPP's current test-migration oracle already owns a strong, self-contained lifecycle: it creates fixture repositories, applies named faults, invokes mutation-specific judges, runs a positive control, proves rollback, and disposes its fixtures. The GroupMe connector also has real mutation-killing tests for historical pagination and cursor-progress faults. + +The current test-accounting authority issues receipts only for manifest-owned complete plans on clean trees. It does not authorize arbitrary focused subsets or dirty mutants. A Git worktree isolates tracked source state but does not constrain filesystem, environment, network, credentials, processes, caches, databases, or Docker access. Local hashes bind content but do not authenticate an issuer. + +## Goals / Non-Goals + +**Goals:** + +- Make one existing oracle's evidence structured without weakening its lifecycle. +- Run one trusted real-domain pilot against the current accounting authority's actual contract. +- Separate requested intent, machine observations, derived projections, and independent triage. +- Measure whether any shared infrastructure would reduce real repeated reasoning and audit cost. + +**Non-Goals:** + +- A generic source-mutating executor or untrusted-code sandbox. +- Arbitrary, packet-authored, or agent-generated patches and commands. +- A new test-accounting subset authority. +- StrykerJS, CI scheduling, blocking gates, mutation scores, or test-deletion authority. +- A reusable execution coordinator, sandbox, or automatic crash-recovery system. +- Product-host portability claims based on this developer-only experiment. + +## Decisions + +### 1. Version one is a trusted local evidence program, not a service + +There is no daemon, queue, server, network API, or remotely supplied executable input. A repository-owned registry names the only permitted adapters and operators. Intent may request a registered risk and a stricter budget; versioned repository policy derives the effective command, working directory, environment allowlist, immutable judge closure, focused evidence, complete backstop, and host limits. + +This prevents a generator from choosing its own judge or safety policy. Arbitrary and agent-generated executable mutations require a separate sandbox design that proves filesystem, network, environment, process-tree, CPU, memory, disk, and output containment. + +### 2. Adapters own mutation mechanics until common structure is earned + +The migration oracle remains self-contained. It gains structured output but keeps its current named cases, fixture repositories, mutation-specific judges, positive control, and rollback proof. It demonstrates evidence shape and crash honesty, not a generic executor. + +The second adapter is a GroupMe cursor/frontier pilot. It uses two or three checked-in declarative operators over `packages/polyfill-connectors/connectors/groupme/index.ts`, such as reintroducing the historical page ceiling or weakening non-progress detection. The operators have exact preimages and permitted postimages; they cannot alter tests, runners, policy, or manifests. + +Only after both adapters run will a decision memo identify whether they share a deep stable boundary. Until then, duplication is preferable to a shallow common executor with adapter-specific escape hatches. + +### 3. Evidence uses three immutable artifact types + +An **intent packet** records requested risk, base identity, adapter/operator descriptor, and requested bounds. Its canonical digest is its identifier; callers do not supply the identifier. + +An **attempt receipt** records one execution's raw observations: issued random attempt ID, deterministic trial key, resolved policy, exact effective plan, environment profile, base/mutant/judge identities, bounded artifact digests and sizes, baseline/materialization/focused/backstop/reachability/cleanup axes, duration, exit or signal, and any referenced accounting receipt digests. + +A **triage receipt** is append-only and binds one attempt digest. It records an independent reviewer's claimed identity, disposition, evidence, reason, and timestamp. Version one does not authenticate that identity. A different reviewer from the operator/test author is required before likely-equivalent or uninteresting evidence is excluded from reported actionable results. + +`killed`, `survived`, and `inconclusive` are computed projections, never caller fields. The projection is total and conservative: + +| Observations | Projection | Additional signal | +| --- | --- | --- | +| Any clean baseline, materialization, protocol, authority, artifact-retention, or cleanup failure | `inconclusive` | Preserve the failing axis. | +| Timeout, signal, resource stop, malformed or partial output, or unexplained nondeterminism | `inconclusive` | Preserve the exact failure; never infer a kill. | +| Validated `not_exercised` reachability | `inconclusive` | Send to independent triage; do not count it as survived. | +| Focused mutant check fails for a mutation-attributable test assertion | `killed` | The complete mutant backstop may be `not_run_focused_kill`. | +| Focused mutant check passes and the complete mutant backstop fails for a mutation-attributable test assertion | `killed` | Record a selector miss and stop selector promotion. | +| Focused mutant check and complete mutant backstop both pass | `survived` | Require independent triage before exclusion from actionable results. | + +A failure is mutation-attributable only when the digest-identical clean command passed and the mutant command produced a recognized owning-test assertion failure, not an infrastructure, protocol, accounting, cleanup, or resource error. Version one performs no automatic retries. A reviewer may request a new attempt; contradictory valid attempts with the same trial key make the aggregate trial inconclusive. `not_exercised` requires adapter-supplied validated reachability evidence; otherwise reachability is `unknown`. + +### 4. Digests provide integrity binding, not authenticity + +RFC 8785 JSON canonicalization, schema version, canonicalization version, and SHA-256 are explicit. A trial key binds the intent digest, repository tree, adapter version, policy version, and mutation identity. Each run gets a random attempt ID, so replay has stable identity but different observations. + +Before spawn, the adapter-specific runner writes an issued attempt marker in a verifier-owned, disk-backed evidence directory outside the disposable workspace. Complete receipts publish atomically only after structured output validation, retained-artifact validation, and cleanup evidence. The writer fsyncs each completed file and its parent directory after publication. Interrupted markers remain incomplete and are discovered at next start. + +The experiment does not claim automatic process recovery or power-loss-proof containment. Before every run it scans its evidence and configured workspace roots. Any incomplete or corrupt marker, unexpected workspace, or prior cleanup failure blocks execution for explicit operator review. Age, PID liveness, and a successful `finally` block never authorize automatic reclamation. Retirement requires a separate append-only recovery receipt that records the operator claim, process and workspace observations, disposition, and retained evidence; it does not convert the interrupted attempt into a completed one. + +Anyone controlling the host can rewrite records and recompute an unkeyed digest. Version one therefore claims internal consistency and tamper evidence relative to a separately retained digest, not issuer authenticity. Authenticated provenance would require a later CI signature or platform attestation. + +### 5. The migration oracle is the evidence pilot + +The first slice adds a structured JSON mode to `scripts/test-migration/mutation-oracle.ts`. Legacy and structured modes must agree on every named case, catching check, hole, positive control, and rollback result. The source checkout must remain unchanged. + +The adapter-specific runner accepts only `test-migration-oracle/v1`; it derives the command and allowlisted environment. It bounds its direct structured-output capture, applies a finite wall deadline, records partial/crash states honestly, and never interprets missing output as success. These are adapter-local protections, not a claim that the runner survives its own crash or contains hostile descendants. It does not claim focused selection, test-accounting authority, or domain value. + +### 6. GroupMe is the real-domain calibration pilot + +The pilot uses the existing hermetic GroupMe cursor/frontier tests as focused adapter evidence. It creates a fresh one-commit mutant descendant for each trusted operator. The clean complete `polyfill-connectors` suite runs through the unchanged test-accounting authority before any mutant is interpreted. The mutant complete suite runs for every focused survivor; after a focused kill, policy may omit it only by recording `not_run_focused_kill`. + +Before a disposable workspace is removed, the verifier copies the complete validated accounting bundle—authority record, transcript, completion record, receipt, and required closure or manifest identity—into its bounded evidence directory. It validates the copied bundle there and records each relative location, size, and digest in the attempt receipt. A digest without retained, revalidatable bytes is invalid evidence. Copy, validation, retention-budget, or publication failure makes the attempt inconclusive and quarantines its workspace. + +Policy declares the evidence root, maximum retained bytes, maximum attempts, and retention deadline before a batch starts. It reserves that capacity without deleting prior evidence. A completed batch remains intact through the decision memo's independent review and for at least 30 days afterward; later deletion is an explicit audited operation. The harness stops before accepting a new attempt that could exceed the declared retained-byte or attempt-count budget. + +The focused clean baseline and complete clean backstop must pass before interpreting mutant evidence. Every focused survivor receives the complete mutant backstop. A focused pass plus backstop failure is a selector miss and blocks selector promotion. If a required backstop cannot complete, the attempt is inconclusive. + +A clean focused baseline and clean complete backstop may be reused only within one locked pilot batch, for at most two hours, while the repository tree, judge and adapter closure, effective commands, allowlisted environment, Node and dependency identities, and budgets remain digest-identical. The retained accounting bundle must still revalidate before every reference. The batch ends on any tree, policy, environment, dependency, or budget change, any missing retained byte, or the first unexplained failure. The next batch runs fresh clean evidence. Reuse changes neither the killed/survived denominator nor the raw count of clean executions; attempt receipts bind the exact baseline digest they used. + +The source and judge closures are separate. Operators can change only predeclared production ranges in GroupMe's implementation. Tests, policy, runner, manifest, lockfile, and receipt validator remain immutable and are digest-bound. + +### 7. Workspaces are one-shot source isolation, not sandboxes + +Domain attempts use a configured disk-backed root with free-space preflight. Each attempt gets an independent local clone made without hard links, a fresh dependency tree, and a clean committed mutant descendant. The clone does not share a Git common directory with the source checkout. All writable attempt state—including `HOME`, `TMPDIR`, XDG directories, pnpm store and virtual store, test temporary files, and test-accounting run files—resolves beneath the attempt root. The evidence directory is the only retained writable path and is outside that root. + +Policy records the repository-pinned Node `v25.8.2`, the actual Node executable digest, repository-pinned `pnpm@10.33.0`, the actual pnpm executable and version, lockfile digest, and materialization command. Dependencies are materialized offline and frozen from the lockfile, with lifecycle scripts disabled for this pure TypeScript pilot. The offline package source is read-only and digest-inventoried; pnpm uses copy import semantics into the attempt-local store and dependency tree rather than links to mutable host content. A preflight runs the exact clean focused and complete commands with this layout before the batch begins. If lifecycle suppression or local materialization cannot satisfy the pilot, execution stops and the design is reviewed rather than silently sharing mutable output. + +Successful workspaces are deleted only after required evidence has been copied and revalidated. The external verifier confirms deletion, then publishes the completed attempt receipt. Interrupted or cleanup-failed workspaces are quarantined and never reused. A later run reports them and refuses automatic recovery. The operator must independently establish that no related process remains before explicitly retiring an incomplete marker. This is source and writable-state isolation for trusted repository code; it is not a sandbox and does not claim containment after verifier death. + +The resource contract is deliberately narrow: + +| Resource | Mechanism | Enforcement and observation | Receipt projection | +| --- | --- | --- | --- | +| Trial count | Repository policy counter | Refuse a new trial after the declared count | No attempt is created. | +| Wall time | Adapter-local deadline and owning-process-group termination | Stop the group when the deadline fires; verify no selected-command process remains; record deadline, signal, and cleanup observation | `inconclusive` | +| Direct structured output | Streaming byte counter before buffering | Stop the adapter when its declared byte cap is crossed; retain a bounded prefix and byte count | `inconclusive` | +| Test-accounting transcript | Unchanged authority plus trusted hermetic command | Observe and record size; no hard byte-cap claim until the authority gains one in a separate change | Over-budget is `inconclusive` and stops the batch. | +| Workspace bytes | Free-space preflight plus periodic observation | Stop and quarantine after the soft threshold; overshoot remains possible | `inconclusive` | +| CPU, memory, and task count | Host observations only | Record peaks when available; no throttle, quota, or kill guarantee | Over-budget is `inconclusive` and stops the batch. | +| Cleanup | Finite verifier wait plus filesystem/process observations | Failure quarantines and blocks the next run | `inconclusive` | + +Initial policy permits one trusted command at a time, two or three domain operators, and 10 wall-clock minutes for the declared pilot batch. Environment construction starts empty and admits only policy-listed non-secret values plus the isolated writable paths. A credential-sentinel test proves ambient values are absent. Live credentials, personal data, third-party network, stateful browsers, Docker sockets, and shared production-like databases are forbidden by policy; because version one is not a sandbox, the adapter preflight must also prove that the selected commands need none of them. This experiment is verified only on its declared Linux developer-host profile. No result changes or narrows PDPP's heterogeneous product-host support contract. + +### 8. The decision gate precedes shared infrastructure + +The pilot predefines valid-trial denominators and reports raw counts for execution axes, projections, selector misses, triage dispositions, runtime, output/workspace size, cleanup, and reviewer minutes. The valid-result denominator is exactly `killed + survived`; inconclusive, invalid, and not-run attempts remain visible as separate raw counts and never enter that denominator. Setup time is reported separately. + +Stop or narrow immediately on a cleanup failure, abandoned process, unexplained selector miss, authority/receipt mismatch, or evidence corruption. Stop generalization if setup consumes most runtime, review exceeds five minutes per disputed attempt, most operators are invalid/trivial, no useful risk evidence appears within the two-or-three-operator/10-minute batch, or the adapters do not expose repeated policy/evidence logic. + +Continue only if evidence is interpretable, costs are acceptable, there are no unexplained selector or cleanup failures, and a proposed shared module would hide substantial repeated invariants across both adapters. Any coordinator, generic executor, Stryker experiment, CI lane, or sandbox is a new reviewed OpenSpec change. + +## Risks / Trade-offs + +- **The narrow slices may not justify a framework** → treat stopping with two useful purpose-fit adapters as success. +- **Local evidence lacks authenticated issuer identity** → state the trust boundary and preserve external attestation as a separate future capability. +- **Full connector backstops may dominate runtime** → measure setup and execution separately; stop rather than weaken the mandatory calibration backstop. +- **A trusted mutant can still affect ambient host state accidentally** → isolate writable paths, constrain operators and environment, use hermetic tests, run sequentially, and stop on any unexpected effect; never call the clone a sandbox. +- **Offline dependency materialization may fail** → refuse the attempt and review the dependency strategy; do not share mutable outputs silently. +- **Version-one evidence is calibrated on one Linux developer profile** → record that profile and make no inference about PDPP product-host support. +- **Small samples cannot prove selector completeness** → report raw counts and treat any observed miss as disqualifying, without claiming zero misses proves completeness. + +## Migration Plan + +1. Land this revised design only after a second independent architecture review returns LAND. +2. Add structured evidence to the existing migration oracle and validate it without generic execution infrastructure. +3. Run the trusted GroupMe cursor/frontier pilot under mandatory clean and mutant backstops. +4. Publish a continue, narrow, or stop decision memo. Do not implement shared infrastructure from this change. + +Rollback removes structured output and pilot artifacts while preserving both existing test suites and their legacy entry points. diff --git a/openspec/changes/add-mutation-falsification-service/proposal.md b/openspec/changes/add-mutation-falsification-evidence/proposal.md similarity index 82% rename from openspec/changes/add-mutation-falsification-service/proposal.md rename to openspec/changes/add-mutation-falsification-evidence/proposal.md index 7c39d2663..d08234f59 100644 --- a/openspec/changes/add-mutation-falsification-service/proposal.md +++ b/openspec/changes/add-mutation-falsification-evidence/proposal.md @@ -8,7 +8,7 @@ PDPP has valuable bespoke falsifiability tests, but their evidence is not consis - Define separate intent, execution-attempt, and reviewer-triage artifacts with honest integrity and provenance claims. - Run one trusted, repository-owned GroupMe cursor/frontier pilot with checked-in fault operators, focused adapter evidence, and mandatory complete test-accounting backstops. - Pre-register resource budgets, selector-miss rules, evidence metrics, and continue, narrow, or stop criteria. -- Keep all mutation work local, sequential, advisory, and limited to reviewed adapters. +- Keep all mutation work local, sequential, operator-supervised, advisory, and limited to reviewed adapters. - Defer a shared coordinator, generic source-mutating executor, arbitrary or agent-generated patches, StrykerJS, CI scheduling, blocking gates, and automatic test deletion until the pilot evidence justifies a separate proposal. ## Capabilities @@ -24,6 +24,7 @@ None. ## Impact - Adds structured output and evidence validation around an existing repository oracle. -- Adds a bounded GroupMe pilot that uses existing tests and the unchanged test-accounting authority. +- Adds a bounded GroupMe pilot that uses existing tests and preserves complete test-accounting evidence outside disposable workspaces. - Does not change product behavior, public APIs, test-accounting authority, or CI requirements. +- Makes no sandbox, crash-containment, or heterogeneous-host claim for the experimental developer harness. - Produces a decision memo before any reusable mutation execution framework is proposed. diff --git a/openspec/changes/add-mutation-falsification-service/specs/mutation-falsification/spec.md b/openspec/changes/add-mutation-falsification-evidence/specs/mutation-falsification/spec.md similarity index 54% rename from openspec/changes/add-mutation-falsification-service/specs/mutation-falsification/spec.md rename to openspec/changes/add-mutation-falsification-evidence/specs/mutation-falsification/spec.md index 42c7d44f7..a4aca42f4 100644 --- a/openspec/changes/add-mutation-falsification-service/specs/mutation-falsification/spec.md +++ b/openspec/changes/add-mutation-falsification-evidence/specs/mutation-falsification/spec.md @@ -20,6 +20,8 @@ Version one SHALL be a local developer harness with no daemon, queue, server, or An intent packet SHALL record the requested risk, trusted adapter and operator descriptors, base identity, and requested budget. An attempt receipt SHALL record immutable machine observations from one execution. A triage receipt SHALL separately record a later reviewer judgment bound to one attempt. The harness SHALL derive conventional killed, survived, or inconclusive projections from valid attempt evidence and SHALL NOT accept those projections or triage dispositions from the intent. +The projection SHALL be conservative and total. Invalid clean evidence, materialization, protocol, authority, retained-artifact, cleanup, timeout, signal, resource, or nondeterminism observations SHALL project to `inconclusive`. A recognized owning-test assertion that passes on the digest-identical clean closure and fails on the mutant SHALL project to `killed`. A focused pass followed by such a complete-backstop failure SHALL also record a selector miss. Only a focused pass plus a complete mutant-backstop pass SHALL project to `survived`. Validated `not_exercised` reachability SHALL project to `inconclusive` pending triage. + #### Scenario: Execution completes - **WHEN** a trusted adapter produces valid execution evidence - **THEN** the attempt receipt SHALL preserve baseline, materialization, focused, backstop, reachability, cleanup, timeout, and error observations without adding a human judgment @@ -28,9 +30,13 @@ An intent packet SHALL record the requested risk, trusted adapter and operator d - **WHEN** an independent reviewer judges an attempt actionable, likely equivalent, uninteresting, deferred, or invalid - **THEN** that judgment SHALL be appended as a separate triage receipt with reviewer claim, evidence reference, reason, and timestamp +#### Scenario: Observations are contradictory +- **WHEN** valid attempts for one trial key disagree or a failure is not attributable to an owning-test assertion +- **THEN** the aggregate trial SHALL be inconclusive and the raw attempts SHALL remain visible + ### Requirement: Evidence identity and integrity claims SHALL be precise -Canonicalization and hash algorithms SHALL be versioned. The deterministic trial key SHALL bind the intent digest, repository tree, adapter version, policy version, and mutation identity. Each execution SHALL receive a distinct random attempt identifier. Content digests SHALL establish internal consistency or tamper evidence relative to an independently retained digest; they SHALL NOT be represented as issuer authentication. +Evidence SHALL use versioned RFC 8785 JSON canonicalization and SHA-256. The deterministic trial key SHALL bind the intent digest, repository tree, adapter version, policy version, and mutation identity. Each execution SHALL receive a distinct random attempt identifier. Content digests SHALL establish internal consistency or tamper evidence relative to an independently retained digest; they SHALL NOT be represented as issuer authentication. #### Scenario: Same trial is repeated - **WHEN** the same bound trial executes more than once @@ -54,7 +60,11 @@ The test-migration oracle SHALL retain its existing named mutations, mutation-sp ### Requirement: Test-accounting authority SHALL NOT be overstated -Focused checks in the domain pilot SHALL be labeled adapter evidence, not test-accounting authority receipts. Repository-owned policy SHALL select the focused check and complete owning-suite backstop. The complete clean backstop SHALL always run. The complete mutant backstop SHALL run for every focused survivor. After a focused kill, policy SHALL either run it or record an explicit `not_run_focused_kill` observation. Every backstop that runs SHALL use the unchanged test-accounting authority on a clean committed tree, and mutation evidence SHALL reference its verified receipt digest. +Focused checks in the domain pilot SHALL be labeled adapter evidence, not test-accounting authority receipts. Repository-owned policy SHALL select the focused check and complete owning-suite backstop. The complete clean backstop SHALL run at the start of each locked pilot batch. The complete mutant backstop SHALL run for every focused survivor. After a focused kill, policy SHALL either run it or record an explicit `not_run_focused_kill` observation. Every backstop that runs SHALL use the unchanged test-accounting authority on a clean committed tree. Before workspace deletion, the verifier SHALL copy and revalidate the complete authority, transcript, completion, receipt, and required closure evidence in its retained bounded store; an attempt SHALL bind the retained locations, sizes, and digests. + +Policy SHALL predeclare the evidence root, retained-byte and attempt-count budgets, and retention deadline without evicting prior evidence to admit a run. Completed batch evidence SHALL remain intact through independent review of the decision memo and for at least 30 days afterward. Later deletion SHALL be an explicit audited action. + +Clean evidence MAY be reused only inside the same locked batch, for at most two hours, while all bound repository, command, judge, environment, runtime, dependency, and budget identities remain equal and the retained bundle still validates. Reuse SHALL NOT change mutant denominators or hide the raw clean-execution count. #### Scenario: Focused mutant check passes - **WHEN** the focused check passes with a trusted mutant present @@ -76,29 +86,33 @@ The GroupMe cursor/frontier pilot SHALL use only checked-in declarative fault op - **WHEN** execution ends before a complete receipt and verified cleanup - **THEN** the issued attempt marker SHALL remain incomplete, its workspace SHALL NOT be reused, and a later run SHALL report or quarantine it -### Requirement: Local execution SHALL use enforceable bounds and honest limitations +### Requirement: Local execution SHALL use bounded trusted inputs and honest limitations + +Repository policy SHALL run one trusted command at a time and SHALL set finite trial-count, wall-time, direct structured-output, workspace-observation, and cleanup budgets. It SHALL distinguish enforced bounds from observations: CPU, memory, task-count, test-accounting transcript bytes, and workspace bytes SHALL NOT be described as hard quotas when the selected mechanism only observes them. The runner SHALL start from an empty environment allowlist and SHALL isolate writable home, temporary, cache, dependency-store, Git, and accounting paths beneath a one-shot attempt root. It SHALL forbid live credentials, personal data, live third-party services, stateful browsers, Docker sockets, and shared production-like databases by policy. + +Version one SHALL be an operator-supervised Linux developer experiment. It SHALL NOT claim to be a sandbox, to recover automatically after verifier death, or to establish PDPP product-host portability. Any incomplete marker, unexpected workspace, unexplained process, or cleanup failure SHALL block later runs pending explicit review. -Repository policy SHALL run one adapter at a time and SHALL set finite hard limits for attempts, wall time, captured output, descendant tasks, CPU, memory, and cleanup time. Workspace bytes SHALL have a preflight reserve and an observed stop threshold, but SHALL NOT be described as a hard quota. Every adapter, focused check, and complete backstop SHALL run inside an owned host containment unit that applies hard limits to the entire descendant process tree and can terminate and verify that tree independently of the launcher process. The containment mechanism and version SHALL be bound into effective policy and the attempt receipt. Version one SHALL support only Linux hosts with the specified cgroup-v2 systemd containment; all other hosts SHALL refuse execution. It SHALL use an environment allowlist and SHALL forbid live credentials, personal data, live third-party services, stateful browsers, and shared production-like databases. +Retiring an interrupted marker SHALL require a separate append-only recovery receipt with operator claim, process and workspace observations, disposition, and retained evidence. It SHALL NOT convert the interrupted attempt into a completed attempt. -#### Scenario: Hard limit is exceeded -- **WHEN** wall time, output, task count, CPU, memory, or another enforced limit is exceeded -- **THEN** the owned containment unit SHALL terminate the descendant tree, verify that no descendant remains, retain bounded evidence, and mark the attempt inconclusive +#### Scenario: Enforced adapter-local bound is exceeded +- **WHEN** the adapter wall deadline or direct structured-output byte cap is exceeded +- **THEN** the runner SHALL stop its owning process group, verify that no selected-command process remains, retain bounded evidence, record cleanup observations, and mark the attempt inconclusive without claiming crash-durable containment #### Scenario: Workspace threshold is observed - **WHEN** workspace observation detects that the advisory byte threshold has been crossed -- **THEN** the harness SHALL stop the contained tree, quarantine the workspace, record the observed overshoot, and SHALL NOT claim that host-disk impact was hard-contained +- **THEN** the harness SHALL stop its owning process group, quarantine the workspace, record the observed overshoot, and SHALL NOT claim that host-disk impact was hard-contained -#### Scenario: Wrapper dies after child start -- **WHEN** the manager-owned attempt wrapper exits after claiming the repository marker and starting adapter descendants but before completion is recorded -- **THEN** the bound unit, invocation, and cgroup identity SHALL let next-start recovery terminate and verify the entire tree before another attempt can begin +#### Scenario: Verifier dies after child start +- **WHEN** the verifier exits after publishing an issued marker but before a complete receipt and cleanup evidence +- **THEN** a later run SHALL refuse automatic recovery and SHALL require explicit operator verification of the marker, workspace, and related processes -#### Scenario: Host cannot enforce required safety -- **WHEN** the host is not a supported Linux cgroup-v2 systemd host or cannot apply the required descendant, task-count, CPU, memory, wall-time, output, and cleanup controls +#### Scenario: Preflight cannot establish the declared profile +- **WHEN** isolated writable paths, empty environment construction, pinned runtime/dependency materialization, hermetic commands, or required adapter-local bounds cannot be established - **THEN** the harness SHALL refuse the attempt instead of silently weakening the policy ### Requirement: Calibration SHALL end with a pre-registered decision -The pilot SHALL remain advisory and SHALL publish raw counts and defined denominators for valid trials, focused-to-backstop misses, invalid faults, execution failures, cleanup failures, runtime, artifact sizes, and reviewer time. It SHALL stop or narrow on any containment or cleanup failure, unexplained selector miss, dominant setup cost, predominantly invalid or trivial faults, or lack of useful evidence within its declared budget. Shared infrastructure SHALL require a later proposal supported by repeated invariants across both adapters and measured reduction in audit cost. +The pilot SHALL remain advisory and SHALL publish raw counts and defined denominators for valid trials, focused-to-backstop misses, invalid faults, execution failures, cleanup failures, runtime, artifact sizes, and reviewer time. Its valid-result denominator SHALL be exactly killed plus survived; inconclusive, invalid, and not-run attempts SHALL remain separate raw counts. It SHALL stop or narrow on any cleanup failure, abandoned process, unexplained selector miss, dominant setup cost, predominantly invalid or trivial faults, or lack of useful evidence within its declared budget. Shared infrastructure SHALL require a later proposal supported by repeated invariants across both adapters and measured reduction in audit cost. #### Scenario: Pilot stays within bounds and produces useful evidence - **WHEN** the two adapters produce interpretable evidence without cleanup failures or unexplained selector misses at acceptable compute and review cost diff --git a/openspec/changes/add-mutation-falsification-evidence/tasks.md b/openspec/changes/add-mutation-falsification-evidence/tasks.md new file mode 100644 index 000000000..f7d4e5713 --- /dev/null +++ b/openspec/changes/add-mutation-falsification-evidence/tasks.md @@ -0,0 +1,25 @@ +## 1. Existing-Oracle Structured Evidence + +- [ ] 1.1 Define versioned adapter-specific intent, attempt, and triage schemas for `test-migration-oracle/v1`, naming RFC 8785 JSON canonicalization plus SHA-256, including golden vectors and explicit integrity-versus-authenticity language. +- [ ] 1.2 Add structured JSON output to the existing test-migration oracle without changing its named mutations, judges, positive control, fixture lifecycle, human output, or rollback proof. +- [ ] 1.3 Add differential tests requiring legacy and structured modes to report identical cases, catching checks, holes, positive-control result, and rollback result. +- [ ] 1.4 Add bounded adapter-local execution evidence with issued, incomplete, and completed states, an external evidence root, finite wall and direct-output limits, and exact source-checkout unchanged proof. +- [ ] 1.5 Add corruption, unknown-version, partial-output, output-flood, timeout, missing-case, missing-control, interrupted-marker, and cleanup-evidence tests; prove a later run blocks instead of automatically reclaiming an incomplete attempt and that retirement emits a separate recovery receipt. +- [ ] 1.6 Run the legacy and structured oracle twice on clean revisions, record runtime and artifact costs, and obtain independent review before starting the domain pilot. + +## 2. Trusted GroupMe Domain Pilot + +- [ ] 2.1 Register two or three reviewed declarative operators for GroupMe page-ceiling and cursor-progress risks, with exact implementation preimages, permitted postimages, and immutable judge closure. +- [ ] 2.2 Implement independent no-hardlink disk-backed clones with isolated `HOME`, `TMPDIR`, XDG, pnpm, Git, accounting, and test paths; use copy-only dependency import semantics; pin and record Node/pnpm executables and dependency identity; preflight the exact clean commands. +- [ ] 2.3 Record focused clean and mutant GroupMe checks as adapter evidence; do not label them test-accounting authority receipts. +- [ ] 2.4 Run the clean complete `polyfill-connectors` backstop at each locked batch start; run the mutant backstop for every focused survivor; copy and revalidate the complete authority bundle outside the disposable clone before deletion; test evidence budgets, 30-day minimum retention, and the two-hour digest-identical reuse rule. +- [ ] 2.5 Implement the total projection table and no-automatic-retry rule; add fault-injection tests for preimage mismatch, forbidden path changes, immutable-judge changes, baseline/backstop absence, selector miss, timeout, direct-output limit, transcript/workspace soft-threshold overshoot, contradictory attempts, abandoned attempt, cleanup failure, missing retained bytes, and altered evidence. +- [ ] 2.6 Run two or three declared operators within the 10-minute locked batch and sequential local policy; capture raw execution axes, projections, selector misses, baseline reuse, runtime, setup time, artifact sizes, cleanup, and reviewer minutes. +- [ ] 2.7 Obtain independent triage for every survivor, likely-equivalent, uninteresting, or invalid-fault disposition. + +## 3. Decision Gate + +- [ ] 3.1 Publish metric definitions and raw pilot results, including invalid trials and excluded denominators rather than only percentages. +- [ ] 3.2 Stop or narrow on any cleanup failure, abandoned process, unexplained selector miss, authority mismatch, dominant setup cost, predominantly invalid or trivial operators, or insufficient useful evidence within budget. +- [ ] 3.3 Compare the migration and GroupMe adapters and identify any substantial repeated policy/evidence invariants; do not generalize superficial lifecycle differences. +- [ ] 3.4 Publish a continue, narrow, or stop memo. Require a new independently reviewed OpenSpec proposal before any shared coordinator, generic executor, StrykerJS experiment, CI scheduling, blocking gate, agent-generated mutant, or test-deletion automation. diff --git a/openspec/changes/add-mutation-falsification-service/design.md b/openspec/changes/add-mutation-falsification-service/design.md deleted file mode 100644 index dbc275a8b..000000000 --- a/openspec/changes/add-mutation-falsification-service/design.md +++ /dev/null @@ -1,113 +0,0 @@ -## Context - -See `proposal.md` for motivation. PDPP's current test-migration oracle already owns a strong, self-contained lifecycle: it creates fixture repositories, applies named faults, invokes mutation-specific judges, runs a positive control, proves rollback, and disposes its fixtures. The GroupMe connector also has real mutation-killing tests for historical pagination and cursor-progress faults. - -The current test-accounting authority issues receipts only for manifest-owned complete plans on clean trees. It does not authorize arbitrary focused subsets or dirty mutants. A Git worktree isolates tracked source state but does not constrain filesystem, environment, network, credentials, processes, caches, databases, or Docker access. Local hashes bind content but do not authenticate an issuer. - -## Goals / Non-Goals - -**Goals:** - -- Make one existing oracle's evidence structured without weakening its lifecycle. -- Run one trusted real-domain pilot against the current accounting authority's actual contract. -- Separate requested intent, machine observations, derived projections, and independent triage. -- Measure whether any shared infrastructure would reduce real repeated reasoning and audit cost. - -**Non-Goals:** - -- A generic source-mutating executor or untrusted-code sandbox. -- Arbitrary, packet-authored, or agent-generated patches and commands. -- A new test-accounting subset authority. -- StrykerJS, CI scheduling, blocking gates, mutation scores, or test-deletion authority. -- Hard CPU or memory containment where the host cannot enforce it. - -## Decisions - -### 1. Version one is a trusted local evidence program, not a service - -There is no daemon, queue, server, network API, or remotely supplied executable input. A repository-owned registry names the only permitted adapters and operators. Intent may request a registered risk and a stricter budget; versioned repository policy derives the effective command, working directory, environment allowlist, immutable judge closure, focused evidence, complete backstop, and host limits. - -This prevents a generator from choosing its own judge or safety policy. Arbitrary and agent-generated executable mutations require a separate sandbox design that proves filesystem, network, environment, process-tree, CPU, memory, disk, and output containment. - -### 2. Adapters own mutation mechanics until common structure is earned - -The migration oracle remains self-contained. It gains structured output but keeps its current named cases, fixture repositories, mutation-specific judges, positive control, and rollback proof. It demonstrates evidence shape and crash honesty, not a generic executor. - -The second adapter is a GroupMe cursor/frontier pilot. It uses two or three checked-in declarative operators over `packages/polyfill-connectors/connectors/groupme/index.ts`, such as reintroducing the historical page ceiling or weakening non-progress detection. The operators have exact preimages and permitted postimages; they cannot alter tests, runners, policy, or manifests. - -Only after both adapters run will a decision memo identify whether they share a deep stable boundary. Until then, duplication is preferable to a shallow common executor with adapter-specific escape hatches. - -### 3. Evidence uses three immutable artifact types - -An **intent packet** records requested risk, base identity, adapter/operator descriptor, and requested bounds. Its canonical digest is its identifier; callers do not supply the identifier. - -An **attempt receipt** records one execution's raw observations: issued random attempt ID, deterministic trial key, resolved policy, exact effective plan, environment profile, base/mutant/judge identities, bounded artifact digests and sizes, baseline/materialization/focused/backstop/reachability/cleanup axes, duration, exit or signal, and any referenced accounting receipt digests. - -A **triage receipt** is append-only and binds one attempt digest. It records an independent reviewer's claimed identity, disposition, evidence, reason, and timestamp. Version one does not authenticate that identity. A different reviewer from the operator/test author is required before likely-equivalent or uninteresting evidence is excluded from reported actionable results. - -`killed`, `survived`, and `inconclusive` are computed projections, never caller fields. `not_exercised` requires adapter-supplied validated reachability evidence; otherwise reachability is `unknown`. Timeout remains a timeout unless a later predeclared repeat policy supports a stronger interpretation. - -### 4. Digests provide integrity binding, not authenticity - -Canonical JSON, schema version, canonicalization version, and hash algorithm are explicit. A trial key binds the intent digest, repository tree, adapter version, policy version, and mutation identity. Each run gets a random attempt ID, so replay has stable identity but different observations. - -Before spawn, the adapter-specific runner writes an issued attempt marker in a verifier-owned run directory. Complete receipts publish atomically only after structured output validation and cleanup evidence. Interrupted markers remain incomplete and are discovered at next start. Referenced transcripts are bounded artifacts whose digests and sizes appear in the receipt. - -Anyone controlling the host can rewrite records and recompute an unkeyed digest. Version one therefore claims internal consistency and tamper evidence relative to a separately retained digest, not issuer authenticity. Authenticated provenance would require a later CI signature or platform attestation. - -### 5. The migration oracle is the evidence pilot - -The first slice adds a structured JSON mode to `scripts/test-migration/mutation-oracle.ts`. Legacy and structured modes must agree on every named case, catching check, hole, positive control, and rollback result. The source checkout must remain unchanged. - -The adapter-specific runner accepts only `test-migration-oracle/v1`; it derives the command and allowlisted environment. It imposes hard wall-time and output-byte limits, records partial/crash states honestly, and never interprets missing output as success. It does not claim focused selection, test-accounting authority, or domain value. - -### 6. GroupMe is the real-domain calibration pilot - -The pilot uses the existing hermetic GroupMe cursor/frontier tests as focused adapter evidence. It creates a fresh one-commit mutant descendant for each trusted operator. The clean complete `polyfill-connectors` suite always runs through the unchanged test-accounting authority. The mutant complete suite runs for every focused survivor; after a focused kill, policy may omit it only by recording `not_run_focused_kill`. Verified accounting receipt digests are referenced rather than duplicated. - -The focused clean baseline and complete clean backstop must pass before interpreting mutant evidence. Every focused survivor receives the complete mutant backstop. A focused pass plus backstop failure is a selector miss and blocks selector promotion. If a required backstop cannot complete, the attempt is inconclusive. - -A clean baseline may be reused across operators only when the repository tree, judge and adapter closure, effective command, allowlisted environment, dependency identity, and budget remain digest-identical. Otherwise it must run again. - -The source and judge closures are separate. Operators can change only predeclared production ranges in GroupMe's implementation. Tests, policy, runner, manifest, lockfile, and receipt validator remain immutable and are digest-bound. - -### 7. Workspaces are one-shot source isolation, not sandboxes - -Domain attempts use a configured disk-backed root with free-space preflight. Each attempt gets a fresh workspace and clean committed mutant descendant. Dependencies are materialized with the repository-pinned `pnpm@10.33.0`, offline and frozen from the lockfile, with lifecycle scripts disabled for this pure TypeScript pilot. If that exact materialization cannot satisfy the pilot, execution stops and the design must be reviewed rather than silently sharing mutable build output. - -Successful workspaces are deleted before the completed attempt receipt is published. Interrupted or cleanup-failed workspaces are quarantined and never reused. Next-start recovery reports owned incomplete markers and quarantines before new work. This handles hard termination honestly; it does not pretend `finally` runs after power loss or `SIGKILL`. - -Version one supports only Linux hosts with cgroup v2 and a usable systemd user manager. Each adapter, focused check, and authority backstop runs as a uniquely named transient systemd user service with `KillMode=control-group`, a finite `RuntimeMaxSec`, and policy-selected cgroup task, CPU, and memory limits. The initial `TasksMax` is 512, not an assumed Node file count; the clean complete backstop records peak tasks so later policy can be calibrated without invalidating the immutable command. macOS, Windows, non-cgroup-v2 Linux, and Linux without the required user-service controls fail closed in version one. - -The launcher only requests a uniquely named transient service; it never starts adapter work itself. The top-level attempt wrapper begins inside the manager-owned service with no adapter descendants. It writes and fsyncs a complete claim record containing the unit name to a unique temporary file, then atomically hard-links that record to the repository-scoped active-marker path. If the link loses to an existing marker, the service exits before spawning adapter work. If it wins, the wrapper queries systemd for its actual unit, `ControlGroup`, and `InvocationID`, atomically publishes the bound record, verifies that its own process is in that control group, and only then starts the adapter. - -This ordering removes the late-start race: a delayed start request can create only a service whose wrapper must win the same marker before doing work. Launcher death is harmless because systemd owns any accepted service. Recovery stops the marker-bound unit and invocation, waits for the stop job, verifies the cgroup absent or unpopulated and the unit inactive, and only then retires the marker. It never relies on a recorded child PID, so PID reuse cannot establish cleanup. An incomplete, mismatched, or unverifiable claim quarantines the attempt and blocks further mutation work until reviewed. - -Initial policy permits one adapter or backstop service at a time, at most 20 domain mutants and 10 wall-clock minutes. It hard-bounds captured output and sets a 60-second cleanup deadline. The cgroup caps the complete descendant tree at 50% of available CPU, a policy-selected memory maximum, and 512 tasks, including full-suite children, threads, and grandchildren. Workspace size uses a disk-free preflight, reserved host headroom, and periodic observation. Crossing the threshold stops and quarantines the attempt, but it is explicitly not a filesystem quota and cannot promise zero overshoot or host-disk impact. Live credentials, personal data, third-party network, stateful browsers, and shared production-like databases are prohibited. Required unsupported controls cause refusal. - -### 8. The decision gate precedes shared infrastructure - -The pilot predefines valid-trial denominators and reports raw counts for execution axes, projections, selector misses, triage dispositions, runtime, output/workspace size, cleanup, and reviewer minutes. Setup time is reported separately. - -Stop or narrow immediately on a cleanup or containment failure, an unexplained selector miss, authority/receipt mismatch, or evidence corruption. Stop generalization if setup consumes most runtime, review exceeds five minutes per disputed attempt, most operators are invalid/trivial, no useful risk evidence appears within 20 mutants/10 minutes, or the adapters do not expose repeated policy/evidence logic. - -Continue only if evidence is interpretable, costs are acceptable, there are no unexplained selector or cleanup failures, and a proposed shared module would hide substantial repeated invariants across both adapters. Any coordinator, generic executor, Stryker experiment, CI lane, or sandbox is a new reviewed OpenSpec change. - -## Risks / Trade-offs - -- **The narrow slices may not justify a framework** → treat stopping with two useful purpose-fit adapters as success. -- **Local evidence lacks authenticated issuer identity** → state the trust boundary and preserve external attestation as a separate future capability. -- **Full connector backstops may dominate runtime** → measure setup and execution separately; stop rather than weaken the mandatory calibration backstop. -- **A trusted mutant can still affect ambient host state accidentally** → constrain paths/operators/environment, use hermetic tests, run sequentially, and never call a worktree a sandbox. -- **Offline dependency materialization may fail** → refuse the attempt and review the dependency strategy; do not share mutable outputs silently. -- **Version-one containment is Linux-specific** → fail closed elsewhere and require separate reviewed host adapters before claiming macOS or Windows support. -- **Small samples cannot prove selector completeness** → report raw counts and treat any observed miss as disqualifying, without claiming zero misses proves completeness. - -## Migration Plan - -1. Land this revised design only after a second independent architecture review returns LAND. -2. Add structured evidence to the existing migration oracle and validate it without generic execution infrastructure. -3. Run the trusted GroupMe cursor/frontier pilot under mandatory clean and mutant backstops. -4. Publish a continue, narrow, or stop decision memo. Do not implement shared infrastructure from this change. - -Rollback removes structured output and pilot artifacts while preserving both existing test suites and their legacy entry points. diff --git a/openspec/changes/add-mutation-falsification-service/tasks.md b/openspec/changes/add-mutation-falsification-service/tasks.md deleted file mode 100644 index 93af9dc83..000000000 --- a/openspec/changes/add-mutation-falsification-service/tasks.md +++ /dev/null @@ -1,29 +0,0 @@ -## 1. Existing-Oracle Structured Evidence - -- [ ] 1.1 Define versioned adapter-specific intent, attempt, and triage schemas for `test-migration-oracle/v1`, including canonicalization golden vectors and explicit integrity-versus-authenticity language. -- [ ] 1.2 Add structured JSON output to the existing test-migration oracle without changing its named mutations, judges, positive control, fixture lifecycle, human output, or rollback proof. -- [ ] 1.3 Add differential tests requiring legacy and structured modes to report identical cases, catching checks, holes, positive-control result, and rollback result. -- [ ] 1.4 Add Linux cgroup-v2/systemd capability probing and fail-closed refusal for unsupported hosts; bind the containment mechanism, version, unit, invocation, cgroup, task, CPU, memory, wall-time, output, workspace-threshold, and cleanup policy into attempt evidence. -- [ ] 1.5 Start the top-level wrapper inside the transient service; before any adapter work, atomically claim the repository marker from a complete record, bind and verify its actual unit/invocation/cgroup identity, and exit on claim loss. -- [ ] 1.6 Add bounded adapter-specific execution evidence with issued, incomplete, and completed attempt states and exact source-checkout unchanged proof. -- [ ] 1.7 Add next-start recovery that stops the marker-bound unit and invocation, waits for the manager job, verifies the cgroup empty and unit inactive, and blocks on unverifiable claims without relying on PID-only liveness. -- [ ] 1.8 Add corruption, unknown-version, partial-output, delayed-start-submission, launcher-death, claim-race, child-and-grandchild timeout, wrapper-death-after-adapter-spawn, PID-reuse, output-flood, task-limit, missing-case, missing-control, no-surviving-descendant, and cleanup-evidence tests. -- [ ] 1.9 Add workspace fill-to-threshold and overshoot tests that prove stop-and-quarantine behavior and explicitly do not claim a hard disk quota. -- [ ] 1.10 Run the legacy and structured oracle twice on clean revisions, record runtime and artifact costs, and obtain independent review before starting the domain pilot. - -## 2. Trusted GroupMe Domain Pilot - -- [ ] 2.1 Register two or three reviewed declarative operators for GroupMe page-ceiling and cursor-progress risks, with exact implementation preimages, permitted postimages, and immutable judge closure. -- [ ] 2.2 Implement one-shot disk-backed mutant workspaces with free-space preflight, pinned offline frozen dependency materialization, issued markers, successful deletion, interruption quarantine, and next-start recovery. -- [ ] 2.3 Record focused clean and mutant GroupMe checks as adapter evidence; do not label them test-accounting authority receipts. -- [ ] 2.4 Always run the clean complete `polyfill-connectors` backstop through the unchanged test-accounting authority; run the mutant backstop for every focused survivor, record `not_run_focused_kill` when policy omits it after a focused kill, and bind every verified receipt digest. -- [ ] 2.5 Add fault-injection tests for preimage mismatch, forbidden path changes, immutable-judge changes, backstop absence, selector miss, timeout, hard output limit, workspace soft-threshold overshoot, abandoned attempt, cleanup failure, and altered evidence. -- [ ] 2.6 Run at most 20 trials within 10 wall-clock minutes and sequential local resource policy; capture raw execution axes, projections, selector misses, runtime, setup time, artifact sizes, cleanup, and reviewer minutes. -- [ ] 2.7 Obtain independent triage for every survivor, likely-equivalent, uninteresting, or invalid-fault disposition. - -## 3. Decision Gate - -- [ ] 3.1 Publish metric definitions and raw pilot results, including invalid trials and excluded denominators rather than only percentages. -- [ ] 3.2 Stop or narrow on any cleanup/containment failure, unexplained selector miss, authority mismatch, dominant setup cost, predominantly invalid or trivial operators, or insufficient useful evidence within budget. -- [ ] 3.3 Compare the migration and GroupMe adapters and identify any substantial repeated policy/evidence invariants; do not generalize superficial lifecycle differences. -- [ ] 3.4 Publish a continue, narrow, or stop memo. Require a new independently reviewed OpenSpec proposal before any shared coordinator, generic executor, StrykerJS experiment, CI scheduling, blocking gate, agent-generated mutant, or test-deletion automation.