diff --git a/openspec/changes/add-mutation-falsification-evidence/.openspec.yaml b/openspec/changes/add-mutation-falsification-evidence/.openspec.yaml new file mode 100644 index 000000000..a8821c74d --- /dev/null +++ b/openspec/changes/add-mutation-falsification-evidence/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-08-11 diff --git a/openspec/changes/add-mutation-falsification-evidence/design.md b/openspec/changes/add-mutation-falsification-evidence/design.md new file mode 100644 index 000000000..cfb87b3fd --- /dev/null +++ b/openspec/changes/add-mutation-falsification-evidence/design.md @@ -0,0 +1,139 @@ +## Context + +See `proposal.md` for motivation. PDPP's current test-migration oracle already owns a strong, self-contained lifecycle: it creates fixture repositories, applies named faults, invokes mutation-specific judges, runs a positive control, proves rollback, and disposes its fixtures. The GroupMe connector also has real mutation-killing tests for historical pagination and cursor-progress faults. + +The current test-accounting authority issues receipts only for manifest-owned complete plans on clean trees. It does not authorize arbitrary focused subsets or dirty mutants. A Git worktree isolates tracked source state but does not constrain filesystem, environment, network, credentials, processes, caches, databases, or Docker access. Local hashes bind content but do not authenticate an issuer. + +## Goals / Non-Goals + +**Goals:** + +- Make one existing oracle's evidence structured without weakening its lifecycle. +- Run one trusted real-domain pilot against the current accounting authority's actual contract. +- Separate requested intent, machine observations, derived projections, and independent triage. +- Measure whether any shared infrastructure would reduce real repeated reasoning and audit cost. + +**Non-Goals:** + +- A generic source-mutating executor or untrusted-code sandbox. +- Arbitrary, packet-authored, or agent-generated patches and commands. +- A new test-accounting subset authority. +- StrykerJS, CI scheduling, blocking gates, mutation scores, or test-deletion authority. +- A reusable execution coordinator, sandbox, or automatic crash-recovery system. +- Product-host portability claims based on this developer-only experiment. + +## Decisions + +### 1. Version one is a trusted local evidence program, not a service + +There is no daemon, queue, server, network API, or remotely supplied executable input. A repository-owned registry names the only permitted adapters and operators. Intent may request a registered risk and a stricter budget; versioned repository policy derives the effective command, working directory, environment allowlist, immutable judge closure, focused evidence, complete backstop, and host limits. + +This prevents a generator from choosing its own judge or safety policy. Arbitrary and agent-generated executable mutations require a separate sandbox design that proves filesystem, network, environment, process-tree, CPU, memory, disk, and output containment. + +### 2. Adapters own mutation mechanics until common structure is earned + +The migration oracle remains self-contained. It gains structured output but keeps its current named cases, fixture repositories, mutation-specific judges, positive control, and rollback proof. It demonstrates evidence shape and crash honesty, not a generic executor. + +The second adapter is a GroupMe cursor/frontier pilot. It uses two or three checked-in declarative operators over `packages/polyfill-connectors/connectors/groupme/index.ts`, such as reintroducing the historical page ceiling or weakening non-progress detection. The operators have exact preimages and permitted postimages; they cannot alter tests, runners, policy, or manifests. + +Only after both adapters run will a decision memo identify whether they share a deep stable boundary. Until then, duplication is preferable to a shallow common executor with adapter-specific escape hatches. + +### 3. Evidence uses three immutable artifact types + +An **intent packet** records requested risk, base identity, adapter/operator descriptor, and requested bounds. Its canonical digest is its identifier; callers do not supply the identifier. + +An **attempt receipt** records one execution's raw observations: issued random attempt ID, deterministic trial key, resolved policy, exact effective plan, environment profile, base/mutant/judge identities, bounded artifact digests and sizes, baseline/materialization/focused/backstop/reachability/cleanup axes, duration, exit or signal, and any referenced accounting receipt digests. + +A **triage receipt** is append-only and binds one attempt digest. It records an independent reviewer's claimed identity, disposition, evidence, reason, and timestamp. Version one does not authenticate that identity. A different reviewer from the operator/test author is required before likely-equivalent or uninteresting evidence is excluded from reported actionable results. + +`killed`, `survived`, and `inconclusive` are computed projections, never caller fields. The projection is total and conservative: + +| Observations | Projection | Additional signal | +| --- | --- | --- | +| Any clean baseline, materialization, protocol, authority, artifact-retention, or cleanup failure | `inconclusive` | Preserve the failing axis. | +| Timeout, signal, resource stop, malformed or partial output, or unexplained nondeterminism | `inconclusive` | Preserve the exact failure; never infer a kill. | +| Validated `not_exercised` reachability | `inconclusive` | Send to independent triage; do not count it as survived. | +| Focused mutant check fails for a mutation-attributable test assertion | `killed` | The complete mutant backstop may be `not_run_focused_kill`. | +| Focused mutant check passes and the complete mutant backstop fails for a mutation-attributable test assertion | `killed` | Record a selector miss and stop selector promotion. | +| Focused mutant check and complete mutant backstop both pass | `survived` | Require independent triage before exclusion from actionable results. | + +A failure is mutation-attributable only when the digest-identical clean command passed and the mutant command produced a recognized owning-test assertion failure, not an infrastructure, protocol, accounting, cleanup, or resource error. Version one performs no automatic retries. A reviewer may request a new attempt; contradictory valid attempts with the same trial key make the aggregate trial inconclusive. `not_exercised` requires adapter-supplied validated reachability evidence; otherwise reachability is `unknown`. + +### 4. Digests provide integrity binding, not authenticity + +RFC 8785 JSON canonicalization, schema version, canonicalization version, and SHA-256 are explicit. A trial key binds the intent digest, repository tree, adapter version, policy version, and mutation identity. Each run gets a random attempt ID, so replay has stable identity but different observations. + +Before spawn, the adapter-specific runner writes an issued attempt marker in a verifier-owned, disk-backed evidence directory outside the disposable workspace. Complete receipts publish atomically only after structured output validation, retained-artifact validation, and cleanup evidence. The writer fsyncs each completed file and its parent directory after publication. Interrupted markers remain incomplete and are discovered at next start. + +The experiment does not claim automatic process recovery or power-loss-proof containment. Before every run it scans its evidence and configured workspace roots. Any incomplete or corrupt marker, unexpected workspace, or prior cleanup failure blocks execution for explicit operator review. Age, PID liveness, and a successful `finally` block never authorize automatic reclamation. Retirement requires a separate append-only recovery receipt that records the operator claim, process and workspace observations, disposition, and retained evidence; it does not convert the interrupted attempt into a completed one. + +Anyone controlling the host can rewrite records and recompute an unkeyed digest. Version one therefore claims internal consistency and tamper evidence relative to a separately retained digest, not issuer authenticity. Authenticated provenance would require a later CI signature or platform attestation. + +### 5. The migration oracle is the evidence pilot + +The first slice adds a structured JSON mode to `scripts/test-migration/mutation-oracle.ts`. Legacy and structured modes must agree on every named case, catching check, hole, positive control, and rollback result. The source checkout must remain unchanged. + +The adapter-specific runner accepts only `test-migration-oracle/v1`; it derives the command and allowlisted environment. It bounds its direct structured-output capture, applies a finite wall deadline, records partial/crash states honestly, and never interprets missing output as success. These are adapter-local protections, not a claim that the runner survives its own crash or contains hostile descendants. It does not claim focused selection, test-accounting authority, or domain value. + +### 6. GroupMe is the real-domain calibration pilot + +The pilot uses the existing hermetic GroupMe cursor/frontier tests as focused adapter evidence. It creates a fresh one-commit mutant descendant for each trusted operator. The clean complete `polyfill-connectors` suite runs through the unchanged test-accounting authority before any mutant is interpreted. The mutant complete suite runs for every focused survivor; after a focused kill, policy may omit it only by recording `not_run_focused_kill`. + +Before a disposable workspace is removed, the verifier copies the complete validated accounting bundle—authority record, transcript, completion record, receipt, and required closure or manifest identity—into its bounded evidence directory. It validates the copied bundle there and records each relative location, size, and digest in the attempt receipt. A digest without retained, revalidatable bytes is invalid evidence. Copy, validation, retention-budget, or publication failure makes the attempt inconclusive and quarantines its workspace. + +Policy declares the evidence root, maximum retained bytes, maximum attempts, and retention deadline before a batch starts. It reserves that capacity without deleting prior evidence. A completed batch remains intact through the decision memo's independent review and for at least 30 days afterward; later deletion is an explicit audited operation. The harness stops before accepting a new attempt that could exceed the declared retained-byte or attempt-count budget. + +The focused clean baseline and complete clean backstop must pass before interpreting mutant evidence. Every focused survivor receives the complete mutant backstop. A focused pass plus backstop failure is a selector miss and blocks selector promotion. If a required backstop cannot complete, the attempt is inconclusive. + +A clean focused baseline and clean complete backstop may be reused only within one locked pilot batch, for at most two hours, while the repository tree, judge and adapter closure, effective commands, allowlisted environment, Node and dependency identities, and budgets remain digest-identical. The retained accounting bundle must still revalidate before every reference. The batch ends on any tree, policy, environment, dependency, or budget change, any missing retained byte, or the first unexplained failure. The next batch runs fresh clean evidence. Reuse changes neither the killed/survived denominator nor the raw count of clean executions; attempt receipts bind the exact baseline digest they used. + +The source and judge closures are separate. Operators can change only predeclared production ranges in GroupMe's implementation. Tests, policy, runner, manifest, lockfile, and receipt validator remain immutable and are digest-bound. + +### 7. Workspaces are one-shot source isolation, not sandboxes + +Domain attempts use a configured disk-backed root with free-space preflight. Each attempt gets an independent local clone made without hard links, a fresh dependency tree, and a clean committed mutant descendant. The clone does not share a Git common directory with the source checkout. All writable attempt state—including `HOME`, `TMPDIR`, XDG directories, pnpm store and virtual store, test temporary files, and test-accounting run files—resolves beneath the attempt root. The evidence directory is the only retained writable path and is outside that root. + +Policy records the repository-pinned Node `v25.8.2`, the actual Node executable digest, repository-pinned `pnpm@10.33.0`, the actual pnpm executable and version, lockfile digest, and materialization command. Dependencies are materialized offline and frozen from the lockfile, with lifecycle scripts disabled for this pure TypeScript pilot. The offline package source is read-only and digest-inventoried; pnpm uses copy import semantics into the attempt-local store and dependency tree rather than links to mutable host content. A preflight runs the exact clean focused and complete commands with this layout before the batch begins. If lifecycle suppression or local materialization cannot satisfy the pilot, execution stops and the design is reviewed rather than silently sharing mutable output. + +Successful workspaces are deleted only after required evidence has been copied and revalidated. The external verifier confirms deletion, then publishes the completed attempt receipt. Interrupted or cleanup-failed workspaces are quarantined and never reused. A later run reports them and refuses automatic recovery. The operator must independently establish that no related process remains before explicitly retiring an incomplete marker. This is source and writable-state isolation for trusted repository code; it is not a sandbox and does not claim containment after verifier death. + +The resource contract is deliberately narrow: + +| Resource | Mechanism | Enforcement and observation | Receipt projection | +| --- | --- | --- | --- | +| Trial count | Repository policy counter | Refuse a new trial after the declared count | No attempt is created. | +| Wall time | Adapter-local deadline and owning-process-group termination | Stop the group when the deadline fires; verify no selected-command process remains; record deadline, signal, and cleanup observation | `inconclusive` | +| Direct structured output | Streaming byte counter before buffering | Stop the adapter when its declared byte cap is crossed; retain a bounded prefix and byte count | `inconclusive` | +| Test-accounting transcript | Unchanged authority plus trusted hermetic command | Observe and record size; no hard byte-cap claim until the authority gains one in a separate change | Over-budget is `inconclusive` and stops the batch. | +| Workspace bytes | Free-space preflight plus periodic observation | Stop and quarantine after the soft threshold; overshoot remains possible | `inconclusive` | +| CPU, memory, and task count | Host observations only | Record peaks when available; no throttle, quota, or kill guarantee | Over-budget is `inconclusive` and stops the batch. | +| Cleanup | Finite verifier wait plus filesystem/process observations | Failure quarantines and blocks the next run | `inconclusive` | + +Initial policy permits one trusted command at a time, two or three domain operators, and 10 wall-clock minutes for the declared pilot batch. Environment construction starts empty and admits only policy-listed non-secret values plus the isolated writable paths. A credential-sentinel test proves ambient values are absent. Live credentials, personal data, third-party network, stateful browsers, Docker sockets, and shared production-like databases are forbidden by policy; because version one is not a sandbox, the adapter preflight must also prove that the selected commands need none of them. This experiment is verified only on its declared Linux developer-host profile. No result changes or narrows PDPP's heterogeneous product-host support contract. + +### 8. The decision gate precedes shared infrastructure + +The pilot predefines valid-trial denominators and reports raw counts for execution axes, projections, selector misses, triage dispositions, runtime, output/workspace size, cleanup, and reviewer minutes. The valid-result denominator is exactly `killed + survived`; inconclusive, invalid, and not-run attempts remain visible as separate raw counts and never enter that denominator. Setup time is reported separately. + +Stop or narrow immediately on a cleanup failure, abandoned process, unexplained selector miss, authority/receipt mismatch, or evidence corruption. Stop generalization if setup consumes most runtime, review exceeds five minutes per disputed attempt, most operators are invalid/trivial, no useful risk evidence appears within the two-or-three-operator/10-minute batch, or the adapters do not expose repeated policy/evidence logic. + +Continue only if evidence is interpretable, costs are acceptable, there are no unexplained selector or cleanup failures, and a proposed shared module would hide substantial repeated invariants across both adapters. Any coordinator, generic executor, Stryker experiment, CI lane, or sandbox is a new reviewed OpenSpec change. + +## Risks / Trade-offs + +- **The narrow slices may not justify a framework** → treat stopping with two useful purpose-fit adapters as success. +- **Local evidence lacks authenticated issuer identity** → state the trust boundary and preserve external attestation as a separate future capability. +- **Full connector backstops may dominate runtime** → measure setup and execution separately; stop rather than weaken the mandatory calibration backstop. +- **A trusted mutant can still affect ambient host state accidentally** → isolate writable paths, constrain operators and environment, use hermetic tests, run sequentially, and stop on any unexpected effect; never call the clone a sandbox. +- **Offline dependency materialization may fail** → refuse the attempt and review the dependency strategy; do not share mutable outputs silently. +- **Version-one evidence is calibrated on one Linux developer profile** → record that profile and make no inference about PDPP product-host support. +- **Small samples cannot prove selector completeness** → report raw counts and treat any observed miss as disqualifying, without claiming zero misses proves completeness. + +## Migration Plan + +1. Land this revised design only after a second independent architecture review returns LAND. +2. Add structured evidence to the existing migration oracle and validate it without generic execution infrastructure. +3. Run the trusted GroupMe cursor/frontier pilot under mandatory clean and mutant backstops. +4. Publish a continue, narrow, or stop decision memo. Do not implement shared infrastructure from this change. + +Rollback removes structured output and pilot artifacts while preserving both existing test suites and their legacy entry points. diff --git a/openspec/changes/add-mutation-falsification-evidence/proposal.md b/openspec/changes/add-mutation-falsification-evidence/proposal.md new file mode 100644 index 000000000..d08234f59 --- /dev/null +++ b/openspec/changes/add-mutation-falsification-evidence/proposal.md @@ -0,0 +1,30 @@ +## Why + +PDPP has valuable bespoke falsifiability tests, but their evidence is not consistently machine-readable or comparable. Before adding shared mutation infrastructure, PDPP needs to prove that structured evidence from one existing oracle and one trusted real-domain pilot produces useful signal at acceptable cost. + +## What Changes + +- Add structured evidence to the existing test-migration oracle without changing its mutation lifecycle, judges, positive control, or rollback proof. +- Define separate intent, execution-attempt, and reviewer-triage artifacts with honest integrity and provenance claims. +- Run one trusted, repository-owned GroupMe cursor/frontier pilot with checked-in fault operators, focused adapter evidence, and mandatory complete test-accounting backstops. +- Pre-register resource budgets, selector-miss rules, evidence metrics, and continue, narrow, or stop criteria. +- Keep all mutation work local, sequential, operator-supervised, advisory, and limited to reviewed adapters. +- Defer a shared coordinator, generic source-mutating executor, arbitrary or agent-generated patches, StrykerJS, CI scheduling, blocking gates, and automatic test deletion until the pilot evidence justifies a separate proposal. + +## Capabilities + +### New Capabilities + +- `mutation-falsification`: Defines trusted local mutation evidence, independent triage, one bounded domain calibration pilot, and the decision gate for any later shared infrastructure. + +### Modified Capabilities + +None. + +## Impact + +- Adds structured output and evidence validation around an existing repository oracle. +- Adds a bounded GroupMe pilot that uses existing tests and preserves complete test-accounting evidence outside disposable workspaces. +- Does not change product behavior, public APIs, test-accounting authority, or CI requirements. +- Makes no sandbox, crash-containment, or heterogeneous-host claim for the experimental developer harness. +- Produces a decision memo before any reusable mutation execution framework is proposed. diff --git a/openspec/changes/add-mutation-falsification-evidence/specs/mutation-falsification/spec.md b/openspec/changes/add-mutation-falsification-evidence/specs/mutation-falsification/spec.md new file mode 100644 index 000000000..a4aca42f4 --- /dev/null +++ b/openspec/changes/add-mutation-falsification-evidence/specs/mutation-falsification/spec.md @@ -0,0 +1,123 @@ +## Purpose + +Provide bounded, reviewable evidence that selected PDPP tests detect plausible trusted faults before the repository invests in shared mutation infrastructure. + +## ADDED Requirements + +### Requirement: Version one SHALL execute only trusted repository adapters + +Version one SHALL be a local developer harness with no daemon, queue, server, or network API. It SHALL execute only reviewed, repository-owned adapter and operator identifiers with validated parameters and exact target preimages. It SHALL NOT execute caller-supplied commands, arbitrary patches, or agent-generated source. + +#### Scenario: Trusted adapter is requested +- **WHEN** an intent names a registered adapter and valid checked-in operator +- **THEN** repository policy SHALL derive the effective command, environment, judge, and limits + +#### Scenario: Arbitrary executable input is requested +- **WHEN** an intent supplies executable source, command arguments, working directory, environment policy, or an unregistered operator +- **THEN** the harness SHALL reject it before execution + +### Requirement: Intent, attempt, and triage evidence SHALL remain separate + +An intent packet SHALL record the requested risk, trusted adapter and operator descriptors, base identity, and requested budget. An attempt receipt SHALL record immutable machine observations from one execution. A triage receipt SHALL separately record a later reviewer judgment bound to one attempt. The harness SHALL derive conventional killed, survived, or inconclusive projections from valid attempt evidence and SHALL NOT accept those projections or triage dispositions from the intent. + +The projection SHALL be conservative and total. Invalid clean evidence, materialization, protocol, authority, retained-artifact, cleanup, timeout, signal, resource, or nondeterminism observations SHALL project to `inconclusive`. A recognized owning-test assertion that passes on the digest-identical clean closure and fails on the mutant SHALL project to `killed`. A focused pass followed by such a complete-backstop failure SHALL also record a selector miss. Only a focused pass plus a complete mutant-backstop pass SHALL project to `survived`. Validated `not_exercised` reachability SHALL project to `inconclusive` pending triage. + +#### Scenario: Execution completes +- **WHEN** a trusted adapter produces valid execution evidence +- **THEN** the attempt receipt SHALL preserve baseline, materialization, focused, backstop, reachability, cleanup, timeout, and error observations without adding a human judgment + +#### Scenario: Reviewer classifies evidence +- **WHEN** an independent reviewer judges an attempt actionable, likely equivalent, uninteresting, deferred, or invalid +- **THEN** that judgment SHALL be appended as a separate triage receipt with reviewer claim, evidence reference, reason, and timestamp + +#### Scenario: Observations are contradictory +- **WHEN** valid attempts for one trial key disagree or a failure is not attributable to an owning-test assertion +- **THEN** the aggregate trial SHALL be inconclusive and the raw attempts SHALL remain visible + +### Requirement: Evidence identity and integrity claims SHALL be precise + +Evidence SHALL use versioned RFC 8785 JSON canonicalization and SHA-256. The deterministic trial key SHALL bind the intent digest, repository tree, adapter version, policy version, and mutation identity. Each execution SHALL receive a distinct random attempt identifier. Content digests SHALL establish internal consistency or tamper evidence relative to an independently retained digest; they SHALL NOT be represented as issuer authentication. + +#### Scenario: Same trial is repeated +- **WHEN** the same bound trial executes more than once +- **THEN** the attempts SHALL share a trial key and SHALL have different attempt identifiers and runtime observations + +#### Scenario: Authenticated provenance is unavailable +- **WHEN** evidence is produced only on a developer-controlled host without a trusted signer or platform attestation +- **THEN** provenance SHALL be recorded as a claim and SHALL NOT be described as authenticated identity + +### Requirement: The existing migration oracle SHALL retain its purpose-fit lifecycle + +The test-migration oracle SHALL retain its existing named mutations, mutation-specific judges, positive control, fixture repositories, and rollback proof. Its structured mode SHALL report the same cases and decisions as its human-readable mode. This adapter SHALL validate structured evidence and crash honesty only; it SHALL NOT be presented as validation of a generic source-mutating executor. + +#### Scenario: Structured mode runs successfully +- **WHEN** the existing oracle is run in structured mode +- **THEN** its named cases, catching checks, holes, positive control, and rollback decision SHALL agree with the legacy result + +#### Scenario: Structured evidence is incomplete +- **WHEN** output is partial, malformed, omits a case or control, exceeds its bound, or the process is interrupted +- **THEN** the attempt SHALL remain incomplete or invalid and SHALL NOT be projected as killed or survived + +### Requirement: Test-accounting authority SHALL NOT be overstated + +Focused checks in the domain pilot SHALL be labeled adapter evidence, not test-accounting authority receipts. Repository-owned policy SHALL select the focused check and complete owning-suite backstop. The complete clean backstop SHALL run at the start of each locked pilot batch. The complete mutant backstop SHALL run for every focused survivor. After a focused kill, policy SHALL either run it or record an explicit `not_run_focused_kill` observation. Every backstop that runs SHALL use the unchanged test-accounting authority on a clean committed tree. Before workspace deletion, the verifier SHALL copy and revalidate the complete authority, transcript, completion, receipt, and required closure evidence in its retained bounded store; an attempt SHALL bind the retained locations, sizes, and digests. + +Policy SHALL predeclare the evidence root, retained-byte and attempt-count budgets, and retention deadline without evicting prior evidence to admit a run. Completed batch evidence SHALL remain intact through independent review of the decision memo and for at least 30 days afterward. Later deletion SHALL be an explicit audited action. + +Clean evidence MAY be reused only inside the same locked batch, for at most two hours, while all bound repository, command, judge, environment, runtime, dependency, and budget identities remain equal and the retained bundle still validates. Reuse SHALL NOT change mutant denominators or hide the raw clean-execution count. + +#### Scenario: Focused mutant check passes +- **WHEN** the focused check passes with a trusted mutant present +- **THEN** the mandatory mutant backstop SHALL run; a focused pass followed by a backstop failure SHALL be recorded as a selector miss + +#### Scenario: Required backstop cannot complete +- **WHEN** a clean or mutant backstop is missing, fails its baseline, exceeds budget, or cannot issue a valid authority receipt +- **THEN** the attempt SHALL be inconclusive and SHALL NOT count as survived + +### Requirement: Domain mutation attempts SHALL be one-shot and bounded + +The GroupMe cursor/frontier pilot SHALL use only checked-in declarative fault operators over permitted production paths while the judge and runner closure remain immutable. Each source-mutating attempt SHALL use a fresh clean committed descendant in a disk-backed workspace. A successful workspace SHALL be deleted; an interrupted or cleanup-failed workspace SHALL be quarantined and detected on the next start. + +#### Scenario: Trusted domain operator runs +- **WHEN** a registered GroupMe operator matches its exact target preimage +- **THEN** the attempt SHALL record base and mutant trees, permitted changed paths, focused and backstop evidence, environment profile, artifacts, resource observations, and cleanup state + +#### Scenario: Attempt is abandoned +- **WHEN** execution ends before a complete receipt and verified cleanup +- **THEN** the issued attempt marker SHALL remain incomplete, its workspace SHALL NOT be reused, and a later run SHALL report or quarantine it + +### Requirement: Local execution SHALL use bounded trusted inputs and honest limitations + +Repository policy SHALL run one trusted command at a time and SHALL set finite trial-count, wall-time, direct structured-output, workspace-observation, and cleanup budgets. It SHALL distinguish enforced bounds from observations: CPU, memory, task-count, test-accounting transcript bytes, and workspace bytes SHALL NOT be described as hard quotas when the selected mechanism only observes them. The runner SHALL start from an empty environment allowlist and SHALL isolate writable home, temporary, cache, dependency-store, Git, and accounting paths beneath a one-shot attempt root. It SHALL forbid live credentials, personal data, live third-party services, stateful browsers, Docker sockets, and shared production-like databases by policy. + +Version one SHALL be an operator-supervised Linux developer experiment. It SHALL NOT claim to be a sandbox, to recover automatically after verifier death, or to establish PDPP product-host portability. Any incomplete marker, unexpected workspace, unexplained process, or cleanup failure SHALL block later runs pending explicit review. + +Retiring an interrupted marker SHALL require a separate append-only recovery receipt with operator claim, process and workspace observations, disposition, and retained evidence. It SHALL NOT convert the interrupted attempt into a completed attempt. + +#### Scenario: Enforced adapter-local bound is exceeded +- **WHEN** the adapter wall deadline or direct structured-output byte cap is exceeded +- **THEN** the runner SHALL stop its owning process group, verify that no selected-command process remains, retain bounded evidence, record cleanup observations, and mark the attempt inconclusive without claiming crash-durable containment + +#### Scenario: Workspace threshold is observed +- **WHEN** workspace observation detects that the advisory byte threshold has been crossed +- **THEN** the harness SHALL stop its owning process group, quarantine the workspace, record the observed overshoot, and SHALL NOT claim that host-disk impact was hard-contained + +#### Scenario: Verifier dies after child start +- **WHEN** the verifier exits after publishing an issued marker but before a complete receipt and cleanup evidence +- **THEN** a later run SHALL refuse automatic recovery and SHALL require explicit operator verification of the marker, workspace, and related processes + +#### Scenario: Preflight cannot establish the declared profile +- **WHEN** isolated writable paths, empty environment construction, pinned runtime/dependency materialization, hermetic commands, or required adapter-local bounds cannot be established +- **THEN** the harness SHALL refuse the attempt instead of silently weakening the policy + +### Requirement: Calibration SHALL end with a pre-registered decision + +The pilot SHALL remain advisory and SHALL publish raw counts and defined denominators for valid trials, focused-to-backstop misses, invalid faults, execution failures, cleanup failures, runtime, artifact sizes, and reviewer time. Its valid-result denominator SHALL be exactly killed plus survived; inconclusive, invalid, and not-run attempts SHALL remain separate raw counts. It SHALL stop or narrow on any cleanup failure, abandoned process, unexplained selector miss, dominant setup cost, predominantly invalid or trivial faults, or lack of useful evidence within its declared budget. Shared infrastructure SHALL require a later proposal supported by repeated invariants across both adapters and measured reduction in audit cost. + +#### Scenario: Pilot stays within bounds and produces useful evidence +- **WHEN** the two adapters produce interpretable evidence without cleanup failures or unexplained selector misses at acceptable compute and review cost +- **THEN** the decision memo MAY recommend a narrowly scoped shared evidence module and SHALL identify the repeated invariants it would hide + +#### Scenario: Pilot does not justify generalization +- **WHEN** signal is weak, costs dominate, safety fails, or the adapters do not share a deep stable boundary +- **THEN** the decision memo SHALL stop or narrow the initiative and retain purpose-fit adapters rather than manufacturing a framework diff --git a/openspec/changes/add-mutation-falsification-evidence/tasks.md b/openspec/changes/add-mutation-falsification-evidence/tasks.md new file mode 100644 index 000000000..f7d4e5713 --- /dev/null +++ b/openspec/changes/add-mutation-falsification-evidence/tasks.md @@ -0,0 +1,25 @@ +## 1. Existing-Oracle Structured Evidence + +- [ ] 1.1 Define versioned adapter-specific intent, attempt, and triage schemas for `test-migration-oracle/v1`, naming RFC 8785 JSON canonicalization plus SHA-256, including golden vectors and explicit integrity-versus-authenticity language. +- [ ] 1.2 Add structured JSON output to the existing test-migration oracle without changing its named mutations, judges, positive control, fixture lifecycle, human output, or rollback proof. +- [ ] 1.3 Add differential tests requiring legacy and structured modes to report identical cases, catching checks, holes, positive-control result, and rollback result. +- [ ] 1.4 Add bounded adapter-local execution evidence with issued, incomplete, and completed states, an external evidence root, finite wall and direct-output limits, and exact source-checkout unchanged proof. +- [ ] 1.5 Add corruption, unknown-version, partial-output, output-flood, timeout, missing-case, missing-control, interrupted-marker, and cleanup-evidence tests; prove a later run blocks instead of automatically reclaiming an incomplete attempt and that retirement emits a separate recovery receipt. +- [ ] 1.6 Run the legacy and structured oracle twice on clean revisions, record runtime and artifact costs, and obtain independent review before starting the domain pilot. + +## 2. Trusted GroupMe Domain Pilot + +- [ ] 2.1 Register two or three reviewed declarative operators for GroupMe page-ceiling and cursor-progress risks, with exact implementation preimages, permitted postimages, and immutable judge closure. +- [ ] 2.2 Implement independent no-hardlink disk-backed clones with isolated `HOME`, `TMPDIR`, XDG, pnpm, Git, accounting, and test paths; use copy-only dependency import semantics; pin and record Node/pnpm executables and dependency identity; preflight the exact clean commands. +- [ ] 2.3 Record focused clean and mutant GroupMe checks as adapter evidence; do not label them test-accounting authority receipts. +- [ ] 2.4 Run the clean complete `polyfill-connectors` backstop at each locked batch start; run the mutant backstop for every focused survivor; copy and revalidate the complete authority bundle outside the disposable clone before deletion; test evidence budgets, 30-day minimum retention, and the two-hour digest-identical reuse rule. +- [ ] 2.5 Implement the total projection table and no-automatic-retry rule; add fault-injection tests for preimage mismatch, forbidden path changes, immutable-judge changes, baseline/backstop absence, selector miss, timeout, direct-output limit, transcript/workspace soft-threshold overshoot, contradictory attempts, abandoned attempt, cleanup failure, missing retained bytes, and altered evidence. +- [ ] 2.6 Run two or three declared operators within the 10-minute locked batch and sequential local policy; capture raw execution axes, projections, selector misses, baseline reuse, runtime, setup time, artifact sizes, cleanup, and reviewer minutes. +- [ ] 2.7 Obtain independent triage for every survivor, likely-equivalent, uninteresting, or invalid-fault disposition. + +## 3. Decision Gate + +- [ ] 3.1 Publish metric definitions and raw pilot results, including invalid trials and excluded denominators rather than only percentages. +- [ ] 3.2 Stop or narrow on any cleanup failure, abandoned process, unexplained selector miss, authority mismatch, dominant setup cost, predominantly invalid or trivial operators, or insufficient useful evidence within budget. +- [ ] 3.3 Compare the migration and GroupMe adapters and identify any substantial repeated policy/evidence invariants; do not generalize superficial lifecycle differences. +- [ ] 3.4 Publish a continue, narrow, or stop memo. Require a new independently reviewed OpenSpec proposal before any shared coordinator, generic executor, StrykerJS experiment, CI scheduling, blocking gate, agent-generated mutant, or test-deletion automation.