Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@
change + normalized evidence + catalog + policy
|
v
check dependency-completeness assessment
|
v
deterministic verification planner
|
v
Expand All @@ -14,9 +17,11 @@ change + normalized evidence + catalog + policy

## Core and adapters

The core owns validation, reverse-impact closure, deterministic policy matching, fail-closed escalation, selection/skip arguments, canonical hashing, Visible Value planning counts, and shadow-result classification.
The core owns validation, reverse-impact closure, deterministic policy matching, check-level dependency-completeness decisions, fail-closed escalation, selection/skip arguments, canonical hashing, Visible Value planning counts including safety additions, and shadow-result classification.

Adapters may translate Git diffs, package/project graphs, imports, coverage, test selectors, CODEOWNERS, schemas, or declared critical boundaries into normalized input. The AV-EXP-003 bounded Python inspector classifies check-local alternate boundaries with source provenance; it remains benchmark-only. This prevents the core from embedding Python, Click, pytest-testmon, or any one repository layout.

Adapters may translate Git diffs, package/project graphs, imports, coverage, test selectors, CODEOWNERS, schemas, or declared critical boundaries into normalized input. V1 implements no production adapter. This prevents the core from embedding Nx, Turbo, Jest, Vitest, testmon, or any one repository layout.
Multiple absence sources do not become complete by agreement. Static graph and native selector omission can coexist while a subprocess, child interpreter, dynamic import, plugin/entry-point discovery, code-generation, runtime-loading, or declared reflection/registration boundary remains open. An open check-local boundary selects that check; it does not globally broaden unrelated checks.

The verification catalog is the planner's universe. It is not a scheduler. Commands are opaque identities; the planner never shells out to them.

Expand All @@ -32,7 +37,7 @@ Affected Verification decides **what should execute**. After execution, Context

## Decision Evidence and Trajectory boundaries

Decision Evidence may later validate change identity, evidence provenance, plan identity, selected/skip arguments, and execution-result bindings. Agent Trajectory Profiler may measure planned selections, actual executions, shadow comparisons, and observed latency/cost when genuinely recorded. V1 records compatible identities but creates no cross-repository package coupling.
Decision Evidence may later validate change identity, evidence provenance, plan identity, selected/skip arguments, and execution-result bindings. Agent Trajectory Profiler may measure planned selections, actual executions, shadow comparisons, and observed latency/cost when genuinely recorded. V2 records compatible identities but creates no cross-repository package coupling.

## Shadow observations

Expand Down
19 changes: 14 additions & 5 deletions BENCHMARK.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,13 @@

AV-EXP-001 completed the first preregistered real-repository `SHADOW`
calibration. AV-EXP-002 completed a second Python/pytest-testmon calibration
and exposed one AV selection miss. See
[benchmark/av-exp-001/REPORT.md](benchmark/av-exp-001/REPORT.md) and
[benchmark/av-exp-002/REPORT.md](benchmark/av-exp-002/REPORT.md) for the bounded
results and immutable artifact identities. The method below remains
authoritative for future runs.
and exposed one AV selection miss. AV-EXP-003 preregistered and evaluated the
opaque-dependency-boundary repair without changing that historical FAIL. See
[benchmark/av-exp-001/REPORT.md](benchmark/av-exp-001/REPORT.md),
[benchmark/av-exp-002/REPORT.md](benchmark/av-exp-002/REPORT.md), and
[benchmark/av-exp-003/REPORT.md](benchmark/av-exp-003/REPORT.md) for the bounded
calibrations, repair, regression matrix, and exact precision cost. The method
below remains authoritative for future runs.

## Central question

Expand Down Expand Up @@ -78,3 +80,10 @@ AV-EXP-002 tested Click at
arms selected 86 of 87 oracle-relevant checks and missed
`tests/test_imports.py::test_light_imports` for a subprocess/runtime-import
hazard. FULL remained authoritative, and the trust stage remains `SHADOW`.

AV-EXP-003 added check-level dependency-completeness evidence and a bounded
Python boundary inspector. The exact AV2-006 replay selected the formerly
missed check in both AV arms; ten adversarial cases and both frozen prior
corpora had zero repaired AV misses. The repair added 6 and 7 test executions
across the two AV-EXP-002 arms and 7 checks across the ten synthetic repair
cases. This is a SHADOW defect-repair result, not a safety or trust claim.
7 changes: 4 additions & 3 deletions LIMITATIONS.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,15 @@
# Limitations and non-goals

- The reusable prototype still consumes normalized evidence. AV-EXP-001 and AV-EXP-002 add narrow benchmark-only adapters for pinned Zustand and Click; neither is a general or production adapter.
- The reusable prototype still consumes normalized evidence. AV-EXP-001, AV-EXP-002, and AV-EXP-003 add narrow benchmark-only adapters and repair evidence; none is a general or production adapter.
- The component graph and catalog completeness flags are caller claims, not independently attested.
- The selection algorithm uses declared component scope and policy tags. It has no symbol/data-flow analysis, runtime coverage collection, weighted set cover, probabilistic model, or learned judgment.
- Commands are opaque identities. The project does not execute, schedule, cache, distribute, retry, or report CI work.
- `SUFFICIENT_*` means sufficient under the supplied model and policy, not globally safe, formally sound, or mathematically minimal.
- Full catalog selection cannot compensate for an incomplete catalog; that state remains `INSUFFICIENT_EVIDENCE`.
- AV-EXP-001 and AV-EXP-002 derive shadow relevance reproducibly from changed outcomes in their frozen full catalogs. Other callers can still supply relevance, and two corpora do not validate every catalog or oracle.
- Test execution counts are declared catalog metadata. They are exact relative to input, not observed executions.
- Two preregistered real-project shadow benchmarks exist in JavaScript and Python. AV-EXP-002 exposed one AV miss for a runtime/subprocess import test that both the static graph and pytest-testmon omitted. There is no historical real-change replay, production-quality adapter, or independent qualifying replication.
- No production trust stage is justified. Both experiments remain `SHADOW`; AV-EXP-002's observed miss blocks a positive cross-ecosystem sufficiency claim.
- Two preregistered real-project shadow benchmarks exist in JavaScript and Python. AV-EXP-002 permanently remains a FAIL after exposing one runtime/subprocess import miss. AV-EXP-003 observed that plan v2 selects the known check and ten generalized repair cases, but does not erase the miss or solve dynamic dependencies generally.
- The bounded Python inspector detects a conservative subset of source-visible boundaries. It does not trace imports across child processes, prove plugin/reflection behavior, or infer closure without identified metadata.
- No production trust stage is justified. All experiments remain `SHADOW`; AV-EXP-003 is a defect repair and cannot authorize `TRUSTED_BOUNDED`.
- No time, token, monetary, correctness, failure-prevention, or causal savings claim is supported.
- Context Firewall, Decision Evidence, Agent Trajectory Profiler, Gearbox, and Opsle Tasks are external consumers or validators, not dependencies.
8 changes: 5 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Affected Verification deterministically selects the smallest verification worklo

The operative claim is **minimum defensible verification**, not mathematical global minimality. Unknown impact is never permission to skip work.

This repository contains a dependency-free Node.js 20 prototype. It consumes normalized change, impact, verification-catalog, and policy data and emits an `opsle.affected-verification.plan.v1` argument containing selected checks, skipped checks, exact reasons, provenance hashes, uncertainty, escalation, and sufficiency. It plans work; it does not run CI.
This repository contains a dependency-free Node.js 20 prototype. It consumes normalized change, impact, check-level dependency-completeness, verification-catalog, and policy data and emits an `opsle.affected-verification.plan.v2` argument containing selected checks, skipped checks, boundary evidence, exact reasons, provenance hashes, uncertainty, escalation, and sufficiency. It plans work; it does not run CI.

## Try it

Expand All @@ -28,12 +28,14 @@ npm run verify
- [BENCHMARK.md](BENCHMARK.md) — controlled research method and calibration results
- [benchmark/av-exp-001/REPORT.md](benchmark/av-exp-001/REPORT.md) — preregistered real-repository shadow calibration
- [benchmark/av-exp-002/REPORT.md](benchmark/av-exp-002/REPORT.md) — cross-ecosystem Python shadow calibration and observed AV miss
- [benchmark/av-exp-003/REPORT.md](benchmark/av-exp-003/REPORT.md) — opaque dependency boundary repair, adversarial corpus, regression matrix, and precision cost
- [LIMITATIONS.md](LIMITATIONS.md) — current claim ceiling and non-goals
- [fixtures/scenarios.js](fixtures/scenarios.js) and [fixtures/negative-cases.js](fixtures/negative-cases.js) — twelve positive/boundary scenarios plus explicit conflicting, malformed, impossible, and tampered cases
- [schemas/plan-v1.schema.json](schemas/plan-v1.schema.json) — plan shape
- [schemas/plan-v2.schema.json](schemas/plan-v2.schema.json) — current check-level dependency-completeness plan shape
- [schemas/plan-v1.schema.json](schemas/plan-v1.schema.json) — immutable historical plan shape

## Status

This is a narrow research prototype, not a trusted replacement for full verification. AV-EXP-001 observed no AV miss in its frozen JavaScript corpus; AV-EXP-002 observed one AV miss in its frozen Python corpus. Both remain `SHADOW`, do not establish general safety, and do not provide production adapters.
This is a narrow research prototype, not a trusted replacement for full verification. AV-EXP-001 observed no AV miss in its frozen JavaScript corpus; AV-EXP-002 permanently observed one AV miss in its frozen Python corpus; AV-EXP-003 selected that known check under the repair and observed zero repaired misses in its generalized and frozen replay corpora. All remain `SHADOW`, do not establish general safety, and do not provide production adapters.

Apache-2.0.
30 changes: 16 additions & 14 deletions SPEC.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Affected Verification specification

Status: normative prototype contract
Plan identity: `opsle.affected-verification.plan.v1`
Input identity: `opsle.affected-verification.input.v1`
Plan identity: `opsle.affected-verification.plan.v2`
Input identity: `opsle.affected-verification.input.v2`

## 1. Canonical definition

Expand All @@ -15,17 +15,17 @@ Affected Verification deterministically selects the smallest verification worklo
One input object contains:

- a base revision, target revision, and nonempty changed-path list, with optional changed regions and explicit risk tags;
- versioned evidence providers, a component dependency graph, path-to-component impact claims, and an explicit completeness bit;
- versioned evidence providers, a component dependency graph, path-to-component impact claims, an explicit graph-completeness bit, and one check-level dependency-completeness assessment per catalog check;
- a complete-or-incomplete verification catalog;
- an identified, versioned policy containing deterministic matching and escalation rules.

A verification catalog entry has a stable check ID, check type, literal command identity, component scope, tags, and a nonnegative integer count of test executions when the entry is a test check. Optional cost metadata is descriptive only in v1 and never drives selection.

The prototype accepts normalized evidence. Git, Nx, Turbo, Jest, Vitest, testmon, coverage, CODEOWNERS, schema tools, and source-graph integrations belong in replaceable adapters that produce this format.
Each check-level assessment declares mechanisms, boundary records, evidence references, provenance, an explanation, and one of `COMPLETE_FOR_CHECK`, `COMPLETE_WITH_DECLARED_BOUNDARIES`, `INCOMPLETE`, `OPAQUE_BOUNDARY`, or `UNKNOWN`. Evidence agreement is separate from evidence completeness. Git, Nx, Turbo, Jest, Vitest, testmon, coverage, CODEOWNERS, schema tools, and source-graph integrations belong in replaceable adapters that produce this normalized format.

## 3. Verification is broader than tests

V1 recognizes unit, integration, end-to-end, lint, typecheck, build, compiler, schema, migration, API-contract, security, snapshot, visual, smoke, release, documentation, and test-infrastructure checks, plus an explicit `other` type. A check is an indivisible catalog entry in v1. The planner does not schedule or execute it.
V2 recognizes unit, integration, end-to-end, lint, typecheck, build, compiler, schema, migration, API-contract, security, snapshot, visual, smoke, release, documentation, and test-infrastructure checks, plus an explicit `other` type. A check is an indivisible catalog entry. The planner does not schedule or execute it.

## 4. Deterministic planning

Expand All @@ -36,13 +36,14 @@ For valid input, the reference planner:
3. matches identified policy rules;
4. selects checks whose declared scope intersects the affected closure;
5. adds all checks carrying tags required by matched policy rules and refuses sufficiency if any required tag has no catalogued check;
6. broadens to the complete catalog when impact is unknown or incomplete;
7. classifies every remaining catalog check as skipped with an evidence-backed reason;
8. hashes canonical input and canonical plan content with SHA-256.
6. selects a check when positive evidence identifies it or its dependency completeness is `INCOMPLETE`, `OPAQUE_BOUNDARY`, or `UNKNOWN`;
7. broadens to the complete catalog when global impact is unknown or incomplete;
8. skips only checks whose dependency mechanisms and boundaries are complete enough for that decision;
9. hashes canonical input and canonical plan content with SHA-256.

Inputs, checks, providers, rules, components, dependencies, selections, skips, and reasons are sorted where order is not semantic. The same input yields byte-equivalent canonical output and the same plan identity.

V1 does not solve weighted set cover and does not choose among alternative commands. If a future catalog expresses equally sufficient alternatives, its policy must define a deterministic tie-break or the planner must report ambiguity. It must not silently use ambient timing or model judgment.
V2 does not solve weighted set cover and does not choose among alternative commands. If a future catalog expresses equally sufficient alternatives, its policy must define a deterministic tie-break or the planner must report ambiguity. It must not silently use ambient timing or model judgment.

## 5. Sufficiency and escalation states

Expand All @@ -61,11 +62,11 @@ V1 does not solve weighted set cover and does not choose among alternative comma
`INVALID_INPUT`
: The input is malformed, contradictory, numerically impossible, references unknown entities, duplicates identity-bearing evidence, or is invalidated by policy. CLI errors use `opsle.affected-verification.error.v1`, exit 2, and emit no success indicator or value receipt. Policy invalidation uses the more specific code `PLAN_INVALIDATED` while retaining the `INVALID_INPUT` semantics.

The invariant is `UNKNOWN != SAFE TO SKIP`. Unknown or incomplete impact selects the full known catalog. If that catalog is incomplete, the result remains `INSUFFICIENT_EVIDENCE` rather than pretending the known full set is sufficient.
The invariants are `UNKNOWN != SAFE TO SKIP` and `NO PROOF OF COMPLETENESS != PROOF OF NO DEPENDENCY`. Unknown or incomplete global impact selects the full known catalog. Unknown, incomplete, or opaque evidence local to one otherwise unrelated check selects that check; it does not imply FULL unless catalog or policy evidence independently requires it.

## 6. Selection and skip arguments

Every selected check contains one or more reason records with a stable code, exact detail, and evidence references. Every skipped check must do the same. V1 emits `OUTSIDE_TRANSITIVE_IMPACT_SET` only when the scope misses the complete affected closure and no matched policy requires the check.
Every selected check contains one or more reason records with a stable code, exact detail, and evidence references. Every skipped check must do the same and expose a complete check-level dependency argument. V2 emits `OUTSIDE_TRANSITIVE_IMPACT_SET` only when scope misses the complete affected closure, no matched policy requires the check, and dependency completeness independently permits the skip.

“Not selected” is not a reason. An implementation must fail conformance if a skipped check has no reason.

Expand All @@ -77,21 +78,22 @@ The plan contains:
- base, target, change identity, paths, and supported changed regions;
- input, evidence, catalog, policy, and provider identities;
- direct and reverse-dependent affected components;
- check-level mechanisms, boundaries, completeness, evidence coverage, provenance, action, and forced-selection state;
- selected and skipped checks with rationale;
- risk tags and matched policy rules;
- uncertainty and escalation states;
- sufficiency classification and the bounded claim.

The exact structural profile is [schemas/plan-v1.schema.json](schemas/plan-v1.schema.json). `plan_identity` is SHA-256 over canonical plan content with that field omitted. `change.identity` is caller supplied or deterministically derived. Provider identities are recorded, not independently attested.
The exact structural profile is [schemas/plan-v2.schema.json](schemas/plan-v2.schema.json). Plan v1 remains historical and is not rewritten. `plan_identity` is SHA-256 over canonical plan content with that field omitted. `change.identity` is caller supplied or deterministically derived. Provider identities are recorded, not independently attested.

## 8. Visible Value

The planner can emit an `opsle.value-receipt.v1` sidecar. Exact measurements are catalog checks available, selected, and skipped; declared test executions available, selected, and skipped; and the rational test-execution reduction. Unlike verification types are not collapsed into a workload percentage. Counts are planning output, not observed execution.

The receipt explicitly makes no time, token, cost, correctness, or causal savings claim. Canonical plan JSON stays on stdout and the named operator indicator stays on stderr.
The receipt also reports exact check and test-execution additions caused by dependency-safety forced selection. It explicitly makes no time, token, cost, correctness, failure-prevention, or causal savings claim. Canonical plan JSON stays on stdout and the named operator indicator stays on stderr.

## 9. Failure behavior and compatibility

Unknown fields are rejected in the normalized input. Duplicate provider, component, impact, check, rule, executed-check, or failure identities are rejected. Unknown graph edges, scopes, and shadow checks are rejected. Negative, noninteger, unsafe-integer, nonfinite, or semantically incompatible numeric values are rejected. Nonempty matcher dimensions within one policy rule compose conjunctively; values inside one dimension are alternatives.

V1 compatibility is exact by schema identity. Additive or semantic changes require a new schema version or a documented compatible profile. No external package or repository is a runtime dependency.
Compatibility is exact by schema identity. Plan/input v2 is the opaque-boundary safety repair; v1 artifacts remain immutable historical evidence. No external package or repository is a runtime dependency.
Loading
Loading