Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
144 changes: 144 additions & 0 deletions .omo/evidence/20260727-v24r11-source-apply-handoff/QA.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
# V24R11 Ordinary Source Apply Handoff QA

## Scope and boundary

V24R11 changes only the Legal Coverage projection of a newly validated ordinary
source proposal. It adds one replay-safe handoff identity so the existing Core
Progress Lease can grant one model turn in which the Agent receives the exact
`source-merge-apply` command.

It does not change Agent Core, O1, validator acceptance, Progress Lease
thresholds `8/2`, Router, Memory, model configuration, corpus, source apply
execution, durable receipts, repair behavior, matrix or authority protocols,
deadlines, or completion authority. Apply readiness remains non-semantic;
verified V24R9 receipts remain the only ordinary source progress identity.

## Counterexample-first verification

The source workflow regression was added before the implementation. Against
V24R10 it failed only at the new handoff assertion:

```text
Expected values to be strictly equal:
0 !== 1
```

Artifact: `red-counterexample.log`, SHA-256
`7705db07e30d8b2926ce1512ffb4d55307b10f9714db2a234c9df547f1e84db3`.

After the 15-line Legal Coverage checkpoint implementation, the same test
passed. Artifact: `green-counterexample.log`, SHA-256
`3dca49fa89146cdeda92dc1e550cbe9c6c61838489f71040cfbf9c7da0eb2a54`.

Why enough: the before/after holds validators, receipts, Core, Lease, test data,
and commands constant. The only changed runtime behavior is the missing
validated-proposal handoff identity.

## Static and patch verification

Commands:

```sh
node --check products/legal/plugins/legal-coverage/hook.mjs
node --check products/legal/plugins/legal-coverage/scripts/legal-coverage.mjs
node --check products/legal/plugins/legal-coverage/scripts/lib/legal-coverage.mjs
node --check .omo/evidence/20260727-v24r11-source-apply-handoff/replay-preserved-case09.mjs
git diff --check
```

Observed: every command exited `0`; `git diff --check` emitted no diagnostics.

## Focused Legal Coverage, Gateway, Lease, and O1 suite

Command:

```sh
node --test --test-force-exit --test-timeout 60000 \
dist/tests/products/legal-coverage.spec.js \
dist/tests/agent/legal-coverage-plugin-runtime.spec.js \
dist/tests/agent/progress-lease.spec.js \
dist/tests/agent/agent-loop-runtime-controls.spec.js \
dist/tests/observability/recorder.spec.js \
dist/tests/observability/local-gateway-progress-handoff.spec.js
```

Observed: `85/85` passed with no failures, cancellations, skips, or todos.
Artifact: `focused-legal-gateway.log`, SHA-256
`cfe8da9140d85ad15728caec45c110eec9a578dad1a7b4d9808af9b87c9979bc`.

The product regression proves invalid proposal state advances no handoff; valid
ordinary apply readiness advances exactly once; replay advances zero; the exact
command remains stable; successful apply advances only progress through the
durable receipt; and applied-state replay advances neither ordinal.

The real local Gateway regression drives proposal write, exact apply, receipt
observation, and completion. It observes:

```text
baseline -> handoff_grace -> renewed -> completed
progressOrdinal: 0 -> 0 -> 1 -> 2
handoffOrdinal: 0 -> 1 -> 1 -> 1
```

Router, Memory, and telemetry are disabled. O1 reports complete model, tool,
and turn pairing, two tool starts and completions, and zero dropped events.

## Preserved V24R10 Case 09 replay

The reviewer-readable driver `replay-preserved-case09.mjs` copies the immutable
V24R10 Gate V2 Case 09 workspace into a disposable directory and replaces only
the copied Legal Coverage plugin. The original campaign is never mutated.

The replay temporarily removes and then byte-for-byte restores the already
validated proposal to establish a same-session pre-handoff baseline. It then
drives the real hook and exact injected command through apply, durable receipt,
and replay.

Observed:

```text
proposal: source-merge-d2979ab3c066.json, 15,355 bytes
handoffOrdinal: 0 -> 1 -> 1 -> 1 -> 1
progressOrdinal: 0 -> 0 -> 0 -> 1 -> 1
applied mutation: 4 sources, 20 facts
durable receipt: source-merge-applied-d2979ab3c066.json
next work group: source-fragment-merge
input tree: unchanged
```

Artifact: `case09-replay-result.json`, SHA-256
`50894e1f6c08cffbc9b14380f831450cd7e9325fec2a977afa28964dd48edeaf`.

Why enough: this is the exact state hash, proposal hash and bytes, source IDs,
and command from the failed 277-second product run. It proves the missing
handoff at the actual dynamic hook boundary, while the unchanged durable
receipt remains the semantic renewal authority.

## Complete repository suite

Command:

```sh
npm test
```

Observed: build plus `309/309` tests passed with no failures, cancellations,
skips, or todos. Artifact: `full-suite.log`, SHA-256
`bfc02836bf9fe56cf859e2267a028956bd343b92e7b13f6935bf07eb37a34d34`.

Why enough: the complete run covers every repository component against this
candidate, including Core Lease and compaction, Gateway, O1, source repair,
ordinary receipts, matrices, authorities, and non-legal behavior. No Core
implementation file changed.

## Omitted and residual risk

No API key, provider URL, authorization header, environment dump, private
source content, report text, prompt text, or model reasoning is included. The
replay result retains only hashes, counts, stable artifact basenames, source
identifiers, and ordinal transitions.

This evidence does not prove that the production model will complete Case 09.
V24R11 is protocol-, Gateway-, O1-, full-suite-, and exact-failure-replay
verified. A fresh immutable campaign must still pass Gate 0, paired smoke,
Case 05, and complete Case 09 before V25 or the 85-case campaign is authorized.
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
{
"passed": true,
"fixture": "preserved-v24r10-case09-valid-source-proposal",
"proposal": "source-merge-d2979ab3c066.json",
"proposalBytes": 15355,
"proposalSha256": "ae2542ae4844585fa77ab54debbf896fbc693fb959b8fa888631bfa0608afb37",
"sourceIds": [
"SRC-99DF1C894092",
"SRC-C2260EF3FDB3",
"SRC-125B91D6E09F",
"SRC-E6DBDE335351"
],
"handoffOrdinal": [
0,
1,
1,
1,
1
],
"progressOrdinal": [
0,
0,
0,
1,
1
],
"appliedSourceCount": 4,
"appliedFactCount": 20,
"stateHashBefore": "6473f1f29eb346dbf0219bbf8c39e75ce143e9555c3d2e3a26fdef51360d8cc8",
"stateHashAfter": "6cfcb3d840075247cdb0295c3240838db6fd69b7f5e349544d6afd0a60cf7410",
"durableReceipt": "source-merge-applied-d2979ab3c066.json",
"nextGroup": "source-fragment-merge",
"inputTreeSha256": "9b63090fe73ccafcd28e77b024759f8acf774409bc23045bad65acbba001980b"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,198 @@
import assert from "node:assert/strict";
import { createHash } from "node:crypto";
import { execFile, spawn } from "node:child_process";
import { cp, mkdir, mkdtemp, readFile, readdir, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { basename, join, resolve } from "node:path";
import { promisify } from "node:util";

const execFileAsync = promisify(execFile);
const sourceRun = resolve(process.argv[2] ?? "");
const candidateRoot = resolve(process.argv[3] ?? "");
const outputPath = resolve(process.argv[4] ?? "");

assert.ok(process.argv[2] && process.argv[3] && process.argv[4],
"usage: replay-preserved-case09.mjs <preserved-workspace> <candidate-root> <output-json>");

const workspace = await mkdtemp(join(tmpdir(), "pilotdeck-v24r11-case09-replay-"));
const pluginRoot = join(workspace, ".pilotdeck", "plugins", "legal-coverage");
const candidatePlugin = join(candidateRoot, "products", "legal", "plugins", "legal-coverage");
const hook = join(pluginRoot, "hook.mjs");

try {
await cp(sourceRun, workspace, { recursive: true });
await rm(pluginRoot, { recursive: true, force: true });
await cp(candidatePlugin, pluginRoot, { recursive: true });

const inputHashBefore = await treeHash(join(workspace, ".pilotdeck", "inputs"));
const inspection = await runHook({
hookEventName: "PreModelRequest",
sessionId: "v24r11-case09-inspection",
transcriptPath: "",
cwd: workspace,
});
const inspectionEnvelope = legalEnvelope(inspection);
assert.equal(inspectionEnvelope.workItems.group, "source-fragment-apply");
assert.equal(inspectionEnvelope.workItems.proposal.validated, true);
assert.match(inspectionEnvelope.sourceMergeApplyCommand, /source-merge-apply/u);

const proposalRelativePath = inspectionEnvelope.workItems.proposal.path;
const proposalPath = join(workspace, proposalRelativePath);
const proposalBytes = await readFile(proposalPath);
const proposalSha256 = sha256(proposalBytes);
assert.equal(proposalSha256, inspectionEnvelope.workItems.proposal.proposalSha256);
await rm(proposalPath);

const sessionId = "v24r11-preserved-case09-source-handoff";
const baseline = await runHook({
hookEventName: "PreModelRequest",
sessionId,
transcriptPath: "",
cwd: workspace,
});
assert.equal(legalEnvelope(baseline).workItems.group, "source-fragment-propose");
const baselineConvergence = convergence(baseline);

await writeFile(proposalPath, proposalBytes);
const applyReady = await runHook({
hookEventName: "PreModelRequest",
sessionId,
transcriptPath: "",
cwd: workspace,
});
const applyEnvelope = legalEnvelope(applyReady);
const applyConvergence = convergence(applyReady);
assert.equal(applyEnvelope.workItems.group, "source-fragment-apply");
assert.equal(applyEnvelope.workItems.proposal.validated, true);
assert.equal(applyConvergence.progressOrdinal, baselineConvergence.progressOrdinal);
assert.equal(applyConvergence.handoffOrdinal, baselineConvergence.handoffOrdinal + 1);
assert.equal(applyEnvelope.sourceMergeApplyCommand, inspectionEnvelope.sourceMergeApplyCommand);

const replayReady = await runHook({
hookEventName: "PreModelRequest",
sessionId,
transcriptPath: "",
cwd: workspace,
});
assert.equal(convergence(replayReady).progressOrdinal, applyConvergence.progressOrdinal);
assert.equal(convergence(replayReady).handoffOrdinal, applyConvergence.handoffOrdinal);
assert.equal(legalEnvelope(replayReady).sourceMergeApplyCommand, applyEnvelope.sourceMergeApplyCommand);

const applied = await execFileAsync("/bin/zsh", ["-lc", applyEnvelope.sourceMergeApplyCommand], {
cwd: workspace,
encoding: "utf8",
});
const appliedResult = JSON.parse(applied.stdout);
assert.equal(appliedResult.applied, true);

const afterApply = await runHook({
hookEventName: "PreModelRequest",
sessionId,
transcriptPath: "",
cwd: workspace,
});
const afterEnvelope = legalEnvelope(afterApply);
const afterConvergence = convergence(afterApply);
assert.ok(afterEnvelope.workItems.appliedSource);
assert.equal(afterConvergence.progressOrdinal, applyConvergence.progressOrdinal + 1);
assert.equal(afterConvergence.handoffOrdinal, applyConvergence.handoffOrdinal);

const replayApplied = await runHook({
hookEventName: "PreModelRequest",
sessionId,
transcriptPath: "",
cwd: workspace,
});
const replayAppliedConvergence = convergence(replayApplied);
assert.equal(replayAppliedConvergence.progressOrdinal, afterConvergence.progressOrdinal);
assert.equal(replayAppliedConvergence.handoffOrdinal, afterConvergence.handoffOrdinal);
assert.equal(sha256(await readFile(proposalPath)), proposalSha256);
const inputHashAfter = await treeHash(join(workspace, ".pilotdeck", "inputs"));
assert.equal(inputHashAfter, inputHashBefore);

const result = {
passed: true,
fixture: "preserved-v24r10-case09-valid-source-proposal",
proposal: basename(proposalRelativePath),
proposalBytes: proposalBytes.byteLength,
proposalSha256,
sourceIds: applyEnvelope.workItems.proposal.sourceIds,
handoffOrdinal: [
baselineConvergence.handoffOrdinal,
applyConvergence.handoffOrdinal,
convergence(replayReady).handoffOrdinal,
afterConvergence.handoffOrdinal,
replayAppliedConvergence.handoffOrdinal,
],
progressOrdinal: [
baselineConvergence.progressOrdinal,
applyConvergence.progressOrdinal,
convergence(replayReady).progressOrdinal,
afterConvergence.progressOrdinal,
replayAppliedConvergence.progressOrdinal,
],
appliedSourceCount: appliedResult.sourceCount,
appliedFactCount: appliedResult.factCount,
stateHashBefore: applyEnvelope.workItems.proposal.expectedStateHash,
stateHashAfter: appliedResult.stateHash,
durableReceipt: basename(afterEnvelope.workItems.appliedSource.path),
nextGroup: afterEnvelope.workItems.group,
inputTreeSha256: inputHashAfter,
};
await mkdir(resolve(outputPath, ".."), { recursive: true });
await writeJson(outputPath, result);
process.stdout.write(`${JSON.stringify(result, null, 2)}\n`);
} finally {
await rm(workspace, { recursive: true, force: true });
}

function sha256(bytes) {
return createHash("sha256").update(bytes).digest("hex");
}

async function treeHash(root) {
const entries = [];
async function visit(directory, prefix = "") {
for (const entry of (await readdir(directory, { withFileTypes: true }))
.sort((left, right) => left.name.localeCompare(right.name))) {
const relativePath = prefix ? `${prefix}/${entry.name}` : entry.name;
const absolutePath = join(directory, entry.name);
if (entry.isDirectory()) await visit(absolutePath, relativePath);
else if (entry.isFile()) entries.push([relativePath, sha256(await readFile(absolutePath))]);
}
}
await visit(root);
return sha256(Buffer.from(JSON.stringify(entries)));
}

function legalEnvelope(output) {
return JSON.parse((output.hookSpecificOutput.additionalContext ?? "")
.replace(/^<legal_coverage_state>\n/u, "")
.replace(/\n<\/legal_coverage_state>$/u, ""));
}

function convergence(output) {
return output.hookSpecificOutput.modelRequestPatch.metadata.pilotdeckConvergence;
}

async function runHook(input) {
return new Promise((resolvePromise, reject) => {
const child = spawn(process.execPath, [hook], { stdio: ["pipe", "pipe", "pipe"] });
let stdout = "";
let stderr = "";
child.stdout.setEncoding("utf8");
child.stderr.setEncoding("utf8");
child.stdout.on("data", (chunk) => { stdout += chunk; });
child.stderr.on("data", (chunk) => { stderr += chunk; });
child.on("error", reject);
child.on("close", (code) => {
if (code !== 0) reject(new Error(stderr || `hook exited with code ${code}`));
else resolvePromise(JSON.parse(stdout));
});
child.stdin.end(JSON.stringify(input));
});
}

async function writeJson(path, value) {
await writeFile(path, `${JSON.stringify(value, null, 2)}\n`);
}
Loading