diff --git a/.omo/evidence/20260727-v24r11-source-apply-handoff/QA.md b/.omo/evidence/20260727-v24r11-source-apply-handoff/QA.md new file mode 100644 index 00000000..dd949652 --- /dev/null +++ b/.omo/evidence/20260727-v24r11-source-apply-handoff/QA.md @@ -0,0 +1,144 @@ +# V24R11 Ordinary Source Apply Handoff QA + +## Scope and boundary + +V24R11 changes only the Legal Coverage projection of a newly validated ordinary +source proposal. It adds one replay-safe handoff identity so the existing Core +Progress Lease can grant one model turn in which the Agent receives the exact +`source-merge-apply` command. + +It does not change Agent Core, O1, validator acceptance, Progress Lease +thresholds `8/2`, Router, Memory, model configuration, corpus, source apply +execution, durable receipts, repair behavior, matrix or authority protocols, +deadlines, or completion authority. Apply readiness remains non-semantic; +verified V24R9 receipts remain the only ordinary source progress identity. + +## Counterexample-first verification + +The source workflow regression was added before the implementation. Against +V24R10 it failed only at the new handoff assertion: + +```text +Expected values to be strictly equal: +0 !== 1 +``` + +Artifact: `red-counterexample.log`, SHA-256 +`7705db07e30d8b2926ce1512ffb4d55307b10f9714db2a234c9df547f1e84db3`. + +After the 15-line Legal Coverage checkpoint implementation, the same test +passed. Artifact: `green-counterexample.log`, SHA-256 +`3dca49fa89146cdeda92dc1e550cbe9c6c61838489f71040cfbf9c7da0eb2a54`. + +Why enough: the before/after holds validators, receipts, Core, Lease, test data, +and commands constant. The only changed runtime behavior is the missing +validated-proposal handoff identity. + +## Static and patch verification + +Commands: + +```sh +node --check products/legal/plugins/legal-coverage/hook.mjs +node --check products/legal/plugins/legal-coverage/scripts/legal-coverage.mjs +node --check products/legal/plugins/legal-coverage/scripts/lib/legal-coverage.mjs +node --check .omo/evidence/20260727-v24r11-source-apply-handoff/replay-preserved-case09.mjs +git diff --check +``` + +Observed: every command exited `0`; `git diff --check` emitted no diagnostics. + +## Focused Legal Coverage, Gateway, Lease, and O1 suite + +Command: + +```sh +node --test --test-force-exit --test-timeout 60000 \ + dist/tests/products/legal-coverage.spec.js \ + dist/tests/agent/legal-coverage-plugin-runtime.spec.js \ + dist/tests/agent/progress-lease.spec.js \ + dist/tests/agent/agent-loop-runtime-controls.spec.js \ + dist/tests/observability/recorder.spec.js \ + dist/tests/observability/local-gateway-progress-handoff.spec.js +``` + +Observed: `85/85` passed with no failures, cancellations, skips, or todos. +Artifact: `focused-legal-gateway.log`, SHA-256 +`cfe8da9140d85ad15728caec45c110eec9a578dad1a7b4d9808af9b87c9979bc`. + +The product regression proves invalid proposal state advances no handoff; valid +ordinary apply readiness advances exactly once; replay advances zero; the exact +command remains stable; successful apply advances only progress through the +durable receipt; and applied-state replay advances neither ordinal. + +The real local Gateway regression drives proposal write, exact apply, receipt +observation, and completion. It observes: + +```text +baseline -> handoff_grace -> renewed -> completed +progressOrdinal: 0 -> 0 -> 1 -> 2 +handoffOrdinal: 0 -> 1 -> 1 -> 1 +``` + +Router, Memory, and telemetry are disabled. O1 reports complete model, tool, +and turn pairing, two tool starts and completions, and zero dropped events. + +## Preserved V24R10 Case 09 replay + +The reviewer-readable driver `replay-preserved-case09.mjs` copies the immutable +V24R10 Gate V2 Case 09 workspace into a disposable directory and replaces only +the copied Legal Coverage plugin. The original campaign is never mutated. + +The replay temporarily removes and then byte-for-byte restores the already +validated proposal to establish a same-session pre-handoff baseline. It then +drives the real hook and exact injected command through apply, durable receipt, +and replay. + +Observed: + +```text +proposal: source-merge-d2979ab3c066.json, 15,355 bytes +handoffOrdinal: 0 -> 1 -> 1 -> 1 -> 1 +progressOrdinal: 0 -> 0 -> 0 -> 1 -> 1 +applied mutation: 4 sources, 20 facts +durable receipt: source-merge-applied-d2979ab3c066.json +next work group: source-fragment-merge +input tree: unchanged +``` + +Artifact: `case09-replay-result.json`, SHA-256 +`50894e1f6c08cffbc9b14380f831450cd7e9325fec2a977afa28964dd48edeaf`. + +Why enough: this is the exact state hash, proposal hash and bytes, source IDs, +and command from the failed 277-second product run. It proves the missing +handoff at the actual dynamic hook boundary, while the unchanged durable +receipt remains the semantic renewal authority. + +## Complete repository suite + +Command: + +```sh +npm test +``` + +Observed: build plus `309/309` tests passed with no failures, cancellations, +skips, or todos. Artifact: `full-suite.log`, SHA-256 +`bfc02836bf9fe56cf859e2267a028956bd343b92e7b13f6935bf07eb37a34d34`. + +Why enough: the complete run covers every repository component against this +candidate, including Core Lease and compaction, Gateway, O1, source repair, +ordinary receipts, matrices, authorities, and non-legal behavior. No Core +implementation file changed. + +## Omitted and residual risk + +No API key, provider URL, authorization header, environment dump, private +source content, report text, prompt text, or model reasoning is included. The +replay result retains only hashes, counts, stable artifact basenames, source +identifiers, and ordinal transitions. + +This evidence does not prove that the production model will complete Case 09. +V24R11 is protocol-, Gateway-, O1-, full-suite-, and exact-failure-replay +verified. A fresh immutable campaign must still pass Gate 0, paired smoke, +Case 05, and complete Case 09 before V25 or the 85-case campaign is authorized. diff --git a/.omo/evidence/20260727-v24r11-source-apply-handoff/case09-replay-result.json b/.omo/evidence/20260727-v24r11-source-apply-handoff/case09-replay-result.json new file mode 100644 index 00000000..31ab8da7 --- /dev/null +++ b/.omo/evidence/20260727-v24r11-source-apply-handoff/case09-replay-result.json @@ -0,0 +1,34 @@ +{ + "passed": true, + "fixture": "preserved-v24r10-case09-valid-source-proposal", + "proposal": "source-merge-d2979ab3c066.json", + "proposalBytes": 15355, + "proposalSha256": "ae2542ae4844585fa77ab54debbf896fbc693fb959b8fa888631bfa0608afb37", + "sourceIds": [ + "SRC-99DF1C894092", + "SRC-C2260EF3FDB3", + "SRC-125B91D6E09F", + "SRC-E6DBDE335351" + ], + "handoffOrdinal": [ + 0, + 1, + 1, + 1, + 1 + ], + "progressOrdinal": [ + 0, + 0, + 0, + 1, + 1 + ], + "appliedSourceCount": 4, + "appliedFactCount": 20, + "stateHashBefore": "6473f1f29eb346dbf0219bbf8c39e75ce143e9555c3d2e3a26fdef51360d8cc8", + "stateHashAfter": "6cfcb3d840075247cdb0295c3240838db6fd69b7f5e349544d6afd0a60cf7410", + "durableReceipt": "source-merge-applied-d2979ab3c066.json", + "nextGroup": "source-fragment-merge", + "inputTreeSha256": "9b63090fe73ccafcd28e77b024759f8acf774409bc23045bad65acbba001980b" +} diff --git a/.omo/evidence/20260727-v24r11-source-apply-handoff/replay-preserved-case09.mjs b/.omo/evidence/20260727-v24r11-source-apply-handoff/replay-preserved-case09.mjs new file mode 100644 index 00000000..5c399331 --- /dev/null +++ b/.omo/evidence/20260727-v24r11-source-apply-handoff/replay-preserved-case09.mjs @@ -0,0 +1,198 @@ +import assert from "node:assert/strict"; +import { createHash } from "node:crypto"; +import { execFile, spawn } from "node:child_process"; +import { cp, mkdir, mkdtemp, readFile, readdir, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { basename, join, resolve } from "node:path"; +import { promisify } from "node:util"; + +const execFileAsync = promisify(execFile); +const sourceRun = resolve(process.argv[2] ?? ""); +const candidateRoot = resolve(process.argv[3] ?? ""); +const outputPath = resolve(process.argv[4] ?? ""); + +assert.ok(process.argv[2] && process.argv[3] && process.argv[4], + "usage: replay-preserved-case09.mjs "); + +const workspace = await mkdtemp(join(tmpdir(), "pilotdeck-v24r11-case09-replay-")); +const pluginRoot = join(workspace, ".pilotdeck", "plugins", "legal-coverage"); +const candidatePlugin = join(candidateRoot, "products", "legal", "plugins", "legal-coverage"); +const hook = join(pluginRoot, "hook.mjs"); + +try { + await cp(sourceRun, workspace, { recursive: true }); + await rm(pluginRoot, { recursive: true, force: true }); + await cp(candidatePlugin, pluginRoot, { recursive: true }); + + const inputHashBefore = await treeHash(join(workspace, ".pilotdeck", "inputs")); + const inspection = await runHook({ + hookEventName: "PreModelRequest", + sessionId: "v24r11-case09-inspection", + transcriptPath: "", + cwd: workspace, + }); + const inspectionEnvelope = legalEnvelope(inspection); + assert.equal(inspectionEnvelope.workItems.group, "source-fragment-apply"); + assert.equal(inspectionEnvelope.workItems.proposal.validated, true); + assert.match(inspectionEnvelope.sourceMergeApplyCommand, /source-merge-apply/u); + + const proposalRelativePath = inspectionEnvelope.workItems.proposal.path; + const proposalPath = join(workspace, proposalRelativePath); + const proposalBytes = await readFile(proposalPath); + const proposalSha256 = sha256(proposalBytes); + assert.equal(proposalSha256, inspectionEnvelope.workItems.proposal.proposalSha256); + await rm(proposalPath); + + const sessionId = "v24r11-preserved-case09-source-handoff"; + const baseline = await runHook({ + hookEventName: "PreModelRequest", + sessionId, + transcriptPath: "", + cwd: workspace, + }); + assert.equal(legalEnvelope(baseline).workItems.group, "source-fragment-propose"); + const baselineConvergence = convergence(baseline); + + await writeFile(proposalPath, proposalBytes); + const applyReady = await runHook({ + hookEventName: "PreModelRequest", + sessionId, + transcriptPath: "", + cwd: workspace, + }); + const applyEnvelope = legalEnvelope(applyReady); + const applyConvergence = convergence(applyReady); + assert.equal(applyEnvelope.workItems.group, "source-fragment-apply"); + assert.equal(applyEnvelope.workItems.proposal.validated, true); + assert.equal(applyConvergence.progressOrdinal, baselineConvergence.progressOrdinal); + assert.equal(applyConvergence.handoffOrdinal, baselineConvergence.handoffOrdinal + 1); + assert.equal(applyEnvelope.sourceMergeApplyCommand, inspectionEnvelope.sourceMergeApplyCommand); + + const replayReady = await runHook({ + hookEventName: "PreModelRequest", + sessionId, + transcriptPath: "", + cwd: workspace, + }); + assert.equal(convergence(replayReady).progressOrdinal, applyConvergence.progressOrdinal); + assert.equal(convergence(replayReady).handoffOrdinal, applyConvergence.handoffOrdinal); + assert.equal(legalEnvelope(replayReady).sourceMergeApplyCommand, applyEnvelope.sourceMergeApplyCommand); + + const applied = await execFileAsync("/bin/zsh", ["-lc", applyEnvelope.sourceMergeApplyCommand], { + cwd: workspace, + encoding: "utf8", + }); + const appliedResult = JSON.parse(applied.stdout); + assert.equal(appliedResult.applied, true); + + const afterApply = await runHook({ + hookEventName: "PreModelRequest", + sessionId, + transcriptPath: "", + cwd: workspace, + }); + const afterEnvelope = legalEnvelope(afterApply); + const afterConvergence = convergence(afterApply); + assert.ok(afterEnvelope.workItems.appliedSource); + assert.equal(afterConvergence.progressOrdinal, applyConvergence.progressOrdinal + 1); + assert.equal(afterConvergence.handoffOrdinal, applyConvergence.handoffOrdinal); + + const replayApplied = await runHook({ + hookEventName: "PreModelRequest", + sessionId, + transcriptPath: "", + cwd: workspace, + }); + const replayAppliedConvergence = convergence(replayApplied); + assert.equal(replayAppliedConvergence.progressOrdinal, afterConvergence.progressOrdinal); + assert.equal(replayAppliedConvergence.handoffOrdinal, afterConvergence.handoffOrdinal); + assert.equal(sha256(await readFile(proposalPath)), proposalSha256); + const inputHashAfter = await treeHash(join(workspace, ".pilotdeck", "inputs")); + assert.equal(inputHashAfter, inputHashBefore); + + const result = { + passed: true, + fixture: "preserved-v24r10-case09-valid-source-proposal", + proposal: basename(proposalRelativePath), + proposalBytes: proposalBytes.byteLength, + proposalSha256, + sourceIds: applyEnvelope.workItems.proposal.sourceIds, + handoffOrdinal: [ + baselineConvergence.handoffOrdinal, + applyConvergence.handoffOrdinal, + convergence(replayReady).handoffOrdinal, + afterConvergence.handoffOrdinal, + replayAppliedConvergence.handoffOrdinal, + ], + progressOrdinal: [ + baselineConvergence.progressOrdinal, + applyConvergence.progressOrdinal, + convergence(replayReady).progressOrdinal, + afterConvergence.progressOrdinal, + replayAppliedConvergence.progressOrdinal, + ], + appliedSourceCount: appliedResult.sourceCount, + appliedFactCount: appliedResult.factCount, + stateHashBefore: applyEnvelope.workItems.proposal.expectedStateHash, + stateHashAfter: appliedResult.stateHash, + durableReceipt: basename(afterEnvelope.workItems.appliedSource.path), + nextGroup: afterEnvelope.workItems.group, + inputTreeSha256: inputHashAfter, + }; + await mkdir(resolve(outputPath, ".."), { recursive: true }); + await writeJson(outputPath, result); + process.stdout.write(`${JSON.stringify(result, null, 2)}\n`); +} finally { + await rm(workspace, { recursive: true, force: true }); +} + +function sha256(bytes) { + return createHash("sha256").update(bytes).digest("hex"); +} + +async function treeHash(root) { + const entries = []; + async function visit(directory, prefix = "") { + for (const entry of (await readdir(directory, { withFileTypes: true })) + .sort((left, right) => left.name.localeCompare(right.name))) { + const relativePath = prefix ? `${prefix}/${entry.name}` : entry.name; + const absolutePath = join(directory, entry.name); + if (entry.isDirectory()) await visit(absolutePath, relativePath); + else if (entry.isFile()) entries.push([relativePath, sha256(await readFile(absolutePath))]); + } + } + await visit(root); + return sha256(Buffer.from(JSON.stringify(entries))); +} + +function legalEnvelope(output) { + return JSON.parse((output.hookSpecificOutput.additionalContext ?? "") + .replace(/^\n/u, "") + .replace(/\n<\/legal_coverage_state>$/u, "")); +} + +function convergence(output) { + return output.hookSpecificOutput.modelRequestPatch.metadata.pilotdeckConvergence; +} + +async function runHook(input) { + return new Promise((resolvePromise, reject) => { + const child = spawn(process.execPath, [hook], { stdio: ["pipe", "pipe", "pipe"] }); + let stdout = ""; + let stderr = ""; + child.stdout.setEncoding("utf8"); + child.stderr.setEncoding("utf8"); + child.stdout.on("data", (chunk) => { stdout += chunk; }); + child.stderr.on("data", (chunk) => { stderr += chunk; }); + child.on("error", reject); + child.on("close", (code) => { + if (code !== 0) reject(new Error(stderr || `hook exited with code ${code}`)); + else resolvePromise(JSON.parse(stdout)); + }); + child.stdin.end(JSON.stringify(input)); + }); +} + +async function writeJson(path, value) { + await writeFile(path, `${JSON.stringify(value, null, 2)}\n`); +} diff --git a/docs/PILOTDECK_CONVERGENCE_V24R11_SOURCE_APPLY_HANDOFF.md b/docs/PILOTDECK_CONVERGENCE_V24R11_SOURCE_APPLY_HANDOFF.md new file mode 100644 index 00000000..59fe5082 --- /dev/null +++ b/docs/PILOTDECK_CONVERGENCE_V24R11_SOURCE_APPLY_HANDOFF.md @@ -0,0 +1,82 @@ +# PilotDeck Convergence V24R11: Ordinary Source Apply Handoff + +## Decision + +V24R11 closes one Legal Coverage protocol handoff gap observed in the V24R10 +Case 09 product gate. It does not change Agent Core, O1, validator acceptance, +Progress Lease thresholds `8/2`, Router, Memory, the model, the corpus, source +apply transactions, durable receipts, repair behavior, matrix selection, +authority closure, or completion authority. + +The failed run produced a valid, state-bound ordinary source proposal as its +final tool result. Legal Coverage exposed the exact `source-merge-apply` +command only on the next hook projection, but the Progress Lease evaluated the +same PostToolUse cycle before the Agent could receive that next model turn and +failed closed. The transaction was ready; the protocol did not acknowledge +the newly available handoff. + +## Protocol + +Legal Coverage adds one replay-safe handoff checkpoint when its derived work +item satisfies all of the following: + +- `group` is exactly `source-fragment-apply`; +- the unchanged proposal validator marked the proposal `validated: true`; +- `expectedStateHash` and `proposalSha256` are valid SHA-256 identities; and +- `sourceIds` is a non-empty, bounded list containing only exact non-empty IDs. + +The checkpoint identity is domain-owned and deterministic: + +```text +kind=source-fragment-apply-ready +expectedStateHash= +proposalSha256= +sourceIds= +``` + +A new identity advances the existing opaque `handoffOrdinal` exactly once. +Agent Core then applies its existing `handoff_grace`, allowing one model turn +to receive the already-generated exact apply command. Replaying the same +projection advances nothing. + +## Boundaries + +- No transaction is auto-applied. The Agent must execute the existing exact + command explicitly. +- Apply readiness is not semantic progress. `progressOrdinal` still advances + only from the verified durable V24R9 applied receipt. +- Invalid, stale, tampered, readiness-only, rejected, out-of-scope, or already + applied proposals do not create a handoff checkpoint. +- No new Lease decision, threshold, retry loop, deadline, or special-case + prompt is introduced. +- Core receives only the existing opaque ordinal and hashes. Legal proposal + fields and validation remain inside Legal Coverage. + +## Counterexamples + +Tests must prove that: + +- an invalid ordinary proposal does not advance `handoffOrdinal` or expose an + apply command; +- a valid state-bound proposal exposes the exact apply command and advances + handoff exactly once; +- replay of the apply-ready state advances neither handoff nor progress; +- stale, tampered, unknown-ID, or rejected proposal state does not advance; +- a successful apply writes the existing durable receipt and advances + semantic progress exactly once; +- applied-state replay advances neither ordinal; and +- repair, matrix, authority, Core Lease, O1, and ordinary non-legal activation + behavior remain unchanged. + +## Verification Gate + +1. Run the focused Legal Coverage, real-Gateway, Progress Lease, Agent runtime, + and O1 suites. +2. Replay the preserved V24R10 Case 09 final source state in a disposable + workspace and prove `valid proposal -> handoff once -> exact apply -> durable + receipt -> semantic progress once`. +3. Run the complete repository suite and patch hygiene checks. +4. Record reviewer-readable QA evidence, commit, push, and open a stacked draft + PR on V24R10. +5. Create a fresh immutable campaign and run Gate 0, paired smoke, Case 05, and + Case 09. V25 and the 85-case campaign remain blocked until Case 09 passes. diff --git a/products/legal/plugins/legal-coverage/hook.mjs b/products/legal/plugins/legal-coverage/hook.mjs index 199f3a35..bc47d3ca 100644 --- a/products/legal/plugins/legal-coverage/hook.mjs +++ b/products/legal/plugins/legal-coverage/hook.mjs @@ -419,6 +419,21 @@ function legalHandoffCheckpointDigest(workItems) { targetEntryId: workItems.proposal.targetEntryId, proposalSha256: workItems.proposal.proposalSha256, }; + } else if (workItems?.group === "source-fragment-apply" + && workItems.proposal?.validated === true + && validStateHash(workItems.proposal.expectedStateHash) + && validStateHash(workItems.proposal.proposalSha256) + && Array.isArray(workItems.proposal.sourceIds)) { + const sourceIds = workItems.proposal.sourceIds.filter(nonEmptyString).slice(0, 12).sort(); + if (sourceIds.length === 0 + || sourceIds.length !== workItems.proposal.sourceIds.length + || new Set(sourceIds).size !== sourceIds.length) return undefined; + checkpoint = { + kind: "source-fragment-apply-ready", + expectedStateHash: workItems.proposal.expectedStateHash, + proposalSha256: workItems.proposal.proposalSha256, + sourceIds, + }; } return checkpointDigest(checkpoint); } diff --git a/tests/agent/legal-coverage-plugin-runtime.spec.ts b/tests/agent/legal-coverage-plugin-runtime.spec.ts index 1b65c7be..6da37769 100644 --- a/tests/agent/legal-coverage-plugin-runtime.spec.ts +++ b/tests/agent/legal-coverage-plugin-runtime.spec.ts @@ -202,6 +202,89 @@ test("real gateway executes the state-bound legal authority closure with complet } }); +test("real gateway hands off a validated ordinary source proposal before renewing from its durable receipt", async () => { + const root = await mkdtemp(join(tmpdir(), "pilotdeck-legal-source-apply-gateway-")); + const projectRoot = join(root, "project"); + const pilotHome = join(root, "home"); + const installedPlugin = join(projectRoot, ".pilotdeck", "plugins", "legal-coverage"); + const requests: CanonicalModelRequest[] = []; + await mkdir(projectRoot, { recursive: true }); + await mkdir(pilotHome, { recursive: true }); + await cp(PLUGIN_ROOT, installedPlugin, { recursive: true }); + await writeFile(join(pilotHome, "pilotdeck.yaml"), SOURCE_APPLY_TEST_CONFIG); + await writeSourceApplyState(projectRoot); + + const runtime = createLocalGateway({ + projectRoot, + fallbackProjectRoot: projectRoot, + pilotHome, + env: { ...process.env, PILOT_HOME: pilotHome, PILOTDECK_BUILD_SHA: "source-apply-test" }, + __testModelFactory: () => sourceApplyModelRuntime(requests, projectRoot), + }); + try { + const events = []; + for await (const event of runtime.gateway.submitTurn({ + sessionKey: "legal-source-apply-session", + channelKey: "test", + projectKey: projectRoot, + message: "Continue the configured legal source review.", + canPrompt: false, + timeoutMs: 60_000, + })) { + events.push(event); + } + + const agentRequests = requests.filter((request) => !isCompactionRequest(request)); + assert.equal(agentRequests.length, 4, JSON.stringify(events)); + assert.match(messageText(agentRequests[0]?.messages ?? []), /"group": "source-fragment-propose"/u); + assert.doesNotMatch(messageText(agentRequests[0]?.messages ?? []), /sourceMergeApplyCommand/u); + assert.match(messageText(agentRequests[1]?.messages ?? []), /"group": "source-fragment-apply"/u); + assert.match(messageText(agentRequests[1]?.messages ?? []), /source-merge-apply/u); + assert.match(messageText(agentRequests[2]?.messages ?? []), /"appliedSource":/u); + assert.match(messageText(agentRequests[3]?.messages ?? []), /"milestone": "COMPLETE"/u); + + const convergence = agentRequests.map((request) => request.metadata?.pilotdeckConvergence as { + progressOrdinal?: number; + handoffOrdinal?: number; + } | undefined); + assert.deepEqual(convergence.map((item) => item?.progressOrdinal), [0, 0, 1, 2]); + assert.deepEqual(convergence.map((item) => item?.handoffOrdinal), [0, 1, 1, 1]); + const decisions = events.flatMap((event) => + event.type === "agent_status" && event.event === "progress_lease_evaluated" + ? [[event.detail?.decision, event.detail?.progressOrdinal, event.detail?.handoffOrdinal]] + : [] + ); + assert.deepEqual(decisions, [ + ["baseline", 0, 0], + ["handoff_grace", 0, 1], + ["renewed", 1, 1], + ["completed", 2, 1], + ]); + assert.equal(events.some((event) => event.type === "turn_completed" && event.finishReason === "completed"), true); + assert.equal(events.some((event) => event.type === "agent_status" + && event.event === "progress_lease_evaluated" + && event.detail?.decision === "fail_closed"), false); + + const storage = createAgentProjectSessionStorage({ + projectRoot, + pilotHome, + sessionId: "legal-source-apply-session", + }); + const observations = await readObservationEvents(join(storage.observabilityDir, "observations.jsonl")); + const integrity = JSON.parse(await readFile(join(storage.observabilityDir, "integrity.json"), "utf8")); + assert.equal(integrity.status, "complete"); + assert.equal(integrity.checks.modelRequestsPaired, true); + assert.equal(integrity.checks.toolCallsPaired, true); + assert.equal(integrity.checks.turnsPaired, true); + assert.equal(integrity.recorder.droppedEvents, 0); + assert.equal(observations.filter((event) => event.type === "tool.call.started").length, 2); + assert.equal(observations.filter((event) => event.type === "tool.call.completed").length, 2); + } finally { + await runtime.dispose(); + await rm(root, { recursive: true, force: true }); + } +}); + test("real gateway applies an immutable source repair before renewing legal progress", async () => { const root = await mkdtemp(join(tmpdir(), "pilotdeck-legal-source-repair-gateway-")); const projectRoot = join(root, "project"); @@ -288,6 +371,163 @@ test("real gateway applies an immutable source repair before renewing legal prog } }); +function sourceApplyModelRuntime( + requests: CanonicalModelRequest[], + projectRoot: string, +): ModelRuntime { + return { + async *stream(request) { + if (isCompactionRequest(request)) { + yield { type: "message_start", role: "assistant" }; + yield { type: "text_delta", text: "Continue the validated ordinary source apply." }; + yield { type: "message_end", finishReason: "stop" }; + return; + } + requests.push(request); + const envelope = legalEnvelopeFromMessages(request.messages); + yield { type: "message_start", role: "assistant" }; + if (envelope?.workItems?.group === "source-fragment-propose") { + const proposal = { + ...envelope.workItems.proposal.template, + facts: envelope.workItems.preparedSlice.sources.map((source: any) => ({ + subject: source.sourceId, + predicate: "contains reviewed evidence", + value: source.facts[0].statement, + missingTimeReason: "The reviewed synthetic source contains no usable date.", + sourceRefs: [{ sourceId: source.sourceId, locator: source.facts[0].locator }], + evidenceClass: source.evidenceClass, + verificationStatus: "verified", + conflictStatus: "none", + material: false, + critical: false, + })), + noMaterialFacts: [], + }; + const toolCall = { + id: "source-proposal-write", + name: "write_file", + input: { + file_path: envelope.workItems.proposal.path, + content: `${JSON.stringify(proposal, null, 2)}\n`, + }, + }; + yield { type: "tool_call_start", id: toolCall.id, name: toolCall.name }; + yield { type: "tool_call_end", toolCall }; + yield { type: "message_end", finishReason: "tool_call" }; + return; + } + if (envelope?.workItems?.group === "source-fragment-apply") { + const toolCall = { + id: "source-proposal-apply", + name: "bash", + input: { command: envelope.sourceMergeApplyCommand }, + }; + yield { type: "tool_call_start", id: toolCall.id, name: toolCall.name }; + yield { type: "tool_call_end", toolCall }; + yield { type: "message_end", finishReason: "tool_call" }; + return; + } + await writeCompletedSourceRepairState(projectRoot); + yield { type: "text_delta", text: "The validated source apply is durable and the synthetic review is complete." }; + yield { type: "usage", usage: { inputTokens: 40, outputTokens: 10, totalTokens: 50 } }; + yield { type: "message_end", finishReason: "stop" }; + }, + async complete() { + return { role: "assistant", content: [{ type: "text", text: '{"title":"Ordinary source apply QA"}' }], finishReason: "stop" }; + }, + getCapabilities: () => ({ ...DEFAULT_MODEL_CAPABILITIES, maxContextTokens: 1_048_576 }), + getMultimodal: () => DEFAULT_MULTIMODAL_CONSTRAINTS, + getProviderProtocol: () => "openai", + getProviderBaseUrl: () => "https://example.invalid", + }; +} + +async function writeSourceApplyState(workspace: string): Promise { + const stateRoot = join(workspace, STATE_ROOT); + const sourceRoot = join(workspace, "source-room"); + await mkdir(sourceRoot, { recursive: true }); + await mkdir(join(workspace, "deliverables"), { recursive: true }); + await mkdir(stateRoot, { recursive: true }); + await writeFile(join(workspace, "deliverables", "opinion.md"), "# Draft legal review\n"); + const sources = []; + for (let index = 1; index <= 5; index += 1) { + const id = `S-${String(index).padStart(3, "0")}`; + const path = `source-room/source-${index}.txt`; + const content = `Reviewed synthetic source ${index}.\n`; + await writeFile(join(workspace, path), content); + sources.push({ id, path, content }); + } + await writeJson(join(stateRoot, "config.json"), { + schemaVersion: 1, + enabled: true, + jurisdiction: "Synthetic jurisdiction", + basisDate: "Synthetic review date", + allowNoMaterialFacts: false, + inputRoots: ["source-room"], + deliverables: [{ id: "opinion", path: "deliverables/opinion.md", required: true }], + }); + await writeJson(join(stateRoot, "sources.json"), { + schemaVersion: 1, + sources: sources.map((source) => ({ + id: source.id, + path: source.path, + sha256: sha256(source.content), + status: "pending", + })), + }); + await writeJson(join(stateRoot, "facts.json"), { schemaVersion: 1, facts: [] }); + await writeJson(join(stateRoot, "matrices.json"), { + schemaVersion: 1, + matrices: REQUIRED_MATRIX_IDS.map((id) => ({ id, status: "pending", entries: [] })), + }); + await writeJson(join(stateRoot, "issues.json"), { schemaVersion: 1, issues: [] }); + await writeJson(join(stateRoot, "authorities.json"), { schemaVersion: 1, authorities: [] }); + await writeJson(join(stateRoot, "coverage.json"), { + schemaVersion: 1, + deliverables: [], + sources: [], + facts: [], + issues: [], + authorities: [], + }); + + const legal = await import(pathToFileURL(join(PLUGIN_ROOT, "scripts", "lib", "legal-coverage.mjs")).href) as any; + const delegated = await legal.pendingSourceReviewPlan(workspace); + assert.equal(delegated.group, "pending-source-review"); + assert.equal(delegated.batches.length, 1); + const batch = delegated.batches[0]; + await mkdir(join(workspace, STATE_ROOT, "fragments"), { recursive: true }); + await writeJson(join(workspace, batch.fragmentPath), { + schemaVersion: 1, + fragmentType: "legal-evidence-source-batch-review", + fragmentId: batch.id, + assignedSourceIds: batch.sourceIds, + sources: sources.map((source) => ({ + sourceId: source.id, + sourcePath: source.path, + inspectionMethod: "plain-text inspection", + facts: [{ locator: "line 1", statement: source.content.trim() }], + evidenceClass: "other", + verificationState: "verified", + conflicts: [], + unresolvedItems: [], + proposedMateriality: "non-material", + })), + }); + const merge = await legal.pendingSourceReviewPlan(workspace); + assert.equal(merge.group, "source-fragment-merge"); + await legal.prepareSourceMergeProposal(workspace, { + readinessPath: merge.readiness.path, + expectedStateHash: merge.proposal.expectedStateHash, + fragmentPath: merge.proposal.fragmentPath, + receiptSha256: merge.proposal.receiptSha256, + sourceIds: merge.proposal.sourceIds, + maxRecords: 4, + maxSerializedBytes: 24576, + }); + assert.equal((await legal.pendingSourceReviewPlan(workspace)).group, "source-fragment-propose"); +} + function sourceRepairModelRuntime( requests: CanonicalModelRequest[], projectRoot: string, @@ -962,3 +1202,8 @@ observability: variant: candidate queueCapacity: 4096 `; + +const SOURCE_APPLY_TEST_CONFIG = SOURCE_REPAIR_TEST_CONFIG.replace( + "campaignId: immutable-source-repair-qa", + "campaignId: ordinary-source-apply-qa", +); diff --git a/tests/products/legal-coverage.spec.ts b/tests/products/legal-coverage.spec.ts index 5a7b2638..981510ee 100644 --- a/tests/products/legal-coverage.spec.ts +++ b/tests/products/legal-coverage.spec.ts @@ -524,6 +524,7 @@ test("legal coverage injects deterministic disjoint worker batches for large pen stateHash?: string; progressOrdinal: number; repairOrdinal: number; + handoffOrdinal: number; } ); const proposeConvergenceHash = proposeConvergence.stateHash; @@ -728,6 +729,7 @@ test("legal coverage injects deterministic disjoint worker batches for large pen progressOrdinal: number; repairOrdinal: number; repairPreparationOrdinal: number; + handoffOrdinal: number; } ); const repairConvergenceHash = repairConvergence.stateHash; @@ -735,6 +737,7 @@ test("legal coverage injects deterministic disjoint worker batches for large pen assert.equal(repairConvergence.repairOrdinal, proposeConvergence.repairOrdinal + 1); assert.equal(repairConvergence.repairPreparationOrdinal, 0); assert.equal(repairConvergence.progressOrdinal, proposeConvergence.progressOrdinal); + assert.equal(repairConvergence.handoffOrdinal, proposeConvergence.handoffOrdinal); const invalidProposalHash = sha256(invalidProposalBytes); const rejectedDirectApply = await runCli( @@ -1242,10 +1245,33 @@ test("legal coverage injects deterministic disjoint worker batches for large pen workItems: { proposal: { path: string; proposalSha256: string } }; }; assert.match(nextApplyEnvelope.sourceMergeApplyCommand, /source-merge-apply/u); - const applyReadyProgress = ( - nextApplyReady.hookSpecificOutput.modelRequestPatch?.metadata?.pilotdeckConvergence as { progressOrdinal: number } - ).progressOrdinal; + const beforeApplyReadyConvergence = replayedAppliedReceipt.hookSpecificOutput.modelRequestPatch?.metadata + ?.pilotdeckConvergence as { progressOrdinal: number; handoffOrdinal: number }; + const applyReadyConvergence = ( + nextApplyReady.hookSpecificOutput.modelRequestPatch?.metadata?.pilotdeckConvergence as { + progressOrdinal: number; + handoffOrdinal: number; + } + ); + const applyReadyProgress = applyReadyConvergence.progressOrdinal; assert.equal(applyReadyProgress, afterRepairProgress.progressOrdinal); + assert.equal(applyReadyConvergence.handoffOrdinal, beforeApplyReadyConvergence.handoffOrdinal + 1); + const replayedApplyReady = await runHook({ + hookEventName: "PreModelRequest", + sessionId: "large-pending-source-plan", + transcriptPath: "", + cwd: workspace, + }); + const replayedApplyReadyConvergence = replayedApplyReady.hookSpecificOutput.modelRequestPatch?.metadata + ?.pilotdeckConvergence as { progressOrdinal: number; handoffOrdinal: number }; + assert.equal(replayedApplyReadyConvergence.progressOrdinal, applyReadyConvergence.progressOrdinal); + assert.equal(replayedApplyReadyConvergence.handoffOrdinal, applyReadyConvergence.handoffOrdinal); + assert.equal( + JSON.parse((replayedApplyReady.hookSpecificOutput.additionalContext ?? "") + .replace(/^\n/u, "") + .replace(/\n<\/legal_coverage_state>$/u, "")).sourceMergeApplyCommand, + nextApplyEnvelope.sourceMergeApplyCommand, + ); const nextApplied = await runCli( workspace, "source-merge-apply", @@ -1333,9 +1359,11 @@ test("legal coverage injects deterministic disjoint worker batches for large pen const observedMergeProgress = ( observedMergeReceipt.hookSpecificOutput.modelRequestPatch?.metadata?.pilotdeckConvergence as { progressOrdinal: number; + handoffOrdinal: number; } - ).progressOrdinal; - assert.equal(observedMergeProgress, applyReadyProgress + 1); + ); + assert.equal(observedMergeProgress.progressOrdinal, applyReadyProgress + 1); + assert.equal(observedMergeProgress.handoffOrdinal, applyReadyConvergence.handoffOrdinal); assert.match(observedMergeReceipt.hookSpecificOutput.additionalContext ?? "", /"appliedSource":/u); const replayedMergeReceipt = await runHook({ hookEventName: "PreModelRequest", @@ -1343,10 +1371,11 @@ test("legal coverage injects deterministic disjoint worker batches for large pen transcriptPath: "", cwd: workspace, }); - assert.equal( + assert.deepEqual( (replayedMergeReceipt.hookSpecificOutput.modelRequestPatch?.metadata?.pilotdeckConvergence as { progressOrdinal: number; - }).progressOrdinal, + handoffOrdinal: number; + }), observedMergeProgress, );