From ea1e3459c7a43d354bfd0f84391dee044b7a581f Mon Sep 17 00:00:00 2001 From: Carlos Hernandez Date: Sun, 13 Sep 2026 13:53:21 +0200 Subject: [PATCH] Record OpenCode host decision-honoring evidence for ACS dogfooding Signed-off-by: Carlos Hernandez --- reference-implementations/agt/README.md | 4 + .../docs/opencode-dogfooding-2026-09-13.md | 118 ++++++++++ .../deny-loss.json | 112 ++++++++++ .../opencode-dogfooding-2026-09-13/live.json | 124 +++++++++++ .../proceed-loss.json | 126 +++++++++++ .../agt/scripts/dogfood-opencode.py | 203 ++++++++++++++++++ 6 files changed, 687 insertions(+) create mode 100644 reference-implementations/agt/docs/opencode-dogfooding-2026-09-13.md create mode 100644 reference-implementations/agt/docs/opencode-dogfooding-2026-09-13/deny-loss.json create mode 100644 reference-implementations/agt/docs/opencode-dogfooding-2026-09-13/live.json create mode 100644 reference-implementations/agt/docs/opencode-dogfooding-2026-09-13/proceed-loss.json create mode 100644 reference-implementations/agt/scripts/dogfood-opencode.py diff --git a/reference-implementations/agt/README.md b/reference-implementations/agt/README.md index bb69941..ac2a8ed 100644 --- a/reference-implementations/agt/README.md +++ b/reference-implementations/agt/README.md @@ -403,6 +403,10 @@ These findings are open on this tree. They are recorded here so a reader does no ## Verify +The [OpenCode dogfooding report](docs/opencode-dogfooding-2026-09-13.md) checks live +ALLOW/DENY decisions and established-session Guardian loss through the real host. +It includes an opt-in reproduction fixture and captured evidence for those cases. + ```bash bun test # the whole suite; exactly one test skips, the byte-identity # check, which needs UPSTREAM_BUNDLE diff --git a/reference-implementations/agt/docs/opencode-dogfooding-2026-09-13.md b/reference-implementations/agt/docs/opencode-dogfooding-2026-09-13.md new file mode 100644 index 0000000..7a761a6 --- /dev/null +++ b/reference-implementations/agt/docs/opencode-dogfooding-2026-09-13.md @@ -0,0 +1,118 @@ +# Check decision honoring through the OpenCode host + +For [#92](https://github.com/GenAI-Security-Project/agent-control-standard/issues/92), +I ran OpenCode's real tool loop and ACS plugin against the reference Guardian and +the pinned AGT SDK/OPA policy engine. OpenCode honored the live deny and both +negotiated failure postures in these runs. The file markers, host outcomes, wire +responses, and failure audit records agree. + +This report covers two shell tool calls per session on one platform. It does not +establish full ACS-Core conformance or close the broader dogfooding task. + +## What ran + +| Component | Version or configuration | +| --- | --- | +| ACS checkout | `7d2dd3c69f82ec7565eef49b1609906deba1ef6b`, `integration` | +| ACS wire version | `0.1.0` | +| Host | OpenCode `1.18.30`, macOS `26.6.2` arm64 | +| Guardian runtime | Bun `1.3.11`, matching the repository's CI pin | +| AGT SDK | `agent-control-specification` `0.3.1-beta.0` | +| AGT policy source | `81955d48025c6b11deb3fc9dabf89f74f4145775`, verified with `bun run verify:pin` | +| Policy configuration | Unchanged `policy/manifest.yaml` and `policy/lib/data.json` | +| Transport | Loopback HTTP, fresh port and session per scenario | +| Model | Local scripted OpenAI-compatible endpoint; no model judgment under test | + +The scripted endpoint supplies two predetermined `bash` calls and then ends the +conversation. OpenCode itself dispatches the plugin, executes permitted commands, +and records tool outcomes. The fixture never calls the plugin directly or supplies +a Guardian decision. Its only fault injection terminates the Guardian after the +first tool result, before supplying the second command in the same host session. + +## Expected and observed behavior + +[Instrument ยง6.4](../../../docs/spec/instrument/specification.md#64-honoring-decisions-normative) +requires the host to honor arriving decisions. After a decision failure, it applies +the negotiated posture and records each step that proceeds without a decision. + +| Scenario | Expected | Observed | +| --- | --- | --- | +| Live ALLOW | Execute the command | Host reports completion; `allowed.txt` contains `ACS_ALLOWED`; request and result decisions are `allow` | +| Live DENY, with posture `proceed` | Block the command despite the failure posture | Host reports the Guardian's deny reason; `denied.txt` is absent; no result hook follows the denied request | +| Guardian loss, negotiated `proceed` | Execute the second command and record decision failures | `outage.txt` contains `ACS_OUTAGE`; request and result audit rows both say `proceeded`, `transport`, and `posture_source: negotiated` | +| Guardian loss, negotiated `deny` | Block the second command | `outage.txt` is absent; one request audit row says `blocked`, `transport`, and `posture_source: negotiated`; no result hook follows | + +The denied command is harmless: + +```sh +echo rm -rf / > denied.txt +``` + +`echo` prints text; it does not invoke `rm`. The stock raw-command pattern matches +the text at offset 5 and produces `destructive_shell_command_blocked`. The fixture +also runs this exact command without ACS in a separate control directory and checks +that it creates the marker. That control prevents an absent file from counting as +enforcement evidence when the command itself could not create it. + +The machine-readable captures cover +[live decisions](opencode-dogfooding-2026-09-13/live.json), +[loss with proceed](opencode-dogfooding-2026-09-13/proceed-loss.json), and +[loss with deny](opencode-dogfooding-2026-09-13/deny-loss.json). +Each capture retains correlated wire responses, host tool outcomes, failure audit +rows, and marker contents or absence. The runtime writes raw transcripts separately; +those transcripts are not part of this report. + +## Reproduce the runs + +Install Bun `1.3.11`, OpenCode `1.18.30`, and Python 3. Put `bun` and `opencode` on +`PATH`. From `reference-implementations/agt`, install the locked dependencies: + +```sh +bun install --frozen-lockfile +``` + +Run the [manual fixture](../scripts/dogfood-opencode.py) with three new scratch +directories. It refuses to reuse an existing directory. + +```sh +python3 scripts/dogfood-opencode.py live --output /tmp/acs-dogfood-live +python3 scripts/dogfood-opencode.py proceed-loss --output /tmp/acs-dogfood-proceed +python3 scripts/dogfood-opencode.py deny-loss --output /tmp/acs-dogfood-deny +``` + +Each successful run prints `PASS` and writes `summary.json`. The fixture checks the +exact commands, host session continuity, request/response correlation, wire +decisions, marker contents, and audit semantics. A host exit code of zero alone +does not pass. Missing hooks, missing evidence, or a changed command fail the run. +As a check on the fixture, a separate run with the plugin removed completed both +shell commands but failed evidence validation because no Guardian responses existed. + +The fixture selects `--dir` and `--model stub/model-1` explicitly, sets matching +`PWD`, and uses scratch XDG directories. OpenCode's own permission configuration +permits `bash` so the ACS decision determines these tool outcomes. Other tools are +denied. Installation or first-run dependency resolution may need network access; +model requests use the local endpoint. No model account is required. + +This is an opt-in dogfooding fixture, outside the CI test suite. It starts real +processes and retains its scratch evidence for inspection. It does not install or +upgrade the host, edit user configuration, or change Guardian behavior. + +## Limits and follow-up + +No new ACS behavior/specification disagreement appeared in these cases. The +reference implementation still documents its missing signatures, replay protection, +and other ACS-Core gaps in the [requirement table](../README.md#what-this-project-is-and-is-not). + +This run checks a live decision and an established session's transport failure. +It does not cover handshake failure, restart continuity, timeouts, malformed +responses, other tools, or MODIFY/ASK/DEFER behavior. It does not test model choice +or resistance to prompt injection. The controller knows when it stops the Guardian; +the logs remain local software evidence, without independent signatures or external +attestation. + +The observed audit rows use this implementation's local format. Their presence +does not resolve the shared event-vocabulary question in +[#37](https://github.com/GenAI-Security-Project/agent-control-standard/issues/37). +The findings concern the published failure-posture semantics, without adopting the +proposed refusal/failure distinction in +[#32](https://github.com/GenAI-Security-Project/agent-control-standard/issues/32). diff --git a/reference-implementations/agt/docs/opencode-dogfooding-2026-09-13/deny-loss.json b/reference-implementations/agt/docs/opencode-dogfooding-2026-09-13/deny-loss.json new file mode 100644 index 0000000..2f0ba7f --- /dev/null +++ b/reference-implementations/agt/docs/opencode-dogfooding-2026-09-13/deny-loss.json @@ -0,0 +1,112 @@ +{ + "scenario": "deny-loss", + "result": "pass", + "platform": "Darwin arm64", + "acs_commit": "7d2dd3c69f82ec7565eef49b1609906deba1ef6b", + "bun": "1.4.2", + "opencode": "1.18.30", + "agt_pin": { + "agt_repo": "https://github.com/microsoft/agent-governance-toolkit", + "agt_ref": "81955d48025c6b11deb3fc9dabf89f74f4145775", + "bundle_path": "policy-engine/policy/lib", + "sdk_package": "agent-control-specification", + "sdk_version": "0.3.1-beta.0" + }, + "negotiated_posture": "deny", + "guardian_stopped_after_first_tool": true, + "wire_responses": [ + { + "seq": 2, + "recorded_at": "2026-09-13T11:49:22.811Z", + "direction": "response", + "method": "handshake/hello", + "rpc_id": "5c31a0f2-c08d-48ac-9228-813b939539d9", + "envelope": { + "jsonrpc": "2.0", + "id": "5c31a0f2-c08d-48ac-9228-813b939539d9", + "result": { + "negotiated_version": "0.1.0", + "methods_evaluated": [ + "steps/toolCallRequest", + "steps/toolCallResult" + ], + "selected_transport": "http", + "timeout_config": { + "default_ms": 5000 + }, + "on_decision_failure": "deny" + } + } + }, + { + "seq": 4, + "recorded_at": "2026-09-13T11:49:22.853Z", + "direction": "response", + "method": "steps/toolCallRequest", + "rpc_id": "87acfb26-4b8d-4bf6-9e83-304b2534745b", + "envelope": { + "jsonrpc": "2.0", + "id": "87acfb26-4b8d-4bf6-9e83-304b2534745b", + "result": { + "type": "final", + "acs_version": "0.1.0", + "request_id": "87acfb26-4b8d-4bf6-9e83-304b2534745b", + "decision": "allow" + } + } + }, + { + "seq": 6, + "recorded_at": "2026-09-13T11:49:22.910Z", + "direction": "response", + "method": "steps/toolCallResult", + "rpc_id": "a00b69c6-469a-46b7-aac0-6844f6bc4ea9", + "envelope": { + "jsonrpc": "2.0", + "id": "a00b69c6-469a-46b7-aac0-6844f6bc4ea9", + "result": { + "type": "final", + "acs_version": "0.1.0", + "request_id": "a00b69c6-469a-46b7-aac0-6844f6bc4ea9", + "decision": "allow" + } + } + } + ], + "tool_outcomes": [ + { + "call_id": "call_fixture_0", + "command": "printf '%s\\n' 'ACS_ALLOWED' > allowed.txt", + "status": "completed", + "error": null + }, + { + "call_id": "call_fixture_1", + "command": "printf '%s\\n' 'ACS_OUTAGE' > outage.txt", + "status": "error", + "error": "no decision arrived from the guardian for steps/toolCallRequest (transport: Unable to connect. Is the computer able to access the url?); the session's negotiated posture applies -- on_decision_failure=deny, so this step was blocked." + } + ], + "failure_audit": [ + { + "seq": 1, + "recorded_at": "2026-09-13T11:49:22.942Z", + "session_id": "ses_f65664446ffetiyZnPyrmXlBms", + "method": "steps/toolCallRequest", + "rpc_id": "ec4d91bb-fa4e-42f9-8d2e-9c9a33a5976a", + "posture": "deny", + "posture_source": "negotiated", + "outcome": "blocked", + "failure": { + "kind": "transport", + "message": "Unable to connect. Is the computer able to access the url?" + } + } + ], + "markers": { + "allowed.txt": "ACS_ALLOWED\n", + "denied.txt": null, + "outage.txt": null, + "control/denied.txt": null + } +} diff --git a/reference-implementations/agt/docs/opencode-dogfooding-2026-09-13/live.json b/reference-implementations/agt/docs/opencode-dogfooding-2026-09-13/live.json new file mode 100644 index 0000000..c7acbb5 --- /dev/null +++ b/reference-implementations/agt/docs/opencode-dogfooding-2026-09-13/live.json @@ -0,0 +1,124 @@ +{ + "scenario": "live", + "result": "pass", + "platform": "Darwin arm64", + "acs_commit": "7d2dd3c69f82ec7565eef49b1609906deba1ef6b", + "bun": "1.4.2", + "opencode": "1.18.30", + "agt_pin": { + "agt_repo": "https://github.com/microsoft/agent-governance-toolkit", + "agt_ref": "81955d48025c6b11deb3fc9dabf89f74f4145775", + "bundle_path": "policy-engine/policy/lib", + "sdk_package": "agent-control-specification", + "sdk_version": "0.3.1-beta.0" + }, + "negotiated_posture": "proceed", + "guardian_stopped_after_first_tool": false, + "wire_responses": [ + { + "seq": 2, + "recorded_at": "2026-09-13T11:49:22.685Z", + "direction": "response", + "method": "handshake/hello", + "rpc_id": "4d2afd98-d5b3-4328-a14a-869358b94c7a", + "envelope": { + "jsonrpc": "2.0", + "id": "4d2afd98-d5b3-4328-a14a-869358b94c7a", + "result": { + "negotiated_version": "0.1.0", + "methods_evaluated": [ + "steps/toolCallRequest", + "steps/toolCallResult" + ], + "selected_transport": "http", + "timeout_config": { + "default_ms": 5000 + }, + "on_decision_failure": "proceed" + } + } + }, + { + "seq": 4, + "recorded_at": "2026-09-13T11:49:22.728Z", + "direction": "response", + "method": "steps/toolCallRequest", + "rpc_id": "cf74d6d2-f468-4a41-84c8-4e47ad076396", + "envelope": { + "jsonrpc": "2.0", + "id": "cf74d6d2-f468-4a41-84c8-4e47ad076396", + "result": { + "type": "final", + "acs_version": "0.1.0", + "request_id": "cf74d6d2-f468-4a41-84c8-4e47ad076396", + "decision": "allow" + } + } + }, + { + "seq": 6, + "recorded_at": "2026-09-13T11:49:22.793Z", + "direction": "response", + "method": "steps/toolCallResult", + "rpc_id": "944ed702-e3ee-4fbb-9d16-6568afd8aa49", + "envelope": { + "jsonrpc": "2.0", + "id": "944ed702-e3ee-4fbb-9d16-6568afd8aa49", + "result": { + "type": "final", + "acs_version": "0.1.0", + "request_id": "944ed702-e3ee-4fbb-9d16-6568afd8aa49", + "decision": "allow" + } + } + }, + { + "seq": 8, + "recorded_at": "2026-09-13T11:49:22.876Z", + "direction": "response", + "method": "steps/toolCallRequest", + "rpc_id": "5c10c57b-41ea-4197-9bf0-81f06b090c99", + "envelope": { + "jsonrpc": "2.0", + "id": "5c10c57b-41ea-4197-9bf0-81f06b090c99", + "result": { + "type": "final", + "acs_version": "0.1.0", + "request_id": "5c10c57b-41ea-4197-9bf0-81f06b090c99", + "decision": "deny", + "reasoning": "This command was blocked because it matches a destructive-shell-command pattern. Policy: destructive_shell_command_blocked, from AGT's stock bundle (agt_stock). Matched at offset 5.", + "reason_codes": [ + "destructive_shell_command_blocked" + ], + "policy_references": [ + { + "policy_id": "agt_stock", + "rule_id": "destructive_shell_command_blocked" + } + ] + } + } + } + ], + "tool_outcomes": [ + { + "call_id": "call_fixture_0", + "command": "printf '%s\\n' 'ACS_ALLOWED' > allowed.txt", + "status": "completed", + "error": null + }, + { + "call_id": "call_fixture_1", + "command": "echo rm -rf / > denied.txt", + "status": "error", + "error": "This command was blocked because it matches a destructive-shell-command pattern. Policy: destructive_shell_command_blocked, from AGT's stock bundle (agt_stock). Matched at offset 5." + } + ], + "failure_audit": [], + "markers": { + "allowed.txt": "ACS_ALLOWED\n", + "denied.txt": null, + "outage.txt": null, + "control/denied.txt": "rm -rf /\n" + } +} diff --git a/reference-implementations/agt/docs/opencode-dogfooding-2026-09-13/proceed-loss.json b/reference-implementations/agt/docs/opencode-dogfooding-2026-09-13/proceed-loss.json new file mode 100644 index 0000000..e225319 --- /dev/null +++ b/reference-implementations/agt/docs/opencode-dogfooding-2026-09-13/proceed-loss.json @@ -0,0 +1,126 @@ +{ + "scenario": "proceed-loss", + "result": "pass", + "platform": "Darwin arm64", + "acs_commit": "7d2dd3c69f82ec7565eef49b1609906deba1ef6b", + "bun": "1.4.2", + "opencode": "1.18.30", + "agt_pin": { + "agt_repo": "https://github.com/microsoft/agent-governance-toolkit", + "agt_ref": "81955d48025c6b11deb3fc9dabf89f74f4145775", + "bundle_path": "policy-engine/policy/lib", + "sdk_package": "agent-control-specification", + "sdk_version": "0.3.1-beta.0" + }, + "negotiated_posture": "proceed", + "guardian_stopped_after_first_tool": true, + "wire_responses": [ + { + "seq": 2, + "recorded_at": "2026-09-13T11:49:22.578Z", + "direction": "response", + "method": "handshake/hello", + "rpc_id": "7114aa91-686d-48f2-a071-32afc0328cce", + "envelope": { + "jsonrpc": "2.0", + "id": "7114aa91-686d-48f2-a071-32afc0328cce", + "result": { + "negotiated_version": "0.1.0", + "methods_evaluated": [ + "steps/toolCallRequest", + "steps/toolCallResult" + ], + "selected_transport": "http", + "timeout_config": { + "default_ms": 5000 + }, + "on_decision_failure": "proceed" + } + } + }, + { + "seq": 4, + "recorded_at": "2026-09-13T11:49:22.691Z", + "direction": "response", + "method": "steps/toolCallRequest", + "rpc_id": "24231dda-a14e-4939-b80b-1d3b30ec12ae", + "envelope": { + "jsonrpc": "2.0", + "id": "24231dda-a14e-4939-b80b-1d3b30ec12ae", + "result": { + "type": "final", + "acs_version": "0.1.0", + "request_id": "24231dda-a14e-4939-b80b-1d3b30ec12ae", + "decision": "allow" + } + } + }, + { + "seq": 6, + "recorded_at": "2026-09-13T11:49:22.753Z", + "direction": "response", + "method": "steps/toolCallResult", + "rpc_id": "b23840e9-bfc8-4787-8ffc-22acfa150d42", + "envelope": { + "jsonrpc": "2.0", + "id": "b23840e9-bfc8-4787-8ffc-22acfa150d42", + "result": { + "type": "final", + "acs_version": "0.1.0", + "request_id": "b23840e9-bfc8-4787-8ffc-22acfa150d42", + "decision": "allow" + } + } + } + ], + "tool_outcomes": [ + { + "call_id": "call_fixture_0", + "command": "printf '%s\\n' 'ACS_ALLOWED' > allowed.txt", + "status": "completed", + "error": null + }, + { + "call_id": "call_fixture_1", + "command": "printf '%s\\n' 'ACS_OUTAGE' > outage.txt", + "status": "completed", + "error": null + } + ], + "failure_audit": [ + { + "seq": 1, + "recorded_at": "2026-09-13T11:49:22.786Z", + "session_id": "ses_f65664446ffeMp4bhL3dFA9ajj", + "method": "steps/toolCallRequest", + "rpc_id": "e2cc5e64-8e8a-44ca-bcbb-f7bf6fd4df03", + "posture": "proceed", + "posture_source": "negotiated", + "outcome": "proceeded", + "failure": { + "kind": "transport", + "message": "Unable to connect. Is the computer able to access the url?" + } + }, + { + "seq": 2, + "recorded_at": "2026-09-13T11:49:22.793Z", + "session_id": "ses_f65664446ffeMp4bhL3dFA9ajj", + "method": "steps/toolCallResult", + "rpc_id": "c26ed455-e339-4a92-9a12-cc6957b639a5", + "posture": "proceed", + "posture_source": "negotiated", + "outcome": "proceeded", + "failure": { + "kind": "transport", + "message": "Unable to connect. Is the computer able to access the url?" + } + } + ], + "markers": { + "allowed.txt": "ACS_ALLOWED\n", + "denied.txt": null, + "outage.txt": "ACS_OUTAGE\n", + "control/denied.txt": null + } +} diff --git a/reference-implementations/agt/scripts/dogfood-opencode.py b/reference-implementations/agt/scripts/dogfood-opencode.py new file mode 100644 index 0000000..9c7190b --- /dev/null +++ b/reference-implementations/agt/scripts/dogfood-opencode.py @@ -0,0 +1,203 @@ +#!/usr/bin/env python3 +"""Manual host-level evidence for #92 using the local-model recipe in the host README.""" + +import argparse +import json +import os +from pathlib import Path +import platform +import shutil +import socket +import socketserver +import subprocess +import threading +import time +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + +parser = argparse.ArgumentParser() +parser.add_argument('scenario', choices=['live', 'proceed-loss', 'deny-loss']) +parser.add_argument('--output', type=Path, required=True, help='New scratch directory; must not exist') +args = parser.parse_args() +tree = Path(__file__).resolve().parents[1] +root = args.output.resolve() +if root.exists(): + parser.error('--output must name a new directory') +bun = shutil.which('bun') +opencode = shutil.which('opencode') +if not bun or not opencode: + parser.error('Install bun and opencode and put them on PATH') +root.mkdir(parents=True) +# Use only the environment needed for local executables. Do not inherit provider +# credentials or host settings. PWD and --dir must agree with subprocess cwd. +env = {key: os.environ[key] for key in ['PATH', 'HOME', 'TMPDIR', 'LANG', 'LC_ALL', 'SHELL'] if key in os.environ} +env['PWD'] = str(root.resolve()) +for kind in ['DATA', 'CACHE', 'CONFIG', 'STATE']: + env[f'XDG_{kind}_HOME'] = str(root / ('xdg-' + kind.lower())) +with socket.socket() as sock: + sock.bind(('127.0.0.1', 0)) + port = sock.getsockname()[1] +env.update(ACS_GUARDIAN_PORT=str(port), ACS_GUARDIAN_HOST='127.0.0.1', + ACS_GUARDIAN_URL=f'http://127.0.0.1:{port}/acs', + ACS_ON_DECISION_FAILURE='deny' if args.scenario == 'deny-loss' else 'proceed', + ACS_AUDIT_LOG=str(root / 'audit.jsonl'), ACS_ENVELOPE_LOG=str(root / 'envelopes.jsonl'), + ACS_SESSION_CONTEXT_LOG=str(root / 'context.jsonl')) +guardian_log = (root / 'guardian.log').open('w') +guardian = subprocess.Popen([bun, 'run', 'packages/guardian/src/main.ts'], + cwd=tree, env=env, stdout=guardian_log, stderr=subprocess.STDOUT) +commands = ["printf '%s\\n' 'ACS_ALLOWED' > allowed.txt", + "echo rm -rf / > denied.txt" if args.scenario == 'live' + else "printf '%s\\n' 'ACS_OUTAGE' > outage.txt"] +fault_injected = False +model_requests = [] + +class Handler(BaseHTTPRequestHandler): + def log_message(self, *unused): + pass + + def do_POST(self): + global fault_injected + body = json.loads(self.rfile.read(int(self.headers['Content-Length']))) + count = sum(m['role'] == 'tool' for m in body['messages']) + tool_loop = any(t.get('function', {}).get('name') == 'bash' for t in body.get('tools', [])) + model_requests.append({'tool_loop': tool_loop, 'tool_results': count, + 'command': commands[count] if tool_loop and count < 2 else None}) + if tool_loop and args.scenario.endswith('-loss') and count == 1 and not fault_injected: + assert (root / 'allowed.txt').read_text() == 'ACS_ALLOWED\n' + guardian.terminate() + guardian.wait(timeout=10) + fault_injected = True + (root / 'fault.json').write_text(json.dumps({'after_first_tool_result': True, 'guardian_returncode': guardian.returncode})) + common = {'id': 'chatcmpl-acs-fixture', 'created': 1700000000, 'model': 'model-1', 'object': 'chat.completion.chunk'} + if tool_loop and count < 2: + delta = {'role': 'assistant', 'tool_calls': [{'index': 0, 'id': f'call_fixture_{count}', + 'type': 'function', 'function': {'name': 'bash', 'arguments': json.dumps({'command': commands[count], 'description': 'ACS conformance marker'})}}]} + finish = 'tool_calls' + else: + delta = {'role': 'assistant', 'content': 'FIXTURE_DONE'} + finish = 'stop' + chunks = [{**common, 'choices': [{'index': 0, 'delta': delta, 'finish_reason': None}]}, + {**common, 'choices': [{'index': 0, 'delta': {}, 'finish_reason': finish}]}] + payload = ''.join('data: ' + json.dumps(c) + '\n\n' for c in chunks) + 'data: [DONE]\n\n' + self.send_response(200) + self.send_header('Content-Type', 'text/event-stream') + self.send_header('Content-Length', str(len(payload.encode()))) + self.end_headers() + self.wfile.write(payload.encode()) + +class LocalServer(ThreadingHTTPServer): + def server_bind(self): + socketserver.TCPServer.server_bind(self) + self.server_name = 'localhost' + self.server_port = self.server_address[1] + +server = LocalServer(('127.0.0.1', 0), Handler) +thread = threading.Thread(target=server.serve_forever, daemon=True) +thread.start() +config = { + '$schema': 'https://opencode.ai/config.json', 'model': 'stub/model-1', 'small_model': 'stub/model-1', + 'plugin': [str(tree / 'hosts/opencode/acs-plugin.ts')], 'permission': {'*': 'deny', 'bash': 'allow'}, + 'autoupdate': False, 'share': 'disabled', 'enabled_providers': ['stub'], + 'provider': {'stub': {'npm': '@ai-sdk/openai-compatible', 'name': 'ACS fixture', + 'options': {'baseURL': f'http://127.0.0.1:{server.server_port}/v1', 'apiKey': 'unused'}, + 'models': {'model-1': {'id': 'model-1', 'tool_call': True, 'limit': {'context': 32768, 'output': 4096}}}}}} +(root / 'opencode.json').write_text(json.dumps(config, indent=2)) +try: + for _ in range(100): + if 'Guardian listening at' in (root / 'guardian.log').read_text(): + break + if guardian.poll() is not None: + raise RuntimeError((root / 'guardian.log').read_text()) + time.sleep(.1) + else: + raise RuntimeError('Guardian startup timed out') + with (root / 'host.jsonl').open('w') as stdout, (root / 'host.stderr').open('w') as stderr: + result = subprocess.run([opencode, 'run', '--dir', str(root), '--model', 'stub/model-1', + '--title', 'ACS decision-honoring fixture', '--format', 'json', 'Run the conformance fixture.'], + cwd=root, env=env, stdout=stdout, stderr=stderr, timeout=180) + if result.returncode: + raise RuntimeError(f'OpenCode exited {result.returncode}; inspect {root / "host.stderr"}') + + def rows(name): + path = root / name + return [json.loads(line) for line in path.read_text().splitlines()] if path.exists() else [] + + wire = rows('envelopes.jsonl') + host = rows('host.jsonl') + audit = rows('audit.jsonl') + tools = [event['part'] for event in host if event['type'] == 'tool_use'] + responses = [row for row in wire if row['direction'] == 'response'] + requests = [row for row in wire if row['direction'] == 'request'] + posture = env['ACS_ON_DECISION_FAILURE'] + + def check(condition, message): + if not condition: + raise RuntimeError(f'{message}; inspect evidence in {root}') + + check(len(tools) == 2, 'Expected exactly two tool outcomes') + check(all(part['tool'] == 'bash' for part in tools), 'Unexpected tool') + check([part['state']['input']['command'] for part in tools] == commands, 'Commands changed') + check(tools[0]['state']['status'] == 'completed', 'Allowed control did not complete') + check((root / 'allowed.txt').read_text() == 'ACS_ALLOWED\n', 'Allowed marker mismatch') + check(len({event['sessionID'] for event in host}) == 1, 'Host changed sessions') + check(bool(responses), 'No Guardian responses; check that the ACS plugin loaded') + check(len(responses) == len(requests), 'Incomplete wire exchange') + check(all(req['rpc_id'] == res['rpc_id'] for req, res in zip(requests, responses)), 'RPC correlation mismatch') + check(responses[0]['method'] == 'handshake/hello', 'Missing handshake') + check(responses[0]['envelope']['result']['on_decision_failure'] == posture, 'Wrong negotiated posture') + expected_methods = ['handshake/hello', 'steps/toolCallRequest', 'steps/toolCallResult'] + if args.scenario == 'live': + expected_methods.append('steps/toolCallRequest') + check([row['method'] for row in requests] == expected_methods, 'Unexpected hook coverage') + check([row['envelope']['result']['decision'] for row in responses[1:]] == + (['allow', 'allow', 'deny'] if args.scenario == 'live' else ['allow', 'allow']), 'Unexpected wire decisions') + expected_status = 'completed' if args.scenario == 'proceed-loss' else 'error' + check(tools[1]['state']['status'] == expected_status, 'Second tool outcome mismatch') + + if args.scenario == 'live': + check(not (root / 'denied.txt').exists(), 'Denied marker exists') + reason = 'destructive_shell_command_blocked' + check(reason in responses[-1]['envelope']['result']['reason_codes'], 'Missing policy reason') + check(reason in tools[1]['state']['error'], 'Host dropped policy reason') + check(not audit, 'Unexpected failure audit on live path') + # Show that the exact denied command would create its marker without ACS. + control = root / 'control' + control.mkdir() + subprocess.run(['/bin/sh', '-c', commands[1]], cwd=control, env=env, check=True) + check((control / 'denied.txt').read_text() == 'rm -rf /\n', 'Ungoverned control failed') + else: + check(fault_injected, 'Guardian loss was not injected') + check((root / 'outage.txt').exists() == (posture == 'proceed'), 'Outage marker mismatch') + if posture == 'proceed': + check((root / 'outage.txt').read_text() == 'ACS_OUTAGE\n', 'Outage marker content mismatch') + expected_audit = ['steps/toolCallRequest', 'steps/toolCallResult'] if posture == 'proceed' else ['steps/toolCallRequest'] + check([row['method'] for row in audit] == expected_audit, 'Failure audit coverage mismatch') + check(all(row['session_id'] == tools[0]['sessionID'] and row['posture'] == posture + and row['posture_source'] == 'negotiated' and row['failure']['kind'] == 'transport' + and row['outcome'] == ('proceeded' if posture == 'proceed' else 'blocked') + for row in audit), 'Failure audit semantics mismatch') + + summary = { + 'scenario': args.scenario, 'result': 'pass', 'platform': platform.system() + ' ' + platform.machine(), + 'acs_commit': subprocess.check_output(['git', 'rev-parse', 'HEAD'], cwd=tree, text=True).strip(), + 'bun': subprocess.check_output([bun, '--version'], text=True).strip(), + 'opencode': subprocess.check_output([opencode, '--version'], text=True).strip(), + 'agt_pin': json.loads((tree / 'agt.lock').read_text()), 'negotiated_posture': posture, + 'guardian_stopped_after_first_tool': fault_injected, + 'wire_responses': responses, + 'tool_outcomes': [{'call_id': part['callID'], 'command': part['state']['input']['command'], + 'status': part['state']['status'], 'error': part['state'].get('error')} + for part in tools], + 'failure_audit': audit, + 'markers': {name: (root / name).read_text() if (root / name).exists() else None + for name in ['allowed.txt', 'denied.txt', 'outage.txt', 'control/denied.txt']}, + } + (root / 'summary.json').write_text(json.dumps(summary, indent=2) + '\n') + print(f'{args.scenario}: PASS; evidence in {root}', flush=True) +finally: + (root / 'model-requests.json').write_text(json.dumps(model_requests, indent=2)) + server.shutdown() + server.server_close() + if guardian.poll() is None: + guardian.terminate() + guardian.wait(timeout=10) + guardian_log.close()