Skip to content

Record OpenCode host decision-honoring evidence for #92 - #113

Open
carloshvp wants to merge 1 commit into
GenAI-Security-Project:integrationfrom
carloshvp:agt-host-conformance-report
Open

carloshvp wants to merge 1 commit into
GenAI-Security-Project:integrationfrom
carloshvp:agt-host-conformance-report

Conversation

@carloshvp

Copy link
Copy Markdown

What changed

Add a host-level dogfooding report for OpenCode 1.18.30 against the ACS reference Guardian and pinned AGT policy engine. The report includes three evidence captures and an opt-in reproduction fixture for live ALLOW/DENY decisions and established-session Guardian loss under both negotiated failure postures.

The real host executes or rejects harmless marker commands. A local scripted model supplies those commands; the Guardian and policy engine remain unchanged. The captures agree with the specification for this slice, without establishing full ACS-Core conformance.

Which issue does this implement

Contributes to #92, which is labeled status:accepted and help wanted. This leaves #92 open for further dogfooding reports.

Base branch

  • integration, because this touches reference-implementation documentation and a manual reproduction fixture

Type of change

  • Documentation
  • Tooling or CI (opt-in local tooling only)

I tested this

  • Synced with integration; tested implementation head 7d2dd3c69f82ec7565eef49b1609906deba1ef6b
  • uv run pytest -v: 278 passed, 1 skipped
  • uv run mkdocs build --strict: passed
  • Bun 1.3.11: bun test: 1,110 passed, 1 skipped; bun run typecheck: passed
  • bun run verify:pin: 5 passed, including upstream Rego byte identity
  • All three manual OpenCode scenarios passed, checking tool outcomes, marker contents, wire decisions, session continuity, and failure-audit fields
  • The exact denied command creates its marker in a separate ungoverned control directory
  • Removing the ACS plugin lets both shell calls complete but makes the fixture fail on missing Guardian evidence
  • git diff --check: passed

The Python skip requires a case-sensitive filesystem. The Bun suite's skipped pin test requires UPSTREAM_BUNDLE; verify:pin supplied the upstream bundle and passed that check separately. The manual host fixture is outside CI because it requires an installed OpenCode host.

Checklist

  • Commits carry a human Signed-off-by line via git commit -s
  • Prose follows STYLE.md
  • No secrets, tokens, or internal URLs in the diff

Security

The fixture uses a scratch directory, loopback servers, a local scripted model, and harmless file markers. It stops only the Guardian process it starts. It does not change production behavior or report a new vulnerability. The report retains the reference implementation's documented authentication and conformance limits; its local logs are not independent attestation.

Signed-off-by: Carlos Hernandez <carloshvp@gmail.com>
@github-project-automation github-project-automation Bot moved this to Needs triage in ACS Project Tracker Sep 17, 2026
rocklambros added a commit that referenced this pull request Sep 17, 2026
Closes the automation half of #97, and picks up the follow-on #104 named for itself.

## What happened

The board drifted for seven days without anyone noticing. Everything from #105 onward, 40 issues and pull requests, was never added. That included the external conformance reports (#109, #111, #113), the second independent implementation and its profile claim (#106), the whole accepted 0.1.3 set (#132 through #136), and every AGT reference implementation bug an outside contributor filed. A stale board looks exactly like a current one, which is why it held for a week.

## The cause was narrower than it looked

Project 9 has six built-in workflows. `Item added to project` is enabled, which is why all 58 items carried a Status and not one was blank. What the project does not have is any auto-add for this repository. The only add workflow present is `Auto-add sub-issues to project`, which fires on sub-issues.

So the board sets Status correctly and never adds anything. A dry run of the reconciler put the split at **40 adds against 1 Status change**.

The reconciler has since been run by hand. The board is at 98 items, every one with a Status.

## What this adds

The real-time path is the project's own `Auto-add to project` workflow, which needs no credential and cannot be configured from here, since the GraphQL API exposes only `deleteProjectV2Workflow` and no way to create or enable one. **That toggle is a manual step and this pull request does not do it.** See below.

This workflow is the safety net under it. Nightly at 04:40 UTC, plus `workflow_dispatch` that defaults to printing the plan rather than applying it. The reconciler only adds an item or changes a Status field, never archives or removes one, which is the property that makes an unattended run acceptable at all.

## On the credential, since #104 rejected exactly this

#104 turned down a workflow for a good reason, quoted from its own message:

> Making it work means storing a personal access token with organization-wide project write as a repository secret, in a repository that opened to a hundred contributors this morning, where anyone who can land a workflow file can reach it.

and named the alternative:

> A GitHub App is the right long-term answer and is separate work.

This is that work, and it narrows the objection twice rather than waving it off.

**The grant is smaller.** `organization_projects: write` plus read on this repository's issues and pull requests. A classic PAT with `project` scope carries write on every project the granting user can see, organization-wide, for as long as the token lives. The App reaches project boards and nothing else. What the job holds is a one-hour installation token, not the key that mints it.

**The key is harder to spend.** It lives in a `board-automation` environment rather than in repository secrets, with the deployment branch rule limited to the default branch. A workflow on a pull request branch, fork included, cannot read it even if it names the environment.

**The residual risk, stated rather than assumed away.** Landing a malicious workflow on the default branch still reaches the key. That is the path CODEOWNERS already guards: `/.github/` carries a narrower owner list than the repository default, and the ruleset on main requires a code owner review. Two controls, not one.

## Manual steps before this does anything

This workflow is inert until all three are done, and it will fail loudly rather than silently if they are not.

1. **Create the GitHub App** on the organization with `organization_projects: write` and read on issues and pull requests. Install it on this repository only.
2. **Create the `board-automation` environment** in repository settings. Add `BOARD_APP_PRIVATE_KEY` as a secret and `BOARD_APP_ID` as a variable. Set the deployment branch rule to the default branch.
3. **Enable `Auto-add to project`** on project 9, filtered to this repository. This is the half that actually prevents the drift in real time. The nightly job only catches what it misses.

Step 3 is worth doing even if 1 and 2 are declined, since it costs nothing and would have prevented 40 of the 41 drift events on its own.

## Verification

- YAML parses, triggers and permissions read as intended.
- Both actions are SHA-pinned with a version comment, matching the convention in the other eleven workflows.
- No GitHub expression is interpolated into a `run:` block. Every value reaches the shell through `env:`, so the workflow-injection class is closed by construction rather than by review.
- Top-level `permissions: {}` with `contents: read` on the job.



Signed-off-by: Rock Lambros <rock@rockcyber.com>
@rocklambros rocklambros moved this from Needs triage to In progress in ACS Project Tracker Sep 17, 2026
rocklambros added a commit that referenced this pull request Sep 17, 2026
… one pass (#158)

Follow-up to #157, fixing a defect I found by testing the merged workflow rather than one a reviewer would have caught from the diff.

## The defect

A reconcile run that adds items cannot also set their Status.

Adding an item triggers project 9's own `Item added to project` workflow, which stamps a default Status asynchronously. `apply_governance.py` builds its entire plan before executing any of it, so on a run that adds anything it has already decided the Status actions before those items exist on the board. The default lands after the run finishes and nothing corrects it until the next run.

## Measured, not theorised

On the 2026-09-17 backfill the first pass made **40 adds and 1 Status change**. A second pass run immediately afterwards found **5 more Status corrections waiting**:

| Item | Why the default was wrong |
| --- | --- |
| #135 | `status:accepted` |
| #131 | `status:accepted` |
| #126 | open pull request, belongs in In progress |
| #113 | open pull request |
| #112 | open pull request |

None of those five has the default as its correct Status, which is precisely the population a single pass strands. A third pass reported zero actions, so the tool converges, it just cannot converge in one run when it is also adding.

Nightly, that is up to 24 hours of a board reading Needs triage for work that is accepted or in flight. That is the failure the board exists to prevent, so shipping #157 without this would have left the automation quietly wrong in the one case it most needed to be right.

## The fix

Run the board step twice under `--apply`. The dry-run path still runs once and returns, since printing the same plan twice helps nobody.

The reconciler is idempotent, so the second pass costs one API listing on a run that added nothing. Once `Auto-add to project` is enabled on the project, that is every run.

## Note on review

I did not admin-merge this one. #157 was merged under the administrative authorisation given for it specifically, and extending that to a second change on my own judgement is the kind of thing that should be asked for rather than assumed. The workflow is inert until the App and environment exist, so there is no urgency in merging this ahead of a normal review.


Signed-off-by: Rock Lambros <rock@rockcyber.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In progress

Development

Successfully merging this pull request may close these issues.

2 participants