Skip to content

Browser toolbox E: consequence-aware acting gate #28

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
josephschorr wants to merge 10 commits into from
Draft

Browser toolbox E: consequence-aware acting gate #28

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
27c8370
browser: add the consequence judge and the web_action resource
josephschorr Oct 11, 2026
a5f9063
browser: gate mutating actions on a caveated web_action grant
josephschorr Oct 11, 2026
9f565cd
browser: tune the acting gate and fix approval-screenshot delivery
josephschorr Oct 11, 2026
6de5828
browser: close acting-gate fail-opens (press-by-key, credential leak,…
josephschorr Oct 10, 2026
23a25f6
fix(browser-authz): runner-side trust gate for web_action stamps + pl…
josephschorr Oct 10, 2026
33ed5f5
Fix BrowserToolbox CEL admission gaps and browser-controller hardening
josephschorr Oct 10, 2026
dda2b46
Fix browser chat delivery: #59 rehydrate deadlock, login-code leak, w…
josephschorr Oct 10, 2026
9392c3f
browser: close pre-PR code-review security findings
josephschorr Oct 10, 2026
1fd30c0
browser: review-pass hardening (screenshot TOCTOU, wait_for refs, sit…
josephschorr Oct 10, 2026
3879c13
browser: make login_human_step code entry work from the chat surface
josephschorr Oct 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 27 additions & 26 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,32 +52,33 @@ transports, artifact renderers, tool CRD kinds, identity setup flows, LLM
providers, memory stores, and so on. The ones already factored as
`interface + registry` (or `interface + DI`):

| Aspect | Interface | Wiring | Backends today |
| --------------------------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| Channel transports | `pkg/channels/channelkinds.Kind` | `pkg/channels/channelkinds/registry` | agent, bento, browser, fake, github, local, slack |
| Artifact renderers | `pkg/channels/channelassets.Renderer` | `pkg/channels/channelassets/registry` | html, css, svg, image, mcpui (operator-only, not agent-selectable) |
| Tool CRD kinds (CLI presentation) | `pkg/tools/contract.Kind` | `pkg/tools/kinds/registry` | SpiceboxToolspec, MCPServer, SidecarToolbox |
| Toolspec parsers | `pkg/tools/toolspec/parser.Parser` | `pkg/tools/toolspec/parser/builtin` registry | declarative (only kind in production use today); builtin registry defined, none registered |
| AgentIdentity setup flows | `pkg/platform/identity/setup/builtins.Flow` | `builtins.Register` (`pkg/platform/identity/setup/builtins/registry.go`) | anthropic-oauth, github-pat, kubectl-kubeconfig, oauth-mcp, onepassword-scim, slack-bot-token, tailscale-authkey |
| AgentIdentity auth kinds | `pkg/platform/identity/authkind.Kind` | `authkind/registry` | cli, mcp, toolbox, toolspec |
| Identity credential types | `pkg/platform/identity/credkind.Kind` | `credkind/registry` (+ the `credkind/imports` bundle) | static, oauth, federated, githubapp |
| Session state kinds | `pkg/agent/session/state.Kind` | `state.Register` (`pkg/agent/session/state/registry.go`) | plans, deliveries, triggerstatus |
| Completion requirements | `pkg/agent/completion.Requirement` | `completion.Register` (`pkg/agent/completion/registry.go`) | artifact-delivered, plan-steps-complete, trigger-status-concluded |
| Agent harnesses | `pkg/agent/harness.Harness` | `pkg/agent/harness/registry` | ap-native |
| Interaction categories | `pkg/channels/channelinteractions.Category` | `pkg/channels/channelinteractions` registry | 44 registered (15 interactive + 29 notice) — see `pkg/channels/channelinteractions/categories` |
| Workspace sources | `pkg/platform/workspacekinds.Kind` | `pkg/platform/workspacekinds/registry` | git |
| Sandbox backends | `pkg/tools/sandboxkinds.Kind` | `pkg/tools/sandboxkinds/registry` | pod, agent-sandbox |
| LLM providers | `pkg/agent/llm.Provider` | `pkg/agent/llm/providers` registry | anthropic, openai, openrouter |
| Web search providers | `pkg/tools/websearch.Provider` | overridable factory var in `cmd/oap/internal/toolscmd` (switch on the `--web-search` value) | anthropic, fake |
| Toolspec LLM | `pkg/tools/toolspec/llm.Provider` | none — unwired since the `oap tools gen` retirement | none in production (interface + `promptlib` adapter retained for a future toolspec-authoring surface) |
| Token broker | `pkg/platform/identity/broker.Broker` | DI in `internal/cmd/runner` + `internal/cmd/operator` | inproc |
| Memory stores | `pkg/memory.Backend` | DI in `internal/cmd/operator` | inmem, sqlite, shadow (postgres dual-write) |
| Search providers | `pkg/memory.SearchProvider` | DI in `internal/cmd/operator` | inmem, postgres, sqlite, graphiti |
| KG providers | `pkg/memory.KGProvider` | DI in `internal/cmd/operator` | graphiti |
| Embedding providers | `pkg/memory.EmbeddingProvider` | DI in `internal/cmd/operator` | openai-compatible |
| Artifact stores | `pkg/platform/artifactstore.Store` | DI (`ARTIFACT_STORE_URL`) | gocloud blob: gs/s3/azblob/file/mem |
| Cluster kinds | `pkg/platform/cloud.Strategy` | `pkg/platform/cloud` registry | local, desktop, default, gke, eks, aks |
| Terminal drivers | `pkg/cli/tui.Driver` | `pkg/cli/tui.DriverFor` (from `Caps`) | tty, plain, noninteractive |
| Aspect | Interface | Wiring | Backends today |
| --------------------------------- | ----------------------------------------------- | ------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| Channel transports | `pkg/channels/channelkinds.Kind` | `pkg/channels/channelkinds/registry` | agent, bento, browser, fake, github, local, slack |
| Artifact renderers | `pkg/channels/channelassets.Renderer` | `pkg/channels/channelassets/registry` | html, css, svg, image, mcpui (operator-only, not agent-selectable) |
| Tool CRD kinds (CLI presentation) | `pkg/tools/contract.Kind` | `pkg/tools/kinds/registry` | SpiceboxToolspec, MCPServer, SidecarToolbox |
| Toolspec parsers | `pkg/tools/toolspec/parser.Parser` | `pkg/tools/toolspec/parser/builtin` registry | declarative (only kind in production use today); builtin registry defined, none registered |
| AgentIdentity setup flows | `pkg/platform/identity/setup/builtins.Flow` | `builtins.Register` (`pkg/platform/identity/setup/builtins/registry.go`) | anthropic-oauth, github-pat, kubectl-kubeconfig, oauth-mcp, onepassword-scim, slack-bot-token, tailscale-authkey |
| AgentIdentity auth kinds | `pkg/platform/identity/authkind.Kind` | `authkind/registry` | cli, mcp, toolbox, toolspec |
| Identity credential types | `pkg/platform/identity/credkind.Kind` | `credkind/registry` (+ the `credkind/imports` bundle) | static, oauth, federated, githubapp |
| Session state kinds | `pkg/agent/session/state.Kind` | `state.Register` (`pkg/agent/session/state/registry.go`) | plans, deliveries, triggerstatus |
| Completion requirements | `pkg/agent/completion.Requirement` | `completion.Register` (`pkg/agent/completion/registry.go`) | artifact-delivered, plan-steps-complete, trigger-status-concluded |
| Agent harnesses | `pkg/agent/harness.Harness` | `pkg/agent/harness/registry` | ap-native |
| Interaction categories | `pkg/channels/channelinteractions.Category` | `pkg/channels/channelinteractions` registry | 44 registered (15 interactive + 29 notice) — see `pkg/channels/channelinteractions/categories` |
| Workspace sources | `pkg/platform/workspacekinds.Kind` | `pkg/platform/workspacekinds/registry` | git |
| Sandbox backends | `pkg/tools/sandboxkinds.Kind` | `pkg/tools/sandboxkinds/registry` | pod, agent-sandbox |
| Consequence judge | `pkg/agent/tool/browser/judge.ConsequenceJudge` | `pkg/agent/tool/browser/judge` registry (same package, `judge.Register`) | deterministic, anthropic-haiku |
| LLM providers | `pkg/agent/llm.Provider` | `pkg/agent/llm/providers` registry | anthropic, openai, openrouter |
| Web search providers | `pkg/tools/websearch.Provider` | overridable factory var in `cmd/oap/internal/toolscmd` (switch on the `--web-search` value) | anthropic, fake |
| Toolspec LLM | `pkg/tools/toolspec/llm.Provider` | none — unwired since the `oap tools gen` retirement | none in production (interface + `promptlib` adapter retained for a future toolspec-authoring surface) |
| Token broker | `pkg/platform/identity/broker.Broker` | DI in `internal/cmd/runner` + `internal/cmd/operator` | inproc |
| Memory stores | `pkg/memory.Backend` | DI in `internal/cmd/operator` | inmem, sqlite, shadow (postgres dual-write) |
| Search providers | `pkg/memory.SearchProvider` | DI in `internal/cmd/operator` | inmem, postgres, sqlite, graphiti |
| KG providers | `pkg/memory.KGProvider` | DI in `internal/cmd/operator` | graphiti |
| Embedding providers | `pkg/memory.EmbeddingProvider` | DI in `internal/cmd/operator` | openai-compatible |
| Artifact stores | `pkg/platform/artifactstore.Store` | DI (`ARTIFACT_STORE_URL`) | gocloud blob: gs/s3/azblob/file/mem |
| Cluster kinds | `pkg/platform/cloud.Strategy` | `pkg/platform/cloud` registry | local, desktop, default, gke, eks, aks |
| Terminal drivers | `pkg/cli/tui.Driver` | `pkg/cli/tui.DriverFor` (from `Caps`) | tty, plain, noninteractive |

**Rules of thumb:**

Expand Down
22 changes: 22 additions & 0 deletions PRIMITIVES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,28 @@ MCP servers, validated against a toolspec.
`channelevents.InteractionRequestPayload.Screenshot` — never placed in the
model's context. Slack uploads it into the approval thread, the web chat
embeds it inline, and the local TUI links it via a signed artifact-view URL.
Phase E gates browser MUTATE actions on CONSEQUENCE, not just on acting-tool
identity: a pluggable `ConsequenceJudge`
(`pkg/agent/tool/browser/judge/judge.go`, backends self-registered —
`deterministic.go`'s fast path and `judge/haiku/`'s Haiku-backed fallback)
classifies every click/type/press/select/submit into observe/navigate/
session/mutate; only `mutate` additionally requires a per-action
`web_action:<key>#act` grant on top of the coarse `web_host#act` — a stable,
readable-but-specific key derived from the call's own facts
(`pkg/agent/tool/browser/actionkey.go`'s `ActionKey`), scoped as its own
SpiceDB resource (`pkg/tools/browserschema/webaction.go`). That grant is
written caveated on a deterministic page-state fingerprint
(`pkg/agent/tool/browser/pagestate.go`'s `PageFingerprint`, checked by the
`pagestate_matches` SpiceDB caveat) captured at approval time, so a page that
meaningfully changed between the human's "yes" and the click re-prompts (the
TOCTOU revalidation) while benign churn elsewhere on the page does not.
Approval persists the grant and appends a `web_action_approval` append-only
audit record of what was approved against which page
(`pkg/agent/runner/web_action_approval.go`'s `persistWebActionApproval`,
`pkg/memory/kinds/webactionapproval/`) — the tamper-evident "what was approved
against what page" trail. The model never authorizes anything here: the judge
only decides whether the per-action grant is required, and the caveat — not
the model's say-so — is what SpiceDB actually enforces at dispatch.
- **Binaries:** `internal/cmd/streamclient/` (claims a streaming ToolCall via
the gateway); tool execution is otherwise driven from `internal/cmd/runner/`

Expand Down
Loading
Loading