From b38294ea49c9d370f325fec98e4be905fccde816 Mon Sep 17 00:00:00 2001 From: Ricardo Vega Date: Tue, 29 Sep 2026 06:33:32 -0500 Subject: [PATCH] Re-aim the contract at a CLI-first, Tyto-owned Chromium. SPEC draft 2: Tyto provisions a pinned Chrome for Testing build and exposes every DevTools capability (pages, input, console, network, cookies, storage, raw CDP) as `tyto` commands. No extension required. ATTACH and the identity vault are deferred, not deleted. Owner access is full but scoped: the CLI holds a power token for POWER_METHODS; Perch and MCP keep the safe token and PERCH_SAFE_METHODS. Secret values are masked unless the request says reveal, and revealed values are never persisted or sent to the model. IMPLEMENTATION gains Slices 15-19 (host hardening, provisioning, CLI, power surface, loop completeness), new ports, and a twelve-item definition of done. README and USAGE now state what runs today versus what is planned. Co-Authored-By: Claude Opus 5.5 (1M context) --- .claude/rules/protocol.md | 7 +- .claude/rules/security.md | 5 +- .cursor/rules/protocol.mdc | 8 +- .cursor/rules/security.mdc | 14 +- AGENTS.md | 7 +- CONTRIBUTING.md | 8 +- README.md | 89 ++++-- docs/DESIGN.md | 95 ++++--- docs/IMPLEMENTATION.md | 230 +++++++++++++-- docs/README.md | 3 +- docs/SPEC.md | 563 ++++++++++++++++++------------------- docs/USAGE.md | 37 ++- extension/README.md | 11 +- 13 files changed, 657 insertions(+), 420 deletions(-) diff --git a/.claude/rules/protocol.md b/.claude/rules/protocol.md index 6b6fed8..14324e0 100644 --- a/.claude/rules/protocol.md +++ b/.claude/rules/protocol.md @@ -2,7 +2,10 @@ JSON-RPC 2.0. Product methods, not CDP names. -- Perch and MCP may only call `PERCH_SAFE_METHODS`. -- No `debug.cdp` / `cdp.send` on that list. Vault: `identity.status` only. +- Two token scopes. **safe** (Perch, MCP) may only call `PERCH_SAFE_METHODS`. + **power** (the `tyto` CLI) may call `PERCH_SAFE_METHODS` ∪ `POWER_METHODS`. +- No `cdp.send` / cookies / storage values on the safe list. Vault: `identity.status` only. +- `POWER_METHODS` is disjoint from the safe set. Power results are masked + (`Redactor.mask`) unless the request says `reveal: true`. - JSON-serializable params. No `backendNodeId` in session payloads. - Errors to clients: code + message, not stacks. diff --git a/.claude/rules/security.md b/.claude/rules/security.md index 8a5e90c..3b9f408 100644 --- a/.claude/rules/security.md +++ b/.claude/rules/security.md @@ -3,5 +3,8 @@ Public repo. Never commit `.env`, profiles, cookies, tokens, keys, vault plaintext. - `npm run secrets:scan` and gitleaks must stay green. Do not print secret values. -- Identity: encrypt at rest; restore into Chrome/Edge only; grant per origin. +- Identity: encrypt at rest; restore into the browser only; grant per origin. +- Cookie/token values reach a caller only via the power scope with `reveal: true`; + never persisted, never to `ModelPort`, never to the safe scope. +- Never serve a token to an unauthenticated request. Check `Host` and `Origin`. - `Redactor` before model and tape. Loopback bind only. Confirm destructive acts. diff --git a/.cursor/rules/protocol.mdc b/.cursor/rules/protocol.mdc index eb24917..1c6f3c1 100644 --- a/.cursor/rules/protocol.mdc +++ b/.cursor/rules/protocol.mdc @@ -8,11 +8,15 @@ alwaysApply: false JSON-RPC 2.0. Methods are the product surface, not CDP method names. -- `PERCH_SAFE_METHODS` is the allowlist for Perch and MCP. +- `PERCH_SAFE_METHODS` is the allowlist for Perch and MCP (safe token). - Do not add `cdp.send`, `debug.cdp`, or credential read/write to that list. +- `POWER_METHODS` (raw CDP, cookies, storage, network bodies) is for the + `tyto` CLI's power token only, disjoint from the safe set, and masked + unless `reveal: true`. - `identity.status` is allowed. `identity.capture` is host-internal. - Errors: JSON-RPC `error.code` + stable `message`. No stack traces to clients. - Params/results are JSON-serializable. No `Map`, no `backendNodeId` in session payloads. -When adding a method: test that MCP tool list still equals the Perch-safe set. +When adding a method: test that MCP tool list still equals the Perch-safe set, +and that a safe token calling a power method is `unauthorized`. diff --git a/.cursor/rules/security.mdc b/.cursor/rules/security.mdc index bdb07e4..478bc98 100644 --- a/.cursor/rules/security.mdc +++ b/.cursor/rules/security.mdc @@ -30,13 +30,19 @@ tape.push("network", redactor.safe(hdr)); ## Identity Auth material is **data AND secret**. Encrypted at rest; DEK in OS keychain -(memory fake in tests). Restore **only** into Chrome/Edge. No OS ticket harvest, -no impersonation, no SDK export of cookies. +(memory fake in tests). Restore **only** into the browser. No OS ticket harvest, +no impersonation. + +Cookie/token values leave the host only to the **power** scope (the `tyto` +CLI) when the request says `reveal: true`. Masked otherwise. Never persisted +raw, never sent to `ModelPort`, never to the safe scope (Perch, MCP). Capture/restore requires an explicit origin grant. Expired bundles yield to the operator — do not replay a dead session. ## Control plane -Host and debug port: `127.0.0.1` + token. Extension: native messaging only; -no `window.tyto`. Confirm submit / purchase / delete / send. +Host and debug port: `127.0.0.1` + token. Reject non-loopback `Host` and +foreign `Origin`. Never serve a token to an unauthenticated request. +Extension (deferred): native messaging only; no `window.tyto`. The agent +confirms submit / purchase / delete / send. diff --git a/AGENTS.md b/AGENTS.md index 2c09acf..6665dad 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -10,14 +10,19 @@ rules live in [`.cursor/rules/`](.cursor/rules/). Claude Code also reads 2. **`npm test` is offline.** No Chrome, no network, no keys. `TYTO_LIVE=1` is opt-in. 3. **No god interfaces.** Ports in `packages/core/src/ports/` — one job each. 4. **`@tyto/core` is pure.** No Playwright, no CDP socket, no vendor LLM SDK, no `LiteLLM` type. -5. **Perch/MCP never see** `RawCdpPort` or `CredentialStorePort`. +5. **Perch/MCP never see** `RawCdpPort` or `CredentialStorePort`. They hold the + **safe** token. Only the `tyto` CLI (power token) reaches `POWER_METHODS`. 6. **Loopback only.** `127.0.0.1` + token. Never `0.0.0.0`. 7. **Trusted CDP input** for clicks. Not `element.click()` as the product path. 8. **No secrets in git.** Cookies/tokens never in session JSON, tape, or model prompts. Run `Redactor` first. + Power results are masked unless `reveal: true`; revealed values are never persisted. 9. After changes: `npm run check`. Do not promote `poc/` (Playwright spike) into product packages. +Product direction (SPEC draft 2): CLI-first, Tyto-owned pinned Chromium, no +extension required. ATTACH and the identity vault are deferred. + Full contract: [docs/IMPLEMENTATION.md](./docs/IMPLEMENTATION.md). ## Noctusoft LLM Relay (optional endpoint) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 35d7f75..03d725d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -49,9 +49,11 @@ To drive a real browser from this clone, follow [docs/USAGE.md](./docs/USAGE.md) | `packages/host` | Composition root, `127.0.0.1` + token | | `packages/sdk` | Client | | `packages/llm` | OpenAI-compatible + Anthropic HTTP | -| `packages/mcp` | Claude Code adapter | -| `packages/perch` | Sidebar view of the session file | -| `packages/poc` | Spike only; harvest algorithms, do not promote | +| `packages/mcp` | MCP adapter (safe scope) | +| `packages/perch` | Local view of the session file (safe scope) | +| `packages/cli` | `tyto` command, power scope (planned, Slice 17) | +| `packages/chromium` | Pinned Chrome for Testing provisioning (planned, Slice 16) | +| `poc/` | Playwright spike only; harvest algorithms, do not promote (`packages/poc` is a pointer README) | ## PRs diff --git a/README.md b/README.md index 044871d..234768c 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,31 @@ # Tyto -**An AI-first browser you drive in prose — without screenshots.** +**An AI-first browser you control from the command line — without screenshots +and without an extension.** Barn owls (*Tyto alba*) hunt in the dark by hearing alone. Tyto reads a page as a **document** (accessibility tree) and clicks as you would (**trusted CDP input**). -The durable object is the **prompt session on disk**, not a Chrome sidebar. +The durable object is the **prompt session on disk**. -You occupy **Google Chrome** or **Microsoft Edge** on macOS, Windows, and Linux. -The site cannot drive Tyto. The only door is a local SDK (`127.0.0.1` + token). +Tyto provisions its **own pinned Chromium** (Chrome for Testing) on macOS, +Windows, and Linux, and exposes everything DevTools can see and do — pages, +frames, input, console, network, cookies, storage, raw CDP — as `tyto` +commands. A human, a script, or Claude Code drives it the same way. The site +cannot drive Tyto. The only door is the local host (`127.0.0.1` + token). + +```bash +tyto start # pinned Chromium, dedicated profile +tyto open https://en.wikipedia.org +tyto type "Search Wikipedia" "barn owl" --enter +tyto extract "conservation status" +tyto logs --follow # console, network, exceptions +tyto cookies --json # values masked unless --reveal +tyto cdp Runtime.evaluate '{"expression":"document.title"}' +tyto run "open the barn owl article and tell me its conservation status" +``` + +> The `tyto` CLI and bundled Chromium are the **v1 target** (SPEC draft 2) and +> are being built now. See [Status](#status) for what runs today. A [YOLOVibeCode](https://github.com/YOLOVibeCode) public repo. Product: Noctusoft, Inc. @@ -43,8 +61,8 @@ last URL, browse, continue. ## What you use it for -Tyto is for operating the **same Chrome/Edge profile you already use**, in -prose, without a screenshot loop. +Tyto is for operating the web from a shell — by command or in prose — without +a screenshot loop and without handing a browser extension your session. | You paste… | Tyto does… | |---|---| @@ -53,11 +71,12 @@ prose, without a screenshot loop. | Forms | Fill from notes in the prompt; confirm-gate on submit / purchase / delete / send. | | Weave | You type name and address; Tyto does the rest of the wizard. You keep the keyboard. | | Resume | Quit the browser. Open the **same prompt file**. `goto` last URL, continue. | -| Claude Code | Same session document over MCP. Losing the MCP socket does not delete the work. | +| Claude Code | Shells out to `tyto … --json`, or connects over MCP. Same host, same session file. | +| Debug | `tyto logs --follow`, `tyto network`, `tyto cookies`, `tyto cdp `. | | Recipes | After a good run, replay **role + accessible name + landmark** — not `backendNodeId`. | -Improper: unattended agent on the profile that holds payroll or bank cookies -with a wide allowlist and no confirm-gates. You pick the profile explicitly. +Improper: unattended agent logged into payroll or banking with a wide +allowlist and no confirm-gates. Profiles are explicit and named. --- @@ -134,36 +153,43 @@ Limits that still apply: default-deny allowlist, confirm-gates, page text is ``` prompt session (disk) - → SDK / MCP / Perch 127.0.0.1 + token - → host (kernel) - ├─ LAUNCH Chrome/Edge + localhost CDP - └─ ATTACH native messaging → extension → chrome.debugger + → tyto CLI (power token) · MCP / Perch (safe token) 127.0.0.1 + → host daemon (kernel) + └─ LAUNCH Tyto Chromium (pinned Chrome for Testing) + loopback CDP + (ATTACH to an everyday Chrome/Edge via extension: deferred) ``` **Core** (`@tyto/core`) is pure TypeScript: ports, recipes, classify, allowlist, redaction, identity classification. Default CI never launches Chrome. -**Identity vault** (browser-scoped): capture *your* cookies/tokens (encrypted, -DEK in the OS keychain), restore into Chrome/Edge only. The model never sees -auth material. No Kerberos TGT harvest, no impersonation, no export to scripts. +**Staying logged in** is the dedicated profile (`~/.tyto/profiles/`): +log in once in the Tyto window. The encrypted identity vault is deferred. +Secret values are masked on every surface unless the owner asks `--reveal`, +and never reach the model. --- ## Status -**Slices 0–9 (contract) + host Perch UI** are in-tree and tested — including host -JSON-RPC on loopback, `GET /` Perch, model HTTP adapters (mocked), and CDP -trusted-click / OOPIF contract tests on a scripted wire. Live Chrome spawn is -`TYTO_LIVE=1` (`npm start`). The MV3 extension is specified; identity restore -is not the first-run path. `poc/` is a Playwright spike used to prove AX + tape; -it is not the product API. +**Works today (`npm start`):** loopback JSON-RPC host, Perch page, launch of an +*installed* Chrome/Edge with a dedicated profile, accessibility snapshot, +model-planned trusted clicks, operator weave. Nightly live E2E covers the +loop, weave, and a cookie capture/restore round-trip. + +**Not yet:** the `tyto` CLI, bundled Chromium, the power surface (cookies, +network, raw CDP), assistant replies and session resume in Perch, recipes, +agent confirm-gates, a runnable MCP server, ATTACH, and a host-wired vault. +Order of work: [SPEC §9](./docs/SPEC.md#9-build-order) · +[IMPLEMENTATION Slices 15–19](./docs/IMPLEMENTATION.md). + +`poc/` is a Playwright spike used to prove AX + tape; it is not the product API. --- ## Use it -There is **no packaged app** yet. You run the host from this repo. Opening -Chrome from the Dock does not start Tyto. +There is **no `tyto` command or packaged app** yet. You run the host from +this repo. Opening Chrome from the Dock does not start Tyto. **Full walkthrough:** [docs/USAGE.md](./docs/USAGE.md). @@ -189,7 +215,8 @@ Kill Perch: session JSON under `~/.tyto/sessions/` remains. ## Develop -Requires **Node 22+**. Tests must pass **offline**. +Requires **Node 24.15+ or 26** (the jsdom test dependency rejects older 22.x). +Tests must pass **offline**. ```bash npm test @@ -221,9 +248,11 @@ npx tsx poc/run.ts --url "https://en.wikipedia.org/wiki/Main_Page" --goal "…" | `@tyto/sdk` | Client | | `@tyto/cdp` | CDP adapter — **not Playwright** | | `@tyto/llm` | OpenAI-compatible + Anthropic HTTP | -| `@tyto/mcp` | Claude Code adapter (Slice 12) | -| `@tyto/perch` | Session sidebar (Slice 10) | -| `extension/` | MV3 ATTACH (Slice 11) | +| `@tyto/mcp` | MCP helpers; server not yet runnable (Slice 12) | +| `@tyto/perch` | Local session view (Slice 10) | +| `@tyto/cli` | `tyto` command — planned (Slice 17) | +| `@tyto/chromium` | Pinned Chrome for Testing provisioning — planned (Slice 16) | +| `extension/` | MV3 side panel; ATTACH deferred (Slice 11) | --- @@ -242,8 +271,10 @@ If you paste a real key: **rotate it**, do not just delete the commit. ## One-line tests (from the spec) +- If you must install an extension to let AI drive the browser, Tyto is not done. +- If DevTools can see it and `tyto` cannot, Tyto is not done. - If a power user would rather screenshot-agent the tab than paste a goal, Tyto is not done. - If they quit the browser and cannot reopen the **same prompt**, Tyto is not done. - If a website can drive the SDK, Tyto is not done. - If they can use it by hand (iframe, DHTML, proxied host) and Tyto cannot, Tyto is not done. -- If the model transcript contains a cookie or token, Tyto is not done. +- If a cookie or token lands in an agent transcript or on disk without `--reveal`, Tyto is not done. diff --git a/docs/DESIGN.md b/docs/DESIGN.md index 6c1f007..2f9cf48 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -2,7 +2,7 @@ **An AI-first browser you drive in prose — without screenshots.** -Noctusoft, Inc. — design doc, draft 2 +Noctusoft, Inc. — design doc, draft 3 (CLI-first, Tyto-owned Chromium) Full specification: [`SPEC.md`](./SPEC.md) (use cases, requirements, security, platforms). Implementation: [`IMPLEMENTATION.md`](./IMPLEMENTATION.md) (TDD + ISP). @@ -17,6 +17,19 @@ the tab, the sidebar, or a Chrome extension — **you must not lose the work.** --- +## Draft 3 in one paragraph + +Tyto ships its **own pinned Chromium** (Chrome for Testing, downloaded on +first run) with the AI plumbing built in, and the **command line is the +steering wheel**: `tyto open`, `tyto snapshot`, `tyto click`, `tyto logs`, +`tyto cookies`, `tyto cdp `. No extension to install, no existing +browser required. A human, a script, or Claude Code drives it the same way. +Everything below still holds — document not screenshot, trusted input, +prompt session on disk, weave — but occupying your everyday Chrome/Edge +(ATTACH) and the identity vault are **deferred**. See [`SPEC.md`](./SPEC.md). + +--- + ## What we are building now The interaction is the same one people already like in **Claude Code** and @@ -177,18 +190,18 @@ This is a product requirement, not a polish item. | Switch tabs, go back | Agent re-browses; never trusts last `ref_N` | | Fill half a form yourself | Agent reads the new AX tree and continues the *rest* | -The Chrome-extension debug banner (`chrome.debugger`) is the tax for -**occupying a browser you already live in** (your Edge/Chrome profiles). -Tyto pays that tax only in **attach** mode, and only because you asked for -100% CDP in *that* process. **Launch** mode starts Chrome or Edge itself with -a localhost debug port — full CDP, your chosen profile copy or path, no -store sandbox. +Tyto launches **its own Chromium** with a loopback debug port — full CDP, +a dedicated persistent profile, no extension, no store sandbox, no debugger +banner. (Draft 2’s ATTACH mode — an extension inside your everyday +Chrome/Edge — is deferred.) Either way, **the open web cannot drive it.** Control is a local SDK, not a page script, not a public API. -`perch` is the Claude Code analog: a thin sidebar. Chat lives there. The -prompt session on disk is the source of truth. The page stays the page. +The **CLI** is the Claude Code analog: you (or Claude Code itself) type +commands and goals in a terminal. `perch` is an optional local view of the +same sessions. The prompt session on disk is the source of truth. The page +stays the page. --- @@ -206,20 +219,19 @@ Not a random extension. Not a WAN-facing port. prompt session (disk) │ ▼ - Tyto SDK (Node / Python / MCP) ← only callers - │ 127.0.0.1 + token + tyto CLI (power token) MCP / Perch (safe token) ← only callers + │ 127.0.0.1 + token, Host/Origin checked ▼ - Tyto host (native, win / mac / linux) + Tyto host daemon (win / mac / linux) │ - ├─ LAUNCH spawn Chrome or Edge - │ user-data-dir = chosen profile - │ --remote-debugging-port=127.0.0.1:… + ├─ LAUNCH spawn Tyto Chromium (pinned Chrome for Testing) + │ or --browser chrome|edge| + │ user-data-dir = ~/.tyto/profiles/ + │ --remote-debugging-address=127.0.0.1 │ full CDP, no MV3 sandbox │ - └─ ATTACH native-messaging ↔ Tyto extension - already running in Chrome or Edge - extension auto-enables debugger on the tab - full CDP in *your* everyday profiles + └─ ATTACH (deferred) native messaging ↔ Tyto extension + inside an everyday Chrome/Edge ``` **SDK** is the product’s exterior: Claude Code, scripts, Perch backend, later @@ -241,9 +253,9 @@ your profiles). Firefox is not a day-one CDP peer; do not pretend. | Caller | Allowed | |---|---| -| Tyto SDK on localhost with session token | Yes — full CDP | -| Tyto extension via native host | Yes — only as the host’s hands | -| Perch UI (same origin as host) | Yes — paste / interrupt, not raw CDP from the page | +| `tyto` CLI on localhost with the power token | Yes — full CDP, cookies, storage, network (secrets masked unless `--reveal`) | +| MCP / Perch with the safe token | Yes — sessions, snapshot, act, tape; no raw CDP, no secret values | +| Tyto extension via native host (deferred) | Only as the host’s hands | | JavaScript on example.com | **No** | | Another extension | **No** | | Bind debug port on `0.0.0.0` | **No** — `127.0.0.1` only | @@ -307,16 +319,21 @@ how we know we failed, without photographing the screen. ## Components -**`tyto` host** — native, cross-platform. Launch or attach, token, allowlist, +**`tyto` CLI** — the primary client. Every browser capability as a command, +`--json` for agents. Power scope. + +**Tyto Chromium** — pinned Chrome for Testing, downloaded to +`~/.tyto/browsers/` and verified. Not a fork. + +**`tyto` host** — local daemon, cross-platform. Launch, tokens, allowlist, prompt sessions on disk. The kernel. **`tyto` SDK** — the only supported way to drive the host from the outside (TypeScript first, MCP as a thin adapter, Python when needed). Full automation surface: tabs, frames, AX snapshot, trusted input, tape, ready. -**`tyto` extension** — Chrome + Edge. Auto debug-attach for 100% CDP in -ATTACH mode. Native messaging only. Not a Store-shaped product that lives -without the host. +**`tyto` extension** (deferred) — Chrome + Edge. Auto debug-attach for ATTACH +mode. Native messaging only. Never required for CLI control. **`perch`** — one *view* of the prompt session. If Perch crashes, the session file is still on disk. Not the source of truth. @@ -347,18 +364,17 @@ CDP actions look like a real user. Nothing downstream will save you. ## Build order (for this product) -**Phase 1 — feel it.** Host + SDK + prompt session on disk. LAUNCH Chrome or -Edge (mac first, same code for win/linux). AX loop, interrupt, Perch paste. -Kill Perch mid-run and resume from the file. - -**Phase 2 — ATTACH.** Extension on Chrome and Edge, auto debugger, native -messaging. Same SDK. Your existing profiles, explicitly picked. - -**Phase 3 — Claude Code / MCP** as another SDK client. Same session file. - -**Phase 4 — weave + recipe replay.** Yield on real user input. +Draft-1 foundation is in-tree (host, AX loop, trusted input, weave, Perch). +Next, in order (SPEC §9): -**Phase 5 — unattended runner.** Exit codes. Not day-one occupancy. +1. **Harden the host** — token scopes, Host/Origin checks. +2. **Provision Chromium** — pinned Chrome for Testing, out of the box. +3. **`tyto` CLI + daemon** — primitives from a shell. +4. **Power surface** — raw CDP, cookies, storage, network, event streams. +5. **Loop completeness** — replies, re-observe, confirm-gate, recipes, resume. +6. **MCP server** on the safe surface. +7. **Unattended runner** live. +8. Later: ATTACH, identity vault. Scholarmancy may later implement `waitReady` / `snapshot` on `IPageDriver`. That is a port, not a merge. @@ -406,7 +422,8 @@ Ruled out: Lumen (vision-first agent already). ## One-line test -If a native power user would rather screenshot-agent the tab than paste into -Perch, the product is not done. If they paste, glance, and type into the same +If you need an extension to let AI drive the browser, the product is not +done. If a native power user would rather screenshot-agent the tab than type +`tyto run`, the product is not done. If they paste, glance, and type into the same field without waiting on a photograph — that is Tyto. If they quit Chrome and cannot open the same prompt and continue — that is not Tyto. diff --git a/docs/IMPLEMENTATION.md b/docs/IMPLEMENTATION.md index ad76237..d278445 100644 --- a/docs/IMPLEMENTATION.md +++ b/docs/IMPLEMENTATION.md @@ -3,7 +3,10 @@ Noctusoft, Inc. Companion to [`SPEC.md`](./SPEC.md) and [`DESIGN.md`](./DESIGN.md) Operator start path: [`USAGE.md`](./USAGE.md) -Status: draft 1 — this is the engineering contract +Status: draft 2 — this is the engineering contract. Draft 2 re-aims the +product at a CLI-first, Tyto-owned Chromium (SPEC draft 2). Slices 0–14 are +unchanged history; Slices 15–19 are the new work. ATTACH (Slice 11) and the +identity vault (Slice 9b) are deferred, not deleted. This plan is how we build the spec without growing a god object or a Playwright script with a chat box glued on. **Tests define behavior. Ports @@ -40,7 +43,9 @@ harvest algorithms from it, do not promote it. ### Control plane (from the spec) -- SDK / debug: `127.0.0.1` + token only. +- SDK / debug: `127.0.0.1` + token only. Two token scopes: **safe** + (Perch, MCP) and **power** (CLI). Secret values are masked unless a power + call asks `reveal: true`. - Page JS is data, never a command channel. - Default-deny allowlist. Explicit profile. Confirm on destructive acts. - Prompt session on disk is source of truth. @@ -65,6 +70,9 @@ harvest algorithms from it, do not promote it. | Perch v1 | Local UI that is an SDK client (side panel later) | Session file outlives the UI | | Extension | MV3, Chrome + Edge, native messaging only | ATTACH mode; no `window` command API | | Monorepo | npm workspaces | ISP at package boundaries | +| Browser | Pinned **Chrome for Testing**, downloaded by Tyto (`@tyto/chromium`). Installed Chrome/Edge is an override | Out of the box, no auto-update drift, `--load-extension` still allowed, no fork | +| Primary client | `@tyto/cli`, bin `tyto`, power scope, `--json` everywhere | The command line is first-class; agents shell out to it | +| Token scopes | `safe` → `PERCH_SAFE_METHODS`; `power` → safe ∪ `POWER_METHODS` | Full access for the owner without widening Perch/MCP | Playwright may remain in `poc/` until the first live adapter is green. It must not appear in `packages/core` or in default CI. @@ -74,9 +82,10 @@ must not appear in `packages/core` or in default CI. ## 2. Package map (dependency direction) ``` -@tyto/sdk ──┐ +@tyto/cli ──┐ (power scope) +@tyto/sdk ──┤ @tyto/mcp ──┼──► @tyto/protocol (JSON-RPC types only) -@tyto/perch ─┘ │ +@tyto/perch ─┘ │ (mcp + perch: safe scope) ▼ @tyto/host ──wires──► adapters │ @@ -91,8 +100,10 @@ must not appear in `packages/core` or in default CI. @tyto/fs implements SessionStore, RecipeArchive @tyto/secrets implements SecretStore (OS keychain; memory fake in tests) IdentityVault (encryption layer + DEK via SecretStore) -@tyto/cdp also implements CredentialStorePort -extension/ native-messaging peer of host (not imported by core) +@tyto/cdp also implements CredentialStorePort, CookieJar, + DomStorageReader, NetworkBodies, CdpEvents +@tyto/chromium implements BrowserProvisioner (download, verify, extract) +extension/ native-messaging peer of host (deferred; not imported by core) ``` **Forbidden imports** @@ -104,9 +115,13 @@ extension/ native-messaging peer of host (not imported by core) | `@tyto/mcp` | cdp, RawCdp method names, CredentialStorePort | | Perch / MCP surfaces | VaultHandle, RawCookie, RawStorageItems | | `@tyto/perch` | cdp | +| `@tyto/cli` | cdp, host, core adapters — only `@tyto/sdk` + `@tyto/protocol` | +| `@tyto/chromium` | cdp, host, llm | | adapters | each other, except host composition root | -The **composition root** is `packages/host/src/main.ts` only. +The **composition root** is `packages/host/src/main.ts` only. (Draft-1 code +also builds CDP adapters in `attach-cdp.ts` after launch; Slice 15 folds that +back into the root.) --- @@ -361,6 +376,54 @@ interface Redactor { - `AuthProfiler` is pure and has no network call; the adapter feeds it recorded `AuthEvidence` (request headers, storage keys, cookie names). +### 3.8 Provisioning and the power surface (draft 2) + +```ts +// BrowserProvisioner — @tyto/chromium. Fetch and extract are injected +// sub-ports so `npm test` never touches the network. +interface BrowserProvisioner { + installed(pin: ChromiumPin, platform: Platform): Promise; + ensure(pin: ChromiumPin, platform: Platform): Promise; // download once, verify, extract +} +// ChromiumPin = { version, downloads: Record } — tracked file +// Platform = "mac-arm64" | "mac-x64" | "linux64" | "win64" + +// CookieJar — the browser's jar (all origins). Not the vault's +// CredentialStorePort; the owner's CLI reads it, masked by default. +interface CookieJar { + list(filter?: { urls?: string[] }): Promise; + set(cookies: CookieRecord[]): Promise; + delete(match: CookieMatch): Promise; + clear(): Promise; +} + +interface DomStorageReader { + read(origin: Origin, kind: "local" | "session"): Promise; +} + +interface NetworkBodies { + body(requestId: string): Promise<{ body: string; base64: boolean } | null>; +} + +interface CdpEvents { + subscribe(domains: string[], fn: (e: CdpEvent) => void): Unsubscribe; +} + +// Redactor gains structured masking for power results: +// mask(value: unknown): unknown // cookie `value`, Cookie/Set-Cookie/Authorization headers +``` + +**ISP rules for the power surface:** + +- `POWER_METHODS` lives in `@tyto/protocol` next to `PERCH_SAFE_METHODS` and + is disjoint from it. MCP tool list and Perch still equal the safe set. +- `RawCdpPort`, `CookieJar`, `DomStorageReader`, `NetworkBodies`, and + `CdpEvents` are wired only into power-scope dispatch. +- Every power result passes through `Redactor.mask` unless the request says + `reveal: true`. Revealed values are returned, never persisted. +- Owner commands (power scope) are not subject to the agent allowlist or + confirm-gate. `AgentLoop` always is. + --- ## 4. Domain modules (pure) @@ -398,17 +461,22 @@ They contain the product brain. ## 5. Test layers ``` -npm test → unit + fake loop < 2s, no Chrome -npm test:contract → adapters vs fixtures no live browser -npm test:live → TYTO_LIVE=1 real Chrome optional -npm test:security → bind, token, allowlist unit + contract +npm test → unit + contract + fake loop no Chrome, no network, no keys +npm run test:e2e → TYTO_E2E=1 TYTO_LIVE=1 real browser optional, nightly ``` +Contract (recorded CDP) and security tests run inside `npm test`; there are +no separate `test:contract` / `test:security` scripts. + Naming: `describe("bind")` / `it("refuses a ref from a previous snapshot")`. Test names are spec sentences. Fixtures live in `packages/core/fixtures/` and `packages/cdp/fixtures/`: +Status: only `ax/react-shell.json`, `ax/wikipedia-search.json`, and +`local-state/chrome.json` exist today (under `packages/cdp/fixtures/`). The +rest are owed; DoD item 3 needs the Wayback fixture. + | Fixture | Use | |---|---| | `ax/wikipedia-search.json` | compact + bind Search | @@ -701,7 +769,10 @@ UI can be ugly. Occupancy and resume are the product. --- -### Slice 11 — ATTACH (extension + native messaging) +### Slice 11 — ATTACH (extension + native messaging) — deferred + +Deferred by SPEC draft 2 (§3.10). Prior work lives on the `ci-deploy` +branch and is not on `main`. **Tests first (protocol)** @@ -745,11 +816,102 @@ Then live attach on a throwaway profile. - exit 0 on `done` - exit 2 on `ShellNotReady` - exit 3 on allowlist deny -- exit 4 on confirm required and `--no-confirm` +- exit 4 on confirm required (the default; there is no `--no-confirm` flag) - no HITL in runner unless `--allow-confirm-fail` --- +### Slice 15 — host hardening and token scopes + +Prerequisite for the power surface: raw CDP and cookies make a leaked +token catastrophic. + +**Tests (write first)** + +- `GET / never returns a token in Set-Cookie or body to an unauthenticated request` +- `safe token calling a POWER_METHODS method → unauthorized, adapter spy 0` +- `power token may call safe and power methods` +- `request with Host other than 127.0.0.1: or localhost: → 421/403` +- `POST with an Origin that is not the host origin → 403` +- `token files are written 0600 and host.json contains no token` +- `body is not read before auth` (401 without consuming a large body) +- `composition: CDP adapters are built in the root, not in dispatch` + +--- + +### Slice 16 — Chromium provisioning (`@tyto/chromium`) + +**Tests (write first; fake fetch + fake extract, tmp dir)** + +- `pin resolves the download URL for mac-arm64, mac-x64, linux64, win64` +- `ensure downloads once; second ensure is a cache hit (fetch spy 0)` +- `checksum mismatch fails closed and leaves no version directory` +- `interrupted extract leaves no version directory` +- `installed returns the executable path per platform` +- `launcher prefers the provisioned binary; --browser chrome uses the installed one` +- `launch args: --remote-debugging-address=127.0.0.1, dedicated user-data-dir, no-first-run` +- `win64 executable discovery uses chrome.exe and ; PATH separator` + +Live (opt-in): `tyto browser install` then launch, `Browser.getVersion` +matches the pin. + +--- + +### Slice 17 — `tyto` CLI and daemon (`@tyto/cli`) + +The CLI is an SDK client. It imports `@tyto/sdk` and `@tyto/protocol` only. + +**Tests (write first; host over loopback with fakes)** + +- `tyto start writes host.json (port, pid) and returns when the host answers` +- `tyto start when already running reports the running host, exit 0` +- `tyto status with no host → exit 69` +- `tyto open grants that origin, then page.goto` +- `tyto snapshot --json prints the compact tree and generation` +- `tyto click "Search" --role button binds role+name then trusted page.act` +- `tyto type --enter fills then presses Enter` +- `tyto run "" writes a session, runs, prints the assistant reply` +- `every command supports --json and prints one JSON document (NDJSON for streams)` +- `unknown command → exit 64 with usage` +- `CLI source imports only @tyto/sdk and @tyto/protocol` (grep test) +- `tokens and settings come from ~/.tyto, not the repo .env` + +--- + +### Slice 18 — power surface + +**Tests (write first; recorded CDP wire)** + +- `POWER_METHODS is disjoint from PERCH_SAFE_METHODS; MCP tools unchanged` +- `cookies.list masks value by default; reveal:true returns it` +- `cookies.list includes httpOnly cookies via CDP, never document.cookie` +- `cookies.set / delete / clear send the Storage/Network CDP methods` +- `storage.read masks values by default` +- `cdp.send forwards any Domain.method (browser or target session)` +- `cdp.send result masks cookie values and Cookie/Set-Cookie/Authorization headers unless reveal` +- `network tape records request/response/failure metadata; headers redacted before persist` +- `network.body fetches Network.getResponseBody on demand` +- `cdp events stream delivers subscribed domains only; unsubscribe on client close` +- `revealed values never appear in session JSON or persisted tape` (grep) + +--- + +### Slice 19 — agent loop completeness + +**Tests (write first; fakes)** + +- `run appends an assistant message with the answer or failure reason` +- `after an act, the loop waits on tape then re-snapshots; old refs do not bind` +- `AgentLoop consults Allowlist and ConfirmGate before every perform` +- `confirm required with no operator → exit 4 / Failed, Actuation spy 0` +- `successful steps are remembered as recipes (role, name, origin), never node ids` +- `second run on the same origin replays recipes with ModelPort spy 0` +- `extract step runs Extractor and stores the answer` +- `system prompt is the fixed core preamble with the plan schema` +- `nested step refs are stripped before the session is saved` + +--- + ## 7. AgentLoop state machine (test every arrow) ``` @@ -785,13 +947,17 @@ optimization; Wayback test forbids model on extract). ## 8. Mapping to product phases -| Spec phase | Slices | Feel it when | +| Spec phase (SPEC §9) | Slices | Feel it when | |---|---|---| -| 1 Host + SDK + LAUNCH + Perch paste + resume | 0–10, **6b**, **9b** | Kill Perch, reopen, continue; Wikipedia-class task; **portal+iframe fake green** | -| 2 ATTACH | 11 | Edge profile you picked (Workday cookies live here), debugger banner, same SDK | -| 3 MCP | 12 | Claude Code on the same session file | -| 4 Weave + recipe replay | 6 (fake) + 13 (live) | Type in the field; second visit skips think | -| 5 Unattended | 14 | Exit codes | +| Draft-1 foundation (done) | 0–10, 6b, 12 (library), 13, 14 (core) | `npm start`, Perch Go, weave live | +| 1 Harden host | 15 | Unauthenticated GET gets nothing; safe token cannot reach power methods | +| 2 Provisioning | 16 | Fresh machine, no Chrome installed, browser comes up | +| 3 CLI | 17 | `tyto open`, `tyto snapshot`, `tyto click` from a shell | +| 4 Power surface | 18 | `tyto cookies`, `tyto logs --follow`, `tyto cdp` anything | +| 5 Loop | 19 | `tyto run` answers, replays recipes, asks before Delete | +| 6 MCP server | 12 (server) | Claude Code connects over stdio | +| 7 Unattended live | 14 (host) | `tyto run --unattended` exit codes | +| Later | 11, 9b | ATTACH, vault | Do not implement Phase 2 UI before Slice 6 is green. That is how god objects are born. @@ -824,6 +990,8 @@ End of week 1: `npm test` tells the Tyto story with no browser — including - Extension `window.tyto = { click }` - Host listen `0.0.0.0` - Silent default to a named Chrome/Edge profile +- Requiring an extension for CLI or agent control +- Forking Chromium instead of pinning Chrome for Testing - One `Browser` interface “for convenience” - MCP tools that are raw CDP method names - Importing `@tyto/cdp` from Perch or MCP @@ -832,7 +1000,9 @@ End of week 1: `npm test` tells the Tyto story with no browser — including - Cookies or tokens in the session JSON (vault handles only) - Any auth material (cookie value, bearer token) in a tape event, model prompt, or log - Harvesting OS credential-store, Kerberos TGT, or LSASS (out of scope) -- Exporting auth material to non-browser callers via the SDK +- Returning unmasked auth material without `reveal: true`, or to the safe scope at all +- Persisting a revealed value (session, tape, log) +- Serving any token to an unauthenticated HTTP request - Logging `Set-Cookie` or `Authorization` header values - Calling `document.cookie` to read httpOnly cookies (use `Network.getAllCookies`) - Replaying an expired bundle without operator re-auth (prompt, do not silently 401) @@ -846,7 +1016,7 @@ End of week 1: `npm test` tells the Tyto story with no browser — including From the spec one-line tests, now as CI: 1. Fake + live: paste-to-trusted-click does not screenshot. -2. Host restart: same session id continues. +2. Host restart: same session id continues (`tyto resume`). 3. Wayback fixture: static extract, `ModelPort` call count 0. 4. Shell fixture: extract blocked. 5. Security: unauthorized RPC, bind loopback, page has no command API @@ -854,14 +1024,20 @@ From the spec one-line tests, now as CI: 6. Occupancy: operator typing prevents `perform` (fake + live). 7. Hosted app: portal shell + tenant iframe; act in the granted child; recipes keyed by tenant origin. -8. Identity vault: quit browser, reopen session, agent resumes authenticated - on a cookie-session origin without operator re-typing a password. +8. Staying logged in: quit the browser, `tyto start` again with the same + profile, agent resumes authenticated without re-typing a password + (profile persistence; the vault is deferred). 9. Redaction: full model transcript from authenticated run contains zero cookie or token values; verified by grep over the session file. -10. Vault at rest: bundle file is ciphertext; no fixture or test writes - plaintext cookies to disk. - -Until those six are green, we are not “done with Phase 1.” +10. Secrets at rest: no fixture, test, session, or tape writes a plaintext + cookie or token to disk. +11. Out of the box: `tyto start` on a machine with no Chrome installed + brings up the pinned Chromium, no extension. +12. Full access: every DevTools domain is reachable via `tyto cdp`; cookies, + storage, console, and network are reachable via first-class commands, + masked unless `--reveal`. + +Until all twelve are green, we are not done with v1. --- diff --git a/docs/README.md b/docs/README.md index d5a951e..42a73f2 100644 --- a/docs/README.md +++ b/docs/README.md @@ -2,6 +2,7 @@ | File | What | |---|---| -| [SPEC.md](./SPEC.md) | Product specification, use cases, security | +| [SPEC.md](./SPEC.md) | Product specification (draft 2: CLI-first, Tyto-owned Chromium) | +| [USAGE.md](./USAGE.md) | How to run a clone today | | [DESIGN.md](./DESIGN.md) | Thesis: prompt-native, weave, control plane | | [IMPLEMENTATION.md](./IMPLEMENTATION.md) | TDD slices, ISP ports, anti-patterns | diff --git a/docs/SPEC.md b/docs/SPEC.md index 22ad77a..608a3db 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -1,45 +1,57 @@ # Tyto — Product specification Noctusoft, Inc. -Status: draft 1 (companion to [`DESIGN.md`](./DESIGN.md)) +Status: draft 2 — CLI-first, Tyto-owned Chromium (supersedes draft 1’s +“occupy your existing Chrome/Edge” framing) +Design: [`DESIGN.md`](./DESIGN.md) Implementation: [`IMPLEMENTATION.md`](./IMPLEMENTATION.md) — TDD + ISP engineering contract -How to run a clone: [`USAGE.md`](./USAGE.md) +How to run a clone today: [`USAGE.md`](./USAGE.md) Audience: anyone building, reviewing, or deciding whether to use Tyto Tyto is a **separate product**. It is not a Scholarmancy feature, not a -generic “AI Chrome extension,” and not a vision-based computer-use agent. +browser extension, and not a vision-based computer-use agent. --- ## 1. What it is -Tyto is an **AI-first browser control system**. You drive the web in prose -(a pasted goal, a conversation) the way you drive Claude Code — except the -hands are on a real Chrome or Edge window, reading the page as a **document** -(accessibility tree), not as a **screenshot**. - -Three properties are load-bearing: - -1. **Prompt-native.** The durable object is the prompt session (goal, plan, - conversation, recipes, answers, last URL). Perch, Claude Code, and the - browser are views and hands. Kill the sidebar or the tab: the work remains. -2. **Native occupancy.** You and the agent share the browser. You click and - type at any time. It yields and re-reads. No screenshot lag. No “possessed - tab” that you cannot touch. -3. **Full control, closed exterior.** Internally, Tyto has 100% CDP (every - DevTools-class capability needed to operate a page). Externally — the page, - other extensions, the public internet — **cannot** issue those commands. - The only door is a **local SDK** (token on `127.0.0.1`). -4. **If you can use it, Tyto can use it.** Anything you can reach and operate - in that Chrome/Edge profile — top-level page, iframe/OOPIF, DHTML, JS- - injected SPA, shadow tree, popup, or a host that reverse-proxies / injects - another app — Tyto must be able to perceive and act on **the same - document**, programmatically, with trusted input. Delivery mechanism is - not a special case. If your hands work and Tyto’s do not, that is a bug. -5. **Identity continuity.** Staying logged in is part of “using” a site. - The browser profile contains the session; the vault makes it durable. - Runs survive restarts; re-auth is silent where the IdP session is warm. - Auth material is encrypted, per-origin, and never reaches the model. +Tyto is an **AI-first browser you control from the command line.** It +provisions its own pinned Chromium, launches it with a dedicated profile, +and exposes everything DevTools can see and do — pages, frames, input, +console, network, cookies, storage — as first-class commands. A human, a +script, or an AI agent drives it the same way. + +You do not install an extension to let AI into a browser. The plumbing is +the product. + +Load-bearing properties: + +1. **Out of the box.** `tyto start` downloads a pinned Chromium on first + run, launches it with a dedicated persistent profile, and opens the + DevTools socket on loopback. No existing Chrome/Edge is required. No + extension. No store. +2. **The command line is first-class.** Every capability is a `tyto` + command with `--json` output: navigate, snapshot, click, type, extract, + logs, network, cookies, storage, tabs, and raw CDP. Claude Code or any + script drives the browser by shelling out to it. +3. **Full access for the owner.** The DevTools protocol is not hidden behind + a narrow API. The owner’s CLI holds a **power token** and can read and + write everything the browser holds. Secret values (cookie values, + auth headers) are **masked unless `--reveal`** so they do not leak into + an agent transcript by accident. +4. **Document, not screenshot.** Perception is the accessibility tree. + Action is trusted CDP `Input`. Waits are page signals, not sleeps. +5. **Prompt-native.** Goal runs persist as a session document on disk + (goal, plan, conversation, recipes, answers, last URL). Kill the CLI, + the host, or the browser: the work remains. +6. **Shared occupancy.** The launched browser is a normal headed window. + You can click and type in it at any time; the agent yields and re-reads. +7. **If you can use it, Tyto can use it.** Top-level page, iframe/OOPIF, + DHTML, SPA shell, shadow tree, popup, reverse proxy — same ports, same + trusted input, on the document you would have used. +8. **Closed exterior.** The only door is the local host on `127.0.0.1` with + a token. Page JavaScript cannot command Tyto. Nothing listens on a + public interface. Hunt in the dark: the owl hears the tree, it does not photograph the field. @@ -49,300 +61,269 @@ Hunt in the dark: the owl hears the tree, it does not photograph the field. | Actor | Role | |---|---| -| **Operator** | You. Paste goals, interrupt, take the keyboard, pick a Chrome/Edge profile, confirm destructive actions. | -| **Host** | Native Tyto process on the machine (macOS, Windows, Linux). Kernel: sessions, token, allowlist, launch/attach. | -| **SDK client** | Perch, Claude Code (MCP), scripts, later other local apps. Speaks only to the host. | -| **Browser** | Google Chrome or Microsoft Edge. Launch (Tyto spawns it) or Attach (extension + auto debugger on a running browser). | -| **Model** | Any OpenAI-compatible (or Anthropic) endpoint you configure. Tyto does not hardcode a vendor or LiteLLM. | +| **Operator** | You. Start the browser, run commands or goals, take the keyboard, confirm destructive agent actions. | +| **CLI** (`tyto`) | The primary client. Holds the **power** token. Every browser capability. | +| **Agent** | Claude Code, a script, CI. Drives Tyto through the CLI (`--json`) or MCP. | +| **Host** | Local daemon. Owns the browser process, tokens, sessions, allowlist, confirm-gates. | +| **Browser** | **Tyto Chromium** — a pinned Chrome for Testing build that Tyto downloads. Installed Chrome/Edge is an optional override. | +| **Model** | Any OpenAI-compatible (or Anthropic) endpoint you configure, used by `tyto run`. No vendor or LiteLLM types. | +| **Safe clients** | Perch (web view) and the MCP adapter. **Safe** token, `PERCH_SAFE_METHODS` only. | -Not an actor: JavaScript on a website, a remote SaaS “drive my Chrome,” -another browser extension. +Not an actor: JavaScript on a website, another extension, a remote service. --- -## 3. Proper use cases +## 3. Use cases -These are in-scope. If a request is not on this list, it is not why Tyto -exists — even if the engine could be abused to do it. +### 3.1 Drive the browser from a shell (primary) -### 3.1 Interactive operator (primary) +```bash +tyto start +tyto open https://en.wikipedia.org +tyto snapshot +tyto type "Search Wikipedia" "barn owl" --enter +tyto extract "conservation status" +tyto logs --follow +tyto cookies --url https://en.wikipedia.org +tyto cdp Page.captureScreenshot '{"format":"png"}' > page.json +``` -You are in the browser, working. You paste a goal. Tyto plans once, acts -with trusted input, waits on real page signals (navigation, injected HTML), -and stays out of your way when you type. +Deterministic primitives. Role + accessible name, not CSS selectors. Refs +from `snapshot` are valid for that snapshot only. -Examples: +### 3.2 An AI agent as the driver -- Research: “Open the barn owl article and extract conservation status.” -- Ops: “Go to this invoice page and download the CSV.” -- Forms: “Fill this form from the notes in the prompt; stop before Submit.” -- Weave: you fill the name and address; Tyto does the rest of the wizard. -- Resume: you quit Edge, open Tyto, same prompt, continue from last URL. +Claude Code (or any agent) runs `tyto … --json` from its shell tool, or +connects over MCP. Same host, same browser, same session file. The agent +sees masked secrets unless the operator’s command says `--reveal`. -Success looks like a native power user: seconds of thinking, milliseconds of -clicking, no photograph of the screen per step. +### 3.3 Goal runs -### 3.2 Claude Code as a second pair of hands +```bash +tyto run "open the barn owl article and tell me its conservation status" +``` -Same session document. You paste in the terminal or in Perch. One host, one -Chrome/Edge, one file. Losing the MCP connection does not delete the prompt. +Tyto plans once, acts with trusted input, waits on real page signals, and +writes the answer into the session. Known steps (recipes) replay with no +model call. `tyto resume ` continues after a restart. -### 3.3 Repeatable local recipes +### 3.4 Full observability -After a successful run, the session stores **recipes** (role + accessible -name + landmark), not CDP node ids. Next time the same origin appears, Tyto -replays without calling the model. That is how it becomes faster than you. - -### 3.4 Authenticated work in *your* profiles (explicit) +Everything a power user glances at in DevTools, as data: -You pick a Chrome or Edge profile (e.g. work vs personal). Launch clones or -opens it with a debug port; Attach uses the live browser via the extension. -You stay logged into the sites that profile already has — **because you -opted into that profile**, not because Tyto scraped your OS silently. +- console messages and uncaught exceptions +- navigations, frame attach/detach, DOM inject events +- network requests and responses (URL, method, status, timing, headers), + failures, and response bodies on demand +- cookies (including httpOnly) and local/session storage -Proper: “Use Edge profile Ambient for this session.” -Improper: unattended agent on the profile that holds payroll, TEA, or bank -cookies with a wide allowlist and no confirm-gates. +Live (`--follow`) and recorded (session tape). Recorded tape is redacted +before it is written. -### 3.5 Static vs dynamic pages +### 3.5 Staying logged in -- **Static HTML** (old sites, Wayback identity captures, server-rendered - articles): classify `static`, snapshot AX, extract. No inject wait. -- **Injected HTML** (React shells, `#root`, hydration): wait until the tree - grows; if it stays a shell, **fail closed** — do not invent data. +The Tyto profile persists at `~/.tyto/profiles/`. Log in once by hand +in the Tyto window; the profile keeps the session across restarts like any +browser. Multiple named profiles separate work from personal. -### 3.6 Observability while operating +### 3.6 Repeatable recipes -Console, exceptions, navigations, failed scripts — a DevTools-class tape so -you (and the planner) know whether the page moved, without a screenshot. -Broken 2008 Wikipedia JS on Wayback is noise; a `TypeError` after your click -is signal. +After a successful run, the session stores **recipes** (role + accessible +name + landmark + frame origin), not CDP node ids. Next time the same +origin appears, Tyto replays without calling the model. -### 3.7 Unattended local runs (secondary, later) +### 3.7 Static vs dynamic pages -A saved session/recipe with exit codes, domain allowlist, and confirm policy -(fail or skip destructive steps). CI-style on a machine you own. Not a -cloud browser farm in v1. +- **Static HTML**: classify `static`, snapshot AX, extract. +- **Injected HTML** (React shells, hydration): wait until the tree grows; + if it stays a shell, **fail closed** — do not invent data. ### 3.8 Whatever you can reach (iframe, DHTML, inject, proxy) -Workday-on-a-portal is one instance of a general rule: **human-reachable -in this profile ⇒ Tyto-reachable through the SDK.** - -The document may arrive as: +**Human-reachable in this profile ⇒ Tyto-reachable.** | How it shows up | What Tyto does | |---|---| -| Top-level navigation | `goto` / you type the URL; snapshot that target | +| Top-level navigation | `open` / `goto`; snapshot that target | | iframe / OOPIF | Auto-attach; focus the working frame; click in **that** session | | DHTML / `innerHTML` / hydrate / CSR | Classify shell vs injected; `waitReady` on **that frame**; fail closed if still empty | -| Shadow DOM / web components | Pierce; AX is truth, not `page.evaluate` of light DOM | +| Shadow DOM / web components | AX is truth, not `page.evaluate` of light DOM | | Popup / new tab (SSO, print) | Related targets; yield for MFA; then operate the app target | -| Reverse proxy / vanity host / injected proxy | Still one or more Chromium documents. Same ports. No “proxy driver.” | +| Reverse proxy / vanity host | Still one or more Chromium documents. Same ports. No “proxy driver.” | There is no Workday package, no DHTML package, no proxy package. There is -`FrameGraph` + `Readiness` + trusted `Actuation` on whatever target you -could have used yourself. - -Limits that still apply: allowlist (you grant origins), confirm-gates, -page text is data, no screenshot loop unless the thing is actually -pictorial (canvas/WebGL). “I can see it” includes those — pictorial -fallback is the exception, not the default. +`FrameGraph` + `Readiness` + trusted `Actuation`. -### 3.9 Identity continuity (browser-scoped) +### 3.9 Unattended local runs -You can use any authenticated site in your profile. Tyto can too — and -it must stay logged in across restarts so the prompt-native promise is real -on authenticated sites. +A saved session/recipe with exit codes, allowlist, and confirm policy +(fail or skip destructive steps). On a machine you own. Not a cloud farm. -The vault identifies how each allowed origin authenticates, captures the -browser-scoped session material, caches it encrypted at rest, and restores -it silently when the session resumes. +### 3.10 Your everyday Chrome/Edge (deferred) -**Auth methods Tyto identifies and preserves:** - -| Method | What is captured | How re-auth works | -|---|---|---| -| Cookie session | `Set-Cookie` sessions (incl. httpOnly, via CDP) | Re-inject cookies before `goto` | -| OAuth / OIDC bearer | Token in localStorage / sessionStorage / IndexedDB | Re-inject; refresh if expiring | -| SAML SSO chain | SP + IdP session cookies | Warm IdP session first; SP auto-redirects | -| Negotiate / Kerberos / IWA | Nothing captured — handled by the real profile | Ensure LAUNCH flags allow IWA; yield for MFA | - -**Rules:** - -- Per-origin grant, default-deny. Discovering an origin through the frame - graph does not grant it vault access. -- Confirm-gate on first capture of any origin and on restore of sensitive - origins (payment, HR, identity provider). -- Expiry-aware: detect dead or expiring bundles and prompt the operator for - re-auth rather than replaying a session that will 401. -- ATTACH mode already inherits the live profile's cookies. Vault capture - there is observation, not escalation — but the consent notice still applies. -- Auth material is **never** in the session JSON, never in the tape, never - in a model prompt, never in git. The session references a vault handle. -- The vault's data-encryption key is stored in the OS keychain, not on disk. - -**Improper:** unattended vault restore on payroll/HR with no confirm-gates -and a wide grant. Shared-machine vault without a host authentication layer. +Occupying an already-running Chrome/Edge profile (ATTACH: extension + +`chrome.debugger` + native messaging) and the encrypted identity vault are +**deferred, optional** features. Nothing in §3.1–3.9 depends on them. They +are specified in §5.12–5.13 so they can return without a redesign. --- -## 4. Out of scope (do not build these as Tyto) +## 4. Out of scope | Not Tyto | Why | |---|---| -| Vision-first computer use (screenshot every step) | That is the latency we exist to kill. Pictorial fallback only for canvas/WebGL/image-only. | -| Consumer AI browser (Comet/Neon) | We do not replace Chrome/Edge; we occupy them. | -| Chrome Web Store extension that works *without* the host | Then the page or the store sandbox owns you. Extension is a hand of the host. | -| Remote “drive my home browser from the cloud” | Exterior stays local unless a later spec adds explicit remote auth. | -| Scholarmancy LMS sync product | Sibling; may borrow `waitReady` / `snapshot`. Different customer. | -| Firefox as a first-class CDP peer | Chromium CDP is the contract. | -| Stealth / anti-detect browser for abuse | Trusted input exists so *real* UI works (file pickers), not to evade banks. | -| Unattended purchasing, wire transfers, production deploys | Confirm-gates; human-in-loop. | +| Vision-first computer use (screenshot every step) | The latency we exist to kill. Pictorial fallback only for canvas/WebGL/image-only. | +| A Chromium fork / custom browser build | Pinned upstream Chrome for Testing + flags is enough. A fork is a build farm and a patch treadmill. | +| A consumer browsing UI (tabs bar, omnibox, sync) | Chromium already has one. Tyto is the control plane. | +| Extension-required control | The extension is optional and deferred. The CLI needs no extension. | +| Remote “drive my browser from the cloud” | Exterior stays loopback unless a later spec adds explicit remote auth. | +| Firefox / Safari as CDP peers | Chromium CDP is the contract. | +| Stealth / anti-detect for abuse | Trusted input exists so real UI works, not to evade fraud checks. | +| Unattended purchasing, wire transfers, production deploys by the agent | Confirm-gates; human-in-loop. | | Treating page text as instructions | Prompt injection. | -| OS credential-store / Kerberos TGT extraction | Out of scope. Negotiate/IWA is handled by operating the real profile; no ticket harvesting. | -| Identity export to non-browser clients | Auth material stays inside the browser boundary. The SDK does not hand cookies to scripts. | -| Impersonation / delegated identity | Acting as a different principal. Not built. A future design would require separate authorization architecture. | -| Shared vault across multiple operator accounts | One vault per operator-machine pair; no multi-tenant credential store in v1. | +| OS credential-store / Kerberos TGT extraction | Out of scope. | +| Impersonation / multi-tenant credential store | Not built. | --- ## 5. Functional specification -### 5.1 Prompt session (source of truth) - -A session is a document on disk (host-managed), not memory in an extension. - -Must persist: +### 5.1 Browser provisioning + +- The repo pins one **Chrome for Testing** version (and per-platform + download + checksum) in a tracked file. Bumping it is a reviewed commit. +- `tyto start` (or `tyto browser install`) downloads that build to + `~/.tyto/browsers///` on first use, verifies the + checksum, and extracts it. A failed or mismatched download leaves nothing + half-installed. No auto-update. +- Platforms: `mac-arm64`, `mac-x64`, `linux64`, `win64`. +- Override: `--browser chrome|edge|` launches an installed binary + instead. Same flags, same ports. +- Launch flags: `--remote-debugging-address=127.0.0.1`, + `--remote-debugging-port=0` (read `DevToolsActivePort`), + `--user-data-dir=~/.tyto/profiles/`, `--no-first-run`, + `--no-default-browser-check`. Headed by default; `--headless` optional. + +### 5.2 Host daemon and CLI + +- `tyto start [--profile ] [--headless] [--browser …]` starts the + host daemon and the browser. `tyto stop`, `tyto status`. +- The daemon writes `~/.tyto/host.json` (port, pid — **no token**) and the + tokens to `~/.tyto/tokens/` with mode `0600`. +- The CLI reads those files, calls the host over JSON-RPC on loopback, and + prints human output by default, `--json` on request (NDJSON for streams). +- Exit codes are stable and documented (0 ok, 1 error, 2 shell not ready, + 3 allowlist deny, 4 confirm required, 64 usage, 69 host not running). + +| Group | Commands | +|---|---| +| Lifecycle | `start`, `stop`, `status`, `browser install\|path\|version` | +| Pages | `open`, `back`, `reload`, `tabs`, `tab new\|close\|focus`, `frames` | +| Perception | `snapshot`, `extract`, `classify`, `wait` | +| Action | `click`, `type`, `press`, `scroll`, `select` (trusted input, role + name) | +| Observability | `logs [--follow] [--kind console\|network\|exception\|nav]`, `network`, `network body ` | +| State | `cookies [list\|set\|delete\|clear]`, `storage local\|session ` | +| Escape hatch | `cdp [params-json] [--target ]`, `cdp events ` | +| Goals | `run ""`, `resume `, `sessions`, `stop-run` | +| Policy | `allow `, `allowlist` | -- Goal and full conversation -- Plan: steps completed vs remaining -- Recipes / anchors (role, name, landmark, origin, route pattern) -- Extracts / answers -- Last URL and tab intent -- Model settings used (provider id + model id, not the raw key) -- Allowlist and confirm policy for that session +### 5.3 Protocol surfaces and token scopes -Must not persist: +One JSON-RPC host, two method sets, two tokens: -- `ref_N`, `backendNodeId`, click coordinates -- Live CDP sockets, extension ports -- Screenshots (except optional operator-debug attachments) +| Scope | Token holder | Methods | +|---|---|---| +| **safe** | Perch, MCP | `PERCH_SAFE_METHODS` (sessions, goto, snapshot, act, extract, frames, tape, operator, models, `identity.status`). No raw CDP, no cookie or storage values. | +| **power** | CLI | `PERCH_SAFE_METHODS` ∪ `POWER_METHODS` (`cdp.send`, CDP event stream, cookies, storage, network bodies, tabs). | -Resume: open session → launch/attach → `goto` last URL → BROWSE → bind -remaining recipes → continue. +A safe token calling a power method is `unauthorized`. The power token is +never served over HTTP; it exists only in the `0600` token file. -### 5.2 Control modes +### 5.4 Observability (tape) -**LAUNCH.** Host starts Chrome or Edge with a chosen `user-data-dir` and -`--remote-debugging-port` bound to `127.0.0.1`. Full CDP. Prefer profile -clone so the user’s daily Chrome can stay open. +CDP events are always on for the launched browser: console, exceptions, +lifecycle, navigation, frame attach/detach, network request/response/ +failure. The host keeps a bounded in-memory tape and appends goal-run tape +to the session. `Redactor` runs **before** tape is persisted or returned. +The power surface can return raw values only with `reveal: true`, and raw +values are never written to disk. -**ATTACH.** Tyto extension in running Chrome/Edge. Auto-enables the debugger -on the target tab (100% CDP). Speaks to the host **only** via native -messaging. Debugger banner is accepted in this mode as the cost of occupying -a live everyday profile. +### 5.5 Cookies and storage -Both modes expose the **same SDK**. The operator does not learn two APIs. +- `cookies` lists the jar (all origins, or `--url`), including httpOnly, + via CDP — never `document.cookie`. +- Values are masked (`name`, `domain`, `path`, `expires`, flags shown; + `value` → `‹redacted›`) unless `--reveal`. +- `set`, `delete`, `clear` write through CDP. +- `storage local|session ` reads DOM storage, masked the same way. -### 5.3 SDK (only exterior) +### 5.6 Raw CDP -The SDK is the only supported way to drive Tyto from “outside” the host: +`tyto cdp [params]` sends any protocol method to the +browser or a chosen target and prints the result. `tyto cdp events` streams +events. Results pass through the redactor’s structured masking (cookie +values, `Cookie` / `Set-Cookie` / `Authorization` headers) unless +`--reveal`. This is the owner’s escape hatch: allowlist and confirm-gates +govern the **agent**, not the owner’s explicit commands. -- TypeScript first -- MCP as a thin adapter for Claude Code -- Python later if needed +### 5.7 Prompt session (source of truth for goal runs) -Capabilities (logical; not a screenshot API): +Must persist: goal and conversation (including the assistant’s replies), +plan (done vs remaining), recipes, extracts/answers, last URL, model id + +base URL (never the key), allowlist and confirm policy. -- List/select Chrome and Edge profiles -- Launch / attach / disconnect -- Tabs, frames, OOPIFs (auto-attach; snapshot and act **per frame origin**) -- Discover cross-origin frames; **do not** auto-grant them -- `classify` / `waitReady` (static vs shell vs injected) **per frame** -- `snapshot` (compact AX + ephemeral refs) -- `act` (trusted click/type/keys) -- `extract` (AX first, JS read, model only if the tree actually contains the field) -- Tape: console, nav, exceptions, network errors, DOM inject events -- Interrupt / yield to operator -- Read/write prompt session +Must not persist: `ref_N`, `backendNodeId`, coordinates, live sockets, +cookie or token values, screenshots (except operator-debug attachments). -The SDK authenticates with a **host token**. It does not listen on a public -interface. +Resume: open session → start/attach browser → `goto` last URL → browse → +bind remaining recipes → continue. -### 5.4 Perception and action +### 5.8 Perception and action | Step | Behavior | |---|---| -| Observe | CDP events always on. Debugger *stepping* off unless inspecting a failure. | -| Classify | `static` / `shell` / `injected` from HTML+AX+main text, not from `load` alone. | +| Observe | CDP events always on. Debugger stepping off. | +| Classify | `static` / `shell` / `injected` from HTML + AX + main text. | | Browse | Compact AX **per focused frame**; refs valid for this snapshot only. | | Think | Model, rare. Emits recipes, not node ids. | -| Act | Trusted `Input` by default. JS click is degraded mode. | +| Act | Trusted `Input`. JS click is degraded mode. | | Wait | Nav, lifecycle, mutation/AX growth — not `sleep(250)`. | | Extract | Fail closed if still a shell. | -Screenshots: only if the chosen node (or page) has no useful AX. - -### 5.5 Weave (operator occupancy) - -- Any real key/mouse from the operator pauses agent dispatch. -- Esc / Stop aborts the current act and drops ephemeral refs. -- Operator-edited fields are truth; the next BROWSE sees them. -- Agent never clicks “on top of” the operator. - -### 5.6 Models - -- Configure `baseUrl` + API key + model id. -- Discover via `GET /v1/models` when the endpoint supports it. -- Manual model id always works. -- No LiteLLM-specific code paths. A LiteLLM proxy is just another URL. +### 5.9 Weave (operator occupancy) -### 5.7 Perch +- Real operator key/mouse input pauses agent dispatch. +- `tyto stop-run` / Esc aborts the current act and drops ephemeral refs. +- Operator-edited fields are truth; the next browse sees them. +- The occupancy signal must not be callable by page JavaScript. -Sidebar view of the prompt session: paste, stream, show next recipe, -interrupt. If Perch dies, the session file does not. Perch is not allowed to -forward page JS into the SDK as commands. +### 5.10 Models -### 5.8 Identity vault +`baseUrl` + API key + model id. Discover via `GET /v1/models` when +supported; a typed model id always works. No LiteLLM code paths. -The vault is a host-owned kernel service. Perch and MCP see only the status -method (is an origin authenticated, fresh or expiring). They never receive -raw cookies, tokens, or vault handles that could be forwarded. +### 5.11 Perch (optional view) -**Lifecycle:** +A local page on the host showing sessions: goal box, transcript, Stop. Safe +scope only. If Perch dies, the session file does not. -1. **Identify** — `AuthProfiler` classifies each allowed origin: - `cookieSession | oauthBearer | samlSso | oidc | negotiateIWA | clientCert | unknown`. - Classification is from observed network events and storage (no heuristic - page-text analysis). The profile is stored in the session doc as metadata, - not as a secret. +### 5.12 Identity vault (deferred, optional) -2. **Capture** — operator confirms (once per origin). `CredentialStorePort` - reads cookies via `Network.getAllCookies` (catches httpOnly), reads token - stores via `Storage.getDOMStorageItems` / `IndexedDB.requestData`. Bundles - are AES-GCM encrypted; the DEK lives in the OS keychain. +The persistent Tyto profile is the primary way to stay logged in. The vault +adds portability (restore a captured session into a fresh profile). -3. **Preserve** — `IdentityBundle` on disk: ciphertext + origin + auth method - + expiry hint + IdP dependency list. No plaintext. No cookies in the session - JSON or in any log. +- Per-origin grant, default-deny; confirm on first capture and on restore + of sensitive origins. +- Capture via CDP, AES-GCM at rest, DEK in the OS keychain. +- Expiry-aware; never replay an expired bundle. +- Never in session JSON, tape, model prompts, or git. Safe clients see + `identity.status` only. -4. **Restore** — on session resume, vault decrypts and re-injects: cookies via - `Network.setCookies`, storage via `Storage.setDOMStorageItem`. For SSO - chains the IdP origin is restored first. For Negotiate/IWA origins, ensure - LAUNCH flags (`--auth-server-allowlist`) permit the host, then `goto`. +### 5.13 ATTACH to an existing Chrome/Edge (deferred, optional) -5. **Expiry / rotation** — `auth/expiry` checks bundle age vs expiry hint. - `expiring` → proactive re-capture prompt. `expired` → yield to operator, - do not proceed until re-authed. Do not replay a dead session at a 401. - -6. **Forget** — explicit operator command. Deletes ciphertext and removes DEK - reference in keychain. Does not log what was deleted. - -**Redaction:** `Redactor` strips all cookie header shapes, bearer-token -shapes, and active vault values from any string before it goes to tape or -to `ModelPort`. The model never learns the auth material and cannot be -prompted to reveal it. +Extension auto-enables `chrome.debugger` on a chosen tab and speaks to the +host only via native messaging. Same method sets. The UI must state that +ATTACH inherits that profile’s cookies before connecting. --- @@ -352,24 +333,18 @@ CDP input is indistinguishable from a human. Policy lives in the host. | Rule | Requirement | |---|---| -| Bind | Debug port and SDK: `127.0.0.1` only | -| Auth | Host token required for SDK | -| Page | No command channel from document JS | -| Extension | Native messaging to host only; no `window` API for control | -| Profile | Explicit pick; empty Tyto profile is default | -| Allowlist | Default-deny **origins**. Grant portal, tenant, and IdP separately. Iframe discovery never auto-grants. | -| Frames | Auto-attach OOPIFs. Act only in an allowed, attached frame. Parent shell is not a substitute for the child tree. | -| Destructive | Confirm submit / purchase / delete / send | +| Bind | Host and debug port on `127.0.0.1` only | +| Host header | Reject requests whose `Host` is not the loopback address + port (DNS rebinding) | +| Origin | Reject browser requests whose `Origin` is not the host’s own origin | +| Tokens | Two scopes (safe, power). Constant-time compare. Never logged. Power token only in a `0600` file | +| Perch | Receives the safe scope only, and never by an unauthenticated request | +| Page | No command channel from document JS (no `window` API, no page-callable bindings) | +| Profile | Dedicated Tyto profile by default; named profiles explicit | +| Allowlist | Default-deny origins **for the agent**. Iframe discovery never auto-grants | +| Destructive | Agent must confirm submit / purchase / delete / send | | Injection | Page content is data, never instructions | -| Secrets | API keys in host/OS keychain, never in the page, never in git | -| Auth material | Vault ciphertext on disk; DEK in OS keychain; never in session JSON, tape, model prompt, or git | -| Identity grant | Default-deny per origin. Explicit operator confirm on first capture and on sensitive-origin restore. | -| Expiry | Detect and prompt. Never silently replay an expired bundle. | -| Redaction | `Redactor` strips cookies and tokens from all tape and model inputs. | - -ATTACH on a live profile **inherits that profile’s cookies**. The UI must -say so before connect. The vault consent notice is required even in ATTACH -mode before any capture occurs. +| Secrets | Masked on every surface unless the power client asks `reveal: true`. Never persisted raw. Never sent to `ModelPort` | +| Model keys | Host env / OS keychain; never in the page, the session, or git | --- @@ -378,55 +353,51 @@ mode before any capture occurs. | | v1 | |---|---| | OS | macOS, Windows, Linux | -| Browsers | Google Chrome, Microsoft Edge (Chromium, CDP) | -| Clients | SDK, Perch, MCP | -| Not v1 | Firefox, Safari, remote hosted browsers | +| Browser | Tyto Chromium (pinned Chrome for Testing). Installed Chrome/Edge as override | +| Clients | CLI (power), MCP and Perch (safe), TypeScript SDK | +| Not v1 | Firefox, Safari, remote hosted browsers, ATTACH, identity vault | --- ## 8. Quality bars (when it is “Tyto”) -- Paste-to-first-trusted-click is obviously faster than a screenshot agent - on a normal HTML page. -- Closing Perch or disconnecting MCP does not lose the prompt session. -- A static Wayback article classifies `static` and extracts from AX without - a model inventing facts. -- A CSR shell either becomes `injected` with real content or extract **blocks**. -- A portal that embeds Workday (or any tenant) in a cross-origin iframe: - Tyto attaches the child, snapshots **that** origin, and does not treat - the parent chrome as the app. -- A site cannot `postMessage` Tyto into navigating or submitting. -- Operator can type in the same field the agent was about to fill. -- The agent resumes on an authenticated site after a browser restart without - the operator re-typing a password — and the model conversation contains no - cookie, token, or credential value. +- On a machine with only Node: `tyto start` brings up a browser with no + extension and no preinstalled Chrome. +- Anything visible in DevTools (console, network, cookies, storage, any CDP + method) is reachable from `tyto` with `--json`. +- `tyto cookies --json` shows the jar with values masked; `--reveal` shows + them; neither writes a value to disk. +- Paste-to-first-trusted-click is obviously faster than a screenshot agent. +- Killing the CLI, the host, or the browser does not lose a goal session. +- A static article classifies `static` and extracts from AX without a model + inventing facts. A CSR shell either becomes `injected` or extract blocks. +- A portal embedding a tenant in a cross-origin iframe: Tyto attaches the + child, snapshots **that** origin, and does not treat the parent as the app. +- A site cannot `postMessage` or call a binding to steer, stop, or grant Tyto. +- The operator can type in the same field the agent was about to fill. --- ## 9. Build order -1. Host + SDK + disk sessions + LAUNCH (Chrome/Edge) + loop + Perch paste + interrupt. -2. ATTACH: extension, auto debugger, native messaging, profile picker. -3. MCP client on the same session file. -4. Weave + recipe replay. -5. Unattended runner with exit codes. - -Proof points already exercised in-repo (`poc/`): AX loop, tape, inject-wait, -fail-closed extract, Wayback 2008 static success. Those are engine spikes, -not the product UI. +1. Harden the host: token scopes, Host/Origin checks, token files. +2. Chromium provisioning (pinned Chrome for Testing). +3. `tyto` CLI + daemon + primitives. +4. Power surface: raw CDP, cookies, storage, network, event streams. +5. Agent loop completeness: replies, re-observe after act, confirm-gate, + recipes, resume. +6. MCP server on the safe surface. +7. Unattended runner live. +8. Later: ATTACH, identity vault. --- ## 10. One-line tests -- If a power user would rather use a vision agent than paste into Perch, - Tyto is not done. -- If they quit the browser and cannot reopen the **same prompt** and - continue, Tyto is not done. -- If a website can drive the SDK, Tyto is not done. -- If they cannot operate whatever they can operate by hand in that - profile (iframe, DHTML, injected app, proxied host), Tyto is not done. -- If they quit the browser and the model transcript from the resumed run - contains a cookie or token value, Tyto is not done. -- If an origin gets vault access without an explicit operator grant, - Tyto is not done. +- If you must install an extension to let AI drive the browser, Tyto is not done. +- If DevTools can see it and `tyto` cannot, Tyto is not done. +- If a power user would rather screenshot-agent the tab, Tyto is not done. +- If they quit the browser and cannot resume the **same session**, Tyto is not done. +- If a website can drive the host, Tyto is not done. +- If a cookie or token value lands in an agent transcript or on disk without + an explicit `--reveal`, Tyto is not done. diff --git a/docs/USAGE.md b/docs/USAGE.md index 39dafcb..efe2476 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -6,7 +6,12 @@ Opening Google Chrome from the Dock does nothing — the **host** has to be runn You occupy a Chrome or Edge window that **Tyto launches**. You steer it from **Perch** (a local page). The site you browse cannot command Tyto. -Operator contract: [SPEC.md](./SPEC.md). This file is the current start path. +Operator contract: [SPEC.md](./SPEC.md). This file is the **current** start path. + +> **Where this is going.** v1 replaces `npm start` + Perch with a `tyto` +> command and a Tyto-downloaded Chromium (no installed Chrome, no extension): +> `tyto start`, `tyto open`, `tyto snapshot`, `tyto logs`, `tyto cookies`, +> `tyto cdp …`. Not built yet — see SPEC §5.2 and IMPLEMENTATION Slices 15–19. --- @@ -27,7 +32,7 @@ the first run. ## Prerequisites -- **Node 22+** +- **Node 24.15+ or 26** (older 22.x fails the jsdom-based UI tests) - **Google Chrome** or **Microsoft Edge** installed in a normal location (`/Applications/Google Chrome.app` on macOS, or on `PATH`) - A running **OpenAI-compatible model**. Default: @@ -60,7 +65,9 @@ npm start The first start: 1. Writes a host token into **local** `.env` if none exists (gitignored; never - printed to the terminal) + printed to the terminal). Note: the host does **not** read `.env` back, so + unless `TYTO_HOST_TOKEN` is exported in your shell, each start uses a fresh + token and the one in `.env` goes stale. 2. Listens on `127.0.0.1:7420` (override with `TYTO_PORT`) 3. Launches Chrome with `--remote-debugging-address=127.0.0.1` 4. Opens Perch in your default browser @@ -105,8 +112,9 @@ It does **not** grant every origin a page happens to load (iframes, SSO popups). | `~/.tyto/profile/` | Chrome/Edge user-data-dir for the launched window (empty on first run). | | `~/.tyto/sessions/` | Prompt session JSON (goal, plan, last URL, model id). This is the durable object. | -Resume later: `npm start` again, then continue from the same session files. -Killing Perch does not delete them. +Killing Perch does not delete session files. Perch cannot reopen an old +session yet — every **Go** starts a new one (resume is planned as +`tyto resume`, Slice 17/19). --- @@ -190,7 +198,8 @@ not running or the catalog endpoint returned an error. ### Send a goal Type your goal in the composer at the bottom and press **Send** (or Enter). -Shift+Enter inserts a newline. The transcript shows your message and Tyto's reply. +Shift+Enter inserts a newline. The transcript shows your message; the agent +loop does not write an assistant reply yet (Slice 19). ### Scope: This tab vs All tabs @@ -244,7 +253,7 @@ Tyto uses four test tiers. Only Tier 1 gates merges. | 1 | JSDOM UI (perch.html + sidepanel.js) | ✅ Merge-gating | `npm test` | | 2 | Live loop — Tyto drives real Chrome | `TYTO_E2E=1 TYTO_LIVE=1` | `npm run test:e2e` | | 3 | Playwright extension side-panel DOM | `TYTO_E2E=1` | `npm run test:e2e` | -| 4 | Ollama nightly — real model round-trip | `TYTO_MODEL_LIVE=1` | `TYTO_MODEL_LIVE=1 npm run test:e2e` | +| 4 | Ollama (manual) — real model round-trip | `TYTO_MODEL_LIVE=1` | `TYTO_MODEL_LIVE=1 npm run test:e2e` | ### Tier 1 — JSDOM (offline, gating) @@ -273,7 +282,9 @@ greps the session file, tape, model prompt, and vault ciphertext for the cookie asks the agent to fill the same box. Mid-keystroke the operator's text stays. After the operator pauses, the loop resumes from a fresh snapshot. -Unattended exit codes (Slice 14) are `runUnattended` in `@tyto/core`. Preflight: +Unattended exit codes (Slice 14) are `runUnattended` in `@tyto/core`. The CLI +entry below parses arguments but is **not usable live yet** — it exits 1 +unless tests inject ports: ```bash npm run run:unattended -- --session [--allow-confirm-fail] @@ -284,7 +295,7 @@ npm run run:unattended -- --session [--allow-confirm-fail] | 0 | done | | 2 | document still a shell (`ShellNotReady`) | | 3 | allowlist deny | -| 4 | confirm required (`--no-confirm`, the default) | +| 4 | confirm required (the default without `--allow-confirm-fail`) | ### Tier 3 — Extension panel (opt-in, Playwright) @@ -305,17 +316,19 @@ visible, token absent from DOM, origin not auto-granted on panel load. Requires `TYTO_E2E=1`. -### Tier 4 — Ollama nightly (non-gating) +### Tier 4 — Ollama (manual, non-gating) `e2e/test/ollama-live.test.ts` hits a local Ollama instance, lists models, then sends a single-step session and asserts the response parses as a valid plan. Catches prompt / -format drift before it affects users. Never blocks a merge. +format drift before it affects users. Never blocks a merge. Not scheduled in +CI; run it by hand. Requires `TYTO_MODEL_LIVE=1` (plus Ollama running with `TYTO_BASE_URL` + `TYTO_MODEL`). ### CI -`.github/workflows/ci.yml` — airplane-mode check, runs on every push / PR. +`.github/workflows/ci.yml` — airplane-mode check plus gitleaks, on push to +`main` and on every PR. `.github/workflows/e2e.yml` — Tiers 2–3, nightly + `workflow_dispatch`, marked `continue-on-error: true`. Not required for merge. Tier 3 runs under `xvfb-run` because the extension test needs a display. diff --git a/extension/README.md b/extension/README.md index a4aa483..3b9c336 100644 --- a/extension/README.md +++ b/extension/README.md @@ -1,7 +1,12 @@ -# Tyto extension (ATTACH) +# Tyto extension (ATTACH — deferred) -Chrome + Edge MV3. Speaks **only** to the host via native messaging. -Auto-enables `chrome.debugger` on the target tab. +**Optional and deferred** (SPEC draft 2 §3.10). Tyto’s CLI and bundled +Chromium need no extension. + +Target design: Chrome + Edge MV3, speaks **only** to the host via native +messaging, auto-enables `chrome.debugger` on the target tab. Today on `main` +the side panel talks HTTP to the loopback host and native messaging / +debugger auto-attach are not wired (work in progress lives on `ci-deploy`). The document must not get a `window.tyto` command API.