Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -65,3 +65,4 @@ blob-report/
!.claude/README.md
canvases/
agent-transcripts/
spikes/own-browser/.core-skill.md
5 changes: 4 additions & 1 deletion .gitleaks.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Gitleaks config for a public Tyto repo.
# Run: gitleaks detect --source . --config .gitleaks.toml
# CI: .github/workflows/gitleaks.yml
# CI: .github/workflows/ci.yml (gitleaks job)

title = "tyto"

Expand Down Expand Up @@ -49,4 +49,7 @@ regexes = [
'''sk-example''',
'''REDACTED''',
'''Bearer \[REDACTED\]''',
# MV3 manifest "key": an RSA-2048 SubjectPublicKeyInfo that pins the
# extension id. It is a public key, not a secret.
'''MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA''',
]
1 change: 1 addition & 0 deletions .npmrc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
engine-strict=true
1 change: 1 addition & 0 deletions .nvmrc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
26
8 changes: 7 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
# Tyto

> **Status: paused (2026-09-29).** For AI-driven browsing we use
> [agent-browser](https://github.com/vercel-labs/agent-browser). Why, with measurements:
> [docs/DECISION-2026-09-29-agent-browser.md](./docs/DECISION-2026-09-29-agent-browser.md).
> The code below is kept for reference.

**An AI-first browser you drive in prose — without screenshots.**

Barn owls (*Tyto alba*) hunt in the dark by hearing alone. Tyto reads a page as a
Expand Down Expand Up @@ -189,7 +194,8 @@ Kill Perch: session JSON under `~/.tyto/sessions/` remains.

## Develop

Requires **Node 22+**. Tests must pass **offline**.
Requires **Node 22.22+, 24.15+, or 26** (`.nvmrc` pins 26; `engine-strict` is on).
Tests must pass **offline**.

```bash
npm test
Expand Down
45 changes: 45 additions & 0 deletions docs/DECISION-2026-09-29-agent-browser.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Decision: use agent-browser; pause Tyto (2026-09-29)

## Goal we were solving

Let any language model work on any web page, any time, from the command line — no browser extension —
with full visibility (network, JavaScript errors, cookies, storage) and native-CLI speed.

## Decision

Use [agent-browser](https://github.com/vercel-labs/agent-browser) (Vercel Labs, Apache-2.0) as the browser
layer instead of building Tyto's own CDP stack. Tyto development is paused. The code in `packages/` stays
for reference; nothing in it is deleted.

## Why

- agent-browser already ships what Tyto's spec promised: its own Chrome for Testing (or any Chromium, e.g.
Edge, via `executablePath`), a fast CLI and daemon, accessibility snapshots with refs, cookies, storage,
console, errors, network requests and bodies, HAR, persistent profiles and saved logins, an encrypted
credential vault, domain allowlists, action confirmation, and an MCP server.
- Rebuilding that would take weeks for parity, with no advantage on the goal above.
- Forking Chromium is unnecessary: the DevTools protocol already exposes all the plumbing.

## Evidence (measured on an M4 Max; details in `spikes/own-browser/README.md`)

| Measurement | Result |
|---|---|
| Where AI browsing time goes | Model turns are ~75–80% of wall time; browser commands ~55 ms each |
| Model driving the browser (Claude Sonnet, 4 tasks) | 8–16 s median per task, 5–9 turns |
| Model calling a precompiled recipe | 4.6–5.8 s, always 2 turns |
| Recipe with no model | 0.23–0.31 s warm, 0.8–0.9 s cold |
| Recipes compiled automatically from one exploration | 18/18 correct on new inputs after one repair pass, 0 wrong |
| One-call page brief (POC) | "Why is this page broken?" in 2 turns for gpt-oss 20B, Qwen3-coder 30B, and Claude |
| Logins across a full browser restart | Kept (local fixture and a real site) |
| Tool routing rule (API → fetch → agent-browser → saved script) | 8/8 correct; browser used only for login and screenshot tasks |

## What carries forward

- Pick the fastest tool per task: API, then plain fetch, then agent-browser only when a page needs
JavaScript, a login, interaction, internals, or pixels.
- Save repeated browser tasks as verified `agent-browser batch` scripts; replay instead of re-browsing.
- If a real gap appears, extend agent-browser (plugins, or upstream contributions) before building anew.

## Known issue found

agent-browser 0.38.1: `errors --clear` does not clear the error buffer (worked around in the POC).
1 change: 1 addition & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@
| [SPEC.md](./SPEC.md) | Product specification, use cases, security |
| [DESIGN.md](./DESIGN.md) | Thesis: prompt-native, weave, control plane |
| [IMPLEMENTATION.md](./IMPLEMENTATION.md) | TDD slices, ISP ports, anti-patterns |
| [DECISION-2026-09-29-agent-browser.md](./DECISION-2026-09-29-agent-browser.md) | Why Tyto is paused in favor of agent-browser |
2 changes: 1 addition & 1 deletion docs/USAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ the first run.

## Prerequisites

- **Node 22+**
- **Node 22.22+, 24.15+, or 26** (older 22.x fails the jsdom-based UI tests)
- **Google Chrome** or **Microsoft Edge** installed in a normal location
(`/Applications/Google Chrome.app` on macOS, or on `PATH`)
- A running **OpenAI-compatible model**. Default:
Expand Down
2 changes: 1 addition & 1 deletion package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
"poc": "npx tsx poc/run.ts"
},
"engines": {
"node": ">=22"
"node": "^22.22.2 || ^24.15.0 || >=26"
},
"devDependencies": {
"@types/node": "^26.3.0",
Expand Down
58 changes: 58 additions & 0 deletions spikes/own-browser/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# Spike: your own AI browser from the command line

Throwaway proof of concept (not product code; see `docs/IMPLEMENTATION.md` for the product laws).
Question it answers: *can any language model drive a page, with full visibility and no browser
extension, at native-CLI speed?* It is a thin layer (`tyto`, Python) over
[agent-browser](https://github.com/vercel-labs/agent-browser), which launches its own Chrome for Testing.

| File | What |
|---|---|
| `tyto` | The CLI: persistent own profile, one-call BRIEF, "what happened" after every action, masked cookies, page-text `find` |
| `card.md` | The ~300-token instructions every model gets (instead of a 37 KB skill) |
| `agent.py` | Minimal model-agnostic agent loop: any OpenAI-compatible model + one `tyto` tool |
| `fixture.py` | Local test site with planted problems (500 API, JS TypeError, login) |
| `matrix.py`, `retest.py` | The graded model matrix |

## Run it

```bash
brew install agent-browser && agent-browser install # own Chrome for Testing, no extension
python3 spikes/own-browser/fixture.py 8765 & # optional test site
export TYTO_HOME=/tmp/tyto-poc-home
spikes/own-browser/tyto start
spikes/own-browser/tyto open http://127.0.0.1:8765/shop # prints the BRIEF
spikes/own-browser/agent.py gpt-oss:20b "The page http://127.0.0.1:8765/shop shows 'No items'. Why?"
```

## Results (2026-09-29, M4 Max, agent-browser 0.38.1, Chrome for Testing 154)

1. **Own browser, no extension** — ✅ `tyto start` launches its own Chrome with a persistent profile: 0.6–1.6 s.
2. **Stays logged in** — ✅ Logged in, fully quit Chrome (processes gone), restarted: still logged in, on the local
fixture (session-only and 30-day cookies) and on a real site (the-internet.herokuapp.com, `rack.session` cookie).
3. **Full visibility in one call** — ✅ `tyto open` returns status, failed requests with response body, JS exceptions
with source location, console errors, API calls, cookies (masked), storage, outline, interactive refs, and text.
143–711 tokens; gathered in 0.08–0.6 s. Every action prints what happened (navigation, requests, errors).
4. **Any model** — ✅ Same card, same tasks, graded:

| Task | gpt-oss 20B (local) | Qwen3-coder 30B (local) | Claude Sonnet + tyto | Claude + plain agent-browser |
|---|---|---|---|---|
| Debug "No items" page | 3/3 · 2 turns · 3.2–4.8 s | 3/3 · 2 turns · 2.9–13.4 s | 2/2 · 2 turns · 5.7 s | 2/2 · 7–8 turns · 13.9–17.9 s |
| Debug Save button | 2/2 · 7.4 s (one 102 s outlier) | 2/2 · 5.8–7.5 s | 2/2 · 10.7–11.6 s | 2/2 · 10.1–13.3 s |
| Log in | 3/3 · 6.7–7.5 s | 3/3 · 6.0 s | 2/2 · 10.1 s | — |
| Wikipedia fact, before `find` | 0/2 | 1/2 | 2/2 | — |
| Wikipedia fact, with `find` | 3/3 · 3 turns · 5.3–5.5 s | 3/3 · 18–22 s | — | — |

5. **Native feel** — ✅ cold start 0.77 s · open+brief 0.98 s (local) / 1.71 s (Wikipedia) · brief alone 0.16–0.62 s ·
raw command 0.1 s · click + feedback 0.47 s. Compiled recipes (earlier test): 0.23–0.31 s warm, no model.

## Findings

- The BRIEF turns "why is this broken?" into **one command** for every model tested, including a local 20B model
(2 turns vs 7–8 turns pulling console/network/errors separately).
- Weak models need a page-text search (`find`), not bigger briefs: it fixed the only failing category.
- agent-browser 0.38.1 bug: `errors --clear` does not clear the error buffer (worked around with offsets). Report upstream.
- Session-only cookies survived a full restart with a persistent profile; do not rely on that for every site.
- Open time includes a ~0.5 s "network quiet" wait; can be made event-driven.
- Refs written without `@` (Qwen) are now accepted.

Not tested: heavy SPAs, iframes/OOPIFs, sites with bot protection, Windows/Linux, long sessions, concurrency.
48 changes: 48 additions & 0 deletions spikes/own-browser/agent.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
#!/usr/bin/env python3
"""Minimal model-agnostic agent: any OpenAI-compatible chat model + one tool (`tyto`).
usage: agent.py <model> <task> env: TYTO_BASE_URL (default Ollama), TYTO_API_KEY
Prints one JSON line: turns, model_s, tool_s, wall_s, answer."""
import json, os, shlex, subprocess, sys, time, urllib.request

HERE = os.path.dirname(os.path.abspath(__file__))
BASE = os.environ.get("TYTO_BASE_URL", "http://127.0.0.1:11434/v1")
KEY = os.environ.get("TYTO_API_KEY", "")
CARD = open(os.path.join(HERE, "card.md")).read()
TOOL = {"type": "function", "function": {
"name": "tyto", "description": "Run one tyto browser command, e.g. 'open https://example.com' or 'click @e3'.",
"parameters": {"type": "object", "properties": {"args": {"type": "string", "description": "arguments after `tyto`"}},
"required": ["args"]}}}

def chat(model, messages):
body = json.dumps({"model": model, "messages": messages, "tools": [TOOL], "temperature": 0}).encode()
req = urllib.request.Request(f"{BASE}/chat/completions", data=body, headers={
"content-type": "application/json", **({"authorization": f"Bearer {KEY}"} if KEY else {})})
return json.load(urllib.request.urlopen(req, timeout=300))["choices"][0]["message"]

def run_tyto(args):
try: argv = shlex.split(args)
except ValueError as e: return f"bad arguments: {e}"
p = subprocess.run([os.path.join(HERE, "tyto"), *argv], capture_output=True, text=True, timeout=90, stdin=subprocess.DEVNULL)
return ((p.stdout or "") + (p.stderr or ""))[:6000] or "(no output)"

def main(model, task, max_turns=15):
msgs = [{"role": "system", "content": CARD}, {"role": "user", "content": task}]
model_s = tool_s = 0.0; t0 = time.time(); calls = []
for turn in range(1, max_turns + 1):
t = time.time(); m = chat(model, msgs); model_s += time.time() - t
tcs = m.get("tool_calls") or []
msgs.append({"role": "assistant", "content": m.get("content") or "", **({"tool_calls": tcs} if tcs else {})})
if not tcs:
return {"turns": turn, "calls": calls, "model_s": round(model_s, 1), "tool_s": round(tool_s, 1),
"wall_s": round(time.time() - t0, 1), "answer": (m.get("content") or "").strip()}
for tc in tcs:
try: args = json.loads(tc["function"]["arguments"] or "{}").get("args", "")
except json.JSONDecodeError: args = ""
t = time.time(); out = run_tyto(args); tool_s += time.time() - t
calls.append(args[:80])
msgs.append({"role": "tool", "tool_call_id": tc.get("id", ""), "content": out})
return {"turns": max_turns, "calls": calls, "model_s": round(model_s, 1), "tool_s": round(tool_s, 1),
"wall_s": round(time.time() - t0, 1), "answer": "(gave up: turn limit)"}

if __name__ == "__main__":
print(json.dumps(main(sys.argv[1], sys.argv[2])))
10 changes: 10 additions & 0 deletions spikes/own-browser/card.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
You control a real web browser through the `tyto` command. Call it with just the arguments.

- `open <url>` — navigate and get a BRIEF: page status, ISSUES (failed requests with response body, JS errors, console errors), network summary, cookies (masked), interactive elements with refs like @e3, and the main text.
- `click <@ref|css>` — click; the output then shows what happened (navigation, new requests, new errors).
- `fill <@ref|css> <text>` · `check <@ref|css>` · `press <key>` — same "what happened" output.
- `brief` — the BRIEF for the current page.
- `find <words>` — search the WHOLE page text for lines containing all the words (with the next lines as context). Use this to locate facts on long pages.
- `get text <@ref|css>` — text of an element. `body <url-part>` — response body of a matching request. `eval <js>` — run JavaScript and print the result.

Refs come from the latest brief. When asked why something fails, read ISSUES first. Quote arguments that contain spaces. When done, answer in one or two sentences.
59 changes: 59 additions & 0 deletions spikes/own-browser/fixture.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
#!/usr/bin/env python3
"""Local test site with planted problems. Loopback only.
/login → sets a session cookie (and a persistent one with 'remember me')
/account → needs the cookie
/shop → fetches /api/items, which returns 500; page logs an error and shows "No items"
/click → the Save button throws a TypeError
"""
import http.server, json, sys, time, urllib.parse
PORT = int(sys.argv[1]) if len(sys.argv) > 1 else 8765
PAGE = "<!doctype html><html><head><meta charset=utf-8><title>{title}</title>{head}</head><body><main><h1>{title}</h1>{body}</main></body></html>"
class H(http.server.BaseHTTPRequestHandler):
def log_message(self, *a): pass
def send(self, code, body, ctype="text/html; charset=utf-8", headers=()):
b = body.encode(); self.send_response(code); self.send_header("content-type", ctype); self.send_header("content-length", str(len(b)))
for k, v in headers: self.send_header(k, v)
self.end_headers(); self.wfile.write(b)
def cookie(self):
c = self.headers.get("cookie", ""); return dict(p.strip().split("=", 1) for p in c.split(";") if "=" in p)
def do_GET(self):
path = urllib.parse.urlparse(self.path).path
if path == "/login":
self.send(200, PAGE.format(title="Sign in", head="", body='<form method=post action=/login><label>Username <input name=user></label>'
'<label>Password <input name=pass type=password></label><label><input type=checkbox name=remember> Remember me</label>'
'<button type=submit>Sign in</button></form>'))
elif path == "/account":
c = self.cookie()
if c.get("sid") == "ada-session" or c.get("remember") == "ada-long":
self.send(200, PAGE.format(title="Account", head="", body="<p id=who>Welcome, Ada</p><a href=/shop>Shop</a>"))
else:
self.send(302, "", headers=[("location", "/login")])
elif path == "/shop":
js = ("<script>fetch('/api/items').then(r=>{if(!r.ok)throw new Error('items API failed: '+r.status);return r.json()})"
".then(items=>{document.querySelector('#list').innerHTML=items.map(i=>'<li>'+i+'</li>').join('')})"
".catch(e=>{console.error(e.message);document.querySelector('#list').innerHTML='<li>No items</li>'});</script>")
self.send(200, PAGE.format(title="Shop", head="", body="<ul id=list><li>Loading…</li></ul>" + js))
elif path == "/spa":
# Content exists only after JS runs: fetched, then decoded client-side (not readable from the HTML).
js = ("<script>fetch('/api/blob').then(r=>r.json()).then(d=>{const t=atob(d.b).split('').reverse().join('');"
"document.querySelector('#app').innerHTML='<h2>Today</h2><p id=price>'+t+'</p>';});</script>")
self.send(200, PAGE.format(title="Deals", head="", body="<div id=app>Loading…</div>" + js))
elif path == "/api/blob":
import base64
self.send(200, json.dumps({"b": base64.b64encode("Pro plan: $41.37 per month"[::-1].encode()).decode()}), "application/json")
elif path == "/api/items":
time.sleep(0.05); self.send(500, json.dumps({"error": "database timeout"}), "application/json")
elif path == "/click":
js = "<script>document.querySelector('#save').addEventListener('click',()=>{const draft=undefined;console.log('saving');return draft.id;});</script>"
self.send(200, PAGE.format(title="Editor", head="", body="<textarea aria-label=Note></textarea><button id=save>Save</button>" + js))
else:
self.send(404, PAGE.format(title="Not found", head="", body="<p>Nothing here</p>"))
def do_POST(self):
n = int(self.headers.get("content-length", 0)); form = urllib.parse.parse_qs(self.rfile.read(n).decode())
if form.get("user") == ["ada"] and form.get("pass") == ["lovelace"]:
h = [("set-cookie", "sid=ada-session; Path=/; HttpOnly")]
if form.get("remember"): h.append(("set-cookie", "remember=ada-long; Path=/; Max-Age=2592000; HttpOnly"))
self.send(303, "", headers=h + [("location", "/account")])
else:
self.send(200, PAGE.format(title="Sign in", head="", body="<p role=alert>Wrong username or password</p><a href=/login>Try again</a>"))
http.server.ThreadingHTTPServer(("127.0.0.1", PORT), H).serve_forever()
Loading
Loading