From c64efd25c56706595c2aa1f516c42e27193da4d5 Mon Sep 17 00:00:00 2001 From: Bailey Dixon Date: Sun, 19 Apr 2026 14:28:59 -0400 Subject: [PATCH] docs(v3): refresh docs for daemon-first architecture MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add four new v3 user-facing docs covering the daemon operator surface, wire protocol, architecture, and v2→v3 migration. Rewrite the Getting Started "Run ARC" section around `arc daemon start`, `ARC_PORT`, and `ARC_DIR`. Add a CHANGELOG entry for 1.0.0-alpha.0 in Keep-a-Changelog format, plus a "v3 in progress" callout atop the README. Every new page links back to `docs/plans/arc-v3-daemon.md` as the source of truth. Co-Authored-By: Claude Opus 4.7 (1M context) --- CHANGELOG.md | 59 ++++++++- README.md | 20 ++- docs/architecture.md | 144 +++++++++++++++++++++ docs/daemon.md | 185 ++++++++++++++++++++++++++ docs/getting-started.md | 99 ++++++++++++++ docs/index.md | 21 ++- docs/protocol.md | 258 +++++++++++++++++++++++++++++++++++++ docs/v2-to-v3-migration.md | 212 ++++++++++++++++++++++++++++++ 8 files changed, 992 insertions(+), 6 deletions(-) create mode 100644 docs/architecture.md create mode 100644 docs/daemon.md create mode 100644 docs/protocol.md create mode 100644 docs/v2-to-v3-migration.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 4b44418..238a31d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,62 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/), and this ## [Unreleased] +## [1.0.0-alpha.0] - 2026-04-19 + +First v3 pre-release. ARC pivots from a collection of short-lived CLI +invocations to a **persistent local daemon** that owns every agent +runtime. The TUI, CLI, web dashboard, Electron desktop app, mobile app, +and self-hosted relay all become thin clients of one binary-mux +WebSocket protocol. Plan of record: +[`docs/plans/arc-v3-daemon.md`](./docs/plans/arc-v3-daemon.md). + +### Added + +- **`@axiom-labs/arc-daemon` package** — long-running local process. HTTP + `/health`, WebSocket binary-mux on `:7272`, SQLite at `~/.arc/arc.db`, + structured JSONL log at `~/.arc/daemon.log`, argon2-hashed per-client + tokens plus a root token in `~/.arc/auth.json`, PID file with + `readPid()` liveness probe, loopback-only host-header enforcement. +- **Binary-mux wire protocol (v1)** — channel byte + flags + `u32be` + length + payload. Channel `0x00` carries a Zod-validated JSON envelope; + `0x01`–`0x03` reserved for terminal, file, and audio streams. Specified + in [`docs/protocol.md`](./docs/protocol.md). +- **`@axiom-labs/arc-client` SDK** — shared client library: frame codec, + Zod envelope, typed RPC wrappers (`health`, `profiles.*`, `agents.*`), + subscription helpers, auto-reconnect with exponential backoff, terminal + channel passthrough. Re-used by every UI surface. +- **`@axiom-labs/arc-relay` placeholder** — package scaffold for the + self-hosted NaCl-box WebSocket multiplexer that lands in Phase 10. +- **`arc daemon` CLI group** — `start` (detached by default), + `start --foreground`, `stop`, `restart`, `status [--json]`, and + `logs [-n ] [--tail]`. `ARC_PORT` and `--port` both respected. +- **Initial SQLite schema** — `agents`, `agent_events`, `chat_rooms`, + `chat_messages`, `loops`, `handoffs`, `clients`, `meta` tables with + additive-only migrations from here on. +- **v3 user docs** — [`docs/architecture.md`](./docs/architecture.md), + [`docs/daemon.md`](./docs/daemon.md), + [`docs/protocol.md`](./docs/protocol.md), and + [`docs/v2-to-v3-migration.md`](./docs/v2-to-v3-migration.md). Getting + Started and the main ToC updated to point at the new daemon-first flow. + +### Changed + +- **Version scheme bumped to `1.x`.** v2 stays recoverable at the + `archive/v0.4.x` tag; new work tracks `1.0.0-alpha.0` onward. +- `packages/cli/src/version.ts` now reads `1.0.0-alpha.0`. + +### Breaking + +- None yet in this pre-release. The v3 wire-protocol version is `1`; + any future breaking protocol change will bump it. Within `1.x` the + additive-only rules documented in [protocol.md](./docs/protocol.md#versioning-and-backward-compat) + apply. + +### Removed + +- Nothing. The v2 JSON stores stay in place and are read-for-read by the + daemon until later phases migrate them into SQLite. + ## [0.4.0] - 2026-04-18 ### Added @@ -223,7 +279,8 @@ All 25 phases of the [v2.0 spec](./docs/spec/SPEC.md) are now implemented. ARC h - **Light mode contrast** — WCAG AA compliant dimmed/border colors, explicit `colors.text` on import hint - **React hooks violation** — `useScreenSize()` moved above conditional returns in DashView -[Unreleased]: https://github.com/Codename-11/ARC/compare/v0.4.0...HEAD +[Unreleased]: https://github.com/Codename-11/ARC/compare/v1.0.0-alpha.0...HEAD +[1.0.0-alpha.0]: https://github.com/Codename-11/ARC/compare/v0.4.0...v1.0.0-alpha.0 [0.4.0]: https://github.com/Codename-11/ARC/compare/v0.2.0...v0.4.0 [0.2.0]: https://github.com/Codename-11/ARC/compare/v0.1.0...v0.2.0 [0.1.0]: https://github.com/Codename-11/ARC/releases/tag/v0.1.0 diff --git a/README.md b/README.md index b875c7d..4d6678a 100644 --- a/README.md +++ b/README.md @@ -26,15 +26,31 @@ --- +> ### 🚧 ARC v3 in progress (`1.0.0-alpha.0`) +> +> ARC is pivoting to a **daemon-first architecture**. A single long-running +> local process now owns every agent runtime; the TUI, CLI, web dashboard, +> and future Electron / mobile / relay clients are thin consumers of one +> binary-mux WebSocket protocol. +> +> - Plan of record: [`docs/plans/arc-v3-daemon.md`](./docs/plans/arc-v3-daemon.md) +> - New-in-v3: [daemon operator guide](./docs/daemon.md) · [architecture](./docs/architecture.md) · [wire protocol](./docs/protocol.md) · [v2 → v3 migration](./docs/v2-to-v3-migration.md) +> - v2 (`0.4.x`) stays recoverable at the `archive/v0.4.x` tag. +

ARC Dashboard

## What is ARC? -ARC started as a profile manager for agent CLIs (v0.1). It has since absorbed the [Axiom-Supervisor](https://github.com/Codename-11/axiom-supervisor) project and implements all 25 phases of the [v2.0 spec](./docs/spec/SPEC.md) — becoming a unified control plane for the full lifecycle of AI coding agents. +ARC started as a profile manager for agent CLIs (v0.1), absorbed the +[Axiom-Supervisor](https://github.com/Codename-11/axiom-supervisor) project +to implement all 25 phases of the v2.0 spec, and is now (v3) pivoting to a +**daemon-first** architecture — a persistent local process that owns every +agent runtime and exposes a single binary-mux WebSocket protocol to every +UI surface. -One binary. One config directory (`~/.arc/`). Every agent runtime — Claude Code, Codex CLI, Gemini CLI, OpenClaw, or anything that speaks MCP/HTTP/stdio. +One daemon. One config directory (`~/.arc/`). Every agent runtime — Claude Code, Codex CLI, Gemini CLI, OpenClaw, or anything that speaks MCP/HTTP/stdio. ## Features diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..45c592e --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,144 @@ +# ARC v3 Architecture + +ARC v3 is **one daemon, many mouths**. A persistent local process owns +every agent runtime, every piece of state, and the single wire protocol +everything else speaks. The CLI, TUI, web dashboard, Electron desktop app, +mobile app, and self-hosted relay are all thin clients. + +> **Source of truth:** [`docs/plans/arc-v3-daemon.md`](./plans/arc-v3-daemon.md). +> This document summarises the architecture that plan puts in motion. + +``` + ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ + │ TUI │ │ CLI │ │ Dashboard│ │ Electron │ │ Mobile │ + │ (Ink) │ │ (short) │ │ (SPA) │ │ (desktop)│ │ (Expo) │ + └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ + │ │ │ │ │ + └─────────────┴──────┬──────┴─────────────┴─────────────┘ + │ + @axiom-labs/arc-client (SDK) + │ + ▼ WebSocket binary-mux @ :7272 (local) + │ or NaCl-box through the relay (remote) + │ + ┌────────┴────────┐ + │ ARC Daemon │ + │ │ + │ HTTP /health ──┤ + │ WS binary-mux │ + │ │ + │ ┌───────────┐ │ + │ │ Router │ │ + │ │ Hub (subs)│ │ + │ │ RPC │ │ + │ │ Agent Mgr │──┼──► adapters spawn CLIs + │ │ Chat │ │ (claude / codex / gemini / ...) + │ │ Orchestr. │ │ + │ │ Hook bus │ │ + │ │ Profile │ │ + │ │ registry │ │ + │ └────┬──────┘ │ + └───────┼─────────┘ + │ + ┌─────────┴──────────┐ + │ ~/.arc/ │ + │ arc.db (SQLite)│ + │ auth.json │ + │ daemon.log │ + │ daemon.pid │ + │ profiles/… │ + │ shared/… │ + └────────────────────┘ +``` + +## Package responsibilities + +| Package | Role | +|----------------------------------|-------------------------------------------------------------------------------------------| +| `@axiom-labs/arc-core` | Agent adapters, agent-client, orchestration, hooks, knowledge, tool registry. **Unchanged from v2**: daemonising is purely a shell around it. | +| `@axiom-labs/arc-daemon` | Long-running process. HTTP + WS bind, router, RPC handlers, Hub (subscriptions), SQLite, structured logger, auth. Owns every agent lifecycle. | +| `@axiom-labs/arc-client` | Wire-protocol SDK — frame codec, Zod envelope, typed RPC wrappers, subscription helpers, auto-reconnect. Shared by every UI surface. | +| `@axiom-labs/arc-cli` | Commander.js CLI. `arc daemon …` lifecycle commands, typed client wrappers around remote ops, legacy passthrough for v2-only commands. | +| `@axiom-labs/arc-dashboard` | Web SPA. Reads from the daemon over WebSocket; no direct adapter spawn any more. | +| `@axiom-labs/arc-relay` | Stateless, zero-knowledge WebSocket multiplexer for remote daemon access. Placeholder in Phase 1 — ships in Phase 10. | +| `@axiom-labs/arc-adapter-claude` | Claude Code adapter — SDK bridge, auth, detect, import, shared. | +| `@axiom-labs/arc-adapter-openclaw` | OpenClaw adapter — plugin manifest, hooks, tools. | +| `@axiom-labs/arc-mcp` | ARC as an MCP server + host manager. | + +## Data flow: client → daemon → adapter → agent + +A typical `agents.run` request follows this path: + +``` +┌─ client (TUI / CLI / dashboard / mobile) +│ +│ 1. client.connect() → WS open → auth.login request +│ ← auth.login response { sessionId, ... } +│ +│ 2. client.agents.run({ profile, prompt, cwd, ... }) +│ └─ encodes Envelope { type: "request", method: "agent.run", params } +│ as a Control frame (channel 0x00) +│ sends over the WebSocket binary message +│ +▼ +┌─ daemon +│ +│ 3. ws/connection.ts reads the binary message, +│ frame.ts decodes channel 0x00 → envelope JSON +│ Zod parses Envelope → { id, type, method, params } +│ +│ 4. router.ts dispatches: +│ - auth gate (requires authenticated session) +│ - rpc/agent.ts → agentRun handler +│ - handler validates AgentRunParams, writes `agents` row in SQLite, +│ tells the Agent Mgr to spawn via the profile's adapter +│ +│ 5. Agent Mgr (Phase 4) spawns the adapter process in worker mode, +│ attaches stdout/stderr/tool-call streams, records agent_events rows. +│ +│ 6. Hub fan-out: +│ - `agents` topic: list-churn event (agent started) +│ - `agent:` topic: stream of stdout / status / tool-call events +│ - responses for future `agent.send` calls go back on the same sock +│ +▼ +┌─ subscribers +│ +│ 7. Any client holding a `subscribe(topic: "agent:")` receives `event` +│ envelopes carrying the recorded payload — terminal frames on channel +│ 0x01 bypass JSON entirely and land raw in the client's renderer. +│ +└─ Client also receives the `response` envelope for request (2) with + { agentId } as soon as the agent is registered. +``` + +Agent lifecycle state, event history, chat messages, and paired-client +tokens all live in `arc.db`. Restarting the daemon (or reconnecting a +client) replays state from SQLite rather than memory. + +## Key properties + +- **Agent survival across UI disconnect.** The daemon owns the child + process, not the UI. Closing the TUI leaves the agent running. +- **Single protocol, many clients.** Every surface uses the same + `@axiom-labs/arc-client` SDK. Local vs. remote is a URL difference + only. +- **Additive-only schema.** Wire protocol rules (see + [protocol.md](./protocol.md#versioning-and-backward-compat)) and SQLite + migrations are additive-only inside a major version. Clients pinned to + `v: 1` keep working as methods and topics accrete. +- **No polling.** UIs receive change events through subscriptions. There + is no "refresh" endpoint; the daemon pushes. +- **Loopback-first security.** The daemon refuses non-loopback hosts; + every paired client carries an argon2-hashed bearer token; remote + access goes through the relay with NaCl-box on every frame. + +## Further reading + +- [Daemon operator guide](./daemon.md) — lifecycle, env vars, filesystem, + troubleshooting. +- [Wire protocol spec](./protocol.md) — frame format, envelopes, method + catalog, error codes, backward-compat rules. +- [v2 → v3 migration](./v2-to-v3-migration.md) — moving from the + pre-daemon layout. +- Plan of record: [`docs/plans/arc-v3-daemon.md`](./plans/arc-v3-daemon.md). diff --git a/docs/daemon.md b/docs/daemon.md new file mode 100644 index 0000000..946bc8f --- /dev/null +++ b/docs/daemon.md @@ -0,0 +1,185 @@ +# ARC Daemon + +The **ARC daemon** (`@axiom-labs/arc-daemon`) is the long-running local +service that hosts agents, profiles, chat rooms, orchestration loops, and +the wire-protocol endpoint. All other surfaces — TUI, CLI commands, web +dashboard, future Electron and mobile apps, and the self-hosted relay — are +thin clients of the daemon. + +> **Source of truth:** [`docs/plans/arc-v3-daemon.md`](./plans/arc-v3-daemon.md). +> This doc stays in lock-step with the Phase 1–3 sections of that plan. + +- **Default bind:** `127.0.0.1:7272` +- **Protocol:** see [protocol.md](./protocol.md) +- **Data home:** `~/.arc/` (overrideable via `ARC_DIR`) + +## Lifecycle commands + +The CLI wires daemon lifecycle management under the `arc daemon` group. + +| Command | Purpose | +|---------------------------------------|------------------------------------------------------| +| `arc daemon start` | Start the daemon, detached (default) | +| `arc daemon start --foreground` | Start in the foreground; blocks the terminal | +| `arc daemon start --port ` | Override bind port (same effect as `ARC_PORT=`) | +| `arc daemon stop` | Send SIGTERM and wait for clean exit | +| `arc daemon restart [--port ]` | Stop, wait, start | +| `arc daemon status [--json]` | Print running pid + host + port, or `stopped` | +| `arc daemon logs [-n ] [--tail]` | Print the last N lines; `--tail` follows | + +The detached form re-execs the current Node binary with the same CLI args +plus `--foreground`, `stdio: "ignore"`, and `detached: true`, then blocks +on the PID file appearing (up to 5 s). + +## Environment variables + +| Variable | Effect | +|--------------|---------------------------------------------------------------------------------| +| `ARC_DIR` | Override the ARC home directory (default: `~/.arc`). Controls where the daemon writes `arc.db`, `daemon.log`, `daemon.pid`, and `auth.json`. | +| `ARC_PORT` | Override the bind port (default `7272`). `--port` on the CLI wins. | +| `ARC_HOST` | Override the bind host (default `127.0.0.1`). Loopback is strongly recommended. | + +The daemon refuses any HTTP/WebSocket request whose `Host` header is not +loopback (`127.0.0.1`, `localhost`, `::1`) plus the configured port. This +is the first line of defence against DNS rebinding. + +## Filesystem layout + +On first start the daemon creates: + +``` +~/.arc/ + arc.db # SQLite — agents, events, chat, loops, handoffs, clients, meta + auth.json # 0600 — { v: 1, rootToken } + daemon.pid # PID of the running daemon (cleared on clean shutdown) + daemon.log # JSONL structured log +``` + +`arc.db` is the canonical runtime store. Its schema (`clients`, `agents`, +`agent_events`, `chat_rooms`, `chat_messages`, `loops`, `handoffs`, `meta`) +is created from +[`packages/daemon/src/db/migrations/001_init.sql`](../packages/daemon/src/db/migrations/001_init.sql). +Profile configuration stays in `~/.arc/config.json` for now; later phases +may migrate it into SQLite. + +`auth.json` contains the **root token** that local clients use for +bootstrap. It is generated on first start with 32 bytes of CSPRNG entropy, +written with `0600` permissions, and rotated by simply deleting the file. + +## Health check + +Over HTTP: + +```bash +curl -s http://127.0.0.1:7272/health | jq +``` + +Returns `DaemonHealth`: + +```jsonc +{ + "ok": true, + "version": "1.0.0-alpha.0", + "protocol": 1, + "uptime_ms": 42113, + "pid": 24580, + "host": "127.0.0.1", + "port": 7272 +} +``` + +Over the wire protocol the same payload comes back from the `health.get` +method (no auth required — see [protocol.md](./protocol.md)). + +## PID file semantics (`readPid`) + +The `readPid(pidPath)` helper (exported from +[`packages/daemon/src/bootstrap.ts`](../packages/daemon/src/bootstrap.ts)) +is what both `arc daemon status` and `arc daemon start` consult before +doing anything destructive: + +1. Read and parse `daemon.pid`. If missing or not a finite positive + integer, return `null` (treated as "not running"). +2. Call `process.kill(pid, 0)` as a liveness probe — this sends no signal + but errors on an unknown pid. If it throws, the pidfile is stale and + `null` is returned; otherwise the parsed pid is returned as-is. + +A `null` return from `readPid` means "it is safe to start a new daemon". +`start` refuses if `readPid` is non-null; `stop` exits early with a +friendly message in the same case. + +## Pairing additional clients + +> This CLI subcommand is planned for Phase 10 ("Self-hosted relay") but +> the underlying primitive (`pairClient` in +> [`packages/daemon/src/auth.ts`](../packages/daemon/src/auth.ts)) is +> already shipped. See the plan for the final UX. + +The intended flow: + +```bash +arc daemon pair --label "laptop-tui" +# prints a one-time bearer token (hex, 64 chars). +# Hand it to the client via QR, paste, or pairing-code flow. +``` + +Server-side this inserts a row into the `clients` table with +`sha256(token)` as the `token_hash` and the label as metadata. Clients +present the token on their first `auth.login` call. + +Tokens are one-time to the operator: revoke by deleting the row +(`arc daemon revoke ` in the same planned CLI group) or +dropping `auth.json` to rotate the root token. + +## Troubleshooting + +### `EADDRINUSE: address already in use 127.0.0.1:7272` + +Something else holds the port. Options: + +1. `arc daemon status` — if it prints `running`, you already have a + daemon. Use the existing one. +2. Kill the process manually: + - Linux/macOS: `lsof -iTCP:7272 -sTCP:LISTEN` + - Windows: `netstat -ano | findstr :7272` +3. Bind elsewhere: `ARC_PORT=7273 arc daemon start` (all clients must + then use the same port). + +### Stale `daemon.pid` + +`readPid` tolerates stale pidfiles automatically (see +[above](#pid-file-semantics-readpid)). If status insists a daemon is +running but connects fail, verify the process is actually alive; if not, +`rm ~/.arc/daemon.pid` and try again. The daemon also clears the pidfile +on clean shutdown. + +### `daemon did not start within 5000ms` + +`arc daemon start` waits up to 5 s for the detached child to write +`daemon.pid`. If it times out, inspect `daemon.log` for a bind failure, +SQLite lock, or missing migrations. Running `arc daemon start +--foreground` surfaces errors to your terminal directly. + +### `unauthorized` on every request + +Either the session did not call `auth.login`, or the token does not match +`rootToken` in `~/.arc/auth.json` / any paired client row. Re-read the +token from the auth file, or rotate it by deleting `auth.json` (all paired +clients keep working; only local "root" access regenerates). + +### Mismatched protocol version + +Clients sending `v` other than `1` in an envelope are rejected with +`bad_request`. This means a client is pinned to a future protocol +revision; pin it to `1` for now (the only shipped version). See +[protocol.md](./protocol.md#versioning-and-backward-compat) for the +compatibility rules. + +## Further reading + +- [Architecture](./architecture.md) — how the daemon fits into the wider + control plane. +- [Wire protocol](./protocol.md) — frame format, envelopes, method catalog. +- [v2 → v3 migration](./v2-to-v3-migration.md) — how to move an existing + `~/.arc/` onto the daemon. +- Plan of record: [`docs/plans/arc-v3-daemon.md`](./plans/arc-v3-daemon.md). diff --git a/docs/getting-started.md b/docs/getting-started.md index 57b2db4..e6fddb7 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -1,5 +1,12 @@ # Getting Started +> **ARC v3 in progress.** `1.0.0-alpha.0` pivots ARC to a **daemon-first** +> architecture — `arc daemon start` is now the default way to run ARC. +> The plan of record is [`docs/plans/arc-v3-daemon.md`](./plans/arc-v3-daemon.md); +> a deeper walk-through lives in [architecture.md](./architecture.md). If +> you are coming from v2, read [v2-to-v3-migration.md](./v2-to-v3-migration.md) +> first. + ## Requirements - **Node.js 20+** @@ -79,6 +86,77 @@ pnpm install:local # Rebuild + refresh shims (idempotent) See [Development](./development.md) for the full local dev workflow. +## Run ARC + +In v3 ARC runs as a **persistent local daemon**. Every UI — TUI, CLI, +web dashboard, Electron desktop, mobile app, or the self-hosted relay — +is a thin client of this daemon and speaks the same +[wire protocol](./protocol.md). + +### Start the daemon + +```bash +arc daemon start # detached by default +arc daemon status # → daemon: running (pid NNNN) on 127.0.0.1:7272 +arc daemon stop +arc daemon logs --tail +``` + +Once running, everything else connects over `ws://127.0.0.1:7272`. +The daemon owns every agent process, so closing a client does not kill +agents — agents now survive UI disconnect. + +### Launch a UI + +```bash +arc # TUI (Ink) — connects to the running daemon +arc web # open the web dashboard in your browser +arc chat # interactive REPL over the active profile +arc status # one-shot profile + daemon status +``` + +If a client RPC cannot reach the daemon, start one with +`arc daemon start` first. A planned auto-probe (Phase 1 of the v3 +plan) will spawn a detached daemon transparently when none is running. + +### `ARC_PORT` + +The daemon binds to `127.0.0.1:7272` by default. Override with either an +environment variable or the `--port` flag — the flag always wins: + +```bash +ARC_PORT=7373 arc daemon start +# …or equivalently: +arc daemon start --port 7373 +``` + +Every client picks up the same `ARC_PORT` automatically, so a single +`export ARC_PORT=7373` in your shell configures the whole stack. + +### `ARC_DIR` + +All daemon state lives under `~/.arc/` — SQLite (`arc.db`), the root +token (`auth.json`), the structured log (`daemon.log`), and the PID +file (`daemon.pid`). Point ARC at a different directory by setting +`ARC_DIR`: + +```bash +ARC_DIR=/opt/arc arc daemon start +``` + +This is useful for per-project isolation, headless server installs, or +running ARC out of a Docker volume. See the +[daemon operator guide](./daemon.md) for the full filesystem layout and +pairing additional clients. + +### Relationship between daemon and TUI + +Before v3 the TUI **was** ARC — it owned every agent lifecycle, and +quitting the TUI killed them all. In v3 the TUI is just a renderer for +state that lives in the daemon. The same is true for the web dashboard, +Electron app, mobile client, and CLI commands. You can swap any of them +in and out freely; the daemon keeps going. + ## First Profile ### Import an existing Claude account @@ -127,3 +205,24 @@ arc launch # launches gemini - [Profiles](./profiles.md) — manage multiple accounts across tools - [Authentication](./authentication.md) — API keys, Bedrock, Vertex AI, Foundry - [Shell Integration](./shell-integration.md) — make tool commands automatically use the active profile +- [Daemon operator guide](./daemon.md) — lifecycle, env vars, log location, + `readPid` semantics +- [Architecture](./architecture.md) — how the daemon, client SDK, and + adapters fit together +- [Wire protocol](./protocol.md) — frame format, envelope, method catalog + +--- + +## v2 deprecation notice + +Everything **below ARC v3** — the pre-daemon 0.x series — still runs, but +is no longer the supported install path. v2 (`0.4.x`) is frozen at the +`archive/v0.4.x` tag for anyone who needs to reproduce the old behaviour. +Moving an existing install forward takes one command: + +```bash +arc migrate v2-to-v3 +``` + +See [v2-to-v3-migration.md](./v2-to-v3-migration.md) for the full guide, +including backup, rollback, and what survives versus what changes. diff --git a/docs/index.md b/docs/index.md index 1195fdf..af93b6a 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,14 +1,29 @@ # ARC Documentation -**ARC — Agent Runtime Control.** Unified profile and environment manager for agent CLIs. Maintains isolated config directories per profile and injects the right credentials before launching any agent tool. +**ARC — Agent Runtime Control.** One persistent local daemon that owns +every agent runtime, backed by a single binary-mux WebSocket protocol. +TUI, CLI, web dashboard, Electron desktop, mobile, and the self-hosted +relay are all thin clients. -> Claude Code is the baseline today. Gemini CLI, Codex CLI, and others are supported via the `--tool` flag. +> **ARC v3 in progress (`1.0.0-alpha.0`).** `arc daemon start` is the new +> default entry point. Plan of record: +> [`docs/plans/arc-v3-daemon.md`](./plans/arc-v3-daemon.md). Moving from +> v2? See [v2-to-v3-migration.md](./v2-to-v3-migration.md). + +## v3 Reference + +| Doc | Description | +|-----|-------------| +| [Architecture](./architecture.md) | Daemon + client SDK + data flow overview | +| [Daemon operator guide](./daemon.md) | Lifecycle, env vars, log location, SQLite layout, pairing, `readPid` semantics, `EADDRINUSE` troubleshooting | +| [Wire protocol](./protocol.md) | Frame format, envelope, channel table, method catalog, subscription topics, error codes, backward-compat rules | +| [v2 → v3 migration](./v2-to-v3-migration.md) | Backup, `arc migrate v2-to-v3`, what survives, rollback to `archive/v0.4.x` | ## Guides | Guide | Description | |-------|-------------| -| [Getting Started](./getting-started.md) | Install, requirements, and first profile | +| [Getting Started](./getting-started.md) | Install, requirements, `arc daemon start`, and first profile | | [Profiles](./profiles.md) | Create, switch, import, and delete profiles | | [Authentication](./authentication.md) | OAuth, API key, Bedrock, Vertex AI, and Foundry | | [Shell Integration](./shell-integration.md) | Bash, zsh, fish, and PowerShell setup | diff --git a/docs/protocol.md b/docs/protocol.md new file mode 100644 index 0000000..ca9049d --- /dev/null +++ b/docs/protocol.md @@ -0,0 +1,258 @@ +# ARC Wire Protocol (v1) + +The ARC v3 daemon exposes a single, versioned, binary-multiplexed WebSocket +protocol. Every client — TUI, CLI, web dashboard, Electron desktop, mobile, +or a future relay — speaks the same wire format. + +> **Source of truth:** [`docs/plans/arc-v3-daemon.md`](./plans/arc-v3-daemon.md) and +> [`packages/client/src/protocol.ts`](../packages/client/src/protocol.ts). +> This document is kept in sync with those by hand. If they disagree, the code wins. + +- **Protocol version:** `1` +- **Transport:** WebSocket (RFC 6455) over loopback TCP, or tunnelled over the + self-hosted relay. +- **Default port:** `7272` (override with `ARC_PORT`). +- **Framing:** custom binary mux — one frame per WebSocket binary message. +- **Control encoding:** UTF-8 JSON, validated against Zod schemas. + +## Frame format + +Each binary message on the socket carries exactly one frame: + +``` + ┌──────┬───────┬──────────────┬────────────────────┐ + │ ch │ flags │ len (u32be) │ payload (N bytes) │ + │ 1B │ 1B │ 4B │ │ + └──────┴───────┴──────────────┴────────────────────┘ +``` + +- `ch` — channel id (see table below). +- `flags` — bit 0 (`0x01`) set when the frame is part of a fragmented sequence; + bits 1–7 are reserved and must be 0. +- `len` — payload length, big-endian `uint32`. +- `payload` — `len` bytes, interpreted per-channel. + +WebSocket already length-prefixes its own messages; the frame header is kept +so the format remains self-describing over other transports (TCP, relay +channels). Implementations live in +[`packages/client/src/frame.ts`](../packages/client/src/frame.ts) and are +shared between daemon and client. + +### Channels + +| Id | Name | Payload | Status | +|--------|------------|----------------------|--------------| +| `0x00` | Control | JSON envelope (UTF-8)| Shipped (v1) | +| `0x01` | Terminal | Raw bytes (PTY pass-through) | Reserved (Phase 4) | +| `0x02` | File | Chunk transfer | Reserved (Phase 12+) | +| `0x03` | Audio | Opus / PCM frames | Reserved (Phase 12+) | + +All other channel ids (`0x04`–`0xFF`) are reserved. Clients must ignore +unknown channels silently; servers must refuse to emit them. + +## Control envelope + +Every control-channel frame is a JSON-encoded envelope validated against the +Zod schema in +[`packages/client/src/protocol.ts`](../packages/client/src/protocol.ts): + +```ts +const Envelope = z.object({ + v: z.literal(1), + id: z.string(), + type: z.enum([ + "request", + "response", + "event", + "subscribe", + "unsubscribe", + "error", + ]), + method: z.string().optional(), // request only + params: z.unknown().optional(), // request only + result: z.unknown().optional(), // response only + topic: z.string().optional(), // subscribe / unsubscribe / event + payload: z.unknown().optional(), // event only + code: z.string().optional(), // error only + message: z.string().optional(), // error only +}); +``` + +### Envelope rules + +- `v` must be `1`. Envelopes that fail schema validation (wrong `v`, + missing required field, wrong type) are silently dropped by the + server and logged as `frame.invalid-envelope`. +- `id` is chosen by the sender. Every `response` or `error` quotes the + `id` of the request it answers. +- Clients originate `request`, `subscribe`, `unsubscribe`. The server + originates `response`, `event`, `error`. +- Envelopes the server does not expect (e.g. a `response` from a client) + are silently dropped. + +## Authentication + +1. Client opens the WebSocket. +2. Client sends `auth.login` as its first request (see + [Method catalog](#method-catalog)). +3. Server validates the token against `rootToken` in `~/.arc/auth.json` or + the hashed token of a paired client in the `clients` table. +4. On success the session is flagged authenticated and the returned + `sessionId` is used for logging and topic routing. +5. All methods other than `auth.login` and `health.get` require an + authenticated session. Calling them earlier returns `unauthorized`. + +## Method catalog + +The authoritative list of v1 methods lives in the `Methods` export of +`packages/client/src/protocol.ts`. Each method takes typed params and +returns a typed result; both are Zod-validated on the boundary. + +| Method | Auth? | Params | Result | +|-----------------|-------|-----------------------------------------------|-------------------------------------------------------------------------------| +| `auth.login` | No | `AuthLoginParams` | `AuthLoginResult` | +| `health.get` | No | – | `HealthGetResult` | +| `profile.list` | Yes | – | `ProfileListResult` | +| `profile.get` | Yes | `{ name: string }` | Profile (shape deferred — returned as unknown) | +| `agent.list` | Yes | – | `AgentListResult` | +| `agent.run` | Yes | `AgentRunParams` | `AgentRunResult` (Phase 4 lands the real handler) | +| `agent.stop` | Yes | `AgentStopParams` | `AgentOkResult` (Phase 4) | +| `agent.send` | Yes | `AgentSendParams` | `AgentOkResult` (Phase 4) | + +### Schemas + +```ts +// auth.login +AuthLoginParams = { token: string } // min length 16 +AuthLoginResult = { + sessionId: string; + clientId: string; + serverVersion: string; + protocol: 1; +} + +// health.get +HealthGetResult = { + ok: true; + version: string; + protocol: 1; + uptime_ms: number; + pid: number; + host: string; + port: number; +} + +// profile.list +ProfileSummary = { name: string; tool: string; active?: boolean } +ProfileListResult = { profiles: ProfileSummary[] } + +// agent.list +AgentSummary = { + id: string; + profile: string; + cwd: string; + status: string; // starting | running | idle | stalled | completed | failed + launchMode: string; // native | worker + createdAt: number; // ms since epoch + updatedAt: number; + completedAt?: number | null; + worktree?: string | null; +} +AgentListResult = { agents: AgentSummary[] } + +// agent.run (Phase 4 executes; Phase 2 returns `unimplemented`) +AgentRunParams = { + profile: string; + prompt?: string; + cwd?: string; + worktree?: string; + launchMode?: "native" | "worker"; +} +AgentRunResult = { agentId: string } + +// agent.stop / agent.send +AgentStopParams = { agentId: string } +AgentSendParams = { agentId: string; text: string } +AgentOkResult = { ok: true } +``` + +Planned methods for later phases (documented here so clients can plan): + +| Method | Phase | +|------------------------------------------------------------------------------------------|-------| +| `profile.create` / `profile.update` / `profile.delete` / `profile.clone` / `profile.switch` | 4 | +| `agent.attach` / `agent.archive` | 4 | +| `chat.post` / `chat.read` / `chat.wait` | 7 | +| `loop.start` / `loop.status` / `loop.stop` | 6 | +| `handoff.create` / `handoff.list` | 9 | +| `roundtable.start` / `roundtable.join` | 9 | +| `doctor.run` / `doctor.fix` | 4 | + +## Subscriptions and topics + +Clients subscribe to a topic by sending: + +```jsonc +{ "v": 1, "id": "...", "type": "subscribe", "topic": "agent:abc" } +``` + +Server acknowledges with a `response` carrying `{ ok: true, topic }` and +begins pushing `event` envelopes for that topic. Unsubscribing is +symmetrical. A connection drop drops all subscriptions; on reconnect the +client is expected to re-subscribe (the reference SDK does this +automatically). + +| Topic | Emitted when | +|------------------|-----------------------------------------------------| +| `agents` | High-level list churn (run / stop / archive) | +| `agent:` | Every event for one agent (stdout, status, tool use)| +| `profiles` | Profile registry changes | +| `chat:` | Chat room messages (Phase 7) | +| `loop:` | Worker/verifier loop transitions (Phase 6) | +| `daemon` | Daemon health / status changes | + +Topic names are strings; dynamic variants (`agent:`, `chat:`, +`loop:`) are built with helpers from `Topics` in +[`packages/client/src/protocol.ts`](../packages/client/src/protocol.ts). + +## Errors + +Errors are envelopes with `type: "error"`, a quoted `id`, a short `code`, +and a human-readable `message`. The current set: + +| Code | Meaning | +|------------------|-----------------------------------------------------------------| +| `unauthorized` | Session is not authenticated, or the presented token is invalid | +| `bad_request` | Envelope failed schema validation or was missing required fields| +| `not_found` | Target entity (profile, agent, room) does not exist | +| `internal` | Unclassified server-side error | +| `unimplemented` | Method known but the handler is a stub (lands in a later phase) | + +Clients must tolerate unknown error codes — new codes may appear in later +minors. + +## Versioning and backward-compat + +The protocol version bumps only on breaking changes. v1 establishes the +following contract between server and client: + +- **Never remove a method.** Deprecate by replacing its handler with a + response that still validates, or bump to v2. +- **Never narrow a schema.** `.optional()` fields may be added, but + required fields cannot be added or their types tightened. +- **Never reuse an error code** with a different meaning; add a new one. +- **Additive events are free.** A new topic does not bump the version. +- **Channel ids are frozen** at this document's revision. New channels + take the next free id. + +Clients should pin to a major protocol version (`v: 1`) and treat new +optional fields / new events as non-breaking. Servers reject envelopes +with a mismatched `v`. + +## Reference implementations + +- Frame codec: [`packages/client/src/frame.ts`](../packages/client/src/frame.ts) +- Envelope + method catalog: [`packages/client/src/protocol.ts`](../packages/client/src/protocol.ts) +- Daemon RPC handlers: [`packages/daemon/src/rpc/`](../packages/daemon/src/rpc) +- Router: [`packages/daemon/src/router.ts`](../packages/daemon/src/router.ts) +- High-level client: [`packages/client/src/client.ts`](../packages/client/src/client.ts) diff --git a/docs/v2-to-v3-migration.md b/docs/v2-to-v3-migration.md new file mode 100644 index 0000000..2170864 --- /dev/null +++ b/docs/v2-to-v3-migration.md @@ -0,0 +1,212 @@ +# Migrating from ARC v2 to v3 + +ARC v3 pivots from a collection of short-lived CLI invocations to a +persistent local **daemon** (see [architecture.md](./architecture.md)). +Your existing profiles, credentials, and shared layer are preserved; what +changes is **who owns the agent** — the daemon, not the UI. + +> **Source of truth:** [`docs/plans/arc-v3-daemon.md`](./plans/arc-v3-daemon.md). +> This guide covers the end-user migration path. Implementation details +> live in the plan. + +- **Current stable:** `0.4.x` (v2) +- **In progress:** `1.0.0-alpha.0` (v3) +- **Pre-daemon archive tag:** `archive/v0.4.x` — recoverable at any time. + +## TL;DR + +```bash +# 1. Back up +cp -r ~/.arc ~/.arc.v2-backup + +# 2. Upgrade ARC itself +npm install -g @axiom-labs/arc-cli@next + +# 3. Migrate the data layout +arc migrate v2-to-v3 + +# 4. Start the daemon +arc daemon start +arc daemon status +``` + +If anything goes wrong, roll back with +`rm -rf ~/.arc && mv ~/.arc.v2-backup ~/.arc && npm install -g @axiom-labs/arc-cli@0.4`. + +## 1. Back up `~/.arc/` + +The whole ARC data home sits under `~/.arc/` (or `$ARC_DIR` if you set +it). Copy the directory wholesale: + +```bash +# Linux / macOS +cp -r ~/.arc ~/.arc.v2-backup +``` + +```powershell +# Windows (PowerShell) +Copy-Item -Recurse $env:USERPROFILE\.arc $env:USERPROFILE\.arc.v2-backup +``` + +Both `arc.db` (v3) and the older JSON stores (v2) are worth keeping; the +migration is designed to be idempotent but backups are free. + +## 2. Upgrade the CLI + +### npm + +```bash +npm install -g @axiom-labs/arc-cli@next +``` + +### From source + +```bash +cd ARC +git fetch +git checkout feature/v3-foundation +pnpm install +pnpm install:local # refresh shims +``` + +Check you have the v3 CLI: + +```bash +arc --version +# → 1.0.0-alpha.0 +arc daemon --help +``` + +## 3. Run the migration + +```bash +arc migrate v2-to-v3 +``` + +> The migration tool lands in Phase 14 of the v3 plan. On pre-release +> builds you can skip it — the daemon will read `~/.arc/config.json` as +> before and lazy-populate `arc.db` as agents run. On stable v3 you must +> run it once. + +What the migration does: + +- Creates `~/.arc/arc.db` from + [`packages/daemon/src/db/migrations/001_init.sql`](../packages/daemon/src/db/migrations/001_init.sql). +- Ingests `~/.arc/history.json` → `agents` + `agent_events` tables. +- Ingests per-profile `chat-sessions/` → `chat_rooms` + `chat_messages`. +- Ingests `~/.arc/activity.log` into the audit trail. +- Writes a fresh `~/.arc/auth.json` (root token) if one does not already + exist. + +## 4. Start the daemon + +```bash +arc daemon start +arc daemon status +# → daemon: running (pid NNNN) on 127.0.0.1:7272 +``` + +See the [daemon operator guide](./daemon.md) for environment variables, +logs, and troubleshooting. + +## What survives + +- **Profiles.** `~/.arc/config.json` remains the authoritative profile + registry. Every profile, auth account, active-profile pointer, shared + layer mapping, and per-profile tool config dir is preserved. +- **Credentials.** Keyring entries and plaintext fallbacks are untouched. + Hot-swap snapshots under `~/.arc/credentials/` continue to work. +- **Shared layer.** `~/.arc/shared/` (MCP servers, CLAUDE.md, memory, + projects) migrates verbatim. Sync semantics are unchanged. +- **Skills, tasks, secrets.** Their JSON stores are read-for-read by the + daemon until Phase 4 moves them into SQLite. +- **Chat sessions.** Existing `~/.arc/profiles//chat-sessions/*.json` + files are copied into the `chat_rooms` / `chat_messages` tables during + `arc migrate v2-to-v3`. + +## What changes + +- **Agent ownership.** In v2 the TUI or CLI spawned the agent directly; in + v3 the daemon does. Closing the UI no longer kills the agent. +- **Chat / roundtable entry points.** `arc chat` and `arc roundtable` + connect to the daemon over the wire protocol instead of running the + session in-process. Set `ARC_PORT` if your daemon binds non-default. +- **Dashboard wire format.** The web dashboard used to speak a bespoke + REST + WebSocket API. Once it migrates onto `@axiom-labs/arc-client` + (Phase 4 of the v3 plan) it will use the same binary-mux WebSocket + every other client uses and point at the daemon on `:7272` instead of + its standalone `:3700` port. +- **Version scheme.** v2 was `0.x`; v3 is `1.x`. `0.4.x` installs keep + working side-by-side if you pin to that version, but the data layout + they understand is the pre-daemon one. +- **Protocol version.** Clients must pin to `v: 1` in the wire envelope. + See [protocol.md](./protocol.md#versioning-and-backward-compat). + +## What does not survive (yet) + +- **Experimental remote agent registry** (`~/.arc/remote-agents.json`). + The v3 pairing flow replaces it; entries are not auto-migrated. Pair + each remote client with `arc daemon pair` once it lands (Phase 10). +- **Legacy dashboard cookies / localStorage.** The new dashboard issues + fresh tokens via the auth flow; clear your browser site data for + `http://127.0.0.1:7272` after first launch. + +## Rolling back to v2 + +The migration is non-destructive — the v2 JSON files remain in place — +but if you want to cleanly revert the CLI as well: + +```bash +# Restore the backup you made in step 1 +rm -rf ~/.arc +mv ~/.arc.v2-backup ~/.arc + +# Re-install the v2 CLI +npm install -g @axiom-labs/arc-cli@0.4 +arc setup +``` + +To build v2 from source, check out the `archive/v0.4.x` tag: + +```bash +git checkout archive/v0.4.x +pnpm install +pnpm install:local +``` + +That tag is frozen — no fixes land there — but it is always recoverable. + +## Troubleshooting + +### `arc daemon start` says "daemon already running" + +You have a daemon (perhaps from a prior session). `arc daemon status` +confirms; use the existing one or `arc daemon stop && arc daemon start`. + +### Profiles look empty after migration + +`arc profile list` reads from `~/.arc/config.json`, not the database. If +the list is empty check `ARC_DIR` and the file directly — migration +never touches this file. + +### `unauthorized` errors from clients + +The daemon generated a fresh root token in `~/.arc/auth.json`. Clients +that cached a v2 token need to re-authenticate; the CLI does this on +first connect. The TUI prompts you through the pairing flow. + +### Anything else + +Check `arc daemon logs --tail`. The structured logger records every +session open, dispatch error, and protocol-validation failure. The +[daemon operator guide](./daemon.md#troubleshooting) covers the common +cases. + +## Further reading + +- [Architecture](./architecture.md) — how the daemon fits together. +- [Daemon operator guide](./daemon.md) — lifecycle, env vars, pairing, + `readPid`. +- [Wire protocol spec](./protocol.md) — frame format, envelopes, method + catalog. +- Plan of record: [`docs/plans/arc-v3-daemon.md`](./plans/arc-v3-daemon.md).