Skip to content

Commit 241c9f1

Browse files
committed
Robustness improvement.
1 parent 4e58fc9 commit 241c9f1

15 files changed

Lines changed: 4081 additions & 835 deletions

File tree

‎README.md‎

Lines changed: 88 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -74,8 +74,9 @@ That separation is the whole point:
7474
│ secrets · usage (rules, persistence, refusals) │
7575
│ machinery manager.py run lifecycle, event fan-out (subscribe) │
7676
│ protocol.py JSONL line parsing │
77-
│ runner.py ServerRunner: one resident harness │
78-
│ process per conversation │
77+
│ runner.py ResidentRunner: the whole sandbox │
78+
│ lifecycle, once; Server/Docker supply │
79+
│ only a SandboxTransport │
7980
│ infra/ config (PAW_* env) · db (engine + migrations) · │
8081
│ security (JWT, Fernet) │
8182
│ models/ ORM entities: users, conversations, runs, │
@@ -105,6 +106,12 @@ That separation is the whole point:
105106
| Backend ↔ harness | JSON-lines over stdin/stdout pipes | ops down, events up |
106107
| Browser ↔ backend | Server-Sent Events (SSE) | events streamed to the UI |
107108

109+
Each SSE frame carries the harness `seq` as its event id, so a
110+
reconnecting client sends `Last-Event-ID` (or `?last_event_id=`) and
111+
resumes from there instead of replaying the run. Lifecycle events have
112+
no `seq` and are always delivered, so a resume can never skip the
113+
terminal event and leave a client hanging.
114+
108115
The controller is a **protocol translator**: it parses each JSONL line
109116
from the harness, fans it out to in-memory subscribers, and re-emits it
110117
as an SSE `data:` frame. The browser therefore sees the same events the
@@ -166,8 +173,10 @@ the harness answers with events.
166173

167174
```
168175
host → harness: {"op": "submit", "prompt": ..., "run_id": ...}
169-
{"op": "answer", "run_id": ..., "answers": [...]}
176+
{"op": "answer", "run_id": ..., "answers": [...], "ask_id": ...}
170177
{"op": "cancel", "run_id": ...} / {"op": "ping"} / {"op": "shutdown"}
178+
{"op": "hello", "protocol_versions": [...]}
179+
... each op carries an "op_id", echoed on what it causes
171180
172181
harness → host: {"type": "ready"} (once, on startup)
173182
start / delta / notify / log (per run, streamed)
@@ -197,7 +206,41 @@ Because the process is **resident** (survives between turns):
197206

198207
`tool_calls` is additive: a harness that doesn't emit it degrades
199208
gracefully to the bare names from `tool_start`. `log` lines are shown
200-
too, except internal session bookkeeping (the generated title).
209+
too, except internal session bookkeeping (the generated title). A
210+
`kind` this build does not know is still relayed, flagged
211+
`unknown_kind` so a client can decline to render it.
212+
213+
### What the driver validates
214+
215+
The protocol's own guarantees are checked rather than assumed:
216+
217+
| field | check |
218+
|---|---|
219+
| `protocol_version` (on `ready`) | refused unless supported — loudly, once per run, instead of misparsing every line |
220+
| `capabilities` (on `ready`) | recorded and logged; features degrade gracefully, so nothing is gated on them |
221+
| `run_id` | a line labelled with another run is dropped, not fanned out under this one |
222+
| `seq` | monotonicity checked per run; gaps and reorders counted (`paw_protocol_seq_anomalies_total`) |
223+
| `errors[].code` | kept beside the text and persisted as `run.error_code`, so callers branch on a code, not on prose |
224+
| `op_id` (on `error`) | echoed back by the harness, so a refusal names the op it refused instead of arriving unattributed |
225+
| `notify` payloads | size-capped before relay (`PAW_MAX_EVENT_BYTES`), type-preserving |
226+
227+
A protocol `error` line (a rejected `answer`, a stale `ask_id`) is kept
228+
in its own trail: it stays visible, but it does not by itself mark an
229+
otherwise successful run as failed.
230+
231+
Negotiation runs both ways: `ready` tells the host what the harness
232+
speaks, and `hello` tells the harness what the host can parse, so a
233+
version mismatch is settled before any run. Both are sent only to a
234+
build advertising the matching capability (`hello`, `op_id`) — an older
235+
harness answers an unknown op with an `error` line, so the host does
236+
not speak features the sandbox never claimed.
237+
238+
A warm sandbox is probed with `ping` before reuse
239+
(`PAW_SANDBOX__PROBE_TIMEOUT`). A resident process that is alive but no
240+
longer reading its stdin is invisible to an exit-status check, and
241+
submitting to it would hang for the host timeout; the probe turns that
242+
into a respawn. Teardown sends `shutdown` before closing stdin, so an
243+
active run can still emit its terminal `result`.
201244

202245
---
203246

@@ -362,6 +405,7 @@ groups use a double underscore (e.g. `PAW_HARNESS__CMD`).
362405
| `PAW_RUNNER` | `server` | `server` (host subprocess) or `docker` (isolated container) |
363406
| `PAW_SANDBOX__TTL_SECONDS` | `300` | idle sandbox reaper TTL (never applied to a sandbox with a live run) |
364407
| `PAW_SANDBOX__REAP_INTERVAL_SECONDS` | `60` | how often the background reaper sweeps; `0` disables it |
408+
| `PAW_SANDBOX__PROBE_TIMEOUT` | `5` | liveness `ping` before reusing a warm sandbox; `0` disables it |
365409

366410
The three budget vars are spawn-time arguments to `harness serve`, not
367411
fields on the submit op: the harness refuses to take a budget off the
@@ -409,6 +453,11 @@ command line entirely when unset, so the default is an unbounded run.
409453
|---|---|---|
410454
| `PAW_SHUTDOWN_DRAIN_SECONDS` | `25` | on SIGTERM, seconds to let in-flight runs finish before exit |
411455
| `PAW_CANCEL_GRACE_SECONDS` | `10` | after a *user* cancels, how long to wait for the harness to honour it before destroying the sandbox; `0` disables escalation |
456+
| `PAW_MAX_EVENT_BYTES` | `65536` | per-event cap on relayed harness payloads; `0` disables it |
457+
| `PAW_MAX_LINE_BYTES` | `4194304` | longest single JSONL line accepted from a sandbox; oversize lines are discarded whole |
458+
| `PAW_MAX_RUN_STDOUT_BYTES` | `8388608` | cap on the raw transcript an exec keeps in memory; past it only outcome-bearing lines are retained |
459+
| `PAW_MAX_STORED_EVENTS` | `2000` | cap on events persisted with a finished run (deltas are merged first) |
460+
| `PAW_MAX_SUBSCRIBER_QUEUE` | `1000` | per-SSE-subscriber queue depth; when full the oldest event is dropped |
412461
| `PAW_INSTANCE_ID` | host+pid | this instance's id (set to pod/task name); scopes run reconciliation |
413462

414463
`PAW_CANCEL_GRACE_SECONDS` is a recovery path, not a run budget: it is
@@ -420,6 +469,24 @@ Escalation closes the sandbox so the run's own worker can finalize it.
420469

421470
---
422471

472+
### Bounded by construction
473+
474+
Everything the sandbox emits is untrusted input, so each hop has a
475+
ceiling rather than growing with whatever the agent decides to print:
476+
477+
| hop | bound |
478+
|---|---|
479+
| one line off the pipe | `PAW_MAX_LINE_BYTES` — an oversize line is drained to its newline and dropped, so its tail is never reparsed as fresh lines |
480+
| one relayed event | `PAW_MAX_EVENT_BYTES` — oversize bodies are replaced, type-preserving, envelope untouched |
481+
| the exec's in-memory transcript | `PAW_MAX_RUN_STDOUT_BYTES` — past it only `result`/`error`/error-and-usage `notify` lines are kept, so the verdict and token counts can never be dropped |
482+
| a subscriber's backlog | `PAW_MAX_SUBSCRIBER_QUEUE` — oldest-first drop keeps a live view current; the `id:` gap tells the client to resume. The terminal event is never dropped |
483+
| the stored transcript | `PAW_MAX_STORED_EVENTS` — consecutive deltas are merged (lossless for rendering), then the middle is elided with a marker |
484+
485+
Deltas are token-sized, and the transcript used to keep one row per
486+
token in a JSON column, so a long answer grew the row without bound.
487+
Merging them is lossless because any client renders deltas by
488+
concatenation.
489+
423490
## Design principles
424491

425492
**Routes are thin; controllers own the rules.** A route resolves
@@ -442,6 +509,23 @@ or writes an ORM entity, or can refuse a request lives in `controllers`.
442509
Keeping that direction means the bottom layer never imports `models` or
443510
`controllers.errors`.
444511

512+
**One resident lifecycle, two transports.** `ServerRunner` and
513+
`DockerRunner` differ only in what hosts the harness: a host
514+
subprocess or a container. That difference is confined to
515+
`SandboxTransport` (spawn, read a line, write a line, check alive,
516+
interrupt a blocked read, tear down). The registry and its locking,
517+
the ready handshake, version negotiation, liveness probing, the idle
518+
reaper, cancel/answer, the exec lifecycle and the event pump live once
519+
in `ResidentRunner`.
520+
521+
They used to live twice, and the two copies drifted five separate
522+
times — `destroy` skipping teardown in one of them, a respawn
523+
re-registering onto a swept sandbox, `hello` never sent, a warm
524+
sandbox never probed, and two different answers for a create-time
525+
spawn failure. Every one was found by reading the pair side by side
526+
rather than by a test. Sharing the logic is what makes that class of
527+
bug unrepresentable.
528+
445529
**The harness is decoupled.** It is a black-box binary driven only by
446530
its JSONL protocol (`seq` for ordering, `run_id` for correlation,
447531
`usage` for billing). No imports, no shared state — the web side is

0 commit comments

Comments
 (0)