Skip to content

persistence.md

Latest commit

 

History

History
346 lines (293 loc) · 19.4 KB

persistence.md

File metadata and controls

346 lines (293 loc) · 19.4 KB
Raw

Persistence and recovery

What the runtime stores, what survives a restart, how a host action that was waiting on the host is recovered, and what is deliberately not promised. Read concepts first for sessions, turns and host tools.

What is stored

State With Postgres Without Postgres
Sessions, the message tree (user, assistant and host messages), steering rows in sessions, messages, steering; written through on every change process memory, until cache eviction or process shutdown
The pending host action sessions.pending_action process memory
Prompt-artifact versions, runtime settings rows process memory
Queued inbox messages rows process memory; listing and marking surfaced through HTTP still need Postgres
Encrypted provider keys rows unavailable
Traces (debug) rows not stored
Background tasks, persistent agents and their messages rows in tasks, agents, agent_messages process memory, lost on restart
Events, actions and their confirmations, host state rows in events, actions, action_confirmations, host_state process memory, lost on restart

Messages, tasks, agent messages, actions, events and voice calls record updated_at, which every write sets to the writing transaction's time (migration 0039; existing rows took the latest time they already recorded). Change cursors read them in (updated_at, id) order, through an index on each table (see change cursors). Without Postgres the in-memory stores keep it from the process clock.

The session store is a write-through cache: every change is written to the row before the in-memory context is updated. The row may be ahead while a write completes, but memory is never durably ahead of the row. A session that is not in the cache is loaded from its rows on first use. Postgres-backed sessions expire after ASSISTANT__SESSION_TTL_HOURS (0 keeps them until they are deleted); expiry deletes the session with its messages, steering and pending action.

Requiring Postgres

The table's right-hand column is a fallback, chosen once at startup. A database that answers at startup but cannot load the runtime settings fails startup, since the next settings change would otherwise overwrite the stored ones. A deployment that must not lose state sets DATABASE__REQUIRED=true: startup then fails with DatabaseUnavailableError when Postgres is unreachable, and nothing falls back to process memory. A host that chooses a database should set it, so that a restart during an outage fails instead of starting in memory mode. It affects startup only: a database lost later leaves /health at 200 (status: degraded) and /health/ready at 503, whether it is set or not (see health). With DATABASE__MIGRATE_ON_START as well, the runtime creates its database when the server is reachable but the database does not exist, then migrates it; a host needs only a running PostgreSQL server.

A database lost after startup is not a fallback, with or without DATABASE__REQUIRED. Each call tries it again, so the runtime recovers as soon as Postgres is back; until then a route that uses it answers 503 with {"error": "Database not reachable", "type": "DatabaseError"}. A runtime-settings change or an inbox message that cannot be saved is refused the same way rather than kept in memory, and a turn whose artifacts cannot be read fails instead of building the prompt from the defaults. Turn traces, the inbox drain, voice checkpoints and OAuth token deletion are best-effort and log a warning; messages a failed drain could not read stay in the inbox for the next turn. A refreshed OAuth login that cannot be saved stays in memory for the process, and its status reports persisted: false.

DatabaseUnavailableError and MigrationError carry a machine-readable cause, so a host can word them for its own users: unreachable (no server answers), auth_refused (the server refused the role), database_missing (the database does not exist and was not, or could not be, created) or migration_failed.

The substrate is deliberately the application's own tables. The upstream options were compared and deferred; see the decision below.

Every assistant message stores the record of the system prompt it was produced with in messages.prompt; the stable part of a prompt (the artifacts, included profiles, documents) is stored once in prompt_snapshots under its SHA-256, since it repeats across turns. Without Postgres the record stays on the in-memory message and the process keeps the most recent snapshots.

Every message also stores what the model was given, as Pydantic AI messages (messages.model_messages, serialised with ModelMessagesTypeAdapter). A user message stores its request: the host context block, its text and its reference attachments. An assistant message stores everything its turn added after that request: responses with their thinking, tool calls and results, delivered steering and ingress messages, and a continuation's result with its host context. Later turns replay these messages unchanged, so each request starts with what the model saw before. Three things are not stored:

  • instructions, since every run sends its own;
  • binary tool output, stored as [Binary content omitted];
  • a look_at_screen image, stored as [Inspected current screen].

Reference attachments are stored with their message, data included, so bound what you send. segments remain the display form. Messages stored before model_messages existed are replayed from their text and segments.

Background tasks are rows in tasks (what was asked, its session, status, result and times). When the runtime starts, a task left queued or running is marked interrupted; its work is not replayed. That recovery covers every task in the database, so run tasks in one process per database; a task's end is recorded once, and a later terminal update changes nothing. Without Postgres, tasks live in process memory and are lost on restart. Persistent agents are rows in agents (their current session, status and config) and their messages rows in agent_messages; a restart keeps the agents and marks every unfinished message in the database interrupted in the same way, so the one-process rule covers agents too.

Host-written records have their own tables: events (unique per source and event id, numbered in arrival order), actions (status history, text revisions, per-recipient results) and action_confirmations (one row per recipient, in insertion order, deleted with their action). Host state is host_state (namespace, key, JSON value, version); a deleted key stays as a tombstone so its version is never reused. Without Postgres they live in process memory and are lost on restart.

Artifact versions are keyed by profile, subject (empty for profile-scoped artifacts) and name. A session created for a subject stores its profile and subject on the row, decided by the session's first saved message (an empty session enforces none), so a restart keeps the binding. keep_versions bounds how many superseded versions an artifact keeps; the rest are deleted as new ones are written.

With DATABASE__REQUIRED, each profile-scoped artifact of a registered profile that has no stored version gets its default stored as version 1 at startup (actor seed), so the text a prompt uses is always a versioned record. A changed default is stored and activated as a new seed only while every stored version is a seed and the newest is active; any host or assistant version (or a rollback) stops that for good, so a shipped default never overwrites anyone's choice.

Concurrent artifact edits

Artifact version checks, writes, activation and rejection run in one transaction, serialized per profile/artifact in Postgres; a unique index allows at most one active version of an artifact. An edit with a stale expected_version returns 409. A failed activation rolls back the proposed version, and a prompt-cache read started before an edit cannot hide that edit from subsequent prompts.

The pending host action

A host tool call ends the turn. The runtime then:

  1. writes the assistant message with the call(s) and no result;
  2. writes pending_action (tool_call_id, tool_name, assistant_message_id, batch: every host call of that response in order) on the session row;
  3. emits final_response.pending_tool_call for the first call and waits.

When a continuation arrives, the accepted result is written on the same assistant message first. For a batch with calls still waiting, pending_action then moves to the next call and the turn ends with that call as pending_tool_call, without a model run; only the final continuation clears pending_action and resumes the model. A crash between steps 1 and 2 leaves a call without a result and no pending action: it is resolved as unknown on the next load. A crash after the result is written but before the row is cleared leaves a pending action whose call already has a result: the load drops it, because the result is already recorded.

On load, the stored pending action is restored only when its assistant message exists and the call still has no result. Every other call without a result is given a synthetic unknown result so the history stays consistent.

GET /api/sessions/{id} returns pending_action with the call id, tool name, arguments and assistant message id, so a host that reconnects or restarts can perform the waiting action and continue without reading the transcript.

Action outcomes

A host tool entry in the stored segments (GET /api/sessions/{id}/messages) carries the native outcome the model sees and, when the action did not complete, a status saying how it was resolved.

status outcome Produced when External effect
completed (no status key) success the host sent its result happened, per the host
failed failed the continuation carried tool_outcome: "failed" did not happen, per the host
cancelled interrupted the turn was cancelled or stopped by a limit before the call returned unknown
superseded interrupted a new user message arrived before the continuation, or core closed an older dangling call when a newer result arrived never recorded
unknown interrupted the session was loaded without a matching pending action, or POST /api/sessions/{id}/repair was called unknown

The runtime never asserts that a cancelled, superseded or unknown action had no effect. If the host knows, it says so with the continuation (tool_outcome) or by telling the user.

Reconnect, restart, duplicates, retries

  • Reconnect. A Socket.IO disconnect leaves the turn running and its events are not replayed. Read the session (messages, and pending_action on the session) to recover the state, then continue.
  • Restart. With Postgres, a pending host action survives: the continuation is accepted by the new process. Without Postgres the session is gone with the process, and the continuation is rejected as a session error because the session does not exist.
  • Duplicate continuations. Once a call's result is recorded (whatever its status), any continuation for that tool_call_id is rejected with Continuation rejected: the result for tool call '...' was already recorded (status: ...). A recorded result is never applied twice, in the same process or after a restart.
  • Retries. The runtime never re-issues a host action: the model asked once, the host performs it once. A host that retries on its own side must make the action idempotent or reconcile an unknown first attempt itself; tool_outcome: "failed" is for actions the host knows did not occur. If the continuation is accepted and the model step after it fails, the result stays recorded and the client continues with a new message rather than resending the continuation.
  • Repair. POST /api/sessions/{id}/repair resolves the pending action and every call without a result as unknown, in memory and on the rows, so a session stuck on a host that will never answer can continue.

Worker and concurrency topology

Supported: one runtime process per set of sessions. Within a process, many sessions run concurrently and each session has at most one live turn; a new message cancels the live turn and waits for its cleanup, a continuation waits without cancelling.

The in-memory context is authoritative for a session while its process runs, and it is refreshed from the rows only on a cache miss. Within a process, turns on one session are serialised: a continuation is planned only after the previous turn has finished and persisted, so two continuations for the same call arriving together are accepted once and rejected once. Two processes serving the same session at the same time would each cache their own copy; the runtime does not coordinate them, and each could accept the same continuation. To run several processes against one database, route each session to one process (sticky sessions by session_id). A process that restarts, or a session that moves to another process after the first stopped, is the supported recovery path and is what tests/compatibility/test_recovery.py exercises: a cold cache over the same rows accepts the pending continuation, rejects duplicates and resolves unanswered calls.

Retention, branching and existing data

  • Migration 0022 adds the nullable pending_action column; nothing else changes. Rows created before it have no pending action and their unanswered calls become unknown on the next load.
  • Branching is unaffected: the pending action names its assistant message, and a branch from an earlier message is a new user message, which supersedes the pending action like any other.
  • Retention is the session TTL; a pending action does not extend it.
  • Migration 0034 allows the host role. Its downgrade is refused while host messages exist, since deleting them would delete every message that follows them.

Local persistent option

The documented local persistent setup is Postgres in Docker: make db-up && make db-upgrade from a checkout, or assistant-runtime migrate from an installed package (the migrations ship in the wheel); see getting started.

A host that builds its settings in code upgrades the schema from Python with the same migrations, and can show the result:

from assistant_runtime.services.database.migrations import migrate, schema_status

await migrate(settings.database)          # raises MigrationError
status = await schema_status(settings.database)  # state, current, head, error

DATABASE__MIGRATE_ON_START=true does the same before the services start. schema_status() never raises; its state is unreachable, auth_refused, database_missing (created at start in required mode with migrate_on_start), behind (upgraded at start with migrate_on_start) or ready, and error keeps the driver's message. assistant-runtime doctor reports reachability and the schema revision against the packaged head. A local server needs no stored password: leave DATABASE__PASSWORD empty for trust authentication, or set DATABASE__HOST to the socket directory (for example /tmp) for peer authentication. peer matches the database user to the operating-system user running the runtime, so set DATABASE__USER to that user (or map it with a PostgreSQL user-name map). Concurrent upgrades, in one process or several, run one at a time. It is the same schema and the same migrations as production. SQLite is not supported: the schema uses JSONB, INSERT ... ON CONFLICT through the Postgres dialect, Postgres server defaults and a partial unique index, and the migrations are written for Postgres. SQLite would need a separate schema and migration path.

Upstream decision

Recorded on 2026-09-06 against Pydantic AI 2.38.0 (2.40.0 as the comparison target, see compatibility).

  • pydantic_ai.durable_exec (Temporal, DBOS, Prefect): deferred. These make one agent run durable by journaling model requests and tool calls in an external workflow engine, and they require that engine and its worker topology. The runtime's recovery unit is different: a host action ends the run, and the continuation starts a new run from the persisted message tree. The pending action is state between runs, which the session row now holds. Upstream also documents that a durable unit may execute more than once after a crash, so it does not provide the exactly-once external effects guarantee either. Applications that want durable model calls can attach a durability capability through AssistantDefinition (see composition); the runtime does not adopt one.
  • Harness StepPersistence: deferred. It persists harness steps, not the graph state of a run, and the runtime does not use the Harness loop (see the compaction evaluation in compatibility).
  • Native message history: used. Every message row stores its Pydantic AI messages (ModelMessagesTypeAdapter), and later turns replay them unchanged, native outcome values and thinking signatures included. The segments are the display form. Provider-side conversation ids are not used, so history does not depend on a provider.

Guarantees, in one place: sessions, messages, steering and the pending host action are durable with Postgres; a continuation is accepted after a restart when the matching pending_action row was committed, and the unanswered call is resolved as unknown otherwise; a recorded result is never applied twice; unanswered calls are distinguishable as cancelled, superseded or unknown; external effects are never assumed, never undone and never retried by the runtime.

Voice checkpoints

Migration 0023 adds voice_calls with a JSON snapshot and a cascading foreign key to the backend session; 0039 adds created_at (when the call started, from the snapshot) and updated_at (the last checkpoint write). The optional voice bridge coalesces transcript, delegation and cumulative duration-usage checkpoints while connected and writes a final snapshot on close. Audio is not retained. Backend tool/message history still uses the normal native persistence path. Database outages leave the call usable in memory; a process crash can lose recent uncheckpointed fragments. Restored active calls are marked interrupted, with provider finalization unconfirmed; neither provider connections nor tool execution resume automatically. Reading cached records also requires access to the current parent session. These snapshots are observation records, not durable execution checkpoints or proof of audio playback.

Queued steering profiles

Queued steering stores its optional profile selector. An explicit selector remains pending across cancellation and restart until a turn with that profile consumes it. Unprofiled legacy and ingress records inherit the consuming turn. Existing Postgres installations must apply migration 0026 with assistant-runtime migrate (or make db-upgrade) before running this version.

Background memory writes

Working-memory extraction runs after the turn completes. Within one runtime, session metadata writes and updates to each message are serialized through transaction commit, so an older background write cannot replace a newer pending action or continuation usage total. Continuation snapshots retain the latest background-owned working-memory usage without counting it twice.