Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions docs/architecture/runtime-and-lifecycle.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,46 @@ claims still block new work, but completed results remain readable. The recorded
actor UID is checked before lifecycle calls; a same-name replacement cannot be
adopted implicitly.

`SessionService.RevertSession` lets an authorized caller restore a confirmed
crashed runtime. It accepts active and idle Sessions, but verifies Substrate's
CRASHED state for the recorded Actor UID and pinned template before admitting
recovery. Quiet turns, transport failures, missing Actors and same-name
replacements cannot authorize a revert. A quiet turn returns `FAILED_PRECONDITION`
with reason `KAGENT_SESSION_RUNTIME_NOT_CRASHED`. A missing or replaced Actor
returns `FAILED_PRECONDITION` with `KAGENT_SESSION_RUNTIME_UNRECOVERABLE`,
because no retry restores it. A transport failure returns `UNAVAILABLE`.
Only an active lifecycle operation returns `ABORTED`. A Session at rest outside
READY returns `KAGENT_SESSION_RUNTIME_NOT_CRASHED`, so a client does not retry
it until its deadline.

Admission records FAILED/RESUME with a RuntimeCrashed failure under the Session
lock. The same transaction publishes a durable terminal result saved before the
crash, or fails nonterminal work absent from the retained snapshot. Waiting tasks
captured by that snapshot remain available. Checkpoint creation excludes competing
recovery.

The caller reads the Session's lifecycle generation before it observes Substrate.
Admission rejects the crash when another lifecycle operation began after that
read. A concurrent revert keeps the Actor UID and returns the Session to READY,
so only the generation exposes a stale observation.

An issued idle suspension does not exclude recovery. It cannot finish against the
crashed Actor, and its worker keeps the claim when suspension fails. Admission
retires the claim in its transaction, so a late finish records nothing.

The worker reverts the same Actor to its last durable snapshot without replaying
input. A runtime without a durable snapshot returns to its initial state.
An interrupted revert continues from REVERTING; a lost response can be reconciled
from SUSPENDED. Already admitted recoveries survive API restarts and retry on the
existing recovery interval in ID-ordered pages of at most 100. The worker does
not inspect READY Sessions or initiate a new recovery.

A successful revert returns the Session to READY with the same Actor suspended.
The next authorized interaction resumes it through Substrate ingress. Runtime
changes after the retained snapshot are lost; the discarded turn receives no
checkpoint at that older snapshot. Another revert against the now-healthy Actor
returns `FAILED_PRECONDITION` with `KAGENT_SESSION_RUNTIME_NOT_CRASHED`.

```mermaid
sequenceDiagram
participant Client
Expand Down
30 changes: 28 additions & 2 deletions docs/lifecycle-retries.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@

Session and Sandbox lifecycle mutations run within the API request. The server
records the intended operation before calling the runtime and records completion
before returning success. Sessions expose Create, Fork, and Delete; explicit Suspend
and Resume are available only for Sandboxes. Session runtime suspension and resumption
before returning success. Sessions expose Create, Fork, Revert, and Delete; explicit
Suspend and Resume are available only for Sandboxes. Session runtime suspension and resumption
are platform-managed. A failed request can have changed the runtime: a timeout
does not mean that nothing happened, and canceling a request does not undo it.

Expand Down Expand Up @@ -39,6 +39,32 @@ for subsequent invocations. Omitting it generates a new ID on each invocation,
so rerunning that command is a new creation rather than a retry. MCP callers
should likewise retain the `request_id` passed to `create_sandbox`.

## Recovering a crashed Session

Call `SessionService.RevertSession` with the Session ID to restore its recorded runtime to the last durable snapshot.
The caller needs Session update access. The operation verifies Substrate's CRASHED state and the recorded Actor identity.
Recovery fails interrupted turns and waiting tasks absent from the retained snapshot.
It preserves durable terminal results and returns the Session to READY with its Actor suspended.
It discards runtime changes since that snapshot and does not replay input.
A runtime without a snapshot restarts from its initial state.

Retry an interrupted revert against the same Session ID. Pending recovery retains its operation across API restarts.
The lifecycle worker retries already admitted recoveries; it does not initiate recovery for other crashed Sessions.
Checkpoint creation returns `ABORTED` until it finishes.
A revert that observed the crash before another lifecycle operation began also returns `ABORTED`.
So does a revert while another lifecycle operation is active.

A revert that can never succeed against the Session as it stands returns `FAILED_PRECONDITION`.
Its `google.rpc.ErrorInfo` detail, in domain `kagent.dev`, names one of two reasons:

| Reason | Meaning | Client action |
| --- | --- | --- |
| `KAGENT_SESSION_RUNTIME_NOT_CRASHED` | The runtime is healthy or suspended. A retry after a revert whose response was lost lands here. | Treat as recovered and continue the conversation. |
| `KAGENT_SESSION_RUNTIME_UNRECOVERABLE` | The recorded Actor is missing, or a same-name Actor replaced it. | Stop retrying. Start another Session, or fork one from a checkpoint. |

A `FAILED_PRECONDITION` without either reason follows the general table below.
Read the reason rather than the message text, which is not stable.

## Errors

| gRPC status | Client action |
Expand Down
Loading
Loading