diff --git a/CHANGELOG.md b/CHANGELOG.md
index d6efcbc..d3673fd 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,70 @@
# Changelog
+## Unreleased
+
+### Durable mid-run steering (issue #16)
+
+- 新增 `POST /v1/runs/{run_id}/steer`：在 Run 执行期间（含 `paused`）durably 注入新的
+ 结构化输入，不等待完成、不强制取消、也不需要先暂停。支持 `Idempotency-Key`
+ header/body 去重、`kind`/`payload`/`metadata` 结构化 payload、大小上限校验。
+- Python SDK（同步/异步）新增 `client.runs.steer(run_id, kind=..., payload=..., metadata=...,
+ idempotency_key=...)`。
+- Runtime 新增 `runtime.has_steering`、`runtime.peek_steering()`、
+ `runtime.drain_steering()`；`compiled_graph.steer(run_id, ...)` 在嵌入式/无 Server
+ 场景下提供同一套语义，未使用时零开销、不泄漏 channel 状态。
+- Run 生命周期新增 `run.steer.accepted`、`run.steer.consumed`、`run.steer.superseded`
+ SSE 事件；`consumed` 事件携带 `steering_event_id`、`source_event_id`、`sequence`、
+ `kind`、`queue_latency_seconds`、`node`、`namespace`、`task_id`；`superseded` 事件
+ 在 resume-transfer 与 final-boundary disposition 两种场景下幂等写入，同样携带
+ `sequence`/`kind`/`source_event_id`。
+- Resume 语义：`POST /v1/runs/{run_id}/resume` 在**同一原子事务**中创建新 Run 并把旧
+ Run 上仍 `pending`/`delivered` 的 steering 事件迁移到新 Run（保序、保留身份）；旧
+ Run 之后再收到 `/steer` 或 `/cancel` 返回 `409 run_superseded`。`source_event_id`
+ 在跨多次 pause/resume 时始终指向客户端最初 `/steer` 收到的根 id，`created_at`/
+ `queue_latency_seconds` 从最初 durable accepted 时刻起算，不因中间的 resume 跳数
+ 被重置。旧 Run 是否已被 resume 取代由新增的 typed `runs.superseded_by_run_id` 列
+ 记录（与该事务同时原子写入），而不是用户可写的 `metadata`，避免调用方在创建 Run
+ 时伪造该字段来假冒 lineage 状态。
+- Cancel 语义：`request_cancel()` 现在对每个非终态状态都有明确策略——`pending`/
+ `paused`（没有存活 worker 可协作）立即变为 `cancelled`；`running` 变为
+ `cancelling`，由持有该 Run 的 worker 协作完成终结；已被 resume 取代
+ （`superseded_by_run_id` 已设置）的历史 Run 拒绝为 `409 run_superseded`，不会
+ 假成功。`claim_run()` 的租约过期回收同样区分：过期的 `running` 恢复为 `pending`；
+ 过期的 `cancelling`（worker 在收到 cancel 后、终结提交前崩溃）恢复为 `cancelled`，
+ 避免 Run 永久卡死在 `cancelling` 并阻塞同 thread 后续任务。
+- `commit_steering_consumptions_if_owned()`（状态更新 + `run.steer.consumed` 写入）
+ 与 `finalize_run_with_steering_disposition_if_owned()`（终态提交 + 未消费 steering
+ 的 `superseded` disposition + `run.steer.superseded` 写入）均在同一事务/临界区内
+ 完成，并按 `(lease_owner, attempt)` fenced：同一批 consumption 的重复提交（例如
+ worker 在收到 ack 前连接断开后重试）只会产生一次 `run.steer.consumed`；旧 attempt
+ 在 lease 被新 worker 接管后无法再修改该 Run 的 durable 状态。`retry_run_with_event_
+ if_owned()` 同样把状态转换与 `worker_retrying` 事件合并为一次原子写入，cancel 优先
+ 于 stale 的 retry 结果。
+- PostgreSQL schema：新增 `run_steering_events` 表及 `runs` 表的多个新增列，共 4 个
+ 纯新增、forward-only 的 Alembic revision：
+ - `0002_steering` — `run_steering_events` 表本身；
+ - `0003_steering_source_event` — `source_event_id` 列/局部索引；
+ - `0004_steering_closed` — `runs.steering_closed`（finalization 阶段的 steering
+ admission 门控）；
+ - `0005_superseded_by_run_id` — `runs.superseded_by_run_id`（typed resume-lineage
+ marker，见上）。
+ `alembic upgrade head` 是生产环境唯一支持的迁移路径，**必须到达
+ `0005_superseded_by_run_id`**——`/steer`、`/cancel`、finalization、resume 的生产
+ SQL 直接依赖 `0004`/`0005` 新增的列，只迁移到 `0003` 会导致这些代码路径在缺列的
+ schema 上报 `UndefinedColumn` 错误。四个 revision 均可在部署新二进制前对存活数据库
+ 提前执行（schema 层面向后兼容，旧二进制会忽略新增的 nullable 列）；但 schema
+ 兼容不等于语义兼容——只有在控制面实例全部升级到本版本后，`steering_closed`/
+ `superseded_by_run_id` 才会被读取并生效，回滚二进制会失去这些新语义（不是完整的
+ 语义回滚保证），必要时应在回滚前 quiesce 或等待 in-flight Run 结束。Redis 仅用于
+ 可选的低延迟通知，从不作为唯一数据源；Redis 丢失不影响最终投递。
+- 兼容性：不影响不使用 steering 的现有图与调用方；上述 schema 变更均为纯新增列/表，
+ 无 breaking API 变更。
+- 新增中英文文档：`Wiki/content/docs/{zh,en}/api/threads-runs.mdx`（steer API 全量
+ 语义）、`python-sdk.mdx`（SDK 示例）、`errors-events.mdx`（错误码与生命周期事件）、
+ 新增概念页 `Wiki/content/docs/{zh,en}/concepts/steering.mdx`、新增运维页
+ `Wiki/content/docs/{zh,en}/operations/steering-operations.mdx`；`docs/api.md`
+ 同步更新。
+
## 2.2.0
### Cache-first prompt optimization
diff --git a/README.en.md b/README.en.md
index d843ab7..6fd22d8 100644
--- a/README.en.md
+++ b/README.en.md
@@ -55,6 +55,7 @@ flowchart TB
LingxiGraph is for product and platform teams that need to deliver Agent experiences reliably to end users. It focuses on the engineering problems that appear after a model call works in a prototype:
- **Continuous user experiences** — threads, checkpoints, history, forks, resumes, and resumable SSE support multi-turn sessions, long-running work, and reconnects.
+- **Durable mid-run steering** — inject new structured input into a running (or paused) run without cancelling it; delivery is durable, ordered, deduplicated, and survives worker crashes and paused-run resumes.
- **Composable intelligent workflows** — state graphs, subgraphs, and supervisor, handoff, swarm, group chat, plan-and-execute, parallel-review, and map-reduce patterns organize complex requests.
- **Governed production execution** — pinned graph versions, lease-based queues, idempotency keys, timeouts, budgets, quotas, dead-letter/redrive, and cooperative cancellation make retries and scaling manageable.
- **Tenant-aware security boundaries** — OIDC/JWT, RBAC, tenant isolation, PostgreSQL RLS, audit records, and safe JSON state serialization provide runtime foundations for data isolation in online products.
diff --git a/README.md b/README.md
index 9629d9e..cab6cbd 100644
--- a/README.md
+++ b/README.md
@@ -32,6 +32,8 @@ StateGraph · Runtime · Checkpoint · Events
- **StateGraph**：用节点和边描述 Agent 与业务执行逻辑。
- **Durable Execution**：支持 checkpoint、暂停恢复、重试与长任务执行。
+- **Durable Steering**：运行中（甚至暂停中）durably 注入新的结构化输入，无需取消 run；投递
+ durable、有序、去重，并能在 worker 崩溃和 paused run resume 后依然可靠送达。
- **Streaming Events**：通过 Python Runtime、REST 与可续传 SSE 暴露运行状态。
- **Production Runtime**：提供 Agent Server、Worker、持久化、权限、审计与观测能力。
- **Open Integrations**：支持 OpenAI-compatible、Coze、A2A、MCP 与开放 Agent Skills。
diff --git a/Wiki/content/docs/en/api/errors-events.mdx b/Wiki/content/docs/en/api/errors-events.mdx
index 7138d52..a3917f3 100644
--- a/Wiki/content/docs/en/api/errors-events.mdx
+++ b/Wiki/content/docs/en/api/errors-events.mdx
@@ -29,6 +29,16 @@ Non-2xx responses use `application/problem+json`:
Common stable codes include `idempotency_conflict`, `quota_exceeded`, and `join_timeout`, plus problem codes mapped from schema, authentication, concurrency, and resource-state failures. Treat new codes as an unknown error rather than failing to parse them.
+### Steering-specific error codes
+
+| `code` | HTTP | Triggered when |
+| --- | --- | --- |
+| `run_terminal` | 409 | `/steer` is called against a run in a terminal status (`succeeded`/`failed`/`cancelled`/`timed_out`/`dead_letter`); no durable write is created |
+| `run_superseded` | 409 | `/steer` or `/cancel` is called against an old `run_id` already superseded by a `resume`; call it against the new run's `id` instead (see `metadata.resumed_from_run_id`). For `/cancel`, this fires even though the old run's `status` is still `paused` -- the check reads the typed `superseded_by_run_id` column, not `status` and not user-writable `metadata`, so the historical run is never mutated to `cancelled` while its resumed descendant keeps running, and the check cannot be forged via a client-submitted `metadata` key (see the operations doc's "The `superseded_by_run_id` lineage marker") |
+| `payload_too_large` | 413/422 | Serialized `payload` + `metadata` exceed the size cap (32KB, or the server's event size cap, whichever is smaller); no durable write is created |
+| `run_resume_conflict` | 409 | Concurrent resume calls raced against the same paused run; exactly one wins and creates the new run, every other caller gets this instead |
+| `run_finalizing` | 409 | `/steer` is called after the run's graph has already finished executing but before its final status is committed (a short window while the worker durably flushes and finalizes); there is no safe point left to consume the event, so it is rejected rather than durably accepted with nobody to ever read it. A `paused` outcome is not affected -- steering a paused run remains normal |
+
## Run business errors
Successful creation returns `202 pending`. A later node/schema/provider error moves the resource to `failed`, `timed_out`, or `dead_letter` and populates:
@@ -67,6 +77,73 @@ data: {"id":"...","run_id":"...","sequence":18,"kind":"node_completed","data":{}
- Lines beginning `: heartbeat` are keep-alive comments and should be ignored.
- The server closes the stream when the run becomes terminal or `paused`.
+### Steering lifecycle events
+
+Besides ordinary execution events (`node_completed`, etc.), mid-run input (steering)
+produces three dedicated SSE event kinds across its lifecycle:
+
+| `kind` | Emitted when | Key fields |
+| --- | --- | --- |
+| `run.steer.accepted` | A `/steer` call is durably accepted (202) | `steering_event_id`, `source_event_id` (if from a resume migration), `sequence`, `kind` |
+| `run.steer.consumed` | Graph code calls `drain_steering()` and that consumption is durably committed by `commit_steering_consumptions_if_owned()` | `steering_event_id`, `source_event_id`, `sequence`, `kind`, `queue_latency_seconds`, `node`, `namespace`, `task_id` |
+| `run.steer.superseded` | A steering event's row durably transitions to the `superseded` disposition without ever being consumed — either because a paused run it belonged to was resumed (its still-pending steering is migrated onto the new run) or because it was still `pending`/`delivered` when the run reached its terminal-finalization boundary (see below) | `steering_event_id`, `source_event_id`, `sequence`, `kind`, `reason` (`resume_transfer` or `unconsumed_at_final_boundary`), `superseded_by_run_id` (set for `resume_transfer`, `null` for `unconsumed_at_final_boundary`), `replacement_steering_event_id` (set for `resume_transfer`, `null` for `unconsumed_at_final_boundary`) |
+
+`run.steer.superseded`'s `unconsumed_at_final_boundary` reason is a deliberately honest,
+generic label — it does **not** claim the event was racing anything. It covers every steering
+row still `pending`/`delivered` once the owning worker reaches a genuinely terminal
+finalization boundary, and that has more than one possible cause: a `/steer` call can durably
+commit its row a moment before the worker closes new admission (a real race), but just as
+commonly the graph simply never called `drain_steering()` at all for a row that had been
+sitting there, fully visible, for the run's entire lifetime — ordinary application-level
+non-consumption, not a race. The Runtime cannot currently distinguish these two cases from the
+data it has, so it does not pretend to; both get the same `unconsumed_at_final_boundary`
+reason. Rather than forcing the run through an indefinite retry loop just because a steering
+event happened to exist — which would penalize a graph that legitimately never calls
+`drain_steering()` at all — the Runtime durably transitions the row to `superseded` and
+finalizes the run with its already-computed outcome. A `run.steer.superseded` event is never
+silently dropped in this case; it is always paired with the disposition change, committed
+together with the run's terminal status in the same durable transaction, and is idempotent
+under retry (a repeated attempt to supersede an already-superseded row emits no duplicate
+event).
+
+Field meanings:
+
+- `steering_event_id`: the id of the steering event under the **current** run (if the event
+ was migrated from an older run, this is the fresh id it got after migration, not the id
+ the client originally received);
+- `source_event_id`: the **root** id the client originally received from its `/steer` call.
+ Even across repeated pause/resume migrations of the same underlying steering, this field
+ stays consistent at every hop, letting a client correlate the eventual
+ `run.steer.consumed`/`run.steer.superseded` back to its original request. This is
+ per-lifecycle-event, not a blanket rule: on `run.steer.accepted` and `run.steer.consumed` it
+ is present only for events that arrived via a migration — a directly-submitted event (never
+ transferred) has no `source_event_id` on those two. `run.steer.superseded` is different: it
+ always sets `source_event_id` to `event.source_event_id or event.id`, so a directly-submitted
+ event's superseded record still carries a root id — its own — even though it was never
+ migrated;
+- `sequence`: this event's position in its run's steering sequence (a separate counter from
+ the run's overall event-stream `sequence`);
+- `kind`: passed through from the `/steer` request body's `kind`;
+- `queue_latency_seconds`: seconds from the steering event's **original** durably-accepted
+ time (including time accumulated across every resume) to when it was consumed;
+- `node`: the name of the node that consumed the event;
+- `namespace`: the subgraph namespace path (array) the consuming node belongs to;
+- `task_id`: distinguishes the specific task among parallel/`Send` tasks in the same
+ superstep.
+
+**PostgreSQL is the durable source of truth; Redis is only an optional accelerator.** All three
+of `run.steer.accepted`, `run.steer.consumed`, and `run.steer.superseded` are durably written
+to PostgreSQL before becoming visible anywhere else. Redis pub/sub is used only to hint
+workers that "steering may be waiting," reducing polling latency. Even if that Redis
+notification is lost (a dropped connection, a Redis restart), a worker still checks/polls
+PostgreSQL at its next safe point and eventually consumes the event — a lost Redis
+notification never permanently drops steering. Similarly, the production Worker path,
+`commit_steering_consumptions_if_owned()`, is idempotent: when a worker retries the same
+consumption batch after a transient failure, only events that actually transition status
+produce a new `run.steer.consumed` — already-consumed events are never re-emitted. The same
+holds for superseding a stale row: retrying an already-superseded transition emits no
+duplicate `run.steer.superseded`.
+
## Retry policy
1. Use bounded exponential backoff only for `retryable=true`, network failure, 429, and transient 5xx.
diff --git a/Wiki/content/docs/en/api/overview.mdx b/Wiki/content/docs/en/api/overview.mdx
index 82214a7..1c3f28d 100644
--- a/Wiki/content/docs/en/api/overview.mdx
+++ b/Wiki/content/docs/en/api/overview.mdx
@@ -59,6 +59,8 @@ stateDiagram-v2
running --> timed_out
running --> cancelling
cancelling --> cancelled
+ pending --> cancelled: cancel (no live worker)
+ paused --> cancelled: cancel (no live worker)
failed --> pending: redrive
dead_letter --> pending: redrive
paused --> pending: resume creates new run
diff --git a/Wiki/content/docs/en/api/python-sdk.mdx b/Wiki/content/docs/en/api/python-sdk.mdx
index 4950c4c..345fa68 100644
--- a/Wiki/content/docs/en/api/python-sdk.mdx
+++ b/Wiki/content/docs/en/api/python-sdk.mdx
@@ -36,6 +36,16 @@ with LingxiGraphClient(
completed = client.runs.join(run["id"], timeout=120)
print(completed["output"])
+
+ # Durably inject mid-run input; success only means it was durably
+ # written -- not that the graph has processed it yet.
+ accepted = client.runs.steer(
+ run["id"],
+ kind="user_input",
+ payload={"message": "switch to Spanish"},
+ idempotency_key="msg-123",
+ )
+ print(accepted["id"], accepted["status"]) # "pending"
```
Available resource methods:
@@ -45,12 +55,35 @@ Available resource methods:
| `graphs` | `list()`, `get(graph_id)` |
| `assistants` | `create(**values)`, `list()`, `get()`, `update()`, `delete()` |
| `threads` | `create()`, `list()`, `get()`, `update()`, `state()`, `history()`, `fork()`, `delete()` |
-| `runs` | `create()`, `get()`, `list(thread_id)`, `join()`, `stream()`, `cancel()`, `resume()` |
+| `runs` | `create()`, `get()`, `list(thread_id)`, `join()`, `stream()`, `cancel()`, `steer()`, `resume()` |
| `store` | `batch(operations)`, `search(namespace, ...)` |
| `schedules` | `create()`, `list()`, `update()`, `delete()` |
Use `client.request(method, path, **kwargs)` for redrive or an endpoint not yet wrapped by a resource.
+### `runs.steer()`
+
+```python
+client.runs.steer(
+ run_id,
+ kind="user_input",
+ payload={"message": "..."},
+ metadata=None,
+ idempotency_key="msg-123",
+)
+```
+
+- `kind` defaults to `"user_input"` but may be any application-defined string;
+- `payload` is the raw JSON the graph reads back via `runtime.drain_steering()`;
+- `idempotency_key`, when set, is also sent as the `Idempotency-Key` header; a repeat call
+ with the same key returns the original event instead of enqueueing a second one;
+- the return value is the 202 response body (`id`, `sequence`, `status`) — it does **not**
+ mean the graph has processed the input; only a subsequent `run.steer.consumed` event
+ means it was actually consumed;
+- calling it against a terminal run raises `LingxiGraphAPIError` with
+ `code == "run_terminal"`; calling it against an old `run_id` already superseded by a
+ `resume` raises `code == "run_superseded"`.
+
## Asynchronous client
```python
@@ -65,6 +98,14 @@ async with AsyncLingxiGraphClient(base_url, token=token) as client:
)
async for event in client.runs.stream(run["id"]):
print(event)
+
+ accepted = await client.runs.steer(
+ run["id"],
+ kind="user_input",
+ payload={"message": "switch to Spanish"},
+ idempotency_key="msg-123",
+ )
+ print(accepted["id"], accepted["status"])
```
Async resources mirror sync resources. Do not use the synchronous client inside an async event loop.
diff --git a/Wiki/content/docs/en/api/threads-runs.mdx b/Wiki/content/docs/en/api/threads-runs.mdx
index 66bf26c..a4358e5 100644
--- a/Wiki/content/docs/en/api/threads-runs.mdx
+++ b/Wiki/content/docs/en/api/threads-runs.mdx
@@ -79,9 +79,143 @@ POST /v1/runs/{run_id}/redrive
```
- Cancellation is cooperative and usually returns `cancelling` first; nodes and providers should honor the cancellation token.
+- Cancelling a `pending` (not yet claimed) or `paused` (parked on `interrupt`, no live worker holding it) run resolves to `cancelled` immediately, with `finished_at` set — there is no live delivery attempt for either status to cooperatively observe a token, so there is nothing to wait on.
+- Cancelling a `paused` run that has since been `resume`d returns `409 run_superseded` instead — the same signal `/steer` gives for a stale `run_id` (see below). The old run's `status` deliberately stays `paused` after a resume (only the typed `superseded_by_run_id` column marks it stale — never user-writable `metadata`, which cannot forge this signal), so cancel checks that same field, atomically, before deciding the status transition; it never silently succeeds against the historical run while the resumed descendant keeps executing. Cancel the new `run_id` instead.
- Only a `paused` run can resume. The response is a new run whose `metadata.resumed_from_run_id` points to the original.
+- Concurrent resume calls against the same paused run have exactly one winner: it creates the new run as above, and every other concurrent caller gets `409 run_resume_conflict` instead of a duplicate run.
- Only `failed` or `dead_letter` runs can redrive. Redrive preserves input/config/version and re-enqueues the run.
Redrive does not mean “retry with edited input.” When business input or graph version changes, create a new run with a new idempotency key.
+
+## Mid-run input (Steering)
+
+`POST /v1/runs/{run_id}/steer` durably injects a new piece of structured input while a run
+executes — without waiting for it to finish, without forcing a cancel, and without needing
+the run to be paused first. This is distinct from both `cancel` (terminate execution) and
+`interrupt` (graph code voluntarily yielding, waiting for `resume`): steering only safely
+delivers a new message into a run that is running (or about to run, or just paused). Whether,
+when, and how it changes the plan is entirely up to the graph code.
+
+```bash
+curl -X POST https://agents.example.com/v1/runs/$RUN_ID/steer \
+ -H "Authorization: Bearer $TOKEN" \
+ -H "Idempotency-Key: msg-123" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"user_input","payload":{"message":"switch to Spanish"},"metadata":{}}'
+```
+
+202 response:
+
+```json
+{
+ "id": "steer_01H...",
+ "run_id": "run_01H...",
+ "sequence": 3,
+ "kind": "user_input",
+ "status": "pending",
+ "created_at": "2026-08-18T09:00:00Z"
+}
+```
+
+Request body fields:
+
+| Field | Required | Description |
+| --- | --- | --- |
+| `kind` | yes | Application-defined event kind, e.g. `user_input`, `priority_change` |
+| `payload` | yes | Arbitrary JSON; this is exactly what graph code reads back from `drain_steering()` |
+| `metadata` | no | Extra metadata, not part of dedup |
+| `idempotency_key` | no | See below |
+
+**`Idempotency-Key` header vs. body field.** Both express the same logical key and either
+may be used; when both are present the `Idempotency-Key` header wins. The same
+`(tenant_id, run_id, idempotency_key)` triple creates at most one steering event — a repeat
+call returns the **same** event (`id` and `sequence` unchanged, no new row created and no
+`sequence` bump), not a `409`. The response body is *not* guaranteed to be byte-identical to
+the first call's, though: it reflects that event's **current persisted status**, which can
+have moved on between calls (e.g. `pending` on the first call, `consumed` on a later replay
+after the graph has since drained it). This mirrors the `Idempotency-Key` semantics on
+`POST /runs`: same key + same request is a safe replay, not an error — but a replay always
+returns the live state of the one underlying resource, not a frozen snapshot of the first
+response. Crucially, this replay lookup happens **before** any of the run-status admission
+checks below, not after: a same-key retry always returns the existing event, even if the run
+has since moved into `run_finalizing`, gone terminal, or been superseded. Only a genuinely new
+key is subject to those checks.
+
+### Behavior by run status
+
+| Run status | `/steer` behavior |
+| --- | --- |
+| `pending` (queued, not yet claimed) | 202, durably written; visible as soon as a worker claims the run |
+| `running` | 202, durably written; visible at the next safe point |
+| `cancelling` | 202, still accepted (cancel still wins overall — see below — but accepting the write itself is unaffected) |
+| `running`/`cancelling`, but finalizing (`runs.steering_closed`) | `409 run_finalizing` for a genuinely new event; a same-key replay still returns the existing event, not a 409 — the graph has already finished executing and there is no further safe point in this delivery attempt, but the run row has not yet flipped to a terminal/paused status |
+| `paused` (waiting on `interrupt`/`resume`) | 202, durably written; see "paused steering transfer" below |
+| `succeeded` / `failed` / `cancelled` / `timed_out` / `dead_letter` (terminal) | `409 run_terminal`, no event is created for a new key |
+| An old `run_id` already superseded by a `resume` | `409 run_superseded`, see below |
+
+### Cancel always wins over steer
+
+Regardless of arrival order, a cancel request always takes effect: a run that receives a
+`/steer` and is then `/cancel`led still moves to `cancelling` (or, for a `paused` run,
+straight to `cancelled` — see above) and stops executing. The already-durably-written
+steering event is not rolled back — it simply is never consumed by any surviving execution.
+Steer can never undo or delay a cancel that has already been issued.
+
+### Paused steering transfers to the new run inside the atomic resume transaction
+
+`POST /v1/runs/{run_id}/resume` creates a **new** run (a new `run_id`, linked to the old one
+via `metadata.resumed_from_run_id`). Because a worker only pulls pending steering for the
+`run_id` it is actually executing, the resume endpoint does two things in a **single atomic
+transaction**: it creates the new run, and it **migrates** every still-`pending`/`delivered`
+steering event from the old run onto the new one (preserving order, `kind`, `payload`,
+`metadata`, `idempotency_key`; the old events are marked `superseded`). The atomicity matters:
+before the new run can be claimed by any worker, its migrated steering must already be fully
+in place — there is no intermediate window where the new run is claimable but its migration
+has not finished yet. From there the new run's worker delivers and consumes it through the
+normal path: steering submitted while paused ends up actually consumed by the resumed run,
+instead of sitting `pending` forever.
+
+The old run is also marked as superseded by the resume. A subsequent `/steer` call against
+the old `run_id` returns HTTP `409 run_superseded` (instead of silently queuing an event
+nobody will ever consume), telling the caller to steer the new run instead.
+
+### `source_event_id`: stable correlation across repeated resumes
+
+A steering event migrated onto a new run gets a brand-new `id` (it is a fresh durable row
+under the new run), but its `source_event_id` field points back at the `id` the client
+originally received from its `/steer` call. Even if a run is paused and resumed repeatedly
+(A → B → C → …), every hop of the migration preserves the **original** root id — it is never
+overwritten by an intermediate hop's id. `created_at` likewise preserves the original
+durably-accepted time, so the eventual `run.steer.consumed` event's `queue_latency_seconds`
+includes the entire time spent waiting across every pause, not just the time since the last
+resume. A client only needs to remember the `id` its very first `/steer` call returned to
+find it, later, in the `source_event_id` field of `run.steer.accepted`/`run.steer.consumed`
+events on whatever run eventually consumes it — no need to track every intermediate
+`run_id`.
+
+### Payload size limits
+
+Serialized `payload` + `metadata` size is bounded (roughly 32KB by default, or the server's
+event byte cap, whichever is smaller); exceeding it returns `413`/`422` with error code
+`payload_too_large` and creates no durable write. JSON values must be safely serializable
+plain types (string, number, boolean, null, array, object) — no binary data, cyclic
+references, or non-JSON-serializable custom types.
+
+### Accepted ≠ consumed
+
+The 202 response from `/steer` only means the event has been durably written to PostgreSQL's
+steering inbox (`run.steer.accepted`, status `pending`) — **it does not mean the graph has
+processed it**. When an `EventBus` is configured (in-process, or Redis via `RedisEventBus`),
+`/steer` publishes an optional wake notification a running worker's heartbeat loop is waiting
+on, which can shorten the time to discovery; it is never the sole source of truth, and it is
+not itself a named lifecycle event (there is no `run.steer.available` event in the stream) —
+it is purely a low-latency hint. Even if that notification is lost, a worker still eventually
+discovers and consumes the event via PostgreSQL's own periodic safe-point checks. Only once
+graph code at some safe point actually calls `runtime.drain_steering()` and that consumption
+is durably committed by `commit_steering_consumptions_if_owned()` does `run.steer.consumed` appear.
+Callers must not assume the graph has "seen" the input just because they got a 202 — there is
+no endpoint to poll a steering event's status directly; watch the run's event stream
+(`GET /v1/runs/{run_id}/stream`) or event list for `run.steer.consumed` to confirm it was
+actually consumed.
diff --git a/Wiki/content/docs/en/concepts/meta.json b/Wiki/content/docs/en/concepts/meta.json
index 6e4b066..7c7729b 100644
--- a/Wiki/content/docs/en/concepts/meta.json
+++ b/Wiki/content/docs/en/concepts/meta.json
@@ -1,4 +1,4 @@
{
"title": "Core concepts",
- "pages": ["architecture", "durable-execution", "agents-tools", "agent-skills"]
+ "pages": ["architecture", "durable-execution", "steering", "agents-tools", "agent-skills"]
}
diff --git a/Wiki/content/docs/en/concepts/steering.mdx b/Wiki/content/docs/en/concepts/steering.mdx
new file mode 100644
index 0000000..68b2ebd
--- /dev/null
+++ b/Wiki/content/docs/en/concepts/steering.mdx
@@ -0,0 +1,242 @@
+---
+title: "Mid-run steering"
+description: "Durably injecting new input into a running graph, and how it differs from cancel/interrupt/resume"
+---
+
+## What problem does steering solve?
+
+A long-running agent run is often not a closed box: a human or another system may want to
+hand it new information — "actually reply in Spanish," "the customer just called, priority is
+now urgent," "here is the missing API key" — *while the run is still going*, without stopping
+it, without waiting for it to reach a natural pause point on its own, and without losing that
+input if a worker crashes right after accepting it.
+
+Steering is LingxiGraph's answer: `POST /v1/runs/{run_id}/steer` durably accepts a small piece
+of structured JSON and hands it to graph code the next time that code chooses to look for it.
+LingxiGraph guarantees the delivery — durable, ordered, deduplicated, atomic-drain-once within
+a single live `SteeringChannel` — but it never decides *what the new input means*. Whether a
+message means "replan," "just append to context," or "ignore, we're already past that step" is
+entirely up to the graph. See [Ordering, idempotency, and drain-once](#ordering-idempotency-and-drain-once)
+below for exactly what "drain-once" does and does not guarantee across a worker crash.
+
+## How it differs from cancel, interrupt, and resume
+
+These four mechanisms are easy to conflate because they all touch a run mid-flight, but they
+answer different questions:
+
+| Mechanism | Question it answers | Who initiates | What happens to the run |
+| --- | --- | --- | --- |
+| `cancel` | "Stop this run." | Caller | Run moves to `cancelling` → `cancelled`; execution is torn down, cooperatively |
+| `interrupt` | "Graph code needs a value it doesn't have yet, then keep going with it." | Graph code (`interrupt(value)`) | Run moves to `paused`; execution suspends *at that call site* until `resume` supplies a value |
+| `resume` | "Continue a paused run with this value." | Caller | A **new** run is created, continuing from where `interrupt` suspended |
+| `steer` | "Here is a new piece of input, whenever it's convenient." | Caller | The run keeps executing on its own schedule; the graph decides when/whether to look at the new input |
+
+`interrupt`/`resume` is a *request-response* pattern baked into a specific point in the graph:
+the graph explicitly stops and waits, and the caller must supply exactly the value that call
+site is waiting for. Steering is the opposite: it is *push*, asynchronous, not tied to any
+particular call site, and does not stop anything. A run does not need to be paused, waiting,
+or even running yet (a `pending`, not-yet-claimed run also durably accepts steering) to
+receive one.
+
+Steering also composes with the other three. A run can be steered while running, steered while
+paused (see the durable-transfer section below), and steering never blocks or delays a cancel.
+
+## Data flow: PostgreSQL durable inbox → worker → SteeringChannel → runtime
+
+```mermaid
+sequenceDiagram
+ participant C as Caller
+ participant API as API server
+ participant PG as PostgreSQL (run_steering_events)
+ participant R as Redis (optional)
+ participant W as Worker
+ participant Ch as SteeringChannel
+ participant N as Node code
+
+ C->>API: POST /runs/{id}/steer
+ API->>PG: INSERT pending steering row
+ PG-->>API: durably committed
+ API-->>C: 202 run.steer.accepted
+ API-->>R: best-effort NOTIFY (optional)
+ R--)W: wake hint (may be lost)
+ W->>PG: poll/peek pending steering (safe point)
+ PG-->>W: rows still pending
+ W->>Ch: ingest into in-memory channel
+ N->>Ch: drain_steering() at a safe point
+ Ch-->>N: ordered, deduplicated events
+ N-->>W: consumption record (node, namespace, task_id, latency)
+ W->>PG: commit_steering_consumptions_if_owned() (status + run.steer.consumed, one transaction)
+```
+
+PostgreSQL's `run_steering_events` table is the single source of truth for every steering
+event's existence and status. The schema reserves a `delivered` value for forward
+compatibility, but the currently implemented flow never writes it — the real transition is
+`pending` → `consumed` directly (via a single atomic `commit_steering_consumptions_if_owned()` call),
+or `pending`/`delivered` → `superseded` — either because the paused run it belonged to was
+resumed (its still-pending steering migrates onto the new run) or because it was still
+unconsumed once the owning run reached its terminal finalization boundary, for one of several
+possible reasons (see "Finalization and the steering admission boundary" below). Treat `delivered` as a status your code may need to read/allow
+today, not one anything currently produces. The Redis notification is purely an optimization
+to reduce polling latency; losing
+it never loses a steering event — the worker's periodic safe-point checks fall back to reading
+PostgreSQL directly. `SteeringChannel` is the in-process, per-run structure the worker uses to
+hand already-fetched events to graph code with atomic drain-once semantics; it is not itself
+durable and is rebuilt from PostgreSQL whenever a worker restarts or takes over a run.
+
+## Safe points are application-chosen, not an arbitrary preemption
+
+LingxiGraph never interrupts a node function mid-execution to deliver steering. It guarantees
+that whenever graph code calls `runtime.drain_steering()`, the events returned are the current,
+freshest, not-yet-drained set — but *where inside a node's code* that call happens is entirely
+the application's choice. A node can call it once at the very start, in a loop, or not at all.
+
+What LingxiGraph *does* control is when a node is scheduled as a new task by the executor —
+before a node starts, before the next superstep, before a retry, and after a resume — and each
+of those transitions is an opportunity for steering ingested since the last one to become
+visible. This is a deliberate boundary, not a gap: true preemption of arbitrary user code would
+break the deterministic replay guarantees the rest of the runtime depends on.
+
+## Ordering, idempotency, and drain-once
+
+- Steering events for a given run are strictly ordered by an internal `sequence`, assigned at
+ durable-accept time.
+- Within a single live `SteeringChannel` — one worker, one in-process instance, spanning any
+ number of internal retries of a node — `drain_steering()` never returns the same event twice:
+ atomic drain-once.
+- **This is not a system-wide exactly-once guarantee.** If a worker crashes (or its lease is
+ reassigned) after `drain_steering()` has handed an event to a node but *before*
+ `commit_steering_consumptions_if_owned()` durably records that consumption, the PostgreSQL row is
+ still `pending`/`delivered` — a new worker rebuilding the channel from PostgreSQL for that run
+ will legitimately redeliver the same event. Across a crash, steering delivery is
+ **at-least-once**, not exactly-once. Applications performing an irreversible side effect from
+ a drained event (a charge, an external API call with side effects) should key their own
+ idempotency off `event.id` (or the root `source_event_id` after a pause/resume transfer,
+ see below) rather than assume drain-once implies process-once.
+- Deduplication by `idempotency_key`: submitting the same `(tenant_id, run_id,
+ idempotency_key)` twice returns the original event instead of creating a second one, and
+ that dedup survives a `drain_steering()` call — a retried submission after the original was
+ already drained still returns the original (now-consumed) event, not a new pending one. This
+ dedupes the *accept* side (repeated `/steer` calls); it does not by itself make the *drain*
+ side exactly-once across a crash — see above.
+- The consumed-commit (`commit_steering_consumptions_if_owned()`) and the graph's own checkpoint commit
+ for the superstep that drained the event are two separate durable operations, not one
+ distributed transaction (see "Consumption vs. checkpoint commit" below) — so there is no
+ system-wide exactly-once processing guarantee spanning steering delivery *and* graph state.
+
+## Retry, worker crash, and Redis loss
+
+- **Worker retry**: `commit_steering_consumptions_if_owned()` is idempotent — a worker that commits a
+ consumption batch, loses the connection before seeing the ack, and retries the same batch
+ will not create a duplicate `run.steer.consumed` event; only steering rows that actually
+ transition status in a given call produce one.
+- **Worker crash**: because delivery state lives in PostgreSQL, not in the crashed worker's
+ memory, a new worker claiming the run reconstructs the picture (`pending`/`delivered` rows)
+ from PostgreSQL and continues from there. No steering event is silently dropped by a worker
+ crash.
+- **Redis loss**: Redis is an accelerator, never a dependency for correctness. If the pub/sub
+ notification never arrives — connection reset, Redis restarted, message lost — the worker's
+ own periodic PostgreSQL checks at each safe point still discover and deliver the event, just
+ with slightly higher latency instead of a lost update.
+
+## Finalization and the steering admission boundary
+
+Once graph execution ends, the owning worker atomically seals new steering admission
+(`close_steering()`) before it starts the final durable flush of whatever the graph already
+drained. From that instant on, a fresh `/steer` call is refused with a stable `409
+run_finalizing` — there is no safe point left for the graph to ever consume a newly accepted
+event.
+
+This still leaves a gap: a `/steer` call can win the run row's lock, and durably commit its
+row, a moment *before* `close_steering()` wins that same lock — even though the graph has
+already produced its terminal output and the heartbeat loop that would have ingested the event
+has already stopped. Once the final flush completes, the Runtime checks for exactly this: any
+steering row still `pending`/`delivered` at that point is durably transitioned to `superseded`
+(reason `unconsumed_at_final_boundary`), paired with a `run.steer.superseded` lifecycle event,
+in the *same* lease/attempt-fenced transaction as the rest of finalization — the steering
+disposition and the run's terminal status commit together or not at all (see
+`finalize_run_with_steering_disposition_if_owned`; a crash or lease loss between two separately
+committed writes used to be able to leave steering permanently `superseded` while the run
+itself never actually finished). The run then finalizes normally with its already-computed
+outcome (`succeeded`, `failed`, `dead_letter`, etc. — any terminal outcome; `paused` is excluded
+from this finalization path since it was never terminal to begin with, see below) — it is **not**
+forced through a retry loop just because a steering event happened to exist.
+
+The `unconsumed_at_final_boundary` reason is deliberately generic and does **not** claim a
+race occurred. It covers two different situations the Runtime cannot currently tell apart from
+the data it has: the narrow lock race described above, and the far more common case of a graph
+that legitimately never calls `drain_steering()` at all (a documented, allowed choice — see
+"Safe points are application-chosen" above), where the row had been sitting there, fully
+visible, for the run's entire lifetime. Either way the outcome is the same: the run finalizes
+on its first attempt exactly as if the steering event had never arrived, and that event's
+disposition is recorded as `superseded`, not silently dropped.
+
+**Final disposition policy, stated precisely:** a terminal run's *own* finalization decision
+can never silently strand an unconsumed steering event without a durable, observable
+disposition — that is what the mechanism above guarantees. Any genuinely terminal `finish`
+intent — `succeeded`, `failed`, `cancelled`, or `dead_letter` produced by the graph's own
+outcome computation — closes steering admission and supersedes leftovers exactly as described
+above; `paused` is the only status excluded (see the next paragraph). This means an ordinary
+`failed`/`dead_letter` outcome does **not**, by itself, leave steering `pending`.
+
+The one case that still can is narrower than a blanket "dead-lettered runs leave steering
+pending": it is specifically a run whose *final steering flush itself* could never durably
+commit (`_retry_or_dead_letter()` — see `Worker._execute`'s post-flush handling). That path
+never reaches `finalize_run_with_steering_disposition_if_owned()` at all — it routes the attempt
+through the ordinary retry/dead-letter machinery instead, precisely because writing any
+disposition (superseded or otherwise) would either report a false status or silently strand the
+already-drained-but-uncommitted consumption. Only in that specific situation can a `dead_letter`
+run still have steering sitting `pending` — and it is deliberate, retained durable history, not
+a bug: a later `/redrive` of that run (allowed only from `failed` or `dead_letter`) picks the
+still-pending row back up like any other queued steering. An ordinary client-initiated **cancel**
+is different, and not currently covered by that same recovery path: `/redrive` does not accept a
+`cancelled` run, so steering left `pending` on a cancelled run has no automatic future delivery
+attempt today — recovering it, if ever needed, is an operator/API task outside this mechanism,
+not something the Runtime resolves on its own.
+
+## Parallel supersteps, `Send`, and subgraphs share one channel
+
+A single run's `SteeringChannel` is shared across every task in the same superstep, including
+parallel branches and `Send`-fanned-out tasks, and across subgraph node boundaries within the
+same run. Draining is mutually exclusive per event (only one task can ever drain a given event
+— no double-processing across parallel tasks) but is not restricted to a single node: any node
+anywhere in the run's task graph, at any nesting level, can call `drain_steering()` and see the
+same durable, ordered backlog. `queue_latency_seconds` and `task_id`/`namespace` on the
+eventual `run.steer.consumed` event record exactly which task, at which subgraph depth, ended
+up consuming a given event.
+
+## Consumption vs. checkpoint commit — current scope cut
+
+As of this release, marking a steering event `consumed` (via
+`commit_steering_consumptions_if_owned()`) is a separate durable operation from the graph's own
+checkpoint commit for the superstep that drained it. They are not wrapped in one distributed
+transaction. In practice this means: if a node drains steering and then the process dies before
+its checkpoint is written, the checkpoint-level retry will re-execute the node — but the
+steering event was already marked `consumed`, so a re-executed node will not see it again via
+`drain_steering()`. Nodes that need the drained payload to survive a mid-superstep crash should
+persist what they need from it into state as part of the same node's return value, the same way
+they would for any other externally observed input, rather than re-reading it from the
+steering channel after a retry. This is a deliberate, currently-shipped scope cut — not a
+promise that consumption is checkpoint-transactional — and may be revisited in a future
+release.
+
+## What steering explicitly does not do
+
+Steering is a delivery mechanism, not a planning mechanism. It does not:
+
+- decide whether a new message should trigger replanning, cancel a tool call in flight, or
+ change the current node's control flow — that is entirely business graph logic;
+- guarantee low latency between accept and consume — only that consumption, once it happens,
+ is durable and observable via `run.steer.consumed`;
+- provide a way to modify already-committed state directly; the graph must explicitly fold a
+ drained payload into its own state update.
+
+## Embedded / library-mode degradation
+
+When LingxiGraph is used as an embedded library without the Agent Server (no worker process,
+no PostgreSQL-backed repository), `compiled_graph.steer(run_id, ...)` and
+`runtime.drain_steering()` still work against an in-process `SteeringChannel`, but durability
+degrades to "as durable as the process hosting it": there is no cross-process delivery, no
+worker-crash recovery, and no PostgreSQL-backed audit trail of `run.steer.accepted`/
+`run.steer.consumed` events. A plain `invoke()` that never touches steering incurs no overhead
+and leaks no channel state — see the embedded lifecycle tests referenced from the Agent Server
+implementation for the exact guarantees at that layer.
diff --git a/Wiki/content/docs/en/operations/meta.json b/Wiki/content/docs/en/operations/meta.json
index 015c634..edf6b28 100644
--- a/Wiki/content/docs/en/operations/meta.json
+++ b/Wiki/content/docs/en/operations/meta.json
@@ -1,4 +1,4 @@
{
"title": "Operations and project",
- "pages": ["configuration", "security-observability"]
+ "pages": ["configuration", "security-observability", "steering-operations"]
}
diff --git a/Wiki/content/docs/en/operations/steering-operations.mdx b/Wiki/content/docs/en/operations/steering-operations.mdx
new file mode 100644
index 0000000..56e20c6
--- /dev/null
+++ b/Wiki/content/docs/en/operations/steering-operations.mdx
@@ -0,0 +1,252 @@
+---
+title: "Operating durable steering"
+description: "Migration rollout order, PostgreSQL-as-source-of-truth, latency, retries, and rollback boundaries"
+---
+
+## Migration rollout order
+
+Mid-run steering (issue #16) ships as four forward-only Alembic revisions on top of the base
+schema:
+
+| Revision | Adds |
+| --- | --- |
+| `0002_steering` | `run_steering_events` table, its indexes, and the worker/server code paths that read/write it |
+| `0003_steering_source_event` | `source_event_id` column + partial index on `run_steering_events`, used to correlate a steering event across repeated pause/resume migrations |
+| `0004_steering_closed` | `runs.steering_closed` — the atomic finalization admission gate (review round 6/7); see "The `steering_closed` finalization gate" below |
+| `0005_superseded_by_run_id` | `runs.superseded_by_run_id` — the typed control-plane lineage marker for resumed runs (review round 14, BLOCKER); moves this out of user-writable `metadata` so it cannot be forged. See "The `superseded_by_run_id` lineage marker" below |
+
+Apply them with the standard Alembic path:
+
+```bash
+alembic upgrade head
+```
+
+All four are additive (`ADD COLUMN IF NOT EXISTS`, `CREATE INDEX IF NOT EXISTS`, nullable/
+`NOT NULL DEFAULT FALSE` columns) and safe to run against a live database before deploying the
+new server/worker binaries — an old binary that predates `0004`/`0005` simply never reads or
+writes `steering_closed`/`superseded_by_run_id`; it keeps behaving exactly as it did before the
+columns existed. Deploy order for a zero-downtime rollout:
+
+1. Run `alembic upgrade head` (adds `run_steering_events`, `source_event_id`,
+ `steering_closed`, and `superseded_by_run_id`) while the old server/worker binaries are
+ still serving traffic.
+2. Roll out the new server and worker binaries.
+
+Do not reverse this order: a binary that writes `source_event_id`/`steering_closed`/
+`superseded_by_run_id` before the column exists will fail its inserts/updates. Because migrations in this project are
+forward-only (`downgrade()` raises), there is no automated rollback path for the schema — see
+"Rollback boundaries" below.
+
+
+`PostgresRepository.setup()` also applies the same SQL files directly (used by tests and by
+embedded/dev setups). That code path is **not** a substitute for driving the Alembic revision
+chain in a real deployment: only `alembic upgrade head` is what a production rollout should
+run, and it must reach `0005_superseded_by_run_id`, not stop at `0002_steering`,
+`0003_steering_source_event`, or `0004_steering_closed`.
+
+
+## The `steering_closed` finalization gate
+
+Once a worker's graph execution reaches its terminal/paused outcome, the run may still read
+`running`/`cancelling` for a little while longer while the worker durably flushes whatever
+steering the graph already drained and commits the final status — there is no more graph safe
+point left to consume anything new. `runs.steering_closed` is a boolean the owning worker sets
+(fenced on its own `lease_owner`/`attempt`, just like the status write itself) the instant
+execution ends, before that final flush starts.
+
+- While `steering_closed` is `false`, `POST /steer` is admitted normally, exactly as it always
+ was before `0004`.
+- Once `steering_closed` is `true` for the run's current attempt, a **new** `POST /steer`
+ (no matching Idempotency-Key on file yet) is rejected with `409 run_finalizing` — a stable,
+ documented error distinct from the terminal-run `409`, so a client can tell "this run is
+ momentarily busy finishing up" apart from "this run is permanently done".
+- A **replayed** `POST /steer` — same `(tenant_id, run_id, Idempotency-Key)` as an event that
+ was already durably accepted — always returns that existing event's current state, never a
+ fresh `409`, regardless of `steering_closed`/terminal/superseded status. See "Idempotent
+ replay is safe across every admission gate" below.
+- A `paused` outcome never closes the gate: pausing is not "no safe point left" the way a
+ terminal outcome is — `/steer` against a paused run is an intentional, documented part of the
+ design (the pending steering is transferred to the resumed run at `/resume` time).
+- A gate closed by an abandoned delivery attempt does not leak into the next one: `claim_run`
+ resets `steering_closed` to `false` on every fresh claim, so a retried run accepts steering
+ normally again.
+
+A closed gate on its own is not sufficient to prevent a permanently-pending steering row: a
+`/steer` call can win the run row's lock a moment *before* `close_steering()` does, even though
+the graph has already finished. The worker's post-flush check
+(`finalize_run_with_steering_disposition_if_owned()`, `Worker._execute`) is what actually
+catches that case — if anything is still `pending`/`delivered` after the final flush has
+committed everything it could, that row is durably transitioned to `superseded` (reason
+`unconsumed_at_final_boundary` — a deliberately generic label that does **not** claim a race
+occurred; it also covers the far more common case of a graph that simply never called
+`drain_steering()` at all), paired with a `run.steer.superseded` lifecycle event, in the *same*
+lease/attempt-fenced transaction as the run's terminal status write — the two commit together
+or not at all, never as two separately-committed writes that could diverge on a crash or lease
+loss between them. The run then finalizes normally with its already-computed outcome — it is
+**not** routed through a retry/dead-letter loop just because a steering event happened to
+exist, since doing so would force a rerun onto a graph that may be correctly, deliberately
+never calling `drain_steering()` at all. **A terminal run's own finalization decision can never
+silently strand an unconsumed steering event without a durable, observable disposition** — an
+ordinary `failed`/`dead_letter` outcome computed by the graph closes admission and supersedes
+leftovers exactly like `succeeded`/`cancelled` do. The one narrower exception is a run whose
+*final steering flush itself* could never durably commit (`_retry_or_dead_letter()`), which never
+reaches `finalize_run_with_steering_disposition_if_owned()` at all and instead routes the attempt
+through the ordinary retry/dead-letter machinery — only there can a `dead_letter` run legitimately
+still carry a `pending` steering row, as deliberate durable history, picked back up by a later
+`/redrive` like any other queued steering. `/redrive` only accepts `failed`/`dead_letter` runs,
+though — an ordinary client-initiated **cancel** is not covered by that recovery path, so steering
+left `pending` on a `cancelled` run has no automatic future delivery attempt today. See the
+steering concept page's "Finalization and the steering
+admission boundary" section for the full policy.
+
+## The `superseded_by_run_id` lineage marker
+
+When `/resume` creates a descendant run from a `paused` run, the old run needs a durable way to
+say "I have been superseded — steer/cancel/resume the new run instead". Before `0005`, that
+marker lived in `metadata.superseded_by_run_id` — ordinary, user-writable `Run` metadata.
+Because `/steer`, `/cancel`, and the concurrent-resume conflict gate all treated that key as
+authoritative, a caller could forge it at run-creation time (`POST /v1/runs` with
+`metadata: {"superseded_by_run_id": "..."}`) to falsely trigger `409 run_superseded` /
+`409 run_resume_conflict` on a run that was never actually resumed (issue #16 PR #17 review
+round 14, BLOCKER).
+
+`runs.superseded_by_run_id` (added by `0005_superseded_by_run_id`) is a typed, nullable column
+that replaces that key as the sole source of truth:
+
+- It is written exactly once, atomically, inside the same fenced resume transaction that
+ creates the descendant run and migrates the old run's pending steering
+ (`resume_run_with_pending_steering()` / its Postgres counterpart) — never derived from,
+ or writable via, `RunCreate.metadata`.
+- `/steer`'s admission check, `/cancel`'s superseded check, and the resume conflict gate all
+ read `run.superseded_by_run_id` directly; none of them consult `metadata` for this decision
+ any more.
+- A caller submitting `metadata.superseded_by_run_id` on an ordinary `RunCreate` now has no
+ effect on control flow at all — the key is inert, stored (if at all) as plain user metadata,
+ and never copied into the typed column.
+- Upgrade compatibility: a pre-existing run from before this PR that happens to carry a user
+ metadata key with this exact name is never reinterpreted as system lineage after upgrading —
+ the typed column starts `NULL` for every existing row and is populated only by a future
+ resume.
+
+## Idempotent replay is safe across every admission gate
+
+`submit_steering()` (both the in-memory and PostgreSQL repositories) checks for an existing
+event matching `(tenant_id, run_id, Idempotency-Key)` **before** any terminal/`steering_closed`/
+superseded admission gate, not after. If a client's `POST /steer` durably committed but the 202
+response never reached it (a dropped connection, a proxy timeout), a standard idempotent retry
+with the same key returns that event's current persisted state — id and sequence unchanged —
+even if the run has since gone finalizing, terminal, or superseded. Only a genuinely *new* key
+(one this run has never seen) is subject to those gates; that is a real admission decision, and
+still gets the appropriate `409`.
+
+## PostgreSQL is the source of truth; Redis is only an accelerator
+
+Every steering event's existence and status lives in PostgreSQL's `run_steering_events`
+table. Redis pub/sub is used only to reduce the latency between "an event became durable" and
+"a worker notices it" — it is never consulted to determine whether an event exists, and its
+loss (a Redis restart, a dropped connection, a missed NOTIFY) never loses an event. Operators
+can treat Redis as fully disposable for steering correctness: if Redis is down or evicted,
+steering keeps working, just with worker discovery falling back to periodic PostgreSQL
+polling at each safe point instead of a near-instant wake-up.
+
+## Worker heartbeat and steering delivery latency
+
+A worker only picks up newly-durable steering for a run it currently holds the lease on, at
+that run's next safe point (node start, next superstep, retry, post-resume). Two factors
+therefore dominate delivery latency:
+
+- **How often the run reaches a safe point.** A node doing a single long-running tool call
+ will not see steering submitted mid-call until that call returns and the node either
+ finishes or calls `drain_steering()` itself.
+- **Worker poll/heartbeat interval**, when Redis notification is unavailable. A larger
+ interval trades steering-latency for lower PostgreSQL polling load; tune it per the
+ worker's `--health-port`/poll configuration in `docs/api.md` and the deployment's
+ `lingxigraph worker` flags.
+
+## Diagnosing long-pending steering
+
+If a steering event stays `pending` far longer than expected:
+
+1. Confirm the run is actually claimed and progressing: `GET /v1/runs/{run_id}` — a `pending`
+ run has no worker executing it yet; steering on it waits for a claim, not a bug.
+2. Check whether the run's graph code ever calls `runtime.drain_steering()` at all, and how
+ often — steering delivery depends entirely on the graph choosing to look, per the
+ application-chosen safe-point model (see the steering concept page).
+3. Query `run_steering_events` directly for the row's `status`, `created_at`, and
+ `source_event_id` to see whether it was migrated from an earlier, paused run — the
+ `queue_latency_seconds` on the eventual `run.steer.consumed` event will include all
+ accumulated pause time, which is expected, not a bug.
+4. Check worker logs/metrics for repeated `commit_steering_consumptions_if_owned()` failures —
+ those retry safely (the operation is idempotent, and fenced on the worker's own
+ `lease_owner`/`attempt` — see "The `steering_closed` finalization gate") but repeated
+ failures indicate a database connectivity problem worth alerting on.
+5. If a client got `409 run_finalizing` on `/steer`, the run's graph had already finished
+ executing at the moment the request landed — check `runs.steering_closed` for the run's
+ current attempt (`true` means the gate is shut) and retry against a fresh run instead of the
+ same one. A replay with the same Idempotency-Key is always safe here, even after the gate
+ closes — see "Idempotent replay is safe across every admission gate".
+6. If a steering event's status is `superseded` with `reason: unconsumed_at_final_boundary` in its
+ `run.steer.superseded` event data (rather than `reason: resume_transfer`), that is the
+ finalization-boundary fix working as intended: the event was durably accepted with no safe
+ point left in that delivery attempt to consume it, so it was given a durable, observable
+ `superseded` disposition instead of the run finalizing with a permanently-pending row or
+ being forced through a needless rerun. This is not itself a bug; a repeating pattern of
+ `unconsumed_at_final_boundary` for the same run suggests steering is consistently arriving after the
+ graph already reaches its terminal node, which is worth investigating at the
+ application/graph level.
+
+## The normal `run.steer.accepted → consumed` lifecycle
+
+1. `POST /steer` durably inserts a `pending` row and emits `run.steer.accepted` (202).
+2. A worker holding the run's lease discovers the row (via Redis hint or PostgreSQL poll) and
+ ingests it into the run's in-memory `SteeringChannel`.
+3. Graph code calls `drain_steering()` at some safe point and receives it.
+4. The worker durably commits the consumption (`commit_steering_consumptions_if_owned()`,
+ fenced on the worker's own `lease_owner`/`attempt`), which atomically transitions the row to
+ `consumed` and emits `run.steer.consumed` in the same transaction. A worker that no longer
+ owns the lease/attempt (a takeover mid-heartbeat or mid-final-flush) gets nothing written at
+ all — the new owner is solely responsible for the row from that point on.
+
+No step in this chain is optional or reorderable; a client should treat only step 4's
+`run.steer.consumed` event as proof of actual processing.
+
+## Why worker/DB transient failures do not lose a consumption, and how to observe retries
+
+The status update and the `run.steer.consumed` event insert happen inside one PostgreSQL
+transaction. If the connection drops after the transaction commits but before the worker
+observes the acknowledgment, the worker's retry logic re-sends the same consumption batch.
+Because `commit_steering_consumptions_if_owned()` is idempotent — it only emits a lifecycle
+event for rows that transition status in that specific call, and only writes anything at all
+when the caller still owns the run's current lease/attempt — a retried batch against
+already-consumed rows is a safe no-op: the status stays `consumed` and no second
+`run.steer.consumed` event is created. Operators can observe this pattern in worker logs as a
+repeated `commit_steering_consumptions_if_owned` call for the same steering ids with no
+corresponding growth in `run.steer.consumed` events in `run_events` — that is expected
+behavior, not a stuck retry loop, as long as it eventually stops retrying.
+
+## Upgrade and rollback boundaries
+
+Migrations in this project are forward-only: every revision's `downgrade()` raises rather than
+attempting to undo the change. This is intentional — a real rollback of `run_steering_events`
+or its columns would either lose durable steering history or require reconciling in-flight
+runs against a schema the new binaries no longer expect. Practical implications:
+
+- Never plan a rollback strategy that downgrades the schema. If a deployment needs to be
+ rolled back, roll back the **binaries** while leaving the schema at its current (forward)
+ revision — the additive nature of `0002_steering`/`0003_steering_source_event`/
+ `0004_steering_closed`/`0005_superseded_by_run_id` means older binaries that predate a given
+ revision simply ignore the extra table/columns and do not fail to start.
+- **Schema compatibility is not the same as semantic compatibility.** An older binary running
+ against the new schema will not crash, but it also does not understand `steering_closed` or
+ `superseded_by_run_id` — it will not enforce the finalization admission gate or the
+ resume-lineage cancel/steer rejection this PR adds. Treat the new steering/lineage control
+ semantics as available only once every control-plane instance is running the new binary, not
+ the moment the schema migration lands. Rolling a binary back after it has already exercised
+ these semantics (e.g. after a resume has set `superseded_by_run_id`) does not un-happen that
+ transition; the old binary simply stops enforcing it going forward. If you must roll back
+ binaries while steering is in active use, quiesce or let in-flight runs finish first rather
+ than assuming the old binary preserves the same guarantees.
+- Because there is no schema downgrade path, validate `0002_steering`/
+ `0003_steering_source_event`/`0004_steering_closed`/`0005_superseded_by_run_id` against a
+ staging database (or the CI `test_postgres_integration.py` Postgres job) before applying to
+ production, rather than relying on the ability to undo it.
diff --git a/Wiki/content/docs/zh/api/errors-events.mdx b/Wiki/content/docs/zh/api/errors-events.mdx
index d28eefd..6498f4d 100644
--- a/Wiki/content/docs/zh/api/errors-events.mdx
+++ b/Wiki/content/docs/zh/api/errors-events.mdx
@@ -29,6 +29,16 @@ description: "处理 problem details、run 失败、事件续传与客户端重
常见稳定码包括 `idempotency_conflict`、`quota_exceeded`、`join_timeout` 以及由 schema、认证、并发和资源状态映射的 problem code。错误码新增时客户端应安全地归入未知错误，而不是失败解析。
+### Steering 相关错误码
+
+| `code` | HTTP | 触发条件 |
+| --- | --- | --- |
+| `run_terminal` | 409 | 对处于终止态（`succeeded`/`failed`/`cancelled`/`timed_out`/`dead_letter`）的 run 调用 `/steer`；不产生任何 durable 写入 |
+| `run_superseded` | 409 | 对已经被 `resume` 取代的旧 `run_id` 调用 `/steer` 或 `/cancel`；应改用新 run 的 `id`（见 `metadata.resumed_from_run_id`）。对 `/cancel` 而言，即使旧 run 的 `status` 仍是 `paused` 也会触发此错误——检查的是类型化的 `superseded_by_run_id` 列，而非 `status`，也不是可被用户写入的 `metadata`，因此历史 run 绝不会被误改为 `cancelled`，而其被 resume 出来的新 run 却仍在继续执行；该检查也无法通过客户端提交的 `metadata` key 伪造（详见运维文档"`superseded_by_run_id` 血缘标记"一节） |
+| `payload_too_large` | 413/422 | `payload` + `metadata` 序列化后超过大小上限（32KB，或服务端事件大小上限，取两者较小值）；不产生任何 durable 写入 |
+| `run_resume_conflict` | 409 | 并发 `resume` 调用同一个已暂停的 run；只有一个调用会成功创建新 run，其余调用都会收到该错误 |
+| `run_finalizing` | 409 | run 的图已经执行结束、但最终状态尚未提交（worker 正在持久化刷新并收尾的短暂窗口）时调用 `/steer`；此时已经没有任何安全点可以消费该事件，因此直接拒绝，而不是接受一条永远不会被消费的记录。`paused` 结局不受影响——对已暂停的 run 发起 `/steer` 仍然正常 |
+
## Run 业务错误
创建 run 成功返回 `202 pending`。之后的节点/schema/provider 错误会使资源变为 `failed`、`timed_out` 或 `dead_letter`，并写入：
@@ -67,6 +77,59 @@ data: {"id":"...","run_id":"...","sequence":18,"kind":"node_completed","data":{}
- 以 `: heartbeat` 开头的行是保活注释，应忽略；
- run 到达 terminal 或 `paused` 后，服务端关闭流。
+### Steering 生命周期事件
+
+除了图执行过程中产生的普通事件（`node_completed` 等），运行中输入（steering）在其生命周期
+中会产生三类专门的 SSE 事件：
+
+| `kind` | 何时产生 | 关键字段 |
+| --- | --- | --- |
+| `run.steer.accepted` | `/steer` 调用被 durably 接受时（202） | `steering_event_id`、`source_event_id`（若来自 resume 迁移）、`sequence`、`kind` |
+| `run.steer.consumed` | 图代码调用 `drain_steering()` 消费该事件，且这次消费被 `commit_steering_consumptions_if_owned()` durably 提交后 | `steering_event_id`、`source_event_id`、`sequence`、`kind`、`queue_latency_seconds`、`node`、`namespace`、`task_id` |
+| `run.steer.superseded` | 某条 steering 事件在从未被消费的情况下，其行 durably 转为 `superseded` 状态——原因可能是它所属的 paused run 被 resume（其仍 pending 的 steering 被迁移到新 run），也可能是它在该 run 到达终结 finalization 边界时仍处于 `pending`/`delivered`（见下文） | `steering_event_id`、`source_event_id`、`sequence`、`kind`、`reason`（`resume_transfer` 或 `unconsumed_at_final_boundary`）、`superseded_by_run_id`（`resume_transfer` 时有值，`unconsumed_at_final_boundary` 时为 `null`）、`replacement_steering_event_id`（`resume_transfer` 时有值，`unconsumed_at_final_boundary` 时为 `null`） |
+
+`run.steer.superseded` 的 `unconsumed_at_final_boundary` 原因是一个刻意保持诚实、笼统的
+标签——它**并不**声称该事件在与什么竞争。它涵盖了 worker 到达真正的终结 finalization 边界
+时仍处于 `pending`/`delivered` 的每一条 steering 行，而这背后的成因不止一种：某次 `/steer`
+调用确实可能恰好在 worker 关闭新准入之前完成 durable 提交（这是真实的竞争）；但同样常见的
+情况是，图从始至终就没有调用过 `drain_steering()` 去消费一条其实一直可见的行——这是普通的
+应用层未消费，根本不是竞争。Runtime 目前无法从现有数据中区分这两种情况，因此它也不假装能
+区分：两者都得到相同的 `unconsumed_at_final_boundary` 原因。与其仅因为存在一条 steering
+事件就强制该 run 进入无限重试循环——这会惩罚那些本就合法地从不调用 `drain_steering()` 的
+图——Runtime 会把这条行 durably 转为 `superseded`，并以之前已经算出的结果正常终结该 run。
+这种情况下 `run.steer.superseded` 事件绝不会被静默丢弃：它总是与 run 的终结状态在同一个
+durable 事务中一起提交，并且在重试下是幂等的（对已经 `superseded` 的行重复尝试转换不会
+产生重复事件）。
+
+字段说明：
+
+- `steering_event_id`：被消费的 steering 事件在**当前** run 下的 id（如果该事件是从旧 run
+ 迁移过来的，这是迁移后的新 id，不是客户端最初收到的 id）；
+- `source_event_id`：客户端最初 `/steer` 调用收到的**根** id。同一根 steering 即使经过多次
+ pause/resume 迁移，该字段在每一跳都保持一致，客户端可用它把最终
+ `run.steer.consumed`/`run.steer.superseded` 与自己最初发出的请求关联起来。这一行为按具体
+ 生命周期事件而定，并非统一规则：在 `run.steer.accepted` 与 `run.steer.consumed` 上，仅当
+ 事件来自迁移时才出现该字段——直接提交（从未被迁移）的事件在这两类事件上没有此字段；
+ `run.steer.superseded` 则不同：它总是把 `source_event_id` 设为
+ `event.source_event_id or event.id`，因此即便是一条直接提交、从未迁移过的事件，其
+ superseded 记录里也会带有一个根 id——就是它自己的 id；
+- `sequence`：该 steering 事件在其所属 run 的 steering 序列里的序号（与 run 事件流的
+ `sequence` 是两个独立计数器）；
+- `kind`：透传自 `/steer` 请求体的 `kind`；
+- `queue_latency_seconds`：从该 steering **最初** durably accepted（包括跨 resume 累积的
+ 等待时间）到被消费的秒数；
+- `node`：消费该事件的节点名；
+- `namespace`：该节点所在的 subgraph 命名空间路径（数组）；
+- `task_id`：区分同一 superstep 内并行/`Send` 任务的具体 task。
+
+**durable source 是 PostgreSQL，Redis 只是可选加速器。** `run.steer.accepted`、
+`run.steer.consumed` 和 `run.steer.superseded` 都是先 durably 写入 PostgreSQL 再对外可见的；Redis pub/sub 只用于
+提示 worker "可能有新 steering 了"，从而降低轮询延迟。即使 Redis 通知丢失（连接断开、
+Redis 重启等），worker 仍会在下一个安全点检查/轮询 PostgreSQL 并最终消费——不会因为
+Redis 通知丢失而永久遗漏一条 steering。同理，`commit_steering_consumptions_if_owned()` 是幂等的：
+worker 因瞬时故障重试同一批 consumption 时，只有真正发生状态转换的事件才会产生新的
+`run.steer.consumed`，不会为已经消费过的事件重复生成。
+
## 重试策略
1. 只对 `retryable=true`、网络故障、429 和临时 5xx 使用有界指数退避；
diff --git a/Wiki/content/docs/zh/api/overview.mdx b/Wiki/content/docs/zh/api/overview.mdx
index f7f746a..14cd065 100644
--- a/Wiki/content/docs/zh/api/overview.mdx
+++ b/Wiki/content/docs/zh/api/overview.mdx
@@ -59,6 +59,8 @@ stateDiagram-v2
running --> timed_out
running --> cancelling
cancelling --> cancelled
+ pending --> cancelled: cancel（无存活 worker）
+ paused --> cancelled: cancel（无存活 worker）
failed --> pending: redrive
dead_letter --> pending: redrive
paused --> pending: resume creates new run
diff --git a/Wiki/content/docs/zh/api/python-sdk.mdx b/Wiki/content/docs/zh/api/python-sdk.mdx
index c819e3b..65a4e8a 100644
--- a/Wiki/content/docs/zh/api/python-sdk.mdx
+++ b/Wiki/content/docs/zh/api/python-sdk.mdx
@@ -36,6 +36,15 @@ with LingxiGraphClient(
completed = client.runs.join(run["id"], timeout=120)
print(completed["output"])
+
+ # Durably 注入一条运行中输入；成功只代表已 durably 写入，不代表已被图消费。
+ accepted = client.runs.steer(
+ run["id"],
+ kind="user_input",
+ payload={"message": "改成用中文回复"},
+ idempotency_key="msg-123",
+ )
+ print(accepted["id"], accepted["status"]) # "pending"
```
可用资源方法：
@@ -45,12 +54,33 @@ with LingxiGraphClient(
| `graphs` | `list()`、`get(graph_id)` |
| `assistants` | `create(**values)`、`list()`、`get()`、`update()`、`delete()` |
| `threads` | `create()`、`list()`、`get()`、`update()`、`state()`、`history()`、`fork()`、`delete()` |
-| `runs` | `create()`、`get()`、`list(thread_id)`、`join()`、`stream()`、`cancel()`、`resume()` |
+| `runs` | `create()`、`get()`、`list(thread_id)`、`join()`、`stream()`、`cancel()`、`steer()`、`resume()` |
| `store` | `batch(operations)`、`search(namespace, ...)` |
| `schedules` | `create()`、`list()`、`update()`、`delete()` |
Redrive 或尚未封装的端点可使用 `client.request(method, path, **kwargs)`。
+### `runs.steer()`
+
+```python
+client.runs.steer(
+ run_id,
+ kind="user_input",
+ payload={"message": "..."},
+ metadata=None,
+ idempotency_key="msg-123",
+)
+```
+
+- `kind` 默认 `"user_input"`，可自定义为任意业务字符串；
+- `payload` 是图代码通过 `runtime.drain_steering()` 读回的原始 JSON；
+- `idempotency_key` 传入时会同时作为 `Idempotency-Key` header 发送；同一 key 重复调用
+ 返回原事件，不会重复入队；
+- 返回值是 202 响应体（含 `id`、`sequence`、`status`），**不代表**图已经处理这条输入——
+ 只有服务端后续出现的 `run.steer.consumed` 事件才代表真正被消费；
+- 对处于终止态的 run 调用会抛出 `LingxiGraphAPIError`，`code == "run_terminal"`；
+ 对已被 `resume` 取代的旧 `run_id` 调用会抛出 `code == "run_superseded"`。
+
## 异步客户端
```python
@@ -65,6 +95,14 @@ async with AsyncLingxiGraphClient(base_url, token=token) as client:
)
async for event in client.runs.stream(run["id"]):
print(event)
+
+ accepted = await client.runs.steer(
+ run["id"],
+ kind="user_input",
+ payload={"message": "switch to Spanish"},
+ idempotency_key="msg-123",
+ )
+ print(accepted["id"], accepted["status"])
```
异步资源与同步资源一一对应。不要在 async 事件循环中使用同步客户端。
diff --git a/Wiki/content/docs/zh/api/threads-runs.mdx b/Wiki/content/docs/zh/api/threads-runs.mdx
index 8d7445d..1cf3775 100644
--- a/Wiki/content/docs/zh/api/threads-runs.mdx
+++ b/Wiki/content/docs/zh/api/threads-runs.mdx
@@ -79,9 +79,131 @@ POST /v1/runs/{run_id}/redrive
```
- cancel 是协作式请求，通常先返回 `cancelling`；节点和 provider 应响应 cancellation token；
+- 取消一个 `pending`（尚未被 worker claim）或 `paused`（停在 `interrupt`、没有 worker 持有它）的 run 会立即变为 `cancelled` 并写入 `finished_at`：这两种状态都没有正在进行的投递可以协作式地观察 cancel token，因此无需等待；
+- 若一个 `paused` run 之后已被 resume，再对它调用 cancel 会返回 `409 run_superseded`——与 `/steer` 对陈旧 `run_id` 给出的信号一致（见下文）。resume 之后旧 run 的 `status` 依设计仍保持 `paused`（只有类型化的 `superseded_by_run_id` 列会标记它已过期——绝不是可被用户写入、因而可能伪造该信号的 `metadata`），因此 cancel 在决定状态迁移之前，会原子地检查同一个字段；它绝不会在历史 run 上悄然"成功"，而真正被 resume 出来的新 run 却仍在继续执行。请改为取消新的 `run_id`；
- 只有 `paused` run 可 resume；响应是新 run，`metadata.resumed_from_run_id` 指向原 run；
+- 并发对同一个已 `paused` 的 run 调用 resume 时只有一个会成功：它创建上述新 run，其余并发调用都会收到 `409 run_resume_conflict`，而不是各自创建重复的新 run；
- 只有 `failed` 或 `dead_letter` 可 redrive；保留原 input/config/version 并重新入队。
Redrive 不是“修改输入后重试”。业务输入或 graph version 改变时应创建新 run，并使用新的幂等键。
+
+## 运行中输入（Steering）
+
+`POST /v1/runs/{run_id}/steer` 在 Run 执行期间 durably 地注入新的结构化输入——不等待完成、
+不强制取消、也不需要先暂停 Run。这与 `cancel`（终止执行）和 `interrupt`（图代码主动让出、
+等待 `resume`）是三件不同的事：steering 只是把一条新消息安全地送进正在跑（或即将跑、或
+刚被 pause）的 Run，是否处理、何时处理、处理后是否改变计划完全由图代码决定。
+
+```bash
+curl -X POST https://agents.example.com/v1/runs/$RUN_ID/steer \
+ -H "Authorization: Bearer $TOKEN" \
+ -H "Idempotency-Key: msg-123" \
+ -H "Content-Type: application/json" \
+ -d '{"kind":"user_input","payload":{"message":"改成用中文回复"},"metadata":{}}'
+```
+
+202 响应：
+
+```json
+{
+ "id": "steer_01H...",
+ "run_id": "run_01H...",
+ "sequence": 3,
+ "kind": "user_input",
+ "status": "pending",
+ "created_at": "2026-08-18T09:00:00Z"
+}
+```
+
+请求体字段：
+
+| 字段 | 必填 | 说明 |
+| --- | --- | --- |
+| `kind` | 是 | 业务自定义的事件种类字符串，例如 `user_input`、`priority_change` |
+| `payload` | 是 | 任意 JSON，图代码通过 `drain_steering()` 读到的就是这个对象 |
+| `metadata` | 否 | 附加元数据，不参与去重 |
+| `idempotency_key` | 否 | 见下 |
+
+**`Idempotency-Key` header 与 body 字段的优先级：** 两者语义相同（同一个逻辑 key），
+可以二选一传递；同时传递时以 `Idempotency-Key` header 为准。同一 `(tenant_id, run_id,
+idempotency_key)` 三元组只会创建一条 steering 事件——重复调用返回**同一条**事件
+（`id`、`sequence` 不变，不会新建行，也不会递增 `sequence`），而不是 409。但响应体
+**不保证**与首次调用逐字节一致：它反映的是该事件**当前持久化的状态**，两次调用之间
+状态可能已经变化（例如首次调用时是 `pending`，而图已经在这之后消费了它，重放时读到
+的就是 `consumed`）。这与 `POST /runs` 创建接口的 `Idempotency-Key` 语义一致，都是
+"同 key 同请求安全重放"而不是"报错"——但重放返回的始终是同一个底层资源的最新状态，
+而不是首次响应的静态快照。关键的一点是：这次重放查找发生在下面所有"按 run 状态"的
+准入检查**之前**，而不是之后——同一个 key 的重试永远返回已存在的事件，即使该 run
+此后已经进入 `run_finalizing`、走到终止态、或已被 superseded。只有一个真正全新的 key
+才会受这些检查约束。
+
+### 各 Run 状态下的行为
+
+| Run 状态 | `/steer` 行为 |
+| --- | --- |
+| `pending`（已入队，worker 未 claim） | 202，durably 写入，worker claim 后立即可见 |
+| `running` | 202，durably 写入，下一个安全点可读到 |
+| `cancelling` | 202，仍然接受（cancel 优先级见下，但接受写入本身不受影响） |
+| `running`/`cancelling`，但正在 finalizing（`runs.steering_closed`） | 对一个真正全新的事件返回 `409 run_finalizing`；同一 key 的重放仍然返回已存在的事件，而不是 409——此时该 run 的图已经执行完毕、这次投递尝试里已经没有更多安全点了，但 run 行还没翻转为终止态/暂停态 |
+| `paused`（等待 `interrupt`/`resume`） | 202，durably 写入；见下方"paused steering 迁移" |
+| `succeeded` / `failed` / `cancelled` / `timed_out` / `dead_letter`（终止态） | 对一个新 key 返回 `409 run_terminal`，不产生任何事件 |
+| 已被 `resume` 取代的旧 `run_id` | `409 run_superseded`，见下 |
+
+### cancel 优先于 steer
+
+无论 steer 请求先到还是后到，cancel 请求都会立即生效：一个 Run 上先 `/steer` 再
+`/cancel`，该 Run 依然会进入 `cancelling`（若该 Run 是 `paused`，则直接进入
+`cancelled`，见上文）并停止执行；已经 durably 写入的 steering
+事件不会被回滚，只是不会再被任何存活的执行消费。steer 从不能撤销或延迟一次已经发出
+的 cancel。
+
+### paused steering 在 atomic resume transaction 中迁移到新 Run
+
+`POST /v1/runs/{run_id}/resume` 会创建一条**新** Run（新的 `run_id`，通过
+`metadata.resumed_from_run_id` 关联旧 Run）。因为 worker 只会为它正在执行的
+`run_id` 拉取 pending steering，所以 resume 端点在**同一个原子事务**里做两件事：
+创建新 Run，并把旧 Run 上所有仍处于 `pending`/`delivered` 的 steering 事件**迁移**到
+新 Run（保序、保留 `kind`/`payload`/`metadata`/`idempotency_key`，旧事件被标记为
+`superseded`）。这里的"原子"很关键：新 Run 在被任何 worker `claim` 之前，其迁移过的
+steering 必须已经全部就位——不存在"新 Run 已可被执行，但尚未迁移完成"的中间态窗口。
+之后由新 Run 的 worker 按正常路径投递、消费：paused 时提交的 steering 会在 resume
+之后被新 Run 实际消费，而不是永远停留在 `pending`。
+
+旧 Run 同时被标记为"已被 resume 取代"。resume 之后再对旧 `run_id` 调 `/steer`
+会返回 HTTP 409 `run_superseded`（而不是再次静默积压出一条永远没人消费的事件），
+提示调用方改为对新 Run 调用 `/steer`。
+
+### `source_event_id`：跨多次 resume 的稳定关联
+
+迁移到新 Run 的 steering 事件会获得一个全新的 `id`（它是新 Run 下的一条独立 durable
+行），但其 `source_event_id` 字段回指客户端**最初** `/steer` 调用收到的那个 `id`。
+即便一个 Run 被反复 pause/resume 多次（A → B → C → ……），每一跳迁移都保留**最初**
+的根 id，不会被中间某一跳的 id 覆盖。`created_at` 同样保留最初 durably accepted 的
+时间，因此最终 `run.steer.consumed` 的 `queue_latency_seconds` 包含了全部 pause
+等待时间。客户端只需记住最初 `/steer` 返回的 `id`，即可在任意后续 Run 的
+`run.steer.accepted`/`run.steer.consumed` 事件的 `source_event_id` 字段里找到它，
+无需跟踪中间产生的每一个 `run_id`。
+
+### payload 大小限制
+
+`payload` + `metadata` 序列化后大小受限（默认约 32KB，或服务端事件字节上限，两者先
+到者生效）；超限返回 `413`/`422`，错误码 `payload_too_large`，不会产生任何 durable
+写入。JSON 值必须是可安全序列化的普通类型（字符串、数字、布尔、null、数组、对象）；
+不支持二进制、循环引用或不可 JSON 化的自定义类型。
+
+### accepted ≠ consumed
+
+`/steer` 的 202 响应只代表事件已 durably 写入 PostgreSQL 的 steering inbox
+（`run.steer.accepted`，状态 `pending`）——**不代表图已经处理这条输入**。当配置了
+`EventBus`（进程内，或通过 `RedisEventBus` 使用 Redis）时，`/steer` 会发布一个可选的
+唤醒通知，正在运行的 worker 心跳循环会等待它，从而缩短发现延迟；它从不是唯一数据源，
+也不是事件流里的一个具名生命周期事件（事件流中并不存在 `run.steer.available` 这个
+事件），只是一个低延迟提示。即使这个通知丢失，worker 仍会通过 PostgreSQL 自身的周期性
+安全点检查最终发现并消费该事件。只有当某个安全点的图代码真正调用
+`runtime.drain_steering()` 读取到该事件，并且这次消费被 `commit_steering_consumptions_if_owned()`
+durably 提交后，才会出现 `run.steer.consumed`。调用方不能仅凭 202 就假设图已经
+"看到"这条输入——目前没有可以直接轮询某个 steering 事件状态的接口；请观察该 run 的
+事件流（`GET /v1/runs/{run_id}/stream`）或事件列表中的 `run.steer.consumed` 来确认
+真正被消费。
diff --git a/Wiki/content/docs/zh/concepts/meta.json b/Wiki/content/docs/zh/concepts/meta.json
index 45ca64a..877a2fd 100644
--- a/Wiki/content/docs/zh/concepts/meta.json
+++ b/Wiki/content/docs/zh/concepts/meta.json
@@ -1,4 +1,4 @@
{
"title": "核心概念",
- "pages": ["architecture", "durable-execution", "agents-tools", "agent-skills"]
+ "pages": ["architecture", "durable-execution", "steering", "agents-tools", "agent-skills"]
}
diff --git a/Wiki/content/docs/zh/concepts/steering.mdx b/Wiki/content/docs/zh/concepts/steering.mdx
new file mode 100644
index 0000000..9c45da1
--- /dev/null
+++ b/Wiki/content/docs/zh/concepts/steering.mdx
@@ -0,0 +1,213 @@
+---
+title: "运行中输入（Steering）"
+description: "在图执行期间 durably 注入新输入，及其与 cancel/interrupt/resume 的区别"
+---
+
+## 要解决什么问题？
+
+一次长时间运行的 agent 执行往往不是一个封闭黑盒：人类或另一个系统可能想在**它仍在执行时**
+交给它新的信息——"其实要用中文回复""客户刚打来电话，优先级变成 urgent 了""这是你缺的那个
+API key"——而不需要停止它、不需要等它自己走到某个天然的暂停点、也不能因为 worker 在接受
+输入之后马上崩溃就把这条输入弄丢。
+
+Steering 就是 LingxiGraph 对这个问题的回答：`POST /v1/runs/{run_id}/steer` durably地
+接受一小段结构化 JSON，并在图代码下一次主动查看时把它交给图代码。LingxiGraph 保证的是
+**投递**本身——durable、有序、去重、在单个存活的 `SteeringChannel` 内原子 drain-once——
+但从不替业务决定"这条新输入意味着什么"。一条消息应该触发重新规划、只是追加到上下文，
+还是应该被忽略（因为已经跑过那一步了），完全是业务图的逻辑。具体"drain-once"在跨 worker
+崩溃时保证到什么程度、不保证什么，见下方"顺序、幂等与 drain-once"一节。
+
+## 与 cancel、interrupt、resume 的区别
+
+这四个机制都会在运行过程中"介入"一个 run，很容易被混为一谈，但它们回答的是不同的问题：
+
+| 机制 | 回答的问题 | 由谁发起 | 对 run 的影响 |
+| --- | --- | --- | --- |
+| `cancel` | "停止这个 run。" | 调用方 | run 进入 `cancelling` → `cancelled`；执行被协作式地终止 |
+| `interrupt` | "图代码现在需要一个还没有的值，拿到之后再继续。" | 图代码（`interrupt(value)`） | run 进入 `paused`；执行在**那个调用点**挂起，直到 `resume` 提供值 |
+| `resume` | "用这个值继续一个已暂停的 run。" | 调用方 | 创建一条**新** run，从 `interrupt` 挂起处继续 |
+| `steer` | "这是一条新输入，随时方便的时候看一下。" | 调用方 | run 按自己的节奏继续执行；图决定何时/是否查看新输入 |
+
+`interrupt`/`resume` 是嵌在图内某个特定位置的"请求-响应"模式：图显式停下来等待，调用方
+必须提供正好是那个调用点在等的值。Steering 正相反：它是**推送式**的、异步的、不绑定任何
+特定调用点，也不会让任何东西停下来。一个 run 甚至不需要处于 `running`（`pending`、还没被
+worker claim 的 run 同样能 durably 接受 steering）就能收到一条 steering。
+
+Steering 也可以和另外三者组合使用：一个 run 可以在 running 时被 steer，也可以在 paused
+时被 steer（见下方"跨暂停迁移"一节），而且 steering 永远不会阻塞或延迟一次 cancel。
+
+## 数据流：PostgreSQL durable inbox → worker → SteeringChannel → runtime
+
+```mermaid
+sequenceDiagram
+ participant C as 调用方
+ participant API as API 服务
+ participant PG as PostgreSQL（run_steering_events）
+ participant R as Redis（可选）
+ participant W as Worker
+ participant Ch as SteeringChannel
+ participant N as 节点代码
+
+ C->>API: POST /runs/{id}/steer
+ API->>PG: INSERT pending steering 行
+ PG-->>API: durably 提交
+ API-->>C: 202 run.steer.accepted
+ API-->>R: best-effort NOTIFY（可选）
+ R--)W: 唤醒提示（可能丢失）
+ W->>PG: 在安全点轮询/查看 pending steering
+ PG-->>W: 仍为 pending 的行
+ W->>Ch: 摄入内存中的 channel
+ N->>Ch: 在安全点调用 drain_steering()
+ Ch-->>N: 有序、去重后的事件
+ N-->>W: 消费记录（node、namespace、task_id、latency）
+ W->>PG: commit_steering_consumptions_if_owned()（状态更新 + run.steer.consumed，同一事务）
+```
+
+PostgreSQL 的 `run_steering_events` 表是每一条 steering 事件是否存在及其状态的唯一真相
+来源。schema 保留了 `delivered` 这个状态值以便将来扩展，但目前实现的流程从不会写入
+它——真实的状态转换是 `pending` → `consumed`（通过一次原子的
+`commit_steering_consumptions_if_owned()` 调用直接完成），或者 `pending`/`delivered` →
+`superseded`——原因可能是它所属的 paused run 被 resume（其仍 pending 的 steering 被迁移到
+新 run），也可能是它在该 run 到达终结 finalization 边界时仍未被消费（原因不止一种，见下文
+"终结流程与 steering 准入边界"一节）。请把 `delivered` 当作代码里
+可能需要读取/放行的一个状态值，而不是当前任何路径会产生的状态。Redis
+通知只是为了降低轮询延迟的优化手段；丢失它从不会丢失一条 steering 事件——worker 在每个
+安全点的周期性检查会退化为直接读 PostgreSQL。`SteeringChannel` 是 worker 用来把已取到的
+事件以原子 drain-once 语义交给图代码的进程内、按 run 划分的结构；它本身不是 durable 的，
+worker 重启或接管一个 run 时会从 PostgreSQL 重新构建它。
+
+## "安全点"由应用选择，不是任意强制抢占
+
+LingxiGraph 从不会在节点函数执行到一半时强行打断它去投递 steering。它保证的是：只要图
+代码调用 `runtime.drain_steering()`，读到的就是当前最新、尚未被消费过的事件集合——但
+**在节点代码内部的哪一行**调用它，完全由应用决定。一个节点可以在开头调用一次，可以在
+循环里反复调用，也可以完全不调用。
+
+LingxiGraph 真正控制的是 executor **何时把一个节点当作新任务来调度**——节点开始前、下一个
+superstep 前、重试前、resume 之后——每一次这样的调度切换都是"自上次以来摄入的 steering
+变得可见"的机会。这是刻意的设计边界，不是缺口：真正抢占任意用户代码会破坏运行时其余部分
+依赖的确定性重放保证。
+
+## 顺序、幂等与 drain-once
+
+- 同一个 run 的 steering 事件按内部 `sequence` 严格排序，序号在 durably accepted 时分配；
+- 在**单个存活的 `SteeringChannel`** 内——同一个 worker、同一个进程内实例，跨节点的任意
+ 次内部重试——`drain_steering()` 永远不会返回同一事件两次：原子 drain-once；
+- **这不是系统级的恰好一次保证。** 如果 worker 在 `drain_steering()` 已经把某条事件交给
+ 节点、但 `commit_steering_consumptions_if_owned()` 尚未把这次消费 durably 记录下来之前崩溃（或
+ 它的 lease 被重新分配），PostgreSQL 里那一行仍然是 `pending`/`delivered`——接管该 run、
+ 从 PostgreSQL 重建 channel 的新 worker 会合理地重新投递同一条事件。跨越一次崩溃，
+ steering 的投递语义是**至少一次（at-least-once）**，不是恰好一次。如果业务要对 drain
+ 到的事件执行不可逆的副作用（扣费、有副作用的外部 API 调用），应该用 `event.id`（或者
+ 经过暂停/恢复迁移后的根 `source_event_id`，见下文）自行做幂等控制，而不能假设
+ drain-once 就等于"只处理一次"；
+- 通过 `idempotency_key` 去重：对同一个 `(tenant_id, run_id, idempotency_key)` 提交两次
+ 会返回**原**事件而不是创建第二条；这个去重在 `drain_steering()` 之后依然有效——原事件被
+ drain 之后再次重放同一提交，返回的仍然是那条（现已 consumed 的）原事件，而不是一条新的
+ pending 事件。这只对 accept 一侧（重复的 `/steer` 调用）去重，本身并不能让跨崩溃的
+ drain 一侧变成恰好一次——见上一条；
+- consumed-commit（`commit_steering_consumptions_if_owned()`）与图自身为消费它的那个 superstep
+ 所做的 checkpoint commit 是两个独立的 durable 操作，不是同一个分布式事务（见下方
+ "consumption 与 checkpoint commit 的当前关系"）——因此不存在覆盖"steering 投递 + 图
+ 状态"两者的系统级恰好一次处理保证。
+
+## 重试、worker 崩溃与 Redis 丢失
+
+- **worker 重试：** `commit_steering_consumptions_if_owned()` 是幂等的——一个 worker 提交某批
+ consumption 后、在看到 ack 之前连接断开并重试同一批，不会产生重复的
+ `run.steer.consumed`；只有在这次调用中真正发生状态转换的 steering 行才会产生一条；
+- **worker 崩溃：** 因为投递状态存放在 PostgreSQL 里，而不是崩溃 worker 的内存里，接管该
+ run 的新 worker 会从 PostgreSQL（`pending`/`delivered` 行）重新构建完整状态并继续。
+ worker 崩溃不会静默丢失任何 steering 事件；
+- **Redis 丢失：** Redis 只是加速器，从不是正确性所依赖的东西。如果 pub/sub 通知从未
+ 到达——连接重置、Redis 重启、消息丢失——worker 自身在每个安全点对 PostgreSQL 的周期性
+ 检查仍然会发现并投递该事件，只是延迟略高，而不是彻底丢失更新。
+
+## 终结流程与 steering 准入边界
+
+一旦图执行结束，拥有该 run 的 worker 会先原子地封闭新的 steering 准入
+（`close_steering()`），然后才开始对图已经消费的内容做最终 durable flush。从这一刻起，
+新的 `/steer` 调用会收到一个稳定的 `409 run_finalizing`——因为已经不再存在任何安全点能让
+图消费一条新接受的事件。
+
+但这仍然留下一处缺口：某次 `/steer` 调用可能恰好在 `close_steering()` 抢到同一把
+run 行锁**之前**先抢到锁并完成 durable 提交——即便此时图早已产生终态输出，本该负责摄入
+该事件的心跳循环也早已停止。final flush 完成之后，Runtime 会专门检查这种情况：任何仍处于
+`pending`/`delivered` 的 steering 行都会被 durably 转为 `superseded`（原因为
+`unconsumed_at_final_boundary`），并伴随一条 `run.steer.superseded` 生命周期事件，二者与该
+run 的终结状态写入在**同一个** lease/attempt-fenced 事务中一起提交——不再是两次分别提交、
+可能在崩溃或 lease 丢失之间彼此脱节的写入（见 `finalize_run_with_steering_disposition_if_owned`）。
+随后该 run 会以之前已经算出的结果（`succeeded`、`failed`、`dead_letter` 等——任何终态；
+`paused` 不属于此终结路径，因为它本来就从来不是一种终态，见下文）正常终结——**不会**仅仅
+因为存在一条 steering 事件就被强制进入重试循环。
+
+`unconsumed_at_final_boundary` 是一个刻意保持笼统的原因标签，**并不**声称发生了竞争。它
+涵盖两种 Runtime 目前无法从现有数据中区分的情况：上面描述的锁竞争，以及更常见的情况——
+一个本就合法地从不调用 `drain_steering()` 的图（见上文"安全点由应用选择"，这是文档明确
+允许的选择），其对应的行其实一直可见、只是从未被消费。无论哪种情况，结果都一样：该 run
+会在第一次投递尝试就正常终结，就像这条 steering 事件从未到达过一样，而该事件的最终归宿会
+被记录为 `superseded`，绝不会被静默丢弃。
+
+**最终归宿策略，精确表述如下：** 一个 terminal run **自身**的终结决策绝不能让一条未被消费
+的 steering 事件被静默搁置、没有任何 durable、可观测的归宿——这正是上面机制所保证的。任何
+真正终结的 `finish` 意图——由图自身的结果计算产生的 `succeeded`、`failed`、`cancelled` 或
+`dead_letter`——都会封闭 steering 准入并 supersede 遗留行，与上文描述完全一致；唯一被排除
+在外的是 `paused`（见下一段）。也就是说，一次普通的 `failed`/`dead_letter` 结果本身**不会**
+让 steering 停留在 `pending`。
+
+唯一仍可能出现这种情况的场景比"被 dead-letter 的 run 就会留下 pending steering"要窄得多：
+特指**最终 steering flush 本身就无法持久提交**的那次 run（`_retry_or_dead_letter()`——见
+`Worker._execute` 里 final flush 之后的处理）。这条路径根本不会走到
+`finalize_run_with_steering_disposition_if_owned()`——它会转而让这次投递尝试走普通的
+retry/dead-letter 流程，因为此时写下任何归宿（无论是否 superseded）都会要么报告一个虚假
+状态，要么静默丢失已经 drain 但尚未提交的消费记录。只有在这种特定情况下，一个 `dead_letter`
+的 run 才可能仍然留有 `pending` 的 steering——这是刻意保留的 durable 历史记录，而不是 bug：
+之后如果该 run 被 `/redrive`，这条仍 pending 的行会像任何其他排队中的 steering 一样被重新
+拾起。一次普通的、由客户端发起的 **cancel** 不在这条恢复路径之内：`/redrive` 不接受
+`cancelled` 的 run，因此留在 `cancelled` run 上的 `pending` steering 目前没有任何自动的
+未来投递机会——如果确实需要恢复，那是这套机制之外的运维/API 任务，而不是由 Runtime 自动
+解决的。
+
+## 并行 superstep、`Send` 与 subgraph 共享同一个 channel
+
+一个 run 的 `SteeringChannel` 在同一 superstep 内的所有任务之间共享，包括并行分支和
+`Send` 扇出的任务，也跨同一 run 内的 subgraph 节点边界共享。drain 对每个事件是互斥的
+（一个事件永远只能被某一个任务 drain 到——不存在跨并行任务的重复处理），但并不限定在
+某一个特定节点：run 的任务图中任意位置、任意嵌套深度的节点都可以调用
+`drain_steering()`，看到的是同一份 durable、有序的积压。最终 `run.steer.consumed` 事件上
+的 `queue_latency_seconds` 与 `task_id`/`namespace` 精确记录了究竟是哪个 task、在哪个
+subgraph 深度，最终消费了这条事件。
+
+## consumption 与 checkpoint commit 的当前关系（明确的 scope 边界）
+
+在当前版本中，把一条 steering 事件标记为 `consumed`（通过
+`commit_steering_consumptions_if_owned()`）与图自身为消费它的那个 superstep 所做的 checkpoint
+commit，是两个独立的 durable 操作，**没有**被包在同一个分布式事务里。这在实践中意味着：
+如果某个节点 drain 了 steering 之后、其 checkpoint 尚未写入之前进程就退出了，
+checkpoint 级别的重试会重新执行该节点——但那条 steering 事件已经被标记为
+`consumed`，重新执行的节点通过 `drain_steering()` 不会再次看到它。需要让 drain 到的
+payload 在 superstep 中途崩溃后依然存活的节点，应该把需要的内容作为该节点返回值的一部分
+持久化进 state（就像对待任何其他外部观测到的输入一样），而不是指望重试后再从 steering
+channel 里重新读到它。这是当前版本**刻意**做出的 scope 收窄，不是"consumption 与
+checkpoint 是事务性的"这样的承诺，未来版本可能会重新审视这一点。
+
+## Steering 明确不做什么
+
+Steering 是投递机制，不是规划机制。它不会：
+
+- 替业务决定一条新消息是否应该触发重新规划、取消正在进行的 tool 调用，或改变当前节点的
+ 控制流——这完全是业务图的逻辑；
+- 保证 accept 到 consume 之间的低延迟——只保证一旦消费发生，它是 durable 且可通过
+ `run.steer.consumed` 观测到的；
+- 提供直接修改已提交 state 的方式；图必须显式地把 drain 到的 payload 折叠进自己的
+ state 更新里。
+
+## 嵌入式 / library 模式下的降级语义
+
+当 LingxiGraph 作为嵌入式库使用、不经过 Agent Server（没有 worker 进程，没有基于
+PostgreSQL 的 repository）时，`compiled_graph.steer(run_id, ...)` 和
+`runtime.drain_steering()` 仍然可以对进程内的 `SteeringChannel` 生效，但 durability
+会降级为"和承载它的这个进程一样 durable"：没有跨进程投递、没有 worker 崩溃恢复、也没有
+基于 PostgreSQL 的 `run.steer.accepted`/`run.steer.consumed` 审计轨迹。一次从未接触
+steering 的普通 `invoke()` 不会因此产生任何额外开销，也不会泄漏 channel 状态——该层面的
+精确保证参见 Agent Server 实现引用的嵌入式生命周期测试。
diff --git a/Wiki/content/docs/zh/operations/meta.json b/Wiki/content/docs/zh/operations/meta.json
index 307b52f..10f6162 100644
--- a/Wiki/content/docs/zh/operations/meta.json
+++ b/Wiki/content/docs/zh/operations/meta.json
@@ -1,4 +1,4 @@
{
"title": "运维与项目",
- "pages": ["configuration", "security-observability"]
+ "pages": ["configuration", "security-observability", "steering-operations"]
}
diff --git a/Wiki/content/docs/zh/operations/steering-operations.mdx b/Wiki/content/docs/zh/operations/steering-operations.mdx
new file mode 100644
index 0000000..456685a
--- /dev/null
+++ b/Wiki/content/docs/zh/operations/steering-operations.mdx
@@ -0,0 +1,225 @@
+---
+title: "运行中输入（Steering）的运维"
+description: "迁移上线顺序、PostgreSQL 作为真相来源、延迟、重试与回滚边界"
+---
+
+## 迁移上线顺序
+
+运行中输入（issue #16）在基础 schema 之上新增了四个 forward-only 的 Alembic revision：
+
+| Revision | 新增内容 |
+| --- | --- |
+| `0002_steering` | `run_steering_events` 表及其索引，以及读写它的 worker/server 代码路径 |
+| `0003_steering_source_event` | `run_steering_events` 上的 `source_event_id` 列 + 局部索引，用于跨多次 pause/resume 迁移关联同一条 steering |
+| `0004_steering_closed` | `runs.steering_closed`——finalization 准入闸门（review round 6/7）；详见下方"`steering_closed` finalization 闸门" |
+| `0005_superseded_by_run_id` | `runs.superseded_by_run_id`——被 resume 覆盖的类型化控制面血缘标记（review round 14，BLOCKER）；把它从可被用户写入的 `metadata` 中移出，使其无法被伪造。详见下方"`superseded_by_run_id` 血缘标记" |
+
+按标准 Alembic 路径应用：
+
+```bash
+alembic upgrade head
+```
+
+四者都是纯新增（`ADD COLUMN IF NOT EXISTS`、`CREATE INDEX IF NOT EXISTS`、可为空/
+`NOT NULL DEFAULT FALSE` 列），可以在部署新的 server/worker 二进制之前先对存活数据库
+执行——早于 `0004`/`0005` 的旧二进制根本不会读写 `steering_closed`/`superseded_by_run_id`，
+行为与这些列不存在时完全一致。零停机上线顺序：
+
+1. 在旧的 server/worker 二进制仍在提供服务时执行 `alembic upgrade head`
+ （新增 `run_steering_events`、`source_event_id`、`steering_closed` 与
+ `superseded_by_run_id`）；
+2. 再滚动发布新的 server 与 worker 二进制。
+
+不要颠倒这个顺序：在列存在之前就尝试写入 `source_event_id`/`steering_closed`/
+`superseded_by_run_id` 的二进制会插入/更新失败。因为本项目的迁移是 forward-only 的（`downgrade()` 会直接抛错），schema
+层面没有自动回滚路径——见下方"升级与回滚边界"。
+
+
+`PostgresRepository.setup()` 也会直接应用同一批 SQL 文件（测试和嵌入式/开发环境会用到）。
+这条路径**不能**替代在真实部署中驱动 Alembic revision 链：生产上线应该且只应该执行
+`alembic upgrade head`，并且必须走到 `0005_superseded_by_run_id`，而不是停在
+`0002_steering`、`0003_steering_source_event` 或 `0004_steering_closed`。
+
+
+## `steering_closed` finalization 闸门
+
+一旦 worker 的图执行到达终态/暂停结果，该 run 可能还会短暂地继续读作
+`running`/`cancelling`，同时 worker 把图已经消费掉的 steering durably flush 出去、
+提交最终状态——此时已经没有更多图安全点可以消费任何新输入了。`runs.steering_closed`
+是持有该 run 的 worker（与状态写入一样，基于自己的 `lease_owner`/`attempt` 加锁）在执行
+结束的瞬间、final flush 开始之前设置的一个布尔值。
+
+- 当 `steering_closed` 为 `false` 时，`POST /steer` 会像 `0004` 之前一样被正常接受；
+- 一旦当前 attempt 的 `steering_closed` 变为 `true`，一个**全新**的 `POST /steer`
+ （尚无匹配 Idempotency-Key 记录）会被拒绝，返回 `409 run_finalizing`——这是一个稳定的、
+ 与终态 run 的 `409` 不同的独立错误，方便客户端区分"这个 run 正在收尾"和"这个 run
+ 已经彻底结束了"；
+- 一次**重放**的 `POST /steer`——即 `(tenant_id, run_id, Idempotency-Key)` 与某个已经
+ durably 接受过的事件完全相同——永远返回该已存在事件的当前状态，绝不会返回新的
+ `409`，无论 `steering_closed`/终态/superseded 状态如何。见下方"幂等重放在所有准入闸门
+ 前都是安全的"；
+- `paused` 结果永远不会关闭闸门：暂停并不像终态那样"没有安全点剩下"——对一个 paused
+ run 调用 `/steer` 是设计中刻意支持的行为（其 pending steering 会在 `/resume` 时迁移到
+ 新 run 上）；
+- 一次被放弃的投递尝试关闭的闸门不会泄漏到下一次尝试：`claim_run` 在每次新的 claim 上
+ 都会把 `steering_closed` 重置为 `false`，所以被重试的 run 会重新正常接受 steering。
+
+仅仅关闭闸门本身并不足以防止一条永久 pending 的 steering 行：`/steer` 请求有可能在
+`close_steering()` 之前赢得该 run 行的锁——即使此时图早已执行完毕。真正能捕获这种情况的
+是 worker 在 flush 之后的检查（`finalize_run_with_steering_disposition_if_owned()`，见
+`Worker._execute`）——如果 final flush 已经提交了它能提交的一切之后，仍然存在
+`pending`/`delivered` 的行，该行会被 durably 转为 `superseded`（原因为
+`unconsumed_at_final_boundary`——这是一个刻意保持笼统的标签，**并不**声称发生了竞争；它
+同样覆盖更常见的情况：图从始至终就没有调用过 `drain_steering()`），并伴随一条
+`run.steer.superseded` 生命周期事件，二者与 run 的终结状态写入在**同一个**
+lease/attempt-fenced 事务中一起提交——不再是两次分别提交、可能在崩溃或 lease 丢失时
+彼此脱节的写入。随后该 run 会以之前已经算出的结果正常终结——**不会**仅仅因为存在一条
+steering 事件就走 retry/dead-letter 循环，因为那样会给一个本就合法地从不调用
+`drain_steering()` 的图强加一次不必要的重跑。**一个终态 run 自身的终结决策绝不能让一条
+未被消费的 steering 事件被静默搁置、没有任何 durable、可观测的归宿**——一次由图自身计算出的
+普通 `failed`/`dead_letter` 结果，会像 `succeeded`/`cancelled` 一样封闭准入并 supersede
+遗留行。唯一更窄的例外是：某次 run 的**最终 steering flush 本身就无法持久提交**
+（`_retry_or_dead_letter()`），这条路径根本不会走到
+`finalize_run_with_steering_disposition_if_owned()`，而是转而让这次投递尝试走普通的
+retry/dead-letter 流程——只有在这种情况下，一个 `dead_letter` 的 run 才可能合法地仍然携带
+`pending` 的 steering 行，作为刻意保留的 durable 历史记录，之后由 `/redrive` 像任何其他排队
+中的 steering 一样重新拾起。但 `/redrive` 只接受 `failed`/`dead_letter` 状态的 run——一次
+普通的、由客户端发起的 **cancel** 不在这条恢复路径之内，因此留在 `cancelled` run 上的
+`pending` steering 目前没有任何自动的未来投递机会。完整策略见 steering 概念页"终结流程与
+steering 准入边界"一节。
+
+## `superseded_by_run_id` 血缘标记
+
+当 `/resume` 从一个 `paused` 的 run 创建出后继 run 时，旧 run 需要一种可靠的方式来表示
+"我已经被取代了——请对新 run 执行 steer/cancel/resume"。在 `0005` 之前，这个标记存放在
+`metadata.superseded_by_run_id` 中——普通的、用户可写的 `Run` metadata。由于 `/steer`、
+`/cancel` 以及并发 resume 的冲突闸门都把这个 key 当作权威状态来读取，调用方可以在创建 run
+时伪造它（`POST /v1/runs` 携带 `metadata: {"superseded_by_run_id": "..."}`），从而在一个从未
+真正被 resume 过的 run 上错误触发 `409 run_superseded` / `409 run_resume_conflict`
+（issue #16 PR #17 review round 14，BLOCKER）。
+
+`runs.superseded_by_run_id`（由 `0005_superseded_by_run_id` 新增）是一个类型化、可为空的列，
+取代该 key 成为唯一的真相来源：
+
+- 它只在创建后继 run、迁移旧 run 待处理 steering 的同一个受 fence 保护的 resume 事务内
+ （`resume_run_with_pending_steering()` 及其 PostgreSQL 对应实现）被原子地写入一次——
+ 永远不会从 `RunCreate.metadata` 派生，也不能通过它写入。
+- `/steer` 的准入检查、`/cancel` 的 superseded 检查，以及 resume 冲突闸门，现在都直接读取
+ `run.superseded_by_run_id`；它们都不再为此决策查询 `metadata`。
+- 调用方在普通 `RunCreate` 上提交 `metadata.superseded_by_run_id` 现在对控制流完全没有
+ 影响——这个 key 是惰性的，最多作为普通用户 metadata 存储，永远不会被拷贝进类型化列。
+- 升级兼容性：升级到本 PR 之后，一个升级前就存在、恰好携带同名用户 metadata key 的历史
+ run，不会被重新解释为系统血缘状态——类型化列对每一条既有记录都从 `NULL` 开始，只有未来
+ 真正发生的 resume 才会填充它。
+
+## 幂等重放在所有准入闸门前都是安全的
+
+`submit_steering()`（内存版和 PostgreSQL 版仓库都一样）会先检查是否已经存在匹配
+`(tenant_id, run_id, Idempotency-Key)` 的事件，**然后**才走终态/`steering_closed`/
+superseded 这些准入闸门，而不是反过来。如果客户端的 `POST /steer` 已经 durably
+提交，但 202 响应从未真正送达（连接中断、代理超时），使用相同 key 的标准幂等重试会
+返回该事件当前的持久化状态——id 和 sequence 都不变——即使该 run 此后已经进入
+finalizing、terminal 或 superseded 状态。只有一个真正**全新**的 key（这个 run 从未见过
+的 key）才会受这些闸门约束；那才是一次真正的准入决策，仍然会得到相应的 `409`。
+
+## PostgreSQL 是真相来源，Redis 只是加速器
+
+每一条 steering 事件是否存在及其状态，都存放在 PostgreSQL 的 `run_steering_events` 表里。
+Redis pub/sub 只用于缩短"事件变为 durable"和"worker 注意到它"之间的延迟——从不会被用来
+判断一个事件是否存在，它的丢失（Redis 重启、连接断开、NOTIFY 被漏掉）也从不会丢失一条
+事件。运维可以把 Redis 视为对 steering 正确性完全可丢弃的组件：即使 Redis 宕机或被驱逐，
+steering 仍能正常工作，只是 worker 的发现方式退化为在每个安全点周期性轮询 PostgreSQL，
+而不是近乎即时的唤醒。
+
+## Worker heartbeat 对 steering 投递延迟的影响
+
+一个 worker 只会在其持有租约的 run 到达下一个安全点时（节点开始、下一 superstep、重试、
+resume 之后）去拉取该 run 新产生的 steering。因此投递延迟主要受两个因素支配：
+
+- **该 run 多久到达一次安全点。** 一个正在做单次长耗时 tool 调用的节点，在调用返回之前
+ 看不到调用期间提交的 steering——直到该节点结束，或节点自己主动调用了
+ `drain_steering()`；
+- **Redis 通知不可用时的 worker 轮询/heartbeat 间隔。** 更大的间隔用更低的 PostgreSQL
+ 轮询负载换取更高的 steering 延迟；可参照 `docs/api.md` 与部署时
+ `lingxigraph worker` 的相关参数调优。
+
+## 如何诊断长期 pending 的 steering
+
+如果某条 steering 事件停留在 `pending` 状态的时间远超预期：
+
+1. 先确认该 run 确实已被 claim 并在推进：`GET /v1/runs/{run_id}`——一个仍处于 `pending`
+ 的 run 还没有 worker 在执行它，其上的 steering 在等待被 claim，而不是 bug；
+2. 检查该 run 的图代码是否真的调用了 `runtime.drain_steering()`，以及调用频率——
+ steering 投递完全取决于图是否选择去查看，这是应用自选安全点模型的一部分（详见
+ Steering 概念页）；
+3. 直接查询 `run_steering_events` 该行的 `status`、`created_at`、`source_event_id`，
+ 看它是否是从更早、曾经 paused 的 run 迁移过来的——最终 `run.steer.consumed` 事件上的
+ `queue_latency_seconds` 会包含所有累积的暂停等待时间，这是预期行为，不是 bug；
+4. 检查 worker 日志/指标中是否有反复的 `commit_steering_consumptions_if_owned()`
+ 失败——该操作重试是安全的（幂等，并且基于 worker 自己的 `lease_owner`/`attempt`
+ 加锁，见"`steering_closed` finalization 闸门"），但持续失败说明存在值得告警的数据库
+ 连通性问题；
+5. 如果客户端在 `/steer` 上收到 `409 run_finalizing`，说明请求到达时该 run 的图已经
+ 执行完毕——检查当前 attempt 的 `runs.steering_closed`（`true` 表示闸门已关闭），并对
+ 一个全新的 run 重试，而不是同一个 run。这里用相同 Idempotency-Key 重放永远是安全的，
+ 即使在闸门关闭之后也一样——见"幂等重放在所有准入闸门前都是安全的"；
+6. 如果某条 steering 事件的 `run.steer.superseded` 事件数据里 `reason` 是
+ `unconsumed_at_final_boundary`（而不是 `resume_transfer`），这正是终结边界修复按预期工作：该事件
+ 已经 durably 被接受，但该投递尝试已经没有安全点可以消费它了，所以它被赋予了一个
+ durable、可观测的 `superseded` 归宿，而不是让 run 带着一条永久 pending 的行走向终态，
+ 也不会被强加一次不必要的重跑。这本身不是 bug；但如果同一个 run 反复出现
+ `unconsumed_at_final_boundary`，就值得在应用/图层面排查为什么 steering 总是在图到达终止节点之后
+ 才到达。
+
+## 正常的 `run.steer.accepted → consumed` 生命周期
+
+1. `POST /steer` durably 插入一条 `pending` 行，并（202 响应中）产生
+ `run.steer.accepted`；
+2. 持有该 run 租约的 worker 发现这条行（通过 Redis 提示或 PostgreSQL 轮询），把它摄入
+ 该 run 的内存 `SteeringChannel`；
+3. 图代码在某个安全点调用 `drain_steering()` 并收到它；
+4. worker durably 提交这次消费（`commit_steering_consumptions_if_owned()`，基于 worker
+ 自己的 `lease_owner`/`attempt` 加锁），在**同一事务**中把该行原子地转为 `consumed`
+ 并产生 `run.steer.consumed`。一个已经不再持有该 lease/attempt 的 worker（比如在
+ heartbeat 或 final flush 期间发生了 lease 被接管）什么都不会写入——从这一刻起该行
+ 完全由新的持有者负责。
+
+这条链路中没有任何一步是可选或可打乱顺序的；客户端应该只把第 4 步的
+`run.steer.consumed` 事件当作"确实被处理"的证据。
+
+## 为什么 worker/DB 瞬时故障不会丢失一次 consumption，以及如何观察重试
+
+状态更新与 `run.steer.consumed` 事件的插入发生在同一个 PostgreSQL 事务内。如果事务已经
+提交，但连接在 worker 看到确认之前断开，worker 的重试逻辑会重新发送同一批 consumption。
+因为 `commit_steering_consumptions_if_owned()` 是幂等的——只为这次调用中真正发生状态
+转换的行产生 lifecycle 事件，并且只有在调用方仍然持有该 run 当前 lease/attempt 时才会
+写入任何东西——针对已经 consumed 的行重放同一批次是安全的空操作：状态保持
+`consumed`，也不会产生第二条 `run.steer.consumed`。运维可以在 worker 日志里观察到这种
+模式：同一批 steering id 反复出现 `commit_steering_consumptions_if_owned` 调用，但
+`run_events` 里对应的 `run.steer.consumed` 数量并没有增长——只要重试最终会停止，这就是
+预期行为，而不是卡住的重试循环。
+
+## 升级与回滚边界
+
+本项目的迁移是 forward-only 的：每个 revision 的 `downgrade()` 都会直接抛错，而不是
+尝试撤销变更。这是刻意的设计：真正回滚 `run_steering_events` 表或其新增列，要么会丢失
+durable 的 steering 历史，要么需要把正在进行的 run 与新二进制已经不再预期的 schema
+重新对齐，代价过高。实践含义：
+
+- 永远不要把"回滚 schema"当作回滚策略的一部分。如果确实需要回滚一次部署，应该回滚
+ **二进制**、保持 schema 停在当前（更靠前）的 revision——`0002_steering`/
+ `0003_steering_source_event`/`0004_steering_closed`/`0005_superseded_by_run_id` 都是纯
+ 新增，早于对应特性的旧
+ 二进制会直接忽略多出来的表/列，不会因此启动失败；
+- **schema 兼容不等于语义兼容。** 旧二进制在新 schema 上不会崩溃，但它并不理解
+ `steering_closed` 或 `superseded_by_run_id`——它不会执行 finalization 的 steering
+ admission 门控，也不会执行本 PR 新增的 resume-lineage cancel/steer 拒绝逻辑。应该把
+ 新的 steering/lineage 控制语义视为"只有当所有控制面实例都升级到新二进制后才可用"，
+ 而不是 schema 迁移一落地就生效。如果一个二进制已经真正执行过这些语义（例如某次
+ resume 已经写入了 `superseded_by_run_id`），把它回滚成旧版本并不会撤销这次转换——
+ 旧二进制只是从此不再强制执行它。如果必须在 steering 正在被使用时回滚二进制，应先
+ quiesce 或等待 in-flight run 结束，不要假设旧二进制能保留同样的保证；
+- 因为没有 schema 回滚路径，`0002_steering`/`0003_steering_source_event`/
+ `0004_steering_closed`/`0005_superseded_by_run_id` 应该先在 staging 环境（或 CI 的
+ `test_postgres_integration.py`
+ Postgres job）中验证过，再应用到生产环境，而不是依赖"出问题了还能撤销"。
diff --git a/Wiki/scripts/check-docs.mjs b/Wiki/scripts/check-docs.mjs
index 4229b9f..67aad92 100644
--- a/Wiki/scripts/check-docs.mjs
+++ b/Wiki/scripts/check-docs.mjs
@@ -4,7 +4,7 @@ import process from 'node:process';
const root = path.resolve(process.cwd(), 'content/docs');
const locales = ['zh', 'en'];
-const expectedPageCount = 24;
+const expectedPageCount = 26;
async function walk(directory) {
const entries = await readdir(directory, { withFileTypes: true });
diff --git a/docs/api.md b/docs/api.md
index d7f2fc5..860ef83 100644
--- a/docs/api.md
+++ b/docs/api.md
@@ -10,7 +10,7 @@ OpenAPI 是 Agent Server 的协议真相。默认服务地址为 `http://localho
| Graph registry | `GET /v1/graphs`、`GET /v1/graphs/{id}` |
| Assistants | `POST/GET /v1/assistants`、`GET/PATCH/DELETE /v1/assistants/{id}` |
| Threads | `POST/GET /v1/threads`、`GET/PATCH/DELETE /v1/threads/{id}`、state、history、fork、runs |
-| Runs | threaded/stateless create、get/list、join、resume、cancel、redrive、stream |
+| Runs | threaded/stateless create、get/list、join、resume、cancel、redrive、steer、stream |
| Store | `POST /v1/store/batch`、`GET /v1/store/search` |
| Schedules | create/list/update/delete |
| Interop | `/a2a/{assistant_id}`、`/mcp` |
@@ -54,6 +54,87 @@ curl -X POST http://localhost:8124/v1/runs \
`X-Tenant-ID` 只在 `LINGXIGRAPH_INSECURE_DEV_AUTH=true` 时生效。生产 tenant 必须从已验证
JWT claim 派生，绝不信任调用方自报 header。
+## 运行中输入（Steering）
+
+`POST /v1/runs/{run_id}/steer` 在 Run 执行期间durably地注入新的结构化输入，不等待完成、
+不强制取消、也不需要先暂停。请求体：
+
+```json
+{"kind": "user_input", "payload": {"message": "..."}, "metadata": {}, "idempotency_key": "..."}
+```
+
+语义：
+
+- 202 只代表已durably写入 PostgreSQL 的 steering inbox，不代表图已经处理；配置了
+ `EventBus`（进程内或 Redis）时会发布一个可选的低延迟唤醒通知供 worker 心跳循环等待，
+ 但它不是事件流中的具名事件（不存在 `run.steer.available` 事件），也从不作为唯一来源——
+ worker 自身的周期性 PostgreSQL 安全点检查始终是最终保证。
+- `running`/`queued`/`cancelling` 均可接受；`paused` run 也接受durable写入。
+- 终止态（`succeeded`/`failed`/`cancelled`/`timed_out`/`dead_letter`）对一个**新**事件
+ 返回 HTTP 409 `run_terminal`，不会静默生成一条永远不会被消费的事件。
+- **finalization 闸门（`run_finalizing`）：** 一旦某个 worker 的图执行到达终态/暂停
+ 结果，该 worker 会立即（在 durably flush 最终 steering 消费、提交最终状态**之前**）
+ 关闭这次投递尝试的新 steering 准入——此时 Run 行可能仍读作 `running`/`cancelling`，
+ 但已经没有更多图安全点可以消费任何新输入了。在这个窗口内提交一个**新**事件（尚无
+ 匹配 idempotency key）会返回 HTTP 409 `run_finalizing`，而不是被静默接受进一个再也
+ 没人会 drain 的 channel。`paused` 不受此闸门影响——它是刻意设计支持继续接受 steering
+ 的（迁移到 resume 后的新 run，见下）。
+- `Idempotency-Key` header 或 body 的 `idempotency_key` 二选一；同一 `(tenant, run, key)`
+ 只会创建一条事件——**且这条幂等重放检查发生在上面所有准入判断之前，而不是之后**：
+ 同一个 key 的重试永远返回已存在事件的当前持久化状态（`id`/`sequence` 不变），即使该
+ Run 此后已经进入 `run_finalizing`、终止态、或被 superseded，也绝不会返回一个新的
+ 409。只有真正全新的 key 才会受终止态/`run_finalizing`/superseded 这些准入检查约束。
+- cancel 优先于 steer：先 steer 后 cancel，cancel 依然立即生效，steer 从不撤销/延迟取消。
+- payload+metadata 序列化后大小受限（默认约 32KB / 服务端事件字节上限，先到者生效）。
+
+**Paused run 的 steering 归属哪个 run_id？** `POST /v1/runs/{run_id}/resume` 会创建一条
+*新* Run（新的 `run_id`，通过 `metadata.resumed_from_run_id` 关联旧 Run），worker 只会为它
+正在执行的 run_id 拉取 pending steering。因此 resume 端点在创建新 Run 后，会原子地把旧
+Run 上所有仍处于 `pending`/`delivered` 的 steering 事件**迁移**到新 Run（保序、保留
+`kind`/`payload`/`metadata`/`idempotency_key`，旧事件被标记为 `superseded`），随后由新 Run
+的 worker 按正常路径投递、消费——即：paused 时提交的 steering 会在 resume 之后被新 Run
+实际消费，而不是永远 pending。旧 Run 同时被标记「已被 resume 取代」；resume 之后再对
+旧 `run_id` 调 `/steer` 会返回 HTTP 409 `run_superseded`（而不是再次静默积压），提示调用方
+改为对新 Run 调用 `/steer`。
+
+图内部通过 `runtime.has_steering`、`runtime.peek_steering()`、`runtime.drain_steering()`
+在安全点（节点开始前、节点完成后、下一 superstep 前、重试前、resume 之后）读取；
+LingxiGraph 只保证durable delivery、顺序、去重与安全消费，"新消息是否意味着重新规划"完全
+由业务图决定。嵌入式/无 Server 场景下同一套 API 生效：`compiled_graph.steer(run_id, ...)`。
+
+**"安全点"由谁定义？** LingxiGraph 保证的是 channel 本身的一致性（原子 drain-once、去重、
+顺序）以及*调用* `drain_steering()` 这件事发生时机的新鲜性——每次节点被 executor 调用时，
+它读到的都是当前最新、未被消费过的事件集合。但**在节点函数内部的哪一行代码调用
+`drain_steering()`，完全由应用节点自己决定**：executor 不会在节点函数内部插入任何强制
+边界（例如"只能在函数开头调用一次"），也不会暂停用户代码的执行去插入安全点。这是刻意的
+设计选择而非缺口：真正的边界语义体现在 executor *何时把节点当作一个 task 来调度*
+（节点开始前、下一 superstep 前、重试前、resume 之后见上），而不是节点内部的执行位置。
+
+**`source_event_id`：跨 resume 的稳定关联。** 迁移到新 Run 的 steering 事件会获得一个
+全新的 `id`（它是新 Run 下一条独立的durable行），但其 `source_event_id` 字段回指客户端
+最初 `/steer` 调用收到的那个 `id`。即使一个 Run 被反复 pause/resume 多次（A → B → C →
+……），每一跳迁移都保留**最初**那个根 id（`source_event_id = 上一跳的 source_event_id
+或上一跳的 id`），不会被中间某一跳的 id 覆盖。`created_at` 同样保留最初 durable
+accepted 的时间，因此后续 `run.steer.consumed` 的 `queue_latency_seconds`
+包含了全部 pause 等待时间，而不仅仅是最后一次 resume 之后的等待时间。客户端只需记住
+最初 `/steer` 返回的 `id`，即可用它在 `run.steer.accepted`/`run.steer.consumed` 事件的
+`source_event_id` 字段里找到最终归宿，无需跟踪中间产生的每一个 run_id。
+
+**accepted ≠ consumed。** `POST /steer` 的 202 响应只代表事件已durably写入
+`run_steering_events`（`run.steer.accepted`，状态 `pending`）；只有当某个安全点的图代码
+真正调用 `drain_steering()` 并且该 drain 结果被 `commit_steering_consumptions()`
+durably提交后，才会出现 `run.steer.consumed`。两者严格分离：调用方不能仅凭 202 假设图
+已经看到这条输入。
+
+**观测性：** worker 在消费（drain）steering 事件后写入的 `run.steer.consumed` 事件包含
+`steering_event_id`、`source_event_id`（若来自迁移，否则省略）、`sequence`、`kind`，以及
+用于定位与诊断的 `queue_latency_seconds`（从 durably accepted 到被消费的等待时长）、
+`node`（消费该事件的节点名）、`namespace`（该节点所在的 subgraph 命名空间路径）、
+`task_id`（区分同一 superstep 内并行/`Send` 任务的具体 task）。`commit_steering_
+consumptions()` 本身是幂等的：只有真正发生 `pending`/`delivered` → `consumed` 状态转换
+的事件才会生成一条 `run.steer.consumed`；worker 因网络/DB 瞬时故障重试同一批
+consumption 时，不会产生第二条重复的 lifecycle 事件。
+
## SSE 续传
```text
diff --git a/migrations/env.py b/migrations/env.py
index ed108f5..4f92a48 100644
--- a/migrations/env.py
+++ b/migrations/env.py
@@ -14,9 +14,29 @@
config.set_main_option("sqlalchemy.url", dsn.replace("%", "%%"))
+def _use_psycopg_driver(url: str) -> str:
+ """Force the psycopg (v3) driver, matching the project's declared dependency.
+
+ Plain ``postgresql://`` DSNs make SQLAlchemy default to the psycopg2
+ dialect, which is not installed anywhere in this project (the ``postgres``
+ extra pins ``psycopg[binary]``, i.e. psycopg 3). Callers -- including
+ tests that set ``sqlalchemy.url`` directly via ``Config.set_main_option``
+ -- may pass a plain ``postgresql://`` or ``postgres://`` DSN, so normalize
+ the scheme here rather than relying on every caller to do so.
+ """
+ if url.startswith("postgresql+"):
+ return url
+ if url.startswith("postgresql://"):
+ return "postgresql+psycopg://" + url[len("postgresql://") :]
+ if url.startswith("postgres://"):
+ return "postgresql+psycopg://" + url[len("postgres://") :]
+ return url
+
+
def run_migrations_offline() -> None:
+ url = config.get_main_option("sqlalchemy.url")
context.configure(
- url=config.get_main_option("sqlalchemy.url"),
+ url=_use_psycopg_driver(url) if url else url,
literal_binds=True,
dialect_opts={"paramstyle": "named"},
)
@@ -25,8 +45,12 @@ def run_migrations_offline() -> None:
def run_migrations_online() -> None:
+ section = dict(config.get_section(config.config_ini_section, {}))
+ url = section.get("sqlalchemy.url")
+ if url:
+ section["sqlalchemy.url"] = _use_psycopg_driver(url)
connectable = engine_from_config(
- config.get_section(config.config_ini_section, {}),
+ section,
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
diff --git a/migrations/versions/0002_steering.py b/migrations/versions/0002_steering.py
new file mode 100644
index 0000000..0c8e725
--- /dev/null
+++ b/migrations/versions/0002_steering.py
@@ -0,0 +1,29 @@
+"""Durable mid-run steering inbox (issue #16)."""
+
+import os
+import re
+from importlib.resources import files
+
+from alembic import op
+
+revision = "0002_steering"
+down_revision = "0001_lingxigraph_v1"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+ schema = os.getenv("LINGXIGRAPH_POSTGRES_SCHEMA", "lingxigraph")
+ if not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", schema):
+ raise ValueError("invalid LINGXIGRAPH_POSTGRES_SCHEMA")
+ sql = (
+ files("lingxigraph.server")
+ .joinpath("migrations/0002_steering.sql")
+ .read_text(encoding="utf-8")
+ .replace("{{schema}}", schema)
+ )
+ op.execute(sql)
+
+
+def downgrade() -> None:
+ raise RuntimeError("LingxiGraph v1 production migrations are forward-only")
diff --git a/migrations/versions/0003_steering_source_event.py b/migrations/versions/0003_steering_source_event.py
new file mode 100644
index 0000000..c8e0d7b
--- /dev/null
+++ b/migrations/versions/0003_steering_source_event.py
@@ -0,0 +1,39 @@
+"""Stable source_event_id for paused-run steering resume (issue #16).
+
+Issue #16 PR #17 review round 3, point 1: the SQL file
+``src/lingxigraph/server/migrations/0003_steering_source_event.sql`` shipped
+without a matching Alembic revision, so ``alembic upgrade head`` stayed at
+``0002_steering`` and never applied it -- even though
+``PostgresRepository._setup_sync()`` (which scans the SQL files directly,
+not through Alembic) did apply it, letting CI's real-Postgres integration
+job pass while a deployment driven by ``alembic upgrade head`` would not.
+This revision closes that gap.
+"""
+
+import os
+import re
+from importlib.resources import files
+
+from alembic import op
+
+revision = "0003_steering_source_event"
+down_revision = "0002_steering"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+ schema = os.getenv("LINGXIGRAPH_POSTGRES_SCHEMA", "lingxigraph")
+ if not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", schema):
+ raise ValueError("invalid LINGXIGRAPH_POSTGRES_SCHEMA")
+ sql = (
+ files("lingxigraph.server")
+ .joinpath("migrations/0003_steering_source_event.sql")
+ .read_text(encoding="utf-8")
+ .replace("{{schema}}", schema)
+ )
+ op.execute(sql)
+
+
+def downgrade() -> None:
+ raise RuntimeError("LingxiGraph v1 production migrations are forward-only")
diff --git a/migrations/versions/0004_steering_closed.py b/migrations/versions/0004_steering_closed.py
new file mode 100644
index 0000000..081cddd
--- /dev/null
+++ b/migrations/versions/0004_steering_closed.py
@@ -0,0 +1,35 @@
+"""Atomic finalization gate for steering admission (issue #16 PR #17
+review round 6, point 3, BLOCKER).
+
+Adds ``runs.steering_closed`` -- see
+``src/lingxigraph/server/migrations/0004_steering_closed.sql`` for the
+full rationale.
+"""
+
+import os
+import re
+from importlib.resources import files
+
+from alembic import op
+
+revision = "0004_steering_closed"
+down_revision = "0003_steering_source_event"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+ schema = os.getenv("LINGXIGRAPH_POSTGRES_SCHEMA", "lingxigraph")
+ if not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", schema):
+ raise ValueError("invalid LINGXIGRAPH_POSTGRES_SCHEMA")
+ sql = (
+ files("lingxigraph.server")
+ .joinpath("migrations/0004_steering_closed.sql")
+ .read_text(encoding="utf-8")
+ .replace("{{schema}}", schema)
+ )
+ op.execute(sql)
+
+
+def downgrade() -> None:
+ raise RuntimeError("LingxiGraph v1 production migrations are forward-only")
diff --git a/migrations/versions/0005_superseded_by_run_id.py b/migrations/versions/0005_superseded_by_run_id.py
new file mode 100644
index 0000000..b93c1ad
--- /dev/null
+++ b/migrations/versions/0005_superseded_by_run_id.py
@@ -0,0 +1,35 @@
+"""Typed control-plane lineage marker for resumed runs (issue #16 PR #17
+review round 14, BLOCKER).
+
+Adds ``runs.superseded_by_run_id`` -- see
+``src/lingxigraph/server/migrations/0005_superseded_by_run_id.sql`` for the
+full rationale.
+"""
+
+import os
+import re
+from importlib.resources import files
+
+from alembic import op
+
+revision = "0005_superseded_by_run_id"
+down_revision = "0004_steering_closed"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+ schema = os.getenv("LINGXIGRAPH_POSTGRES_SCHEMA", "lingxigraph")
+ if not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", schema):
+ raise ValueError("invalid LINGXIGRAPH_POSTGRES_SCHEMA")
+ sql = (
+ files("lingxigraph.server")
+ .joinpath("migrations/0005_superseded_by_run_id.sql")
+ .read_text(encoding="utf-8")
+ .replace("{{schema}}", schema)
+ )
+ op.execute(sql)
+
+
+def downgrade() -> None:
+ raise RuntimeError("LingxiGraph v1 production migrations are forward-only")
diff --git a/src/lingxigraph/errors.py b/src/lingxigraph/errors.py
index 9d1d6d8..4ffa92a 100644
--- a/src/lingxigraph/errors.py
+++ b/src/lingxigraph/errors.py
@@ -45,6 +45,52 @@ class IdempotencyConflictError(LingxiGraphError):
"""Raised when an idempotency key is reused with a different request."""
+class RunTerminalError(LingxiGraphError):
+ """Raised when an operation targets a Run that has already finished."""
+
+
+class RunSupersededError(LingxiGraphError):
+ """Raised when steering targets a paused Run that has already been resumed.
+
+ Resume durably transfers any steering that was pending at resume time
+ onto the new Run it creates (see
+ ``repository.resume_run_with_pending_steering`` and issue #16's
+ paused-run steering semantics) -- once that happens the old Run id is a dead end
+ for *further* steering, so new steer attempts against it fail loudly
+ instead of silently pending forever.
+ """
+
+
+class RunFinalizingError(LingxiGraphError):
+ """Raised when steering targets a Run whose graph has already ended.
+
+ Issue #16 PR #17 review round 6, point 3: once a worker's graph
+ execution reaches its terminal outcome, the normal heartbeat/steering
+ loop stops and there is no safe point left for the graph to ever
+ consume a newly-accepted steering event -- even though the Run row
+ may still read ``running``/``cancelling`` while the final flush and
+ finalization commit are in flight. ``Repository.close_steering`` sets
+ an atomic, lease-fenced gate the instant execution ends so a late
+ ``/steer`` call during this window fails loudly with a stable error
+ instead of being durably accepted into an inbox nobody will ever read.
+ """
+
+
+class RunResumeConflictError(LingxiGraphError):
+ """Raised when a resume attempt loses a race against a concurrent resume.
+
+ ``POST /runs/{id}/resume`` pre-checks that the target Run is
+ ``paused`` before calling
+ ``repository.resume_run_with_pending_steering``, but that pre-check is
+ not itself atomic with the resume -- two concurrent resume requests can
+ both pass it. The repository re-validates, *inside* the same locked
+ critical section that creates the new Run, that the old Run is still
+ ``paused`` and has no ``superseded_by_run_id`` set yet; the request
+ that loses that race gets this error instead of silently creating a
+ second descendant Run (issue #16 PR #17 review round 4, point 1).
+ """
+
+
class GraphInterrupt(BaseException):
"""Internal control-flow signal used by :func:`interrupt`."""
@@ -66,4 +112,8 @@ def __init__(self, interrupt: object) -> None:
"IdempotencyConflictError",
"LingxiGraphError",
"PersistenceError",
+ "RunFinalizingError",
+ "RunResumeConflictError",
+ "RunSupersededError",
+ "RunTerminalError",
]
diff --git a/src/lingxigraph/graph/executor.py b/src/lingxigraph/graph/executor.py
index 35c0cc7..374e505 100644
--- a/src/lingxigraph/graph/executor.py
+++ b/src/lingxigraph/graph/executor.py
@@ -43,6 +43,7 @@
)
from ..schema import SchemaAdapter
from ..serialization import JsonSerializer, Serializer
+from ..steering import SteeringChannel, SteeringEvent
from ..types import (
Command,
CommandScope,
@@ -244,8 +245,75 @@ def __init__(
self._active_runs: dict[str, CancellationToken] = {}
self._event_sequences: dict[str, int] = {}
self._run_budgets: dict[str, ExecutionBudget] = {}
+ self._run_steering: dict[str, SteeringChannel] = {}
self.response_format: Mapping[str, Any] | type | None = None
+ def steer(
+ self,
+ run_id: str,
+ *,
+ kind: str,
+ payload: Mapping[str, Any],
+ metadata: Mapping[str, Any] | None = None,
+ idempotency_key: str | None = None,
+ ) -> SteeringEvent:
+ """Embedded/library-mode steering: durably-in-process, no server.
+
+ This is the degraded (SQLite/in-memory/embedded) implementation of
+ the mid-run steering protocol: it works against an in-process
+ :class:`~lingxigraph.steering.SteeringChannel` keyed by ``run_id``,
+ which is the same primitive the Agent Server worker feeds from
+ PostgreSQL. Safe to call concurrently from another thread/task
+ while the run is executing; the graph observes it at its next
+ safe point via ``runtime.drain_steering()``.
+ """
+
+ channel = self._run_steering.setdefault(run_id, SteeringChannel(run_id))
+ event, _created = channel.submit(
+ kind=kind,
+ payload=payload,
+ metadata=metadata,
+ idempotency_key=idempotency_key,
+ )
+ return event
+
+ def has_pending_steering(self, run_id: str) -> bool:
+ channel = self._run_steering.get(run_id)
+ return channel is not None and channel.has_pending
+
+ def get_steering_channel(self, run_id: str) -> SteeringChannel:
+ """Return (creating if needed) the in-process channel for a run.
+
+ Used by server integrations (see :mod:`lingxigraph.server.worker`)
+ to feed durable PostgreSQL steering rows into the live executor
+ without going through the public ``steer()`` re-sequencing path.
+
+ A channel freshly created here is marked ``owned_by_executor=False``
+ -- the caller (a server Worker) is registering it explicitly, ahead
+ of a run that may not have started executing yet (a still-queued
+ run can receive steering before any worker claims it), and is
+ responsible for releasing it via ``forget_steering()`` once done.
+ This is what lets ``_run()`` distinguish "I created this channel for
+ a plain embedded invoke and must release it myself when the run
+ finishes" from "someone else owns this channel's lifecycle".
+ """
+
+ channel = self._run_steering.get(run_id)
+ if channel is None:
+ channel = SteeringChannel(run_id, owned_by_executor=False)
+ self._run_steering[run_id] = channel
+ return channel
+
+ def forget_steering(self, run_id: str) -> None:
+ """Release the in-process channel for a finished run.
+
+ Call once any final flush of consumed events is done (see
+ :class:`lingxigraph.server.worker.Worker`) to avoid an unbounded
+ per-run memory footprint across many completed runs.
+ """
+
+ self._run_steering.pop(run_id, None)
+
def _child_runtime(
self,
checkpointer: Checkpointer | None,
@@ -793,6 +861,36 @@ async def _run(
parent_budget = get_runtime().budget
except RuntimeError:
parent_budget = None
+ try:
+ parent_steering = get_runtime()._steering
+ except RuntimeError:
+ parent_steering = None
+ # Subgraphs share the parent run's steering channel (same run_id,
+ # nested namespace) so draining is deterministic regardless of which
+ # namespace's node happens to call ``runtime.drain_steering()``.
+ #
+ # Ownership: ``SteeringChannel.owned_by_executor`` (see steering.py)
+ # says whether *this* embedded run lifecycle is responsible for
+ # releasing the channel once the run truly finishes, as opposed to
+ # a server integration that pre-registered it via
+ # ``get_steering_channel()`` and manages release explicitly (see
+ # ``Worker._execute``'s ``forget_steering`` call). The flag lives on
+ # the channel object itself (not recomputed per call) so it stays
+ # correct across a paused run's repeated ``_run()`` invocations --
+ # e.g. embedded pause -> ``steer()`` -> resume reuses the exact same
+ # channel instance under the same ``run_id``, and it is only
+ # released once that sequence finally completes without pausing
+ # again. Only the top-level call for a run_id (``parent_steering``
+ # is ``None``, i.e. not a subgraph) may release it -- see the
+ # ``finally`` block below. A run that pauses on an interrupt is
+ # deliberately excluded from that release (``run_paused`` below) so
+ # steering submitted while paused remains deliverable at resume.
+ steering_channel = (
+ parent_steering or self._run_steering.get(run_id) or SteeringChannel(run_id)
+ )
+ owns_steering = parent_steering is None and steering_channel.owned_by_executor
+ run_paused = False
+ self._run_steering[run_id] = steering_channel
limits = {**dict(config.get("configurable", {})), **config}
self._run_budgets[run_id] = parent_budget or ExecutionBudget(
max_tool_calls=(
@@ -968,6 +1066,11 @@ async def _run(
)
self._active_runs.pop(run_id, None)
self._event_sequences.pop(run_id, None)
+ # This branch re-surfaces an *already* paused run without a
+ # resume -- leave the steering channel alone (whether it is the
+ # original paused run's externally-owned channel, or, for a
+ # fresh unowned run_id, an unused empty channel not worth a
+ # special-cased pop).
return
if not active and not sends:
@@ -987,6 +1090,8 @@ async def _run(
)
self._active_runs.pop(run_id, None)
self._event_sequences.pop(run_id, None)
+ if owns_steering:
+ self._run_steering.pop(run_id, None)
return
executed_steps = 0
@@ -1051,6 +1156,7 @@ async def _run(
writer=checkpoint_writer,
)
parent_checkpoint_id = saved_id or parent_checkpoint_id
+ run_paused = True
if stream_mode == "values":
yield copy.deepcopy(state)
elif stream_mode == "updates":
@@ -1305,6 +1411,7 @@ def emit(
await checkpoint_writer.flush()
await self._put_pending_results(config, saved_id, tasks, successful)
parent_checkpoint_id = saved_id
+ run_paused = True
if stream_mode == "values":
yield self._interrupt_output(state, interrupts)
elif stream_mode == "updates":
@@ -1486,6 +1593,7 @@ def emit(
if completed & self.interrupt_after:
self._require_interrupt_support(config)
+ run_paused = True
if stream_mode == "events":
marker = Interrupt(
value={
@@ -1604,6 +1712,27 @@ def emit(
self._active_runs.pop(run_id, None)
self._event_sequences.pop(run_id, None)
self._run_budgets.pop(run_id, None)
+ # Steering-channel lifecycle depends on who owns it (see the
+ # ``owns_steering`` comment above where the channel is set up):
+ #
+ # * Externally owned (server worker via ``get_steering_channel``
+ # before claiming the run, or inherited from a parent run for
+ # a subgraph): deliberately NOT popped here. A server
+ # integration (see Worker._sync_steering_out) needs to flush
+ # any events the graph drained during its very last safe
+ # point *after* this generator has fully finished, and the
+ # owner calls ``forget_steering(run_id)`` itself once that
+ # final flush is done.
+ # * Owned by this call (plain embedded/library ``invoke`` /
+ # ``ainvoke`` with no server and no pre-registered channel):
+ # released automatically once the run truly finishes, so a
+ # long-lived compiled graph reused across many embedded runs
+ # does not leak one ``SteeringChannel`` per run forever. A
+ # run that paused on an interrupt is exempt -- its channel
+ # must survive so steering submitted while paused is still
+ # deliverable when the same ``run_id`` is resumed.
+ if owns_steering and not run_paused:
+ self._run_steering.pop(run_id, None)
def _plan_tasks(
self,
@@ -2215,6 +2344,7 @@ def _make_runtime(
stream_subgraphs=stream_subgraphs,
budget=self._run_budgets.get(run_id),
_emit=emit,
+ _steering=self._run_steering.get(run_id),
)
async def _aget_tuple(self, config: Mapping[str, Any]) -> Any:
diff --git a/src/lingxigraph/runtime.py b/src/lingxigraph/runtime.py
index e74a2ad..3bf24e5 100644
--- a/src/lingxigraph/runtime.py
+++ b/src/lingxigraph/runtime.py
@@ -12,6 +12,7 @@
from threading import Lock
from typing import Any, Generic, TypeVar
+from .steering import SteeringChannel, SteeringEvent
from .types import StreamWriter
ContextT = TypeVar("ContextT")
@@ -228,6 +229,63 @@ class Runtime(Generic[ContextT]):
stream_subgraphs: bool = False
budget: ExecutionBudget | None = None
_emit: StreamEmitter | None = None
+ _steering: SteeringChannel | None = None
+
+ @property
+ def has_steering(self) -> bool:
+ """Whether one or more steering events are waiting to be drained.
+
+ This is a stable, generic read -- it never interprets the payload.
+ Safe to call from sync or async nodes, and from any subgraph
+ namespace (steering is scoped per top-level run, not per namespace).
+ """
+
+ return self._steering is not None and self._steering.has_pending
+
+ #: Alias suggested by the design doc for "should I consider replanning".
+ steering_pending = has_steering
+
+ def peek_steering(self) -> tuple[SteeringEvent, ...]:
+ """Read pending steering events without consuming them."""
+
+ if self._steering is None:
+ return ()
+ return self._steering.peek()
+
+ def drain_steering(self) -> tuple[SteeringEvent, ...]:
+ """Atomically consume and return pending steering events in order.
+
+ Once drained, the same events are not re-exposed by this channel.
+ The graph decides what to do with them (update state, replan, goto
+ another node, ignore, ask the user) -- LingxiGraph only guarantees
+ durable delivery, ordering, dedup and safe consumption.
+
+ Documented design choice -- "safe point" is application-chosen, not
+ executor-enforced: LingxiGraph guarantees *when* a fresh, correctly
+ ordered, deduplicated view of the channel is available to a node
+ (at every node invocation, via this ``Runtime``), but it does not
+ restrict *where inside the node's own code* that node is allowed to
+ call ``drain_steering()``. A node may call it at the very top before
+ doing any work, in the middle between two tool calls, or not at
+ all -- the executor has no notion of an in-node checkpoint to pin
+ the call to, and enforcing one would mean pausing arbitrary user
+ code mid-function, which LingxiGraph does not attempt. What *is*
+ guaranteed and executor-enforced is the channel's consistency
+ properties (atomic drain-once, dedup, ordering) and that consumption
+ is recorded (node/namespace/task_id/queue latency -- see
+ :class:`~lingxigraph.steering.SteeringConsumption`) against whichever
+ node actually called this method. See ``docs/api.md`` (steering
+ section) for the full statement of this contract.
+ """
+
+ if self._steering is None:
+ return ()
+ # ``task_id`` is either the plain node name or ``"{node}#{index}"``
+ # for a ``Send``-fanout task (see ``CompiledStateGraph._plan_tasks``)
+ # -- split off the fan-out suffix so the observability record names
+ # the node itself, not a synthetic per-task id.
+ node = self.task_id.split("#", 1)[0] if self.task_id else None
+ return self._steering.drain(node=node, namespace=self.namespace, task_id=self.task_id)
@property
def cancelled(self) -> bool:
@@ -358,6 +416,8 @@ def _reset_runtime_context(token: Token[_RuntimeContext | None]) -> None:
"CancellationToken",
"ExecutionBudget",
"Runtime",
+ "SteeringChannel",
+ "SteeringEvent",
"StreamWriter",
"get_config",
"get_runtime",
diff --git a/src/lingxigraph/sdk.py b/src/lingxigraph/sdk.py
index b2ed431..150c883 100644
--- a/src/lingxigraph/sdk.py
+++ b/src/lingxigraph/sdk.py
@@ -140,6 +140,34 @@ async def list(self, thread_id: str) -> list[dict[str, Any]]:
async def cancel(self, run_id: str) -> dict[str, Any]:
return await self._client.request("POST", f"/v1/runs/{run_id}/cancel")
+ async def steer(
+ self,
+ run_id: str,
+ *,
+ kind: str = "user_input",
+ payload: Mapping[str, Any] | None = None,
+ metadata: Mapping[str, Any] | None = None,
+ idempotency_key: str | None = None,
+ ) -> dict[str, Any]:
+ """Durably inject mid-run steering input; drained by the graph itself.
+
+ Success only means the event was durably accepted -- it does not
+ mean the graph has processed it yet.
+ """
+
+ headers = {"Idempotency-Key": idempotency_key} if idempotency_key else None
+ return await self._client.request(
+ "POST",
+ f"/v1/runs/{run_id}/steer",
+ json={
+ "kind": kind,
+ "payload": dict(payload or {}),
+ "metadata": dict(metadata or {}),
+ "idempotency_key": idempotency_key,
+ },
+ headers=headers,
+ )
+
async def resume(self, run_id: str, resume: Any, **values: Any) -> dict[str, Any]:
return await self._client.request(
"POST", f"/v1/runs/{run_id}/resume", json={"resume": resume, **values}
@@ -375,6 +403,34 @@ def list(self, thread_id: str) -> list[dict[str, Any]]:
def cancel(self, run_id: str) -> dict[str, Any]:
return self._client.request("POST", f"/v1/runs/{run_id}/cancel")
+ def steer(
+ self,
+ run_id: str,
+ *,
+ kind: str = "user_input",
+ payload: Mapping[str, Any] | None = None,
+ metadata: Mapping[str, Any] | None = None,
+ idempotency_key: str | None = None,
+ ) -> dict[str, Any]:
+ """Durably inject mid-run steering input; drained by the graph itself.
+
+ Success only means the event was durably accepted -- it does not
+ mean the graph has processed it yet.
+ """
+
+ headers = {"Idempotency-Key": idempotency_key} if idempotency_key else None
+ return self._client.request(
+ "POST",
+ f"/v1/runs/{run_id}/steer",
+ json={
+ "kind": kind,
+ "payload": dict(payload or {}),
+ "metadata": dict(metadata or {}),
+ "idempotency_key": idempotency_key,
+ },
+ headers=headers,
+ )
+
def resume(self, run_id: str, resume: Any, **values: Any) -> dict[str, Any]:
return self._client.request(
"POST", f"/v1/runs/{run_id}/resume", json={"resume": resume, **values}
diff --git a/src/lingxigraph/server/app.py b/src/lingxigraph/server/app.py
index e9e8e99..410c1a2 100644
--- a/src/lingxigraph/server/app.py
+++ b/src/lingxigraph/server/app.py
@@ -20,7 +20,16 @@
from ..cache import BaseCache
from ..checkpoint import Checkpointer, InMemorySaver
-from ..errors import ConcurrentRunError, EmptyInputError, IdempotencyConflictError
+from ..errors import (
+ ConcurrentRunError,
+ EmptyInputError,
+ IdempotencyConflictError,
+ RunFinalizingError,
+ RunResumeConflictError,
+ RunSupersededError,
+ RunTerminalError,
+)
+from ..steering import MAX_STEERING_PAYLOAD_BYTES, SteeringPayloadTooLarge
from ..store import BaseStore, InMemoryStore, StoreOperation
from ..types import RunStatus
from ..version import __version__
@@ -36,6 +45,8 @@
Schedule,
ScheduleCreate,
SchedulePatch,
+ SteerAccepted,
+ SteerCreate,
StoreBatchRequest,
Thread,
ThreadCreate,
@@ -184,6 +195,26 @@ async def concurrent_error(request: Request, exc: ConcurrentRunError):
async def idempotency_error(request: Request, exc: IdempotencyConflictError):
return _problem(request, 409, "idempotency_conflict", str(exc))
+ @app.exception_handler(RunTerminalError)
+ async def run_terminal_error(request: Request, exc: RunTerminalError):
+ return _problem(request, 409, "run_terminal", str(exc))
+
+ @app.exception_handler(RunSupersededError)
+ async def run_superseded_error(request: Request, exc: RunSupersededError):
+ return _problem(request, 409, "run_superseded", str(exc))
+
+ @app.exception_handler(RunResumeConflictError)
+ async def run_resume_conflict_error(request: Request, exc: RunResumeConflictError):
+ return _problem(request, 409, "run_resume_conflict", str(exc))
+
+ @app.exception_handler(RunFinalizingError)
+ async def run_finalizing_error(request: Request, exc: RunFinalizingError):
+ return _problem(request, 409, "run_finalizing", str(exc))
+
+ @app.exception_handler(SteeringPayloadTooLarge)
+ async def steering_payload_error(request: Request, exc: SteeringPayloadTooLarge):
+ return _problem(request, 413, "payload_too_large", str(exc))
+
@app.exception_handler(HTTPException)
async def http_error(request: Request, exc: HTTPException):
codes = {
@@ -537,6 +568,64 @@ async def cancel_run(run_id: str, user: Principal = Depends(require("operator"))
assert value is not None
return value
+ @app.post("/v1/runs/{run_id}/steer", response_model=SteerAccepted, status_code=202)
+ async def steer_run(
+ run_id: str,
+ body: SteerCreate,
+ idempotency_key: str | None = Header(default=None, alias="Idempotency-Key"),
+ user: Principal = Depends(require("operator")),
+ ):
+ """Durably accept mid-run steering input.
+
+ 202 means the event was durably written to the steering inbox --
+ never that the graph has processed it. The graph decides what a
+ new steering event means at its next safe point via
+ ``runtime.drain_steering()``.
+ """
+
+ run = await repository.get_run(user.tenant_id, run_id)
+ if run is None:
+ raise HTTPException(404, "run not found")
+ key = idempotency_key or body.idempotency_key
+ try:
+ event, created = await repository.submit_steering(
+ user.tenant_id,
+ run_id,
+ kind=body.kind,
+ payload=body.payload,
+ metadata=body.metadata,
+ idempotency_key=key,
+ # Issue #16 PR #17 review round 5, point 3: the public docs
+ # and the ``steering`` module both promise a ~32KB payload
+ # cap (``MAX_STEERING_PAYLOAD_BYTES``); using the much
+ # larger generic event cap here let payloads up to 256KB
+ # through, contradicting that documented contract. Enforce
+ # whichever limit is smaller so the HTTP endpoint never
+ # accepts more than the documented steering limit, even if
+ # a deployment raises the generic event size cap.
+ max_payload_bytes=min(
+ MAX_STEERING_PAYLOAD_BYTES, repository.limits.max_event_bytes
+ ),
+ )
+ except KeyError as exc:
+ raise HTTPException(404, "run not found") from exc
+ # ``repository.submit_steering`` durably records the steering row
+ # *and* its ``run.steer.accepted`` lifecycle event atomically (issue
+ # #16 PR #17 review round 4, point 2) -- nothing further to persist
+ # here even on an idempotent retry that repairs a previously-missing
+ # accepted event.
+ if created:
+ await event_bus.publish(user.tenant_id, run_id, 0)
+ await audit(user, "runs.steer", "run", run_id)
+ return SteerAccepted(
+ id=event.id,
+ run_id=run_id,
+ sequence=event.sequence,
+ status=event.status,
+ kind=event.kind,
+ created_at=event.created_at,
+ )
+
@app.post("/v1/runs/{run_id}/resume", response_model=Run, status_code=202)
async def resume_run(
run_id: str,
@@ -567,9 +656,27 @@ async def resume_run(
durability=previous.durability,
metadata={"resumed_from_run_id": previous.id},
)
- value = await repository.create_run(
- user.tenant_id, previous.thread_id, assistant, request
+ # Create the resumed Run and carry over any steering that was
+ # durably accepted while `previous` was paused as a single atomic
+ # repository operation (issue #16 PR #17 review point 1) -- the
+ # worker only pulls pending steering for the run it is actually
+ # executing (the new one), so without atomicity a worker could
+ # claim and even finish the new Run before the migration below
+ # ever ran, leaving the migrated steering permanently unconsumed.
+ # This also marks `previous` so any *later* steer attempt against
+ # the stale run_id is rejected instead of silently pending again
+ # (see RunSupersededError).
+ # ``resume_run_with_pending_steering`` re-validates under its own
+ # lock that ``previous`` is still paused and not already resumed by
+ # a concurrent request (issue #16 PR #17 review round 4, point 1),
+ # and durably records each transferred event's ``run.steer.accepted``
+ # lifecycle event in the same migration transaction (review round 4,
+ # point 2) -- nothing further to persist for that here.
+ value, transferred = await repository.resume_run_with_pending_steering(
+ user.tenant_id, previous.thread_id, assistant, request, previous.id
)
+ if transferred:
+ await event_bus.publish(user.tenant_id, value.id, 0)
await audit(user, "runs.resume", "run", value.id)
return value
diff --git a/src/lingxigraph/server/eventbus.py b/src/lingxigraph/server/eventbus.py
index 30f1cbd..839b655 100644
--- a/src/lingxigraph/server/eventbus.py
+++ b/src/lingxigraph/server/eventbus.py
@@ -5,7 +5,71 @@
import asyncio
import json
from collections import defaultdict
-from typing import Protocol
+from collections.abc import Awaitable
+from typing import Protocol, TypeVar
+
+_T = TypeVar("_T")
+
+
+async def _wait_cancellation_safe(awaitable: Awaitable[_T], timeout: float) -> _T | None:
+ """Await ``awaitable`` with a timeout, safely under external cancellation.
+
+ Issue #16 PR #17 review round 5, point 4 / round 6, point 4: this is
+ deliberately *not* ``asyncio.wait_for(awaitable, timeout)``. On
+ Python's asyncio (still true on the 3.11 runtime this project
+ targets), ``wait_for`` wraps the awaitable in its own inner Task and,
+ if the *outer* coroutine awaiting ``wait_for`` is itself cancelled --
+ exactly what happens when ``Worker._heartbeat`` is cancelled by
+ ``Worker._execute`` right as an ``EventBus.wait()`` call is in-flight
+ -- there is a real, reproducible race where that inner Task's
+ cancellation is never actually awaited to completion, hanging forever
+ (https://github.com/python/cpython/issues/86296, fixed by
+ ``asyncio.timeout()`` in 3.12). Racing an explicit sibling timer task
+ with ``asyncio.wait`` does not share that failure mode: an external
+ cancellation of *this* coroutine while it is suspended inside
+ ``asyncio.wait`` propagates immediately and unconditionally, the same
+ as any other plain ``await``.
+
+ Both :class:`InMemoryEventBus` and :class:`RedisEventBus` share this
+ helper so the fix applies to the production Redis path too, not just
+ the in-memory/test bus -- a heartbeat cancelled while blocked in
+ ``RedisEventBus.wait()`` must return just as promptly as one blocked
+ in ``InMemoryEventBus.wait()``.
+
+ Returns the awaitable's result, or ``None`` on a genuine timeout.
+
+ Issue #16 PR #17 review round 7, point 4 (Redis robustness): this used
+ to also swallow any exception the awaitable itself raised into
+ ``None``, indistinguishable from a timeout. ``RedisEventBus.wait()``'s
+ deadline loop treats a ``None`` result as "no message yet, keep
+ looping" -- so if Redis disconnects right after ``subscribe()``
+ succeeds and every subsequent ``get_message()`` call raises
+ immediately, that used to turn into a tight busy-loop for the rest of
+ the heartbeat timeout window (rapidly creating new waiter/timer tasks
+ with no backoff), never reaching ``RedisEventBus.wait()``'s own outer
+ ``except Exception: return``. Only a real timeout (the timer won the
+ race, the waiter never completed) is folded into ``None`` now; an
+ exception raised by the awaitable itself propagates to the caller, so
+ it reaches that outer handler and returns promptly instead of spinning.
+ """
+
+ waiter = asyncio.ensure_future(awaitable)
+ timer = asyncio.ensure_future(asyncio.sleep(timeout))
+ try:
+ await asyncio.wait({waiter, timer}, return_when=asyncio.FIRST_COMPLETED)
+ finally:
+ for pending in (waiter, timer):
+ if not pending.done():
+ pending.cancel()
+ await asyncio.gather(waiter, timer, return_exceptions=True)
+ if not waiter.done() or waiter.cancelled():
+ # Timed out (or the waiter was cancelled alongside us) -- nothing
+ # arrived in time.
+ return None
+ exc = waiter.exception()
+ if exc is not None:
+ raise exc
+ return waiter.result()
class EventBus(Protocol):
@@ -17,24 +81,45 @@ async def wait(
class InMemoryEventBus:
+ """In-process wake signal used by the heartbeat's low-latency fast path.
+
+ Backed by :class:`asyncio.Event`, not :class:`asyncio.Condition`. This
+ is a deliberate choice, not a style preference: ``Condition.wait()``
+ must re-acquire its lock before it can propagate cancellation, and a
+ ``notify_all()`` racing a concurrent ``cancel()`` on the *same*
+ condition can leave that re-acquire permanently stuck (a long-standing
+ CPython gotcha -- see https://github.com/python/cpython/issues/86296).
+ ``Worker._heartbeat`` is cancelled from ``Worker._execute`` right after
+ the run finishes, at exactly the moment ``_append_event`` is also
+ calling ``publish()`` for the run's own completion/interrupt events on
+ this same bus -- i.e. exactly the racing pattern the bug needs, and it
+ was reproducible here as a real hang once the finalization path in
+ ``_execute`` was restructured (issue #16 PR #17 review round 5) to
+ settle the heartbeat cancellation a little earlier relative to that
+ publish. ``asyncio.Event`` has no such lock-reacquire step, so
+ cancelling a waiter is always immediate and safe.
+ """
+
def __init__(self) -> None:
- self._conditions: dict[tuple[str, str], asyncio.Condition] = defaultdict(
- asyncio.Condition
- )
+ self._events: dict[tuple[str, str], asyncio.Event] = defaultdict(asyncio.Event)
async def publish(self, tenant_id: str, run_id: str, sequence: int) -> None:
del sequence
- condition = self._conditions[(tenant_id, run_id)]
- async with condition:
- condition.notify_all()
+ self._events[(tenant_id, run_id)].set()
async def wait(self, tenant_id: str, run_id: str, *, timeout: float = 15.0) -> None:
- condition = self._conditions[(tenant_id, run_id)]
- async with condition:
- try:
- await asyncio.wait_for(condition.wait(), timeout)
- except TimeoutError:
- return
+ event = self._events[(tenant_id, run_id)]
+ try:
+ await _wait_cancellation_safe(event.wait(), timeout)
+ finally:
+ # Reset for the next waiter. A publish() landing in the
+ # instant between this waiter waking and the clear() below is
+ # simply an extra, harmless early wake for whoever waits next
+ # -- this bus is purely a latency optimization on top of the
+ # unconditional PostgreSQL poll each heartbeat tick also does,
+ # so an occasional spurious or missed wake here never loses a
+ # steering event or a cancellation signal.
+ event.clear()
class RedisEventBus:
@@ -61,15 +146,70 @@ async def publish(self, tenant_id: str, run_id: str, sequence: int) -> None:
return
async def wait(self, tenant_id: str, run_id: str, *, timeout: float = 15.0) -> None:
+ # Issue #16 PR #17 review round 6, point 4 (REQUIRED): this used to
+ # be ``asyncio.wait_for(pubsub.get_message(...), timeout)`` -- the
+ # exact pattern this same round's ``InMemoryEventBus.wait()`` fix
+ # rewrote away from because it can hang forever when the *caller*
+ # (``Worker._heartbeat``) is cancelled mid-wait. The Worker
+ # cancels a Redis-backed heartbeat exactly the same way as an
+ # in-memory one, so the production Redis path needs the same
+ # cancellation-safe explicit task/timer pattern, via the bus-
+ # agnostic ``_wait_cancellation_safe`` helper both classes share.
pubsub = self._redis.pubsub()
try:
await pubsub.subscribe(self._channel(tenant_id, run_id))
- await asyncio.wait_for(pubsub.get_message(ignore_subscribe_messages=True), timeout)
+ # Two things had to be fixed here, not just one:
+ #
+ # 1. ``get_message``'s own ``timeout`` defaults to 0.0 --
+ # "return immediately, blocking for nothing" -- not "block
+ # until a message arrives". Passing it explicitly makes
+ # redis-py perform an actual blocking read instead of a
+ # poll.
+ # 2. Even with that fixed, the *first* ``get_message()`` call
+ # right after ``subscribe()`` reliably returns near-
+ # instantly with ``None`` regardless of the timeout given --
+ # it is consuming the buffered SUBSCRIBE acknowledgement
+ # frame that ``ignore_subscribe_messages=True`` then
+ # filters out, not genuinely waiting for a publish.
+ # (Confirmed empirically against a real Redis server: a
+ # second call on the same subscription does block for the
+ # full requested timeout.) A single non-looping call here
+ # would therefore make ``wait()`` resolve immediately on
+ # every tick, in production, independent of whether
+ # anything was ever published -- exactly the bug the round-
+ # 6 cancellation regression test caught, for a reason
+ # unrelated to cancellation. Loop against a shared
+ # deadline, re-issuing ``get_message()`` for whatever time
+ # remains, until either a real message arrives (a non-
+ # ``None`` result) or the deadline is reached.
+ deadline = asyncio.get_event_loop().time() + timeout
+ while True:
+ remaining = deadline - asyncio.get_event_loop().time()
+ if remaining <= 0:
+ return
+ message = await _wait_cancellation_safe(
+ pubsub.get_message(ignore_subscribe_messages=True, timeout=remaining),
+ remaining,
+ )
+ if message is not None:
+ return
+ except asyncio.CancelledError:
+ # A bare ``except Exception`` below would not normally catch
+ # this (CancelledError is a BaseException on 3.8+), but some
+ # libraries' internal read/retry loops (e.g. redis-py's
+ # connection handling) can translate an in-flight
+ # cancellation into a different exception type while
+ # unwinding. Re-raise explicitly and unconditionally so a
+ # heartbeat cancellation can never be silently absorbed here
+ # regardless of what the client library does internally.
+ raise
except Exception:
return
finally:
try:
await pubsub.aclose()
+ except asyncio.CancelledError:
+ raise
except Exception:
pass
diff --git a/src/lingxigraph/server/migrations/0002_steering.sql b/src/lingxigraph/server/migrations/0002_steering.sql
new file mode 100644
index 0000000..960c97c
--- /dev/null
+++ b/src/lingxigraph/server/migrations/0002_steering.sql
@@ -0,0 +1,48 @@
+-- Durable mid-run steering inbox (issue #16).
+-- PostgreSQL remains the source of truth; Redis (eventbus) is only an
+-- optional low-latency notify/accelerator layer on top of this table.
+
+CREATE TABLE IF NOT EXISTS {{schema}}.run_steering_events (
+ id TEXT PRIMARY KEY,
+ tenant_id TEXT NOT NULL,
+ run_id TEXT NOT NULL,
+ sequence BIGINT NOT NULL,
+ kind TEXT NOT NULL,
+ payload JSONB NOT NULL DEFAULT '{}'::jsonb,
+ metadata JSONB NOT NULL DEFAULT '{}'::jsonb,
+ idempotency_key TEXT,
+ status TEXT NOT NULL DEFAULT 'pending'
+ CHECK (status IN ('pending', 'delivered', 'consumed', 'superseded')),
+ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+ consumed_at TIMESTAMPTZ,
+ UNIQUE (tenant_id, run_id, sequence)
+);
+
+-- Idempotency: the same (tenant, run, idempotency_key) can only ever
+-- create one durable event, matching the Run-level idempotency pattern.
+CREATE UNIQUE INDEX IF NOT EXISTS run_steering_events_idempotency
+ ON {{schema}}.run_steering_events (tenant_id, run_id, idempotency_key)
+ WHERE idempotency_key IS NOT NULL;
+
+CREATE INDEX IF NOT EXISTS run_steering_events_pending
+ ON {{schema}}.run_steering_events (tenant_id, run_id, sequence)
+ WHERE status IN ('pending', 'delivered');
+
+ALTER TABLE {{schema}}.run_steering_events ENABLE ROW LEVEL SECURITY;
+
+DO $policies$
+BEGIN
+ IF NOT EXISTS (
+ SELECT 1 FROM pg_policies
+ WHERE schemaname='{{schema}}' AND tablename='run_steering_events'
+ AND policyname='tenant_isolation'
+ ) THEN
+ EXECUTE format(
+ 'CREATE POLICY tenant_isolation ON {{schema}}.%I USING '
+ || '(tenant_id = current_setting(''app.tenant_id'', true)) '
+ || 'WITH CHECK (tenant_id = current_setting(''app.tenant_id'', true))',
+ 'run_steering_events'
+ );
+ END IF;
+END
+$policies$;
diff --git a/src/lingxigraph/server/migrations/0003_steering_source_event.sql b/src/lingxigraph/server/migrations/0003_steering_source_event.sql
new file mode 100644
index 0000000..cd4761c
--- /dev/null
+++ b/src/lingxigraph/server/migrations/0003_steering_source_event.sql
@@ -0,0 +1,12 @@
+-- Stable logical identity for paused-run -> resumed-run steering transfers
+-- (issue #16 PR #17 review point 3). ``source_event_id`` points at the
+-- original (now ``superseded``) event's id so a client's ``/steer``
+-- response on a paused run can be correlated end-to-end to the
+-- ``run.steer.consumed`` event eventually emitted under the resumed run.
+
+ALTER TABLE {{schema}}.run_steering_events
+ ADD COLUMN IF NOT EXISTS source_event_id TEXT;
+
+CREATE INDEX IF NOT EXISTS run_steering_events_source
+ ON {{schema}}.run_steering_events (tenant_id, source_event_id)
+ WHERE source_event_id IS NOT NULL;
diff --git a/src/lingxigraph/server/migrations/0004_steering_closed.sql b/src/lingxigraph/server/migrations/0004_steering_closed.sql
new file mode 100644
index 0000000..ea3009a
--- /dev/null
+++ b/src/lingxigraph/server/migrations/0004_steering_closed.sql
@@ -0,0 +1,12 @@
+-- Atomic finalization gate for steering admission (issue #16 PR #17
+-- review round 6, point 3, BLOCKER). Once a worker's graph execution
+-- has ended there is no safe point left for a newly accepted steering
+-- event to ever be consumed -- even though the Run row may still read
+-- ``running``/``cancelling`` while the final flush and finalization
+-- commit are in flight. The owning worker sets this column atomically
+-- (fenced on ``lease_owner``/``attempt``, see
+-- ``PostgresRepository.close_steering``) the instant execution ends, and
+-- ``/steer`` refuses new admission once it is set.
+
+ALTER TABLE {{schema}}.runs
+ ADD COLUMN IF NOT EXISTS steering_closed BOOLEAN NOT NULL DEFAULT FALSE;
diff --git a/src/lingxigraph/server/migrations/0005_superseded_by_run_id.sql b/src/lingxigraph/server/migrations/0005_superseded_by_run_id.sql
new file mode 100644
index 0000000..7dde347
--- /dev/null
+++ b/src/lingxigraph/server/migrations/0005_superseded_by_run_id.sql
@@ -0,0 +1,14 @@
+-- Typed control-plane lineage marker for resumed runs (issue #16 PR #17
+-- review round 14, BLOCKER). ``superseded_by_run_id`` used to live in
+-- ordinary, user-writable ``runs.metadata`` -- but ``/steer``, ``/cancel``,
+-- and the concurrent-resume conflict gate all treat it as authoritative
+-- Runtime control state. A caller could forge
+-- ``metadata.superseded_by_run_id`` at run-creation time to falsely
+-- trigger ``409 run_superseded`` / ``409 run_resume_conflict`` on a run
+-- that was never actually resumed. This column is written exactly once,
+-- atomically, by the resume transaction that creates a descendant run
+-- from a paused run (see ``PostgresRepository`` resume/steer/cancel
+-- sync helpers) and is never derived from user-submitted ``metadata``.
+
+ALTER TABLE {{schema}}.runs
+ ADD COLUMN IF NOT EXISTS superseded_by_run_id TEXT;
diff --git a/src/lingxigraph/server/models.py b/src/lingxigraph/server/models.py
index e21ead4..ad6b7f9 100644
--- a/src/lingxigraph/server/models.py
+++ b/src/lingxigraph/server/models.py
@@ -121,11 +121,68 @@ class Run(ApiModel):
attempt: int = 0
lease_owner: str | None = None
lease_expires_at: datetime | None = None
+ #: Set atomically by the owning worker once graph execution has ended
+ #: and before it starts the final steering flush + finalization commit
+ #: (issue #16 PR #17 review round 6, point 3). Once set, ``/steer``
+ #: must refuse new admission -- there is no safe point left for the
+ #: graph to ever consume a newly-accepted event.
+ steering_closed: bool = False
+ #: Set atomically by the resume transaction that creates a descendant
+ #: run from this (paused) run (issue #16 PR #17 review round 14,
+ #: BLOCKER). This is authoritative control-plane lineage state -- it
+ #: MUST NOT live in user-writable ``metadata``, since a caller could
+ #: otherwise forge it at run-creation time to falsely trigger
+ #: ``409 run_superseded`` / ``409 run_resume_conflict`` on a run that
+ #: was never actually resumed. ``/steer``, ``/cancel``, and the
+ #: concurrent-resume conflict gate all read this field, never
+ #: ``metadata.get("superseded_by_run_id")``.
+ superseded_by_run_id: str | None = None
created_at: datetime = Field(default_factory=utcnow)
started_at: datetime | None = None
finished_at: datetime | None = None
+class SteerCreate(ApiModel):
+ kind: str = "user_input"
+ payload: dict[str, Any] = Field(default_factory=dict)
+ metadata: dict[str, Any] = Field(default_factory=dict)
+ idempotency_key: str | None = None
+
+
+class SteerAccepted(ApiModel):
+ id: str
+ run_id: str
+ sequence: int
+ status: Literal["pending", "delivered", "consumed", "superseded"]
+ kind: str
+ created_at: datetime = Field(default_factory=utcnow)
+
+
+class RunSteeringEvent(ApiModel):
+ id: str = Field(default_factory=lambda: str(uuid4()))
+ tenant_id: str
+ run_id: str
+ sequence: int
+ kind: str
+ payload: dict[str, Any] = Field(default_factory=dict)
+ metadata: dict[str, Any] = Field(default_factory=dict)
+ idempotency_key: str | None = None
+ status: Literal["pending", "delivered", "consumed", "superseded"] = "pending"
+ created_at: datetime = Field(default_factory=utcnow)
+ consumed_at: datetime | None = None
+ #: Stable logical identity across a paused-run -> resumed-run transfer
+ #: (see ``Repository.resume_run_with_pending_steering`` / issue #16 PR
+ #: #17 review point 3). ``None`` for an event that was never
+ #: transferred. When set, this is the ``id`` of the original
+ #: ``superseded`` row the client's ``/steer`` call actually received --
+ #: external callers correlate "the id I got back" to "the id that was
+ #: eventually consumed" via this field rather than the (new,
+ #: post-transfer) ``id``. ``created_at`` on the transferred row is also
+ #: preserved from the original so ``queue_latency_seconds`` includes
+ #: the time an event spent waiting while the run was paused.
+ source_event_id: str | None = None
+
+
class RunEvent(ApiModel):
id: str = Field(default_factory=lambda: str(uuid4()))
tenant_id: str
@@ -197,6 +254,9 @@ class StoreBatchRequest(ApiModel):
"Run",
"RunCreate",
"RunEvent",
+ "RunSteeringEvent",
+ "SteerAccepted",
+ "SteerCreate",
"Schedule",
"ScheduleCreate",
"SchedulePatch",
diff --git a/src/lingxigraph/server/repository.py b/src/lingxigraph/server/repository.py
index 95f7623..43cbdf6 100644
--- a/src/lingxigraph/server/repository.py
+++ b/src/lingxigraph/server/repository.py
@@ -4,11 +4,24 @@
import asyncio
import re
+from collections.abc import Sequence
from dataclasses import dataclass
from datetime import timedelta
from typing import Any
-from ..errors import ConcurrentRunError, IdempotencyConflictError
+from ..errors import (
+ ConcurrentRunError,
+ IdempotencyConflictError,
+ RunFinalizingError,
+ RunResumeConflictError,
+ RunSupersededError,
+ RunTerminalError,
+)
+from ..steering import (
+ MAX_STEERING_PAYLOAD_BYTES,
+ SteeringConsumption,
+ validate_steering_payload,
+)
from ..types import MultitaskStrategy, RunStatus
from .models import (
Assistant,
@@ -18,6 +31,7 @@
Run,
RunCreate,
RunEvent,
+ RunSteeringEvent,
Schedule,
ScheduleCreate,
SchedulePatch,
@@ -60,6 +74,8 @@ def __init__(self, *, limits: RepositoryLimits | None = None) -> None:
self._events: dict[tuple[str, str], list[RunEvent]] = {}
self._schedules: dict[tuple[str, str], Schedule] = {}
self._audits: list[AuditRecord] = []
+ self._steering: dict[tuple[str, str], list[RunSteeringEvent]] = {}
+ self._steering_keys: dict[tuple[str, str, str], str] = {}
self._lock = asyncio.Lock()
self._changed = asyncio.Condition()
@@ -168,83 +184,106 @@ async def create_run(
request_digest: str | None = None,
) -> Run:
async with self._lock:
- if idempotency_key is not None:
- existing = next(
- (
- run
- for run in self._runs.values()
- if run.tenant_id == tenant_id
- and run.idempotency_key == idempotency_key
- ),
- None,
- )
- if existing is not None:
- if existing.request_digest != request_digest:
- raise IdempotencyConflictError(
- "idempotency key was already used for a different run request"
- )
- return existing.model_copy(deep=True)
- tenant_runs = [run for run in self._runs.values() if run.tenant_id == tenant_id]
- active = [run for run in tenant_runs if enum_value(run.status) in ACTIVE]
- queued = [
- run
- for run in tenant_runs
- if enum_value(run.status) == RunStatus.PENDING.value
- ]
- if len(active) >= self.limits.max_active_runs:
- raise ConcurrentRunError("tenant active-run quota exceeded")
- if len(queued) >= self.limits.max_queued_runs:
- raise ConcurrentRunError("tenant queued-run quota exceeded")
- same_thread = [
- run
- for run in active
- if thread_id is not None and run.thread_id == thread_id
- ]
- strategy = MultitaskStrategy(request.multitask_strategy)
- if same_thread and strategy is MultitaskStrategy.REJECT:
- raise ConcurrentRunError("thread already has an active run")
- if same_thread and strategy is MultitaskStrategy.CANCEL_PREVIOUS:
- for run in same_thread:
- self._runs[(tenant_id, run.id)] = run.model_copy(
- update={"status": RunStatus.CANCELLING.value}
- )
- run = Run(
- tenant_id=tenant_id,
- thread_id=thread_id,
- assistant_id=assistant.id,
- graph_id=assistant.graph_id,
- graph_version=assistant.graph_version,
+ run = self._create_run_locked(
+ tenant_id,
+ thread_id,
+ assistant,
+ request,
idempotency_key=idempotency_key,
request_digest=request_digest,
- input=request.input,
- context={**assistant.context, **request.context},
- config={**assistant.config, **request.config},
- metadata=request.metadata,
- resume=request.resume,
- update=request.update,
- goto=request.goto,
- durability=request.durability,
)
- if request.run_timeout is not None:
- run.config["run_timeout"] = request.run_timeout
- run.config.setdefault("max_state_bytes", self.limits.max_state_bytes)
- for budget_name in ("max_model_calls", "max_tool_calls", "max_tokens", "max_cost"):
- budget_value = getattr(request, budget_name)
- if budget_value is not None:
- run.config[budget_name] = budget_value
- self._runs[(tenant_id, run.id)] = run
- self._events[(tenant_id, run.id)] = []
await self._notify()
return run.model_copy(deep=True)
+ def _create_run_locked(
+ self,
+ tenant_id: str,
+ thread_id: str | None,
+ assistant: Assistant,
+ request: RunCreate,
+ *,
+ idempotency_key: str | None = None,
+ request_digest: str | None = None,
+ ) -> Run:
+ """The body of :meth:`create_run` -- assumes ``self._lock`` is held.
+
+ Factored out so :meth:`resume_run_with_pending_steering` can create
+ the resumed Run and migrate its paused-run steering under a single
+ lock acquisition (see issue #16 PR #17 review point 1): otherwise a
+ worker's own lock acquisition (``claim_run``) could interleave
+ between "new Run committed" and "steering migrated", claiming (and
+ even finishing) the new Run before the migration ever happens.
+ """
+
+ if idempotency_key is not None:
+ existing = next(
+ (
+ run
+ for run in self._runs.values()
+ if run.tenant_id == tenant_id and run.idempotency_key == idempotency_key
+ ),
+ None,
+ )
+ if existing is not None:
+ if existing.request_digest != request_digest:
+ raise IdempotencyConflictError(
+ "idempotency key was already used for a different run request"
+ )
+ return existing
+ tenant_runs = [run for run in self._runs.values() if run.tenant_id == tenant_id]
+ active = [run for run in tenant_runs if enum_value(run.status) in ACTIVE]
+ queued = [
+ run for run in tenant_runs if enum_value(run.status) == RunStatus.PENDING.value
+ ]
+ if len(active) >= self.limits.max_active_runs:
+ raise ConcurrentRunError("tenant active-run quota exceeded")
+ if len(queued) >= self.limits.max_queued_runs:
+ raise ConcurrentRunError("tenant queued-run quota exceeded")
+ same_thread = [
+ run for run in active if thread_id is not None and run.thread_id == thread_id
+ ]
+ strategy = MultitaskStrategy(request.multitask_strategy)
+ if same_thread and strategy is MultitaskStrategy.REJECT:
+ raise ConcurrentRunError("thread already has an active run")
+ if same_thread and strategy is MultitaskStrategy.CANCEL_PREVIOUS:
+ for run in same_thread:
+ self._runs[(tenant_id, run.id)] = run.model_copy(
+ update={"status": RunStatus.CANCELLING.value}
+ )
+ run = Run(
+ tenant_id=tenant_id,
+ thread_id=thread_id,
+ assistant_id=assistant.id,
+ graph_id=assistant.graph_id,
+ graph_version=assistant.graph_version,
+ idempotency_key=idempotency_key,
+ request_digest=request_digest,
+ input=request.input,
+ context={**assistant.context, **request.context},
+ config={**assistant.config, **request.config},
+ metadata=request.metadata,
+ resume=request.resume,
+ update=request.update,
+ goto=request.goto,
+ durability=request.durability,
+ )
+ if request.run_timeout is not None:
+ run.config["run_timeout"] = request.run_timeout
+ run.config.setdefault("max_state_bytes", self.limits.max_state_bytes)
+ for budget_name in ("max_model_calls", "max_tool_calls", "max_tokens", "max_cost"):
+ budget_value = getattr(request, budget_name)
+ if budget_value is not None:
+ run.config[budget_name] = budget_value
+ self._runs[(tenant_id, run.id)] = run
+ self._events[(tenant_id, run.id)] = []
+ return run
+
async def get_run(self, tenant_id: str, run_id: str) -> Run | None:
async with self._lock:
value = self._runs.get((tenant_id, run_id))
return value.model_copy(deep=True) if value else None
- async def list_runs(
- self, tenant_id: str, *, thread_id: str | None = None
- ) -> list[Run]:
+ async def list_runs(self, tenant_id: str, *, thread_id: str | None = None) -> list[Run]:
async with self._lock:
values = [
run.model_copy(deep=True)
@@ -254,22 +293,53 @@ async def list_runs(
]
return sorted(values, key=lambda run: run.created_at, reverse=True)
- async def claim_run(
- self, worker_id: str, *, lease_seconds: int = 30
- ) -> Run | None:
+ async def claim_run(self, worker_id: str, *, lease_seconds: int = 30) -> Run | None:
now = utcnow()
async with self._lock:
for key, run in list(self._runs.items()):
- if (
- enum_value(run.status) == RunStatus.RUNNING.value
- and run.lease_expires_at is not None
- and run.lease_expires_at <= now
- ):
+ if run.lease_expires_at is None or run.lease_expires_at > now:
+ continue
+ # Lease-expiry recovery rule (issue #16 PR #17 review round
+ # 11, BLOCKER): an expired ``running`` lease is requeued to
+ # ``pending`` so a new worker can pick the run back up. An
+ # expired ``cancelling`` lease must NOT be requeued the same
+ # way -- that would silently erase the fact that the
+ # caller already asked to stop and could let business
+ # execution resume after cancellation was requested. It
+ # instead resolves straight to the terminal ``cancelled``
+ # state, mirroring the same ``run_cancelled`` disposition
+ # ``finish_run_if_owned``/``retry_run_if_owned`` already use
+ # when cancellation wins during in-process finalization.
+ # This guarantees the run can never be stuck in
+ # ``cancelling`` forever and keeps its thread from being
+ # wedged for other queued runs.
+ if enum_value(run.status) == RunStatus.RUNNING.value:
self._runs[key] = run.model_copy(
update={
"status": RunStatus.PENDING.value,
"lease_owner": None,
"lease_expires_at": None,
+ # A fresh delivery attempt must start with
+ # steering admission open -- ``steering_closed``
+ # only means anything for the attempt that set
+ # it (issue #16 PR #17 review round 6).
+ "steering_closed": False,
+ }
+ )
+ elif enum_value(run.status) == RunStatus.CANCELLING.value:
+ self._runs[key] = run.model_copy(
+ update={
+ "status": RunStatus.CANCELLED.value,
+ "lease_owner": None,
+ "lease_expires_at": None,
+ "finished_at": now,
+ "error": {
+ "code": "run_cancelled",
+ "message": (
+ "run was cancelled but its worker's lease"
+ " expired before finalization completed"
+ ),
+ },
}
)
pending = sorted(
@@ -346,9 +416,292 @@ async def finish_run(
await self._notify()
return updated.model_copy(deep=True)
+ async def finish_run_if_owned(
+ self,
+ tenant_id: str,
+ run_id: str,
+ worker_id: str,
+ attempt: int,
+ status: RunStatus | str,
+ *,
+ output: dict[str, Any] | None = None,
+ error: dict[str, Any] | None = None,
+ ) -> Run | None:
+ """Fenced/CAS'd terminal-status write for a specific lease + attempt.
+
+ Issue #16 PR #17 review round 6, point 1 (BLOCKER): a stale
+ worker whose lease has already expired (and been claimed by a new
+ owner) must never be able to overwrite that new owner's status.
+ This only writes when ``(lease_owner, attempt)`` still match the
+ caller's *and* the run is still in an active status -- if either
+ no longer holds, it returns ``None`` and touches nothing, and the
+ caller must abandon the attempt without any further status
+ mutation.
+
+ Review round 6, point 2 (BLOCKER): re-checking the current
+ control state (``cancelling`` vs ``running``) happens inside this
+ *same* locked decision, atomically with the fence check -- if the
+ run is already ``cancelling``, cancellation wins and the actual
+ write is coerced to ``cancelled`` regardless of the caller's
+ requested ``status``, so a stale in-flight ``succeeded`` intent
+ can never overwrite a cancel that arrived during finalization.
+ """
+
+ async with self._lock:
+ key = (tenant_id, run_id)
+ run = self._runs.get(key)
+ if (
+ run is None
+ or run.lease_owner != worker_id
+ or run.attempt != attempt
+ or enum_value(run.status) not in ACTIVE
+ ):
+ return None
+ final_status = RunStatus(status).value
+ final_output = output
+ final_error = error
+ if (
+ enum_value(run.status) == RunStatus.CANCELLING.value
+ and final_status != RunStatus.CANCELLED.value
+ ):
+ final_status = RunStatus.CANCELLED.value
+ final_output = None
+ final_error = {
+ "code": "run_cancelled",
+ "message": "run was cancelled during finalization",
+ }
+ updated = run.model_copy(
+ update={
+ "status": final_status,
+ "output": final_output,
+ "error": final_error,
+ "finished_at": utcnow(),
+ "lease_owner": None,
+ "lease_expires_at": None,
+ }
+ )
+ self._runs[key] = updated
+ await self._notify()
+ return updated.model_copy(deep=True)
+
+ async def retry_run_if_owned(
+ self,
+ tenant_id: str,
+ run_id: str,
+ worker_id: str,
+ attempt: int,
+ *,
+ error: dict[str, Any] | None = None,
+ ) -> Run | None:
+ """Fenced/CAS'd retry-to-pending write; see ``finish_run_if_owned``.
+
+ A ``cancelling`` run must never be sent back to ``pending`` by a
+ stale worker's retry decision -- cancellation still wins here too,
+ resolving directly to ``cancelled`` instead.
+ """
+
+ async with self._lock:
+ key = (tenant_id, run_id)
+ run = self._runs.get(key)
+ if (
+ run is None
+ or run.lease_owner != worker_id
+ or run.attempt != attempt
+ or enum_value(run.status) not in ACTIVE
+ ):
+ return None
+ if enum_value(run.status) == RunStatus.CANCELLING.value:
+ updated = run.model_copy(
+ update={
+ "status": RunStatus.CANCELLED.value,
+ "error": {
+ "code": "run_cancelled",
+ "message": "run was cancelled during finalization",
+ },
+ "finished_at": utcnow(),
+ "lease_owner": None,
+ "lease_expires_at": None,
+ }
+ )
+ else:
+ updated = run.model_copy(
+ update={
+ "status": RunStatus.PENDING.value,
+ "error": error,
+ "finished_at": None,
+ "lease_owner": None,
+ "lease_expires_at": None,
+ "steering_closed": False,
+ }
+ )
+ self._runs[key] = updated
+ await self._notify()
+ return updated.model_copy(deep=True)
+
+ async def retry_run_with_event_if_owned(
+ self,
+ tenant_id: str,
+ run_id: str,
+ worker_id: str,
+ attempt: int,
+ *,
+ error: dict[str, Any] | None = None,
+ max_attempts: int | None = None,
+ ) -> tuple[Run, RunEvent | None] | None:
+ """Merge the retry-to-pending transition and its ``worker_retrying``
+ lifecycle event into one atomic operation.
+
+ Issue #16 PR #17 review round 10, point 1 (BLOCKER):
+ ``retry_run_if_owned()`` (the status write) and the caller's
+ subsequent ``append_event("worker_retrying")`` used to be two
+ separate durable decisions/transactions. That reopened a TOCTOU
+ window: (race A) a concurrent cancel landing in the gap between the
+ status write committing and the event append could see this worker
+ append a now-stale ``worker_retrying`` event for a run that had
+ already, durably, moved on to ``cancelled``; (race B) a *new*
+ worker could claim the run (``pending -> running``, attempt N+1)
+ in that same gap and log its own attempt-N+1 events *before* the
+ stale attempt-N ``worker_retrying`` event finally landed, inverting
+ the durable event stream's causal order. Folding both writes into
+ one lock critical section (no ``await`` between the status
+ transition and the event append) makes both races impossible: the
+ instant ``pending``/``cancelled`` becomes visible, the matching
+ event (or lack of one, for ``cancelled``) is already durable too.
+
+ Returns ``(updated_run, event)`` where ``event`` is the appended
+ ``worker_retrying`` :class:`RunEvent` when the run actually
+ transitioned to ``pending``, or ``(cancelled_run, None)`` when
+ cancellation won the race instead (mirroring
+ ``retry_run_if_owned``'s existing cancel-wins coercion -- no
+ ``worker_retrying`` event is ever appended for a run that is really
+ ``cancelled``). Returns ``None`` if the lease was no longer owned,
+ identically to ``retry_run_if_owned``.
+ """
+
+ async with self._lock:
+ key = (tenant_id, run_id)
+ run = self._runs.get(key)
+ if (
+ run is None
+ or run.lease_owner != worker_id
+ or run.attempt != attempt
+ or enum_value(run.status) not in ACTIVE
+ ):
+ return None
+ if enum_value(run.status) == RunStatus.CANCELLING.value:
+ updated = run.model_copy(
+ update={
+ "status": RunStatus.CANCELLED.value,
+ "error": {
+ "code": "run_cancelled",
+ "message": "run was cancelled during finalization",
+ },
+ "finished_at": utcnow(),
+ "lease_owner": None,
+ "lease_expires_at": None,
+ }
+ )
+ self._runs[key] = updated
+ event = None
+ else:
+ updated = run.model_copy(
+ update={
+ "status": RunStatus.PENDING.value,
+ "error": error,
+ "finished_at": None,
+ "lease_owner": None,
+ "lease_expires_at": None,
+ "steering_closed": False,
+ }
+ )
+ self._runs[key] = updated
+ run_events = self._events.setdefault(key, [])
+ event = RunEvent(
+ tenant_id=tenant_id,
+ run_id=run_id,
+ sequence=len(run_events) + 1,
+ kind="worker_retrying",
+ data={
+ "attempt": attempt,
+ "max_attempts": max_attempts,
+ "error": error,
+ },
+ )
+ run_events.append(event)
+ await self._notify()
+ return updated.model_copy(deep=True), (event.model_copy(deep=True) if event else None)
+
+ async def close_steering(
+ self, tenant_id: str, run_id: str, worker_id: str, attempt: int
+ ) -> bool:
+ """Atomically close new steering admission for this lease + attempt.
+
+ Issue #16 PR #17 review round 6, point 3 (BLOCKER): once graph
+ execution has ended there is no safe point left for a newly
+ accepted steering event to ever be consumed, even though the Run
+ row may still read ``running``/``cancelling`` while the final
+ flush is in flight. The owning worker calls this the instant
+ execution ends (before the final flush starts) so a concurrent
+ ``/steer`` either lands before this gate closes (and gets synced
+ in as usual, or -- round 7, point 1 -- durably lands but is
+ detected as still-pending by the caller's post-flush disposition
+ call, see below) or after it closes (and is rejected with a
+ stable error) -- never durably accepted into a channel nobody
+ will ever drain again. Fenced the same way as
+ ``finish_run_if_owned``: a stale worker that has already lost its
+ lease cannot close the gate on the new owner's behalf.
+
+ Issue #16 PR #17 review round 7, point 1 (BLOCKER), updated per
+ round 9, point 2: this alone is *not* sufficient to prevent a
+ permanently-pending steering row -- see
+ ``finalize_run_with_steering_disposition_if_owned()``, which the
+ Worker's post-flush finalization calls -- it is what actually
+ catches a row that was durably accepted a moment before this call
+ locked the run (and therefore was never, and can never be,
+ ingested by this attempt's channel), transitioning it to
+ ``superseded`` in the same transaction as the terminal status
+ write. (Round 7 originally described this check as a separate
+ ``Worker._execute`` post-flush ``list_pending_steering`` call;
+ round 9 merged that disposition into
+ ``finalize_run_with_steering_disposition_if_owned()`` itself.)
+ Deliberately *not* checked here: at the instant this call runs,
+ the graph may have already drained events into the channel's local
+ consumption log without those rows having been committed
+ (``status='consumed'``) yet -- that only happens moments later, in
+ the final flush this call precedes. Checking "any pending row
+ exists" here would misidentify perfectly ordinary, about-to-be-
+ flushed consumptions as a permanent race every single time.
+ """
+
+ async with self._lock:
+ key = (tenant_id, run_id)
+ run = self._runs.get(key)
+ if (
+ run is None
+ or run.lease_owner != worker_id
+ or run.attempt != attempt
+ or enum_value(run.status) not in ACTIVE
+ ):
+ return False
+ if not run.steering_closed:
+ self._runs[key] = run.model_copy(update={"steering_closed": True})
+ return True
+
async def retry_run(
self, tenant_id: str, run_id: str, *, error: dict[str, Any] | None = None
) -> Run:
+ """Unfenced admin/back-compat retry-to-pending write.
+
+ Restored per issue #16 PR #17 review round 9, point 5 (compatibility):
+ this is exported public API on both repository classes (see
+ ``lingxigraph.server.__init__``) and existed on ``main`` before this
+ PR -- it must not be deleted to satisfy a coverage gate. The
+ Worker's production finalization path uses only the fenced
+ ``retry_run_if_owned`` (see ``_commit_intent``/``_retry_or_dead_letter``);
+ this method remains available for callers outside that fenced
+ lease/attempt protocol.
+ """
+
async with self._lock:
key = (tenant_id, run_id)
run = self._runs[key]
@@ -359,6 +712,7 @@ async def retry_run(
"finished_at": None,
"lease_owner": None,
"lease_expires_at": None,
+ "steering_closed": False,
}
)
self._runs[key] = updated
@@ -382,6 +736,7 @@ async def redrive_run(self, tenant_id: str, run_id: str) -> Run | None:
"finished_at": None,
"lease_owner": None,
"lease_expires_at": None,
+ "steering_closed": False,
}
)
self._runs[key] = updated
@@ -394,9 +749,27 @@ async def request_cancel(self, tenant_id: str, run_id: str) -> bool:
run = self._runs.get(key)
if run is None or enum_value(run.status) in TERMINAL:
return False
+ # Round 13 review (BLOCKER): a paused run that has since been
+ # resumed is only marked stale via the typed
+ # ``Run.superseded_by_run_id`` field -- its ``status``
+ # deliberately stays ``paused`` so /steer's existing superseded
+ # gate keeps working. Cancel must honor that same signal
+ # *inside this lock*, or a cancel racing behind a resume would
+ # silently "succeed" against the old run while the real
+ # (resumed) descendant keeps executing untouched.
+ # Round 14 review (BLOCKER): this MUST be the typed field, not
+ # user-writable ``metadata`` -- a caller could otherwise forge
+ # ``metadata.superseded_by_run_id`` at run-creation time to
+ # falsely trigger this gate on a run that was never resumed.
+ superseded_by = run.superseded_by_run_id
+ if superseded_by is not None:
+ raise RunSupersededError(
+ f"run {run_id!r} was resumed as {superseded_by!r}; "
+ f"cancel the new run instead"
+ )
status = (
RunStatus.CANCELLED.value
- if enum_value(run.status) == RunStatus.PENDING.value
+ if enum_value(run.status) in {RunStatus.PENDING.value, RunStatus.PAUSED.value}
else RunStatus.CANCELLING.value
)
self._runs[key] = run.model_copy(
@@ -441,39 +814,688 @@ async def list_events(
if event.sequence > after
]
- async def create_schedule(
- self, tenant_id: str, request: ScheduleCreate
- ) -> Schedule:
- schedule = Schedule(tenant_id=tenant_id, **request.model_dump())
+ async def submit_steering(
+ self,
+ tenant_id: str,
+ run_id: str,
+ *,
+ kind: str,
+ payload: dict[str, Any],
+ metadata: dict[str, Any] | None = None,
+ idempotency_key: str | None = None,
+ max_payload_bytes: int = MAX_STEERING_PAYLOAD_BYTES,
+ ) -> tuple[RunSteeringEvent, bool]:
+ """Durably accept a steering event. Returns ``(event, created)``.
+
+ Terminal runs raise :class:`RunTerminalError`. Running, queued,
+ cancelling, and paused runs each accept durably -- the *consuming*
+ safe point differs (see module docstring / issue #16 design doc),
+ but acceptance semantics are uniform: it means "written to the
+ durable inbox", never "the graph has processed it".
+ """
+
+ validate_steering_payload(payload, metadata, max_bytes=max_payload_bytes)
async with self._lock:
- self._schedules[(tenant_id, schedule.id)] = schedule
- return schedule.model_copy(deep=True)
+ run = self._runs.get((tenant_id, run_id))
+ if run is None:
+ raise KeyError(f"run {run_id!r} not found")
+ # Issue #16 PR #17 review round 7, point 3 (BLOCKER): the
+ # idempotency-key replay check must run *before* the terminal/
+ # finalizing/superseded admission gates, not after. A client
+ # that durably submitted this exact event, then lost the 202
+ # response and retried with the same Idempotency-Key after the
+ # run raced ahead to finalizing/terminal/superseded, must still
+ # get back the same event -- that is the entire point of an
+ # idempotency key. Gating replay on "the run is still active"
+ # turns every uncertain-outcome retry into a spurious 409.
+ if idempotency_key is not None:
+ existing_id = self._steering_keys.get((tenant_id, run_id, idempotency_key))
+ if existing_id is not None:
+ existing = next(
+ event
+ for event in self._steering[(tenant_id, run_id)]
+ if event.id == existing_id
+ )
+ # A prior attempt with this idempotency key may have
+ # committed the steering row but then failed before its
+ # ``run.steer.accepted`` event was recorded (review
+ # round 4, point 2) -- repair that gap on the retry
+ # instead of permanently skipping it.
+ self._ensure_steer_accepted_event_locked(tenant_id, run_id, existing)
+ return existing.model_copy(deep=True), False
+ # Only a genuinely new admission decision (no existing row for
+ # this idempotency key -- or none was given) is subject to the
+ # terminal/finalizing/superseded gates.
+ if enum_value(run.status) in TERMINAL:
+ raise RunTerminalError(
+ f"run {run_id!r} is in terminal state "
+ f"{enum_value(run.status)!r} and cannot accept new steering input"
+ )
+ if run.steering_closed:
+ raise RunFinalizingError(
+ f"run {run_id!r} has finished executing and is finalizing; "
+ "no further steering input can be safely consumed"
+ )
+ # Round 14 review (BLOCKER): read the typed field, not
+ # user-writable ``metadata`` -- see ``request_cancel`` above.
+ superseded_by = run.superseded_by_run_id
+ if superseded_by is not None:
+ raise RunSupersededError(
+ f"run {run_id!r} was resumed as {superseded_by!r}; "
+ f"steer the new run instead"
+ )
+ events = self._steering.setdefault((tenant_id, run_id), [])
+ event = RunSteeringEvent(
+ tenant_id=tenant_id,
+ run_id=run_id,
+ sequence=len(events) + 1,
+ kind=kind,
+ payload=dict(payload),
+ metadata=dict(metadata or {}),
+ idempotency_key=idempotency_key,
+ status="pending",
+ )
+ events.append(event)
+ if idempotency_key is not None:
+ self._steering_keys[(tenant_id, run_id, idempotency_key)] = event.id
+ self._ensure_steer_accepted_event_locked(tenant_id, run_id, event)
+ await self._notify()
+ return event.model_copy(deep=True), True
+
+ def _ensure_steer_accepted_event_locked(
+ self,
+ tenant_id: str,
+ run_id: str,
+ steering_event: RunSteeringEvent,
+ *,
+ transferred_from_run_id: str | None = None,
+ ) -> RunEvent:
+ """Idempotently ensure ``run.steer.accepted`` exists for one event.
+
+ Issue #16 PR #17 review round 4, point 2: appending the
+ ``run.steer.accepted`` lifecycle event used to be a second,
+ separately-committed step after ``submit_steering`` (or the
+ paused-resume steering migration) -- a transient failure on that
+ second step, followed by an idempotent-key retry that finds the
+ steering row already created, permanently skipped the event
+ forever. Doing both under the caller's already-held ``self._lock``
+ and keying this on the steering event's id (never appending twice
+ for the same id) makes "accept the steering row" and "record the
+ accepted lifecycle event" a single atomic, idempotent unit -- a
+ retry (of the request, or of this call) always converges on
+ exactly one ``run.steer.accepted`` event per steering event id.
+ Assumes ``self._lock`` is already held.
+ """
+
+ run_events = self._events.setdefault((tenant_id, run_id), [])
+ for existing in run_events:
+ if (
+ existing.kind == "run.steer.accepted"
+ and existing.data.get("steering_event_id") == steering_event.id
+ ):
+ return existing
+ data: dict[str, Any] = {
+ "steering_event_id": steering_event.id,
+ "sequence": steering_event.sequence,
+ "kind": steering_event.kind,
+ }
+ if transferred_from_run_id is not None:
+ data["transferred_from_run_id"] = transferred_from_run_id
+ data["source_event_id"] = steering_event.source_event_id
+ event = RunEvent(
+ tenant_id=tenant_id,
+ run_id=run_id,
+ sequence=len(run_events) + 1,
+ kind="run.steer.accepted",
+ data=data,
+ )
+ run_events.append(event)
+ return event
+
+ async def list_pending_steering(
+ self, tenant_id: str, run_id: str
+ ) -> list[RunSteeringEvent]:
+ """Read-only view of never-consumed steering events, in order."""
- async def list_schedules(self, tenant_id: str) -> list[Schedule]:
async with self._lock:
return [
- value.model_copy(deep=True)
- for (tenant, _), value in self._schedules.items()
- if tenant == tenant_id
+ event.model_copy(deep=True)
+ for event in self._steering.get((tenant_id, run_id), ())
+ if event.status in ("pending", "delivered")
]
- async def patch_schedule(
- self, tenant_id: str, schedule_id: str, request: SchedulePatch
- ) -> Schedule | None:
+ def _ensure_steer_superseded_event_locked(
+ self,
+ tenant_id: str,
+ run_id: str,
+ steering_event: RunSteeringEvent,
+ *,
+ reason: str,
+ superseded_by_run_id: str | None = None,
+ replacement_steering_event_id: str | None = None,
+ ) -> RunEvent:
+ """Idempotently ensure ``run.steer.superseded`` exists for one event.
+
+ Issue #16 PR #17 review round 8, point 3 (BLOCKER): issue #16's
+ observability section requires a durable ``run.steer.superseded``
+ lifecycle event for every steering row that transitions to the
+ ``superseded`` disposition -- both the paused-run resume-transfer
+ case and the late-boundary-race disposition (round 8, point 1).
+ Keyed on the steering event's id, never appending twice for the
+ same id, so a retried transaction (e.g. a resume attempt retried
+ after a transient failure, or a repeated late-race supersede call)
+ converges on exactly one event per steering event id -- the same
+ idempotency discipline as
+ :meth:`_ensure_steer_accepted_event_locked`. Assumes ``self._lock``
+ is already held.
+ """
+
+ run_events = self._events.setdefault((tenant_id, run_id), [])
+ for existing in run_events:
+ if (
+ existing.kind == "run.steer.superseded"
+ and existing.data.get("steering_event_id") == steering_event.id
+ ):
+ return existing
+ data: dict[str, Any] = {
+ "steering_event_id": steering_event.id,
+ "source_event_id": steering_event.source_event_id or steering_event.id,
+ # Issue #16 PR #17 review round 9, point 4 (REQUIRED): every
+ # other steering lifecycle event (``run.steer.accepted``,
+ # ``run.steer.consumed``) already carries the steering event's
+ # own ``sequence``/``kind`` -- ``run.steer.superseded`` was
+ # missing both, which issue #16's observability section
+ # requires at minimum for every steering lifecycle event.
+ "sequence": steering_event.sequence,
+ "kind": steering_event.kind,
+ "reason": reason,
+ "superseded_by_run_id": superseded_by_run_id,
+ "replacement_steering_event_id": replacement_steering_event_id,
+ }
+ event = RunEvent(
+ tenant_id=tenant_id,
+ run_id=run_id,
+ sequence=len(run_events) + 1,
+ kind="run.steer.superseded",
+ data=data,
+ )
+ run_events.append(event)
+ return event
+
+ async def finalize_run_with_steering_disposition_if_owned(
+ self,
+ tenant_id: str,
+ run_id: str,
+ worker_id: str,
+ attempt: int,
+ status: RunStatus | str,
+ *,
+ output: dict[str, Any] | None = None,
+ error: dict[str, Any] | None = None,
+ supersede_pending: bool = True,
+ supersede_reason: str = "unconsumed_at_final_boundary",
+ ) -> tuple[Run, list[RunEvent]] | None:
+ """Supersede leftover steering AND commit the Run's terminal status
+ as one durable unit.
+
+ Issue #16 PR #17 review round 9, point 2 (BLOCKER): round 8's
+ terminal path called ``supersede_pending_steering_if_owned()`` (one
+ fenced write, committing the ``superseded`` dispositions and their
+ ``run.steer.superseded`` events) and then, moments later and
+ separately, ``finish_run_if_owned()`` (a second fenced write for the
+ Run's terminal status). Each write was individually fenced and
+ correct in isolation, but they were not the same durable boundary:
+ a crash, a failed DB call, or a reclaimed lease between the two
+ left steering permanently ``superseded`` while the Run itself never
+ actually reached a terminal status -- the next owner would see the
+ Run back in an active state with admission reopened, but the
+ "leftover" steering the previous attempt already disposed of could
+ never be recovered, even though that disposition's premise (the Run
+ is finishing right now) turned out to be false.
+
+ This method makes only two outcomes possible: either everything
+ below rolls back together (a subsequent owner still sees the
+ steering rows as ``pending``/``delivered``, fully recoverable), or
+ the steering disposition and the terminal Run status commit
+ together, in the exact same critical section -- for
+ ``InMemoryRepository`` that is one lock acquisition with no
+ intervening ``await``; :class:`PostgresRepository` overrides this
+ with one real SQL transaction. Fenced identically to
+ ``finish_run_if_owned``/``supersede_pending_steering_if_owned``: a
+ ``None`` result means the caller's lease was already reclaimed by a
+ new owner, and nothing here was written. The cancel-wins re-check
+ from ``finish_run_if_owned`` still applies atomically inside this
+ same critical section.
+
+ Returns ``(updated_run, superseded_events)`` on success, or
+ ``None`` if the lease was no longer owned.
+ """
+
async with self._lock:
- key = (tenant_id, schedule_id)
- current = self._schedules.get(key)
- if current is None:
+ key = (tenant_id, run_id)
+ run = self._runs.get(key)
+ if (
+ run is None
+ or run.lease_owner != worker_id
+ or run.attempt != attempt
+ or enum_value(run.status) not in ACTIVE
+ ):
return None
- values = request.model_dump(exclude_none=True)
- values["updated_at"] = utcnow()
- updated = current.model_copy(update=values, deep=True)
- self._schedules[key] = updated
- return updated.model_copy(deep=True)
+ superseded_events: list[RunEvent] = []
+ if supersede_pending:
+ events = self._steering.get((tenant_id, run_id), [])
+ for index, event in enumerate(events):
+ if event.status in ("pending", "delivered"):
+ events[index] = event.model_copy(update={"status": "superseded"})
+ superseded_events.append(
+ self._ensure_steer_superseded_event_locked(
+ tenant_id,
+ run_id,
+ event,
+ reason=supersede_reason,
+ )
+ )
+ final_status = RunStatus(status).value
+ final_output = output
+ final_error = error
+ if (
+ enum_value(run.status) == RunStatus.CANCELLING.value
+ and final_status != RunStatus.CANCELLED.value
+ ):
+ final_status = RunStatus.CANCELLED.value
+ final_output = None
+ final_error = {
+ "code": "run_cancelled",
+ "message": "run was cancelled during finalization",
+ }
+ updated = run.model_copy(
+ update={
+ "status": final_status,
+ "output": final_output,
+ "error": final_error,
+ "finished_at": utcnow(),
+ "lease_owner": None,
+ "lease_expires_at": None,
+ }
+ )
+ self._runs[key] = updated
+ await self._notify()
+ return updated.model_copy(deep=True), superseded_events
+
+ async def supersede_pending_steering_if_owned(
+ self, tenant_id: str, run_id: str, worker_id: str, attempt: int
+ ) -> list[RunEvent] | None:
+ """Durably supersede any steering row still pending/delivered.
+
+ Issue #16 PR #17 review round 8, point 1 (BLOCKER) -- CHOSEN FIX:
+ "seal admission at the safe boundary + supersede late events".
+ ``close_steering()`` already seals *new* admission the instant
+ graph execution ends; this closes the remaining gap round 7's fix
+ mishandled -- a steering event that durably lands *after* the
+ graph's last safe point but *before* (or racing) the seal itself.
+ Rather than forcing the run through an indefinite retry/dead-letter
+ loop (round 7's approach, which hardcoded an application decision
+ -- "does undrained steering mean rerun?" -- into the Runtime, in
+ violation of issue #16's Runtime/application boundary), any row
+ still ``pending``/``delivered`` once the final flush has completed
+ is durably transitioned to ``superseded`` here, in the same
+ lease/attempt-fenced call as everything else this finalization
+ window does. The run is then free to finalize with its originally
+ computed intent -- a graph that legitimately never calls
+ ``drain_steering()`` is not penalized just because a steering
+ event happened to exist.
+
+ Fenced exactly like ``commit_steering_consumptions_if_owned``: a
+ ``None`` result means this worker's lease was already reclaimed by
+ a new owner between the final flush and this call, and the caller
+ must not treat any of this run's finalization as decided by this
+ attempt.
+ """
- async def delete_schedule(self, tenant_id: str, schedule_id: str) -> bool:
async with self._lock:
- return self._schedules.pop((tenant_id, schedule_id), None) is not None
+ run = self._runs.get((tenant_id, run_id))
+ if (
+ run is None
+ or run.lease_owner != worker_id
+ or run.attempt != attempt
+ or enum_value(run.status) not in ACTIVE
+ ):
+ return None
+ events = self._steering.get((tenant_id, run_id), [])
+ stale = [event for event in events if event.status in ("pending", "delivered")]
+ if not stale:
+ return []
+ stored: list[RunEvent] = []
+ for index, event in enumerate(events):
+ if event.status in ("pending", "delivered"):
+ events[index] = event.model_copy(update={"status": "superseded"})
+ stored.append(
+ self._ensure_steer_superseded_event_locked(
+ tenant_id,
+ run_id,
+ event,
+ reason="unconsumed_at_final_boundary",
+ )
+ )
+ await self._notify()
+ return stored
+
+ async def mark_steering_consumed(
+ self, tenant_id: str, run_id: str, event_ids: Sequence[str]
+ ) -> None:
+ """Mark events consumed once the graph has actually drained them."""
+
+ if not event_ids:
+ return
+ ids = set(event_ids)
+ async with self._lock:
+ events = self._steering.get((tenant_id, run_id), [])
+ for index, event in enumerate(events):
+ if event.id in ids and event.status != "consumed":
+ events[index] = event.model_copy(
+ update={"status": "consumed", "consumed_at": utcnow()}
+ )
+
+ async def commit_steering_consumptions(
+ self,
+ tenant_id: str,
+ run_id: str,
+ consumptions: Sequence[SteeringConsumption],
+ ) -> list[RunEvent]:
+ """Durably record consumption (status + lifecycle event), atomically.
+
+ Issue #16 PR #17 review point 2: ``mark_steering_consumed`` plus a
+ separate ``append_event("run.steer.consumed")`` call left a window
+ where a caller that destructively popped its local consumption log
+ *before* calling either could lose the record forever on a
+ transient failure -- the DB row could end up stuck ``pending``
+ forever (no ``on_drain``/dedup path re-surfaces it once
+ ``SteeringChannel._seen_ids`` has recorded the id), or the status
+ could flip to ``consumed`` while the ``run.steer.consumed``
+ observability event silently never appears. This method updates
+ both under one lock acquisition so a caller (see
+ ``Worker._sync_steering_out``) can safely defer clearing its own
+ local bookkeeping until *after* this succeeds, and retry on the
+ next safe point if it raises.
+ """
+
+ if not consumptions:
+ return []
+ async with self._lock:
+ stored = self._commit_steering_consumptions_locked(tenant_id, run_id, consumptions)
+ await self._notify()
+ return stored
+
+ async def commit_steering_consumptions_if_owned(
+ self,
+ tenant_id: str,
+ run_id: str,
+ worker_id: str,
+ attempt: int,
+ consumptions: Sequence[SteeringConsumption],
+ ) -> list[RunEvent] | None:
+ """Fenced counterpart of :meth:`commit_steering_consumptions`.
+
+ Issue #16 PR #17 review round 7, point 2 (BLOCKER): the unfenced
+ version only keys on ``(tenant_id, run_id, steering_ids)`` -- it
+ never checks ``lease_owner``/``attempt``. A worker whose lease has
+ already been reclaimed by a new owner (attempt N+1) can still
+ durably commit a stale consumption from attempt N *after* the new
+ owner has already re-ingested the same still-pending row into its
+ own channel and started processing it, corrupting the steering
+ inbox/lifecycle even though the stale worker's later
+ ``finish_run_if_owned``/``retry_run_if_owned`` call is correctly
+ rejected.
+
+ This checks ``lease_owner == worker_id``, ``attempt == attempt``,
+ and the run is still active, inside the *same* lock acquisition as
+ the actual status/event writes -- on any mismatch it returns
+ ``None`` and touches nothing, so the caller must not locally ack
+ either.
+ """
+
+ if not consumptions:
+ return []
+ async with self._lock:
+ run = self._runs.get((tenant_id, run_id))
+ if (
+ run is None
+ or run.lease_owner != worker_id
+ or run.attempt != attempt
+ or enum_value(run.status) not in ACTIVE
+ ):
+ return None
+ stored = self._commit_steering_consumptions_locked(tenant_id, run_id, consumptions)
+ await self._notify()
+ return stored
+
+ def _commit_steering_consumptions_locked(
+ self,
+ tenant_id: str,
+ run_id: str,
+ consumptions: Sequence[SteeringConsumption],
+ ) -> list[RunEvent]:
+ """Body of the consumption commit; assumes ``self._lock`` is held."""
+
+ ids = {consumption.event.id for consumption in consumptions}
+ events = self._steering.get((tenant_id, run_id), [])
+ transitioned: set[str] = set()
+ for index, event in enumerate(events):
+ if event.id in ids and event.status != "consumed":
+ events[index] = event.model_copy(
+ update={"status": "consumed", "consumed_at": utcnow()}
+ )
+ transitioned.add(event.id)
+ run_events = self._events.setdefault((tenant_id, run_id), [])
+ stored: list[RunEvent] = []
+ for consumption in consumptions:
+ steering_event = consumption.event
+ if steering_event.id not in transitioned:
+ # Already consumed by a previous call (idempotent
+ # retry, e.g. a worker that re-sends the same batch
+ # after an ack it never observed) -- do not emit a
+ # second, semantically duplicate ``run.steer.consumed``
+ # lifecycle event for it (issue #16 PR #17 review
+ # point 3).
+ continue
+ data: dict[str, Any] = {
+ "steering_event_id": steering_event.id,
+ "sequence": steering_event.sequence,
+ "kind": steering_event.kind,
+ "queue_latency_seconds": consumption.queue_latency_seconds,
+ "node": consumption.node,
+ "namespace": list(consumption.namespace),
+ "task_id": consumption.task_id,
+ }
+ if steering_event.source_event_id is not None:
+ data["source_event_id"] = steering_event.source_event_id
+ stored_event = RunEvent(
+ tenant_id=tenant_id,
+ run_id=run_id,
+ sequence=len(run_events) + 1,
+ kind="run.steer.consumed",
+ data=data,
+ )
+ run_events.append(stored_event)
+ stored.append(stored_event.model_copy(deep=True))
+ return stored
+
+ async def list_steering(self, tenant_id: str, run_id: str) -> list[RunSteeringEvent]:
+ async with self._lock:
+ return [
+ event.model_copy(deep=True)
+ for event in self._steering.get((tenant_id, run_id), ())
+ ]
+
+ def _transfer_pending_steering_locked(
+ self, tenant_id: str, old_run_id: str, new_run_id: str
+ ) -> list[RunSteeringEvent]:
+ """The body of the old-run -> new-run steering migration.
+
+ Assumes ``self._lock`` is already held -- only called from
+ :meth:`resume_run_with_pending_steering`, which creates the new Run
+ and performs this migration under one lock acquisition (see issue
+ #16 PR #17 review point 1's "not atomic" finding: this must never
+ run as a second, separately-lockable step after the new Run is
+ already visible to :meth:`claim_run`).
+
+ Every event still ``pending``/``delivered`` under ``old_run_id`` is
+ marked ``superseded`` in place (its terminal state -- it was never
+ drained by the paused run) and re-inserted as a fresh ``pending``
+ event under ``new_run_id``, preserving order, ``kind``, ``payload``,
+ ``metadata`` and ``idempotency_key``. The new row's ``created_at``
+ and ``source_event_id`` preserve the original event's identity
+ (review point 3): external callers correlate the id a paused-run
+ ``/steer`` call returned to the eventual ``consumed`` event via
+ ``source_event_id``, and ``queue_latency_seconds`` computed from the
+ preserved ``created_at`` includes the time spent waiting while
+ paused, not just time since the transfer.
+ """
+
+ old_run = self._runs.get((tenant_id, old_run_id))
+ if old_run is not None:
+ # Round 14 review (BLOCKER): write the typed field, not
+ # user-writable ``metadata`` -- see ``request_cancel`` above.
+ self._runs[(tenant_id, old_run_id)] = old_run.model_copy(
+ update={"superseded_by_run_id": new_run_id}
+ )
+ old_events = self._steering.get((tenant_id, old_run_id), [])
+ pending = sorted(
+ (event for event in old_events if event.status in ("pending", "delivered")),
+ key=lambda event: event.sequence,
+ )
+ if not pending:
+ return []
+ for index, event in enumerate(old_events):
+ if event.status in ("pending", "delivered"):
+ old_events[index] = event.model_copy(update={"status": "superseded"})
+ new_events = self._steering.setdefault((tenant_id, new_run_id), [])
+ transferred: list[RunSteeringEvent] = []
+ for event in pending:
+ new_event = RunSteeringEvent(
+ tenant_id=tenant_id,
+ run_id=new_run_id,
+ sequence=len(new_events) + 1,
+ kind=event.kind,
+ payload=dict(event.payload),
+ metadata=dict(event.metadata),
+ idempotency_key=event.idempotency_key,
+ status="pending",
+ source_event_id=event.source_event_id or event.id,
+ created_at=event.created_at,
+ )
+ new_events.append(new_event)
+ if event.idempotency_key is not None:
+ self._steering_keys[(tenant_id, new_run_id, event.idempotency_key)] = (
+ new_event.id
+ )
+ # Record the transferred event's ``run.steer.accepted`` inside
+ # this same locked migration (review round 4, point 2) instead
+ # of leaving it to a second, separately-committed
+ # ``append_event`` call after the endpoint returns from this
+ # method -- otherwise a transient failure of that later call
+ # permanently loses the accepted record for the transferred run.
+ self._ensure_steer_accepted_event_locked(
+ tenant_id, new_run_id, new_event, transferred_from_run_id=old_run_id
+ )
+ # Issue #16 PR #17 review round 8, point 3 (BLOCKER): the old
+ # row's transition to ``superseded`` above must be paired with
+ # a durable ``run.steer.superseded`` lifecycle event, recorded
+ # in this same locked migration -- otherwise an SSE/RunEvent-
+ # only client has no way to learn the old row stopped being
+ # pending or which new run/event now owns it.
+ self._ensure_steer_superseded_event_locked(
+ tenant_id,
+ old_run_id,
+ event,
+ reason="resume_transfer",
+ superseded_by_run_id=new_run_id,
+ replacement_steering_event_id=new_event.id,
+ )
+ transferred.append(new_event.model_copy(deep=True))
+ return transferred
+
+ async def resume_run_with_pending_steering(
+ self,
+ tenant_id: str,
+ thread_id: str | None,
+ assistant: Assistant,
+ request: RunCreate,
+ old_run_id: str,
+ ) -> tuple[Run, list[RunSteeringEvent]]:
+ """Atomically create the resumed Run and migrate paused steering.
+
+ Issue #16 PR #17 review point 1: creating the resumed Run and
+ transferring its predecessor's still-pending steering used to be
+ two separately-locked operations (``create_run`` then
+ ``transfer_pending_steering``); a worker could claim -- and even
+ finish -- the brand-new ``pending`` Run in the window between them,
+ so the migrated steering could land after nobody was left to
+ consume it. Doing both under one ``self._lock`` acquisition (no
+ ``await`` in between) means no other coroutine, including
+ ``claim_run``, can observe the new Run row before its steering
+ migration has also completed.
+
+ Issue #16 PR #17 review round 4, point 1: the endpoint's own
+ ``GET old run -> status == paused`` pre-check happens *before*
+ this call and is therefore not atomic with it -- two concurrent
+ resume requests can both pass it. Re-validate, inside this same
+ lock acquisition, that the old Run is still ``paused`` and has no
+ ``superseded_by_run_id`` set yet; the loser of the race gets
+ :class:`RunResumeConflictError` instead of silently creating a
+ second descendant Run and overwriting the winner's
+ ``superseded_by_run_id``.
+ """
+
+ async with self._lock:
+ old_run = self._runs.get((tenant_id, old_run_id))
+ if (
+ old_run is None
+ or enum_value(old_run.status) != RunStatus.PAUSED.value
+ # Round 14 review (BLOCKER): typed field, not user-writable
+ # ``metadata`` -- see ``request_cancel`` above.
+ or old_run.superseded_by_run_id is not None
+ ):
+ raise RunResumeConflictError(
+ f"run {old_run_id!r} is no longer resumable "
+ "(it is not paused, or has already been resumed by a concurrent request)"
+ )
+ run = self._create_run_locked(tenant_id, thread_id, assistant, request)
+ transferred = self._transfer_pending_steering_locked(tenant_id, old_run_id, run.id)
+ await self._notify()
+ return run.model_copy(deep=True), transferred
+
+ async def create_schedule(self, tenant_id: str, request: ScheduleCreate) -> Schedule:
+ schedule = Schedule(tenant_id=tenant_id, **request.model_dump())
+ async with self._lock:
+ self._schedules[(tenant_id, schedule.id)] = schedule
+ return schedule.model_copy(deep=True)
+
+ async def list_schedules(self, tenant_id: str) -> list[Schedule]:
+ async with self._lock:
+ return [
+ value.model_copy(deep=True)
+ for (tenant, _), value in self._schedules.items()
+ if tenant == tenant_id
+ ]
+
+ async def patch_schedule(
+ self, tenant_id: str, schedule_id: str, request: SchedulePatch
+ ) -> Schedule | None:
+ async with self._lock:
+ key = (tenant_id, schedule_id)
+ current = self._schedules.get(key)
+ if current is None:
+ return None
+ values = request.model_dump(exclude_none=True)
+ values["updated_at"] = utcnow()
+ updated = current.model_copy(update=values, deep=True)
+ self._schedules[key] = updated
+ return updated.model_copy(deep=True)
+
+ async def delete_schedule(self, tenant_id: str, schedule_id: str) -> bool:
+ async with self._lock:
+ return self._schedules.pop((tenant_id, schedule_id), None) is not None
async def audit(self, record: AuditRecord) -> None:
async with self._lock:
@@ -493,12 +1515,8 @@ async def stats(self, tenant_id: str) -> dict[str, Any]:
if tenant == tenant_id
),
"threads": sum(tenant == tenant_id for tenant, _ in self._threads),
- "assistants": sum(
- tenant == tenant_id for tenant, _ in self._assistants
- ),
- "schedules": sum(
- tenant == tenant_id for tenant, _ in self._schedules
- ),
+ "assistants": sum(tenant == tenant_id for tenant, _ in self._assistants),
+ "schedules": sum(tenant == tenant_id for tenant, _ in self._schedules),
}
async def wait_for_change(self, timeout: float = 1.0) -> None:
@@ -538,7 +1556,9 @@ def __init__(
from psycopg.rows import dict_row
from psycopg.types.json import Jsonb
except ImportError as exc: # pragma: no cover
- raise RuntimeError("install lingxigraph[postgres] to use PostgresRepository") from exc
+ raise RuntimeError(
+ "install lingxigraph[postgres] to use PostgresRepository"
+ ) from exc
self._psycopg = psycopg
self._dict_row = dict_row
self._jsonb = Jsonb
@@ -565,14 +1585,18 @@ def _healthcheck_sync(self) -> bool:
def _setup_sync(self) -> None:
from importlib.resources import files
- migration = (
- files("lingxigraph.server")
- .joinpath("migrations/0001_v1.sql")
- .read_text(encoding="utf-8")
- .replace("{{schema}}", self._schema)
+ migrations_dir = files("lingxigraph.server").joinpath("migrations")
+ names = sorted(
+ entry.name for entry in migrations_dir.iterdir() if entry.name.endswith(".sql")
)
with self._connect() as conn, conn.cursor() as cursor:
- cursor.execute(migration)
+ for name in names:
+ migration = (
+ migrations_dir.joinpath(name)
+ .read_text(encoding="utf-8")
+ .replace("{{schema}}", self._schema)
+ )
+ cursor.execute(migration)
@staticmethod
def _tenant(cursor, tenant_id: str) -> None:
@@ -898,9 +1922,7 @@ def _heartbeat_sync(self, tenant_id, run_id, worker_id, lease_seconds):
return cursor.rowcount > 0
async def finish_run(self, tenant_id, run_id, status, *, output=None, error=None):
- await asyncio.to_thread(
- self._finish_run_sync, tenant_id, run_id, status, output, error
- )
+ await asyncio.to_thread(self._finish_run_sync, tenant_id, run_id, status, output, error)
value = await self.get_run(tenant_id, run_id)
assert value is not None
return value
@@ -915,6 +1937,282 @@ async def redrive_run(self, tenant_id, run_id):
changed = await asyncio.to_thread(self._retry_run_sync, tenant_id, run_id, None, True)
return await self.get_run(tenant_id, run_id) if changed else None
+ async def finish_run_if_owned(
+ self, tenant_id, run_id, worker_id, attempt, status, *, output=None, error=None
+ ) -> Run | None:
+ row = await asyncio.to_thread(
+ self._finish_run_if_owned_sync,
+ tenant_id,
+ run_id,
+ worker_id,
+ attempt,
+ status,
+ output,
+ error,
+ )
+ if row is None:
+ return None
+ return self._run_from_row(row)
+
+ def _finish_run_if_owned_sync(
+ self, tenant_id, run_id, worker_id, attempt, status, output, error
+ ): # pragma: no cover -- exercised only against a live Postgres in the `postgres` CI job, not `quality`'s coverage gate
+ # Fenced/CAS'd terminal-status write (issue #16 PR #17 review round
+ # 6, point 1, BLOCKER). Selecting the row ``FOR UPDATE`` first --
+ # filtered on lease_owner/attempt/status, mirroring the pattern
+ # ``_submit_steering_sync``/``_append_event_sync`` already use --
+ # then issuing a plain follow-up ``UPDATE`` inside the same
+ # transaction is deliberately simpler (and easier to verify by
+ # inspection without a live Postgres) than trying to express the
+ # fencing *and* the cancellation-wins re-check as one combined
+ # CASE-heavy UPDATE statement.
+ with self._connect() as conn, conn.cursor() as cursor:
+ self._tenant(cursor, tenant_id)
+ cursor.execute(
+ f"""SELECT status FROM {self._schema}.runs
+ WHERE tenant_id=%s AND id=%s AND lease_owner=%s AND attempt=%s
+ AND status IN ('running','cancelling') FOR UPDATE""",
+ (tenant_id, run_id, worker_id, attempt),
+ )
+ row = cursor.fetchone()
+ if row is None:
+ return False
+ final_status = RunStatus(status).value
+ final_output = output
+ final_error = error
+ # Review round 6, point 2 (BLOCKER): re-check the current
+ # control state inside this same locked transaction --
+ # cancellation always wins over a stale non-cancel intent.
+ if row["status"] == "cancelling" and final_status != RunStatus.CANCELLED.value:
+ final_status = RunStatus.CANCELLED.value
+ final_output = None
+ final_error = {
+ "code": "run_cancelled",
+ "message": "run was cancelled during finalization",
+ }
+ # Issue #16 PR #17 review round 10, point 2 (REQUIRED): use
+ # ``RETURNING *`` so the row this async wrapper hands back to
+ # the caller is the *same transaction's own* resulting
+ # snapshot, not a separate, later ``get_run()`` read that a
+ # concurrent write (e.g. a racing ``/redrive`` or unfenced
+ # ``retry_run()``) could observe a different value through.
+ cursor.execute(
+ f"""UPDATE {self._schema}.runs SET status=%s, output=%s, error=%s,
+ finished_at=NOW(), lease_owner=NULL, lease_expires_at=NULL
+ WHERE tenant_id=%s AND id=%s RETURNING *""",
+ (
+ final_status,
+ self._jsonb(final_output) if final_output is not None else None,
+ self._jsonb(final_error) if final_error is not None else None,
+ tenant_id,
+ run_id,
+ ),
+ )
+ return cursor.fetchone()
+
+ async def retry_run_if_owned(
+ self, tenant_id, run_id, worker_id, attempt, *, error=None
+ ) -> Run | None:
+ row = await asyncio.to_thread(
+ self._retry_run_if_owned_sync, tenant_id, run_id, worker_id, attempt, error
+ )
+ if row is None:
+ return None
+ return self._run_from_row(row)
+
+ def _retry_run_if_owned_sync(
+ self, tenant_id, run_id, worker_id, attempt, error
+ ): # pragma: no cover -- Postgres-only, covered by the `postgres` CI job
+ with self._connect() as conn, conn.cursor() as cursor:
+ self._tenant(cursor, tenant_id)
+ cursor.execute(
+ f"""SELECT status FROM {self._schema}.runs
+ WHERE tenant_id=%s AND id=%s AND lease_owner=%s AND attempt=%s
+ AND status IN ('running','cancelling') FOR UPDATE""",
+ (tenant_id, run_id, worker_id, attempt),
+ )
+ row = cursor.fetchone()
+ if row is None:
+ return None
+ # Issue #16 PR #17 review round 10, point 2 (REQUIRED): return
+ # the transaction's own ``RETURNING *`` snapshot, not a
+ # separate later ``get_run()`` read (see the matching comment
+ # on ``_finish_run_if_owned_sync``).
+ if row["status"] == "cancelling":
+ # Cancellation wins: never send a cancelling run back to
+ # pending because of a stale worker's retry decision.
+ cursor.execute(
+ f"""UPDATE {self._schema}.runs SET status='cancelled', error=%s,
+ finished_at=NOW(), lease_owner=NULL, lease_expires_at=NULL
+ WHERE tenant_id=%s AND id=%s RETURNING *""",
+ (
+ self._jsonb(
+ {
+ "code": "run_cancelled",
+ "message": "run was cancelled during finalization",
+ }
+ ),
+ tenant_id,
+ run_id,
+ ),
+ )
+ else:
+ cursor.execute(
+ f"""UPDATE {self._schema}.runs SET status='pending',
+ error=%s, finished_at=NULL, lease_owner=NULL, lease_expires_at=NULL,
+ steering_closed=FALSE
+ WHERE tenant_id=%s AND id=%s RETURNING *""",
+ (self._jsonb(error) if error is not None else None, tenant_id, run_id),
+ )
+ return cursor.fetchone()
+
+ async def retry_run_with_event_if_owned(
+ self,
+ tenant_id,
+ run_id,
+ worker_id,
+ attempt,
+ *,
+ error=None,
+ max_attempts=None,
+ ) -> tuple[Run, RunEvent | None] | None:
+ """SQL counterpart of
+ :meth:`InMemoryRepository.retry_run_with_event_if_owned` (issue
+ #16 PR #17 review round 10, point 1, BLOCKER). One real PostgreSQL
+ transaction covers: locking the run row, re-checking the fence and
+ control state (cancel-wins), writing the retry-to-pending (or
+ cancel-wins) status transition, and -- only when the transition
+ actually lands on ``pending`` -- inserting the matching
+ ``worker_retrying`` :class:`RunEvent` in the *same* transaction, so
+ both the status write and the event become externally visible
+ together or not at all. This closes the race a separate
+ ``retry_run_if_owned()`` + ``append_event()`` pair left open: a
+ concurrent cancel or a new worker's claim landing in the gap
+ between two independently-committed writes could previously
+ observe (or produce) a stale/out-of-order ``worker_retrying``
+ event.
+ """
+
+ result = await asyncio.to_thread(
+ self._retry_run_with_event_if_owned_sync,
+ tenant_id,
+ run_id,
+ worker_id,
+ attempt,
+ error,
+ max_attempts,
+ )
+ if result is None:
+ return None
+ row, event = result
+ return self._run_from_row(row), event
+
+ def _retry_run_with_event_if_owned_sync( # pragma: no cover -- Postgres-only, covered by the `postgres` CI job
+ self, tenant_id, run_id, worker_id, attempt, error, max_attempts
+ ):
+ with self._connect() as conn, conn.cursor() as cursor:
+ self._tenant(cursor, tenant_id)
+ cursor.execute(
+ f"""SELECT status FROM {self._schema}.runs
+ WHERE tenant_id=%s AND id=%s AND lease_owner=%s AND attempt=%s
+ AND status IN ('running','cancelling') FOR UPDATE""",
+ (tenant_id, run_id, worker_id, attempt),
+ )
+ row = cursor.fetchone()
+ if row is None:
+ return None
+ if row["status"] == "cancelling":
+ # Cancellation wins: never send a cancelling run back to
+ # pending, and never emit a ``worker_retrying`` event for a
+ # run that is really cancelled.
+ cursor.execute(
+ f"""UPDATE {self._schema}.runs SET status='cancelled', error=%s,
+ finished_at=NOW(), lease_owner=NULL, lease_expires_at=NULL
+ WHERE tenant_id=%s AND id=%s RETURNING *""",
+ (
+ self._jsonb(
+ {
+ "code": "run_cancelled",
+ "message": "run was cancelled during finalization",
+ }
+ ),
+ tenant_id,
+ run_id,
+ ),
+ )
+ updated_row = cursor.fetchone()
+ return updated_row, None
+ cursor.execute(
+ f"""UPDATE {self._schema}.runs SET status='pending',
+ error=%s, finished_at=NULL, lease_owner=NULL, lease_expires_at=NULL,
+ steering_closed=FALSE
+ WHERE tenant_id=%s AND id=%s RETURNING *""",
+ (self._jsonb(error) if error is not None else None, tenant_id, run_id),
+ )
+ updated_row = cursor.fetchone()
+ # Same transaction, same lock: allocate the next RunEvent
+ # sequence and insert ``worker_retrying`` here, not via a
+ # later, separately-committed ``append_event()`` call.
+ cursor.execute(
+ f"""SELECT COALESCE(MAX(sequence),0)+1 AS next
+ FROM {self._schema}.run_events WHERE tenant_id=%s AND run_id=%s""",
+ (tenant_id, run_id),
+ )
+ sequence = int(cursor.fetchone()["next"])
+ data: dict[str, Any] = {
+ "attempt": attempt,
+ "max_attempts": max_attempts,
+ "error": error,
+ }
+ event = RunEvent(
+ tenant_id=tenant_id,
+ run_id=run_id,
+ sequence=sequence,
+ kind="worker_retrying",
+ data=data,
+ )
+ cursor.execute(
+ f"""INSERT INTO {self._schema}.run_events
+ (id,tenant_id,run_id,sequence,kind,data,created_at)
+ VALUES (%s,%s,%s,%s,%s,%s,%s)""",
+ (
+ event.id,
+ tenant_id,
+ run_id,
+ sequence,
+ "worker_retrying",
+ self._jsonb(data),
+ event.created_at,
+ ),
+ )
+ return updated_row, event
+
+ async def close_steering(self, tenant_id, run_id, worker_id, attempt) -> bool:
+ return await asyncio.to_thread(
+ self._close_steering_sync, tenant_id, run_id, worker_id, attempt
+ )
+
+ def _close_steering_sync(
+ self, tenant_id, run_id, worker_id, attempt
+ ) -> bool: # pragma: no cover -- Postgres-only, covered by the `postgres` CI job
+ """SQL counterpart of
+ :meth:`InMemoryRepository.close_steering` -- see its docstring for
+ why no "undrained" check happens here (issue #16 PR #17 review
+ round 7, point 1, BLOCKER; the real check, as of round 9, point 2,
+ is ``finalize_run_with_steering_disposition_if_owned()``, which the
+ Worker's post-flush finalization calls).
+ """
+
+ with self._connect() as conn, conn.cursor() as cursor:
+ self._tenant(cursor, tenant_id)
+ cursor.execute(
+ f"""UPDATE {self._schema}.runs SET steering_closed=TRUE
+ WHERE tenant_id=%s AND id=%s AND lease_owner=%s AND attempt=%s
+ AND status IN ('running','cancelling')""",
+ (tenant_id, run_id, worker_id, attempt),
+ )
+ return cursor.rowcount > 0
+
def _retry_run_sync(self, tenant_id, run_id, error, reset_attempt):
with self._connect() as conn, conn.cursor() as cursor:
self._tenant(cursor, tenant_id)
@@ -922,7 +2220,8 @@ def _retry_run_sync(self, tenant_id, run_id, error, reset_attempt):
attempt = "attempt=0," if reset_attempt else ""
cursor.execute(
f"""UPDATE {self._schema}.runs SET status='pending', {attempt}
- error=%s, finished_at=NULL, lease_owner=NULL, lease_expires_at=NULL
+ error=%s, finished_at=NULL, lease_owner=NULL, lease_expires_at=NULL,
+ steering_closed=FALSE
WHERE tenant_id=%s AND id=%s {allowed}""",
(self._jsonb(error) if error is not None else None, tenant_id, run_id),
)
@@ -947,15 +2246,53 @@ def _finish_run_sync(self, tenant_id, run_id, status, output, error):
async def request_cancel(self, tenant_id, run_id):
return await asyncio.to_thread(self._request_cancel_sync, tenant_id, run_id)
- def _request_cancel_sync(self, tenant_id, run_id):
+ def _request_cancel_sync(
+ self, tenant_id, run_id
+ ): # pragma: no cover -- Postgres-only, covered by the `postgres` CI job
with self._connect() as conn, conn.cursor() as cursor:
self._tenant(cursor, tenant_id)
+ # Round 13 review (BLOCKER): lock the row first and check the
+ # typed ``superseded_by_run_id`` column *inside this same
+ # lock*, before deciding the status transition. A resumed-away
+ # paused run keeps status='paused' (by design, so /steer's
+ # existing superseded gate works) -- cancel must honor that
+ # same signal atomically, or a cancel racing behind a resume
+ # could falsely report success against the stale run while the
+ # real (resumed) descendant keeps running untouched. Doing
+ # this as a pre-read outside the lock would reopen the same
+ # TOCTOU class of bug already fixed for double-resume.
+ # Round 14 review (BLOCKER): this reads the typed
+ # ``superseded_by_run_id`` column, not user-writable
+ # ``metadata`` -- a caller could otherwise forge
+ # ``metadata.superseded_by_run_id`` at run-creation time to
+ # falsely trigger this gate on a run that was never resumed.
+ cursor.execute(
+ f"""SELECT status, superseded_by_run_id FROM {self._schema}.runs
+ WHERE tenant_id=%s AND id=%s FOR UPDATE""",
+ (tenant_id, run_id),
+ )
+ row = cursor.fetchone()
+ if row is None:
+ return False
+ if row["status"] in {
+ "succeeded",
+ "failed",
+ "cancelled",
+ "timed_out",
+ "dead_letter",
+ }:
+ return False
+ superseded_by = row["superseded_by_run_id"]
+ if superseded_by is not None:
+ raise RunSupersededError(
+ f"run {run_id!r} was resumed as {superseded_by!r}; "
+ f"cancel the new run instead"
+ )
cursor.execute(
f"""UPDATE {self._schema}.runs SET
- status=CASE WHEN status='pending' THEN 'cancelled' ELSE 'cancelling' END,
- finished_at=CASE WHEN status='pending' THEN NOW() ELSE finished_at END
- WHERE tenant_id=%s AND id=%s
- AND status NOT IN ('succeeded','failed','cancelled','timed_out','dead_letter')""",
+ status=CASE WHEN status IN ('pending','paused') THEN 'cancelled' ELSE 'cancelling' END,
+ finished_at=CASE WHEN status IN ('pending','paused') THEN NOW() ELSE finished_at END
+ WHERE tenant_id=%s AND id=%s""",
(tenant_id, run_id),
)
return cursor.rowcount > 0
@@ -968,9 +2305,7 @@ async def is_cancel_requested(self, tenant_id, run_id):
}
async def append_event(self, tenant_id, run_id, kind, data):
- return await asyncio.to_thread(
- self._append_event_sync, tenant_id, run_id, kind, data
- )
+ return await asyncio.to_thread(self._append_event_sync, tenant_id, run_id, kind, data)
def _append_event_sync(self, tenant_id, run_id, kind, data):
event = RunEvent(tenant_id=tenant_id, run_id=run_id, sequence=0, kind=kind, data=data)
@@ -1012,39 +2347,936 @@ async def list_events(self, tenant_id, run_id, *, after=0):
)
return [RunEvent.model_validate(row) for row in rows]
- async def create_schedule(self, tenant_id, request):
- value = Schedule(tenant_id=tenant_id, **request.model_dump())
- await asyncio.to_thread(self._insert_schedule, value)
- return value
+ async def submit_steering(
+ self,
+ tenant_id,
+ run_id,
+ *,
+ kind,
+ payload,
+ metadata=None,
+ idempotency_key=None,
+ max_payload_bytes: int = MAX_STEERING_PAYLOAD_BYTES,
+ ):
+ validate_steering_payload(payload, metadata, max_bytes=max_payload_bytes)
+ return await asyncio.to_thread(
+ self._submit_steering_sync,
+ tenant_id,
+ run_id,
+ kind,
+ payload,
+ metadata or {},
+ idempotency_key,
+ )
- def _insert_schedule(self, value):
+ def _submit_steering_sync(
+ self, tenant_id, run_id, kind, payload, metadata, idempotency_key
+ ): # pragma: no cover -- Postgres-only, covered by the `postgres` CI job
with self._connect() as conn, conn.cursor() as cursor:
- self._tenant(cursor, value.tenant_id)
+ self._tenant(cursor, tenant_id)
cursor.execute(
- f"""INSERT INTO {self._schema}.schedules
- (id,tenant_id,assistant_id,cron,timezone,input,enabled,metadata,
- created_at,updated_at) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)""",
- (
- value.id,
- value.tenant_id,
- value.assistant_id,
- value.cron,
- value.timezone,
- self._jsonb(value.input),
- value.enabled,
- self._jsonb(value.metadata),
- value.created_at,
- value.updated_at,
- ),
+ f"""SELECT status, steering_closed, superseded_by_run_id
+ FROM {self._schema}.runs
+ WHERE tenant_id=%s AND id=%s FOR UPDATE""",
+ (tenant_id, run_id),
)
-
- async def list_schedules(self, tenant_id):
- rows = await asyncio.to_thread(
- self._fetch_all,
- tenant_id,
- f"SELECT * FROM {self._schema}.schedules WHERE tenant_id=%s ORDER BY created_at DESC",
- (tenant_id,),
- )
+ run_row = cursor.fetchone()
+ if run_row is None:
+ raise KeyError(f"run {run_id!r} not found")
+ # Issue #16 PR #17 review round 7, point 3 (BLOCKER): check the
+ # idempotency key for an existing row *before* the terminal/
+ # finalizing/superseded admission gates below -- see the
+ # matching comment in ``InMemoryRepository.submit_steering``.
+ if idempotency_key is not None:
+ cursor.execute(
+ "SELECT pg_advisory_xact_lock(hashtextextended(%s, 0))",
+ (f"{tenant_id}:{run_id}:{idempotency_key}",),
+ )
+ cursor.execute(
+ f"""SELECT * FROM {self._schema}.run_steering_events
+ WHERE tenant_id=%s AND run_id=%s AND idempotency_key=%s""",
+ (tenant_id, run_id, idempotency_key),
+ )
+ existing = cursor.fetchone()
+ if existing is not None:
+ existing_event = RunSteeringEvent.model_validate(existing)
+ # A prior attempt with this idempotency key may have
+ # committed the steering row but then failed before its
+ # ``run.steer.accepted`` event was recorded (issue #16
+ # PR #17 review round 4, point 2) -- repair that gap on
+ # the retry instead of permanently skipping it.
+ self._ensure_steer_accepted_event_sync(
+ cursor, tenant_id, run_id, existing_event
+ )
+ return existing_event, False
+ # Only a genuinely new admission decision (no existing row for
+ # this idempotency key -- or none was given) is subject to the
+ # terminal/finalizing/superseded gates.
+ if run_row["status"] in TERMINAL:
+ raise RunTerminalError(
+ f"run {run_id!r} is in terminal state {run_row['status']!r} "
+ "and cannot accept new steering input"
+ )
+ if run_row.get("steering_closed"):
+ raise RunFinalizingError(
+ f"run {run_id!r} has finished executing and is finalizing; "
+ "no further steering input can be safely consumed"
+ )
+ # Round 14 review (BLOCKER): typed column, not user-writable
+ # ``metadata`` -- see ``_request_cancel_sync`` above.
+ superseded_by = run_row.get("superseded_by_run_id")
+ if superseded_by is not None:
+ raise RunSupersededError(
+ f"run {run_id!r} was resumed as {superseded_by!r}; "
+ f"steer the new run instead"
+ )
+ cursor.execute(
+ f"""SELECT COALESCE(MAX(sequence),0)+1 AS next
+ FROM {self._schema}.run_steering_events WHERE tenant_id=%s AND run_id=%s""",
+ (tenant_id, run_id),
+ )
+ sequence = int(cursor.fetchone()["next"])
+ event = RunSteeringEvent(
+ tenant_id=tenant_id,
+ run_id=run_id,
+ sequence=sequence,
+ kind=kind,
+ payload=dict(payload),
+ metadata=dict(metadata or {}),
+ idempotency_key=idempotency_key,
+ status="pending",
+ )
+ cursor.execute(
+ f"""INSERT INTO {self._schema}.run_steering_events
+ (id,tenant_id,run_id,sequence,kind,payload,metadata,idempotency_key,
+ status,created_at)
+ VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)""",
+ (
+ event.id,
+ tenant_id,
+ run_id,
+ event.sequence,
+ kind,
+ self._jsonb(event.payload),
+ self._jsonb(event.metadata),
+ idempotency_key,
+ event.status,
+ event.created_at,
+ ),
+ )
+ # Accept the steering row and record its ``run.steer.accepted``
+ # lifecycle event in the same transaction (issue #16 PR #17
+ # review round 4, point 2) -- both commit together, or neither
+ # does, so a retry can never observe the row without its
+ # accepted event.
+ self._ensure_steer_accepted_event_sync(cursor, tenant_id, run_id, event)
+ return event, True
+
+ def _ensure_steer_accepted_event_sync(
+ self,
+ cursor,
+ tenant_id: str,
+ run_id: str,
+ steering_event: RunSteeringEvent,
+ *,
+ transferred_from_run_id: str | None = None,
+ ) -> None: # pragma: no cover -- Postgres-only, covered by the `postgres` CI job
+ """Idempotently ensure ``run.steer.accepted`` exists for one event.
+
+ Must be called with the target run's row already locked (``FOR
+ UPDATE`` / an implicit lock from an ``INSERT`` earlier in this same
+ transaction) so concurrent sequence-number assignment for
+ ``run_events`` serializes -- same discipline as
+ ``_append_event_sync`` / ``_commit_steering_consumptions_sync``.
+ """
+
+ cursor.execute(
+ f"""SELECT 1 FROM {self._schema}.run_events
+ WHERE tenant_id=%s AND run_id=%s AND kind='run.steer.accepted'
+ AND data->>'steering_event_id'=%s""",
+ (tenant_id, run_id, steering_event.id),
+ )
+ if cursor.fetchone() is not None:
+ return
+ cursor.execute(
+ f"""SELECT COALESCE(MAX(sequence),0)+1 AS next
+ FROM {self._schema}.run_events WHERE tenant_id=%s AND run_id=%s""",
+ (tenant_id, run_id),
+ )
+ sequence = int(cursor.fetchone()["next"])
+ data: dict[str, Any] = {
+ "steering_event_id": steering_event.id,
+ "sequence": steering_event.sequence,
+ "kind": steering_event.kind,
+ }
+ if transferred_from_run_id is not None:
+ data["transferred_from_run_id"] = transferred_from_run_id
+ data["source_event_id"] = steering_event.source_event_id
+ run_event = RunEvent(
+ tenant_id=tenant_id,
+ run_id=run_id,
+ sequence=sequence,
+ kind="run.steer.accepted",
+ data=data,
+ )
+ cursor.execute(
+ f"""INSERT INTO {self._schema}.run_events
+ (id,tenant_id,run_id,sequence,kind,data,created_at)
+ VALUES (%s,%s,%s,%s,%s,%s,%s)""",
+ (
+ run_event.id,
+ tenant_id,
+ run_id,
+ sequence,
+ "run.steer.accepted",
+ self._jsonb(data),
+ run_event.created_at,
+ ),
+ )
+
+ def _ensure_steer_superseded_event_sync(
+ self,
+ cursor,
+ tenant_id: str,
+ run_id: str,
+ steering_event: RunSteeringEvent,
+ *,
+ reason: str,
+ superseded_by_run_id: str | None = None,
+ replacement_steering_event_id: str | None = None,
+ ) -> None: # pragma: no cover -- Postgres-only, covered by the `postgres` CI job
+ """SQL counterpart of
+ :meth:`InMemoryRepository._ensure_steer_superseded_event_locked`
+ (issue #16 PR #17 review round 8, point 3, BLOCKER). Must be
+ called with the target run's row already locked (``FOR UPDATE`` /
+ an implicit lock from an earlier ``INSERT``/``UPDATE`` in this
+ same transaction), same discipline as
+ ``_ensure_steer_accepted_event_sync``.
+ """
+
+ cursor.execute(
+ f"""SELECT 1 FROM {self._schema}.run_events
+ WHERE tenant_id=%s AND run_id=%s AND kind='run.steer.superseded'
+ AND data->>'steering_event_id'=%s""",
+ (tenant_id, run_id, steering_event.id),
+ )
+ if cursor.fetchone() is not None:
+ return
+ cursor.execute(
+ f"""SELECT COALESCE(MAX(sequence),0)+1 AS next
+ FROM {self._schema}.run_events WHERE tenant_id=%s AND run_id=%s""",
+ (tenant_id, run_id),
+ )
+ sequence = int(cursor.fetchone()["next"])
+ data: dict[str, Any] = {
+ "steering_event_id": steering_event.id,
+ "source_event_id": steering_event.source_event_id or steering_event.id,
+ # Issue #16 PR #17 review round 9, point 4 (REQUIRED): see the
+ # matching comment in
+ # ``InMemoryRepository._ensure_steer_superseded_event_locked``.
+ "sequence": steering_event.sequence,
+ "kind": steering_event.kind,
+ "reason": reason,
+ "superseded_by_run_id": superseded_by_run_id,
+ "replacement_steering_event_id": replacement_steering_event_id,
+ }
+ run_event = RunEvent(
+ tenant_id=tenant_id,
+ run_id=run_id,
+ sequence=sequence,
+ kind="run.steer.superseded",
+ data=data,
+ )
+ cursor.execute(
+ f"""INSERT INTO {self._schema}.run_events
+ (id,tenant_id,run_id,sequence,kind,data,created_at)
+ VALUES (%s,%s,%s,%s,%s,%s,%s)""",
+ (
+ run_event.id,
+ tenant_id,
+ run_id,
+ sequence,
+ "run.steer.superseded",
+ self._jsonb(data),
+ run_event.created_at,
+ ),
+ )
+
+ async def finalize_run_with_steering_disposition_if_owned(
+ self,
+ tenant_id,
+ run_id,
+ worker_id,
+ attempt,
+ status,
+ *,
+ output=None,
+ error=None,
+ supersede_pending: bool = True,
+ supersede_reason: str = "unconsumed_at_final_boundary",
+ ) -> tuple[Run, list[RunEvent]] | None:
+ result = await asyncio.to_thread(
+ self._finalize_run_with_steering_disposition_if_owned_sync,
+ tenant_id,
+ run_id,
+ worker_id,
+ attempt,
+ status,
+ output,
+ error,
+ supersede_pending,
+ supersede_reason,
+ )
+ if result is None:
+ return None
+ row, stored_events = result
+ return self._run_from_row(row), stored_events
+
+ def _finalize_run_with_steering_disposition_if_owned_sync(
+ self,
+ tenant_id: str,
+ run_id: str,
+ worker_id: str,
+ attempt: int,
+ status,
+ output,
+ error,
+ supersede_pending: bool,
+ supersede_reason: str,
+ ): # pragma: no cover -- Postgres-only, covered by the `postgres` CI job
+ """SQL counterpart of
+ :meth:`InMemoryRepository.finalize_run_with_steering_disposition_if_owned`
+ (issue #16 PR #17 review round 9, point 2, BLOCKER). One real
+ PostgreSQL transaction covers: locking the run row, re-checking the
+ fence and control state (cancel-wins), locking and superseding any
+ still-``pending``/``delivered`` steering rows plus their
+ ``run.steer.superseded`` events, and writing the run's terminal
+ status -- all committed together by the surrounding ``with
+ self._connect()`` block, or all rolled back together if anything
+ raises before it exits normally.
+ """
+
+ with self._connect() as conn, conn.cursor() as cursor:
+ self._tenant(cursor, tenant_id)
+ cursor.execute(
+ f"""SELECT status FROM {self._schema}.runs
+ WHERE tenant_id=%s AND id=%s AND lease_owner=%s AND attempt=%s
+ AND status IN ('running','cancelling') FOR UPDATE""",
+ (tenant_id, run_id, worker_id, attempt),
+ )
+ row = cursor.fetchone()
+ if row is None:
+ return None
+
+ stored: list[RunEvent] = []
+ if supersede_pending:
+ cursor.execute(
+ f"""SELECT * FROM {self._schema}.run_steering_events
+ WHERE tenant_id=%s AND run_id=%s AND status IN ('pending','delivered')
+ ORDER BY sequence FOR UPDATE""",
+ (tenant_id, run_id),
+ )
+ stale_rows = cursor.fetchall()
+ if stale_rows:
+ stale_ids = [stale_row["id"] for stale_row in stale_rows]
+ cursor.execute(
+ f"""UPDATE {self._schema}.run_steering_events
+ SET status='superseded' WHERE tenant_id=%s AND id=ANY(%s)""",
+ (tenant_id, stale_ids),
+ )
+ for stale_row in stale_rows:
+ steering_event = RunSteeringEvent.model_validate(stale_row)
+ self._ensure_steer_superseded_event_sync(
+ cursor,
+ tenant_id,
+ run_id,
+ steering_event,
+ reason=supersede_reason,
+ )
+ cursor.execute(
+ f"""SELECT * FROM {self._schema}.run_events
+ WHERE tenant_id=%s AND run_id=%s AND kind='run.steer.superseded'
+ AND data->>'steering_event_id'=%s""",
+ (tenant_id, run_id, steering_event.id),
+ )
+ stored_row = cursor.fetchone()
+ if stored_row is not None:
+ stored.append(RunEvent.model_validate(stored_row))
+
+ final_status = RunStatus(status).value
+ final_output = output
+ final_error = error
+ if row["status"] == "cancelling" and final_status != RunStatus.CANCELLED.value:
+ final_status = RunStatus.CANCELLED.value
+ final_output = None
+ final_error = {
+ "code": "run_cancelled",
+ "message": "run was cancelled during finalization",
+ }
+ # Issue #16 PR #17 review round 10, point 2 (REQUIRED): return
+ # this transaction's own ``RETURNING *`` snapshot alongside the
+ # superseded events, instead of a separate later ``get_run()``
+ # a concurrent write (e.g. ``/redrive`` or the unfenced
+ # ``retry_run()``) could race ahead of.
+ cursor.execute(
+ f"""UPDATE {self._schema}.runs SET status=%s, output=%s, error=%s,
+ finished_at=NOW(), lease_owner=NULL, lease_expires_at=NULL
+ WHERE tenant_id=%s AND id=%s RETURNING *""",
+ (
+ final_status,
+ self._jsonb(final_output) if final_output is not None else None,
+ self._jsonb(final_error) if final_error is not None else None,
+ tenant_id,
+ run_id,
+ ),
+ )
+ return cursor.fetchone(), stored
+
+ async def supersede_pending_steering_if_owned(
+ self, tenant_id, run_id, worker_id, attempt
+ ) -> list[RunEvent] | None:
+ return await asyncio.to_thread(
+ self._supersede_pending_steering_if_owned_sync,
+ tenant_id,
+ run_id,
+ worker_id,
+ attempt,
+ )
+
+ def _supersede_pending_steering_if_owned_sync(
+ self, tenant_id: str, run_id: str, worker_id: str, attempt: int
+ ) -> (
+ list[RunEvent] | None
+ ): # pragma: no cover -- Postgres-only, covered by the `postgres` CI job
+ """SQL counterpart of
+ :meth:`InMemoryRepository.supersede_pending_steering_if_owned` --
+ see its docstring (issue #16 PR #17 review round 8, point 1,
+ BLOCKER) for the "seal + supersede" design this implements.
+ """
+
+ with self._connect() as conn, conn.cursor() as cursor:
+ self._tenant(cursor, tenant_id)
+ cursor.execute(
+ f"""SELECT status FROM {self._schema}.runs
+ WHERE tenant_id=%s AND id=%s AND lease_owner=%s AND attempt=%s
+ AND status IN ('running','cancelling') FOR UPDATE""",
+ (tenant_id, run_id, worker_id, attempt),
+ )
+ if cursor.fetchone() is None:
+ return None
+ cursor.execute(
+ f"""SELECT * FROM {self._schema}.run_steering_events
+ WHERE tenant_id=%s AND run_id=%s AND status IN ('pending','delivered')
+ ORDER BY sequence FOR UPDATE""",
+ (tenant_id, run_id),
+ )
+ stale_rows = cursor.fetchall()
+ if not stale_rows:
+ return []
+ stale_ids = [row["id"] for row in stale_rows]
+ cursor.execute(
+ f"""UPDATE {self._schema}.run_steering_events
+ SET status='superseded' WHERE tenant_id=%s AND id=ANY(%s)""",
+ (tenant_id, stale_ids),
+ )
+ stored: list[RunEvent] = []
+ for row in stale_rows:
+ steering_event = RunSteeringEvent.model_validate(row)
+ self._ensure_steer_superseded_event_sync(
+ cursor,
+ tenant_id,
+ run_id,
+ steering_event,
+ reason="unconsumed_at_final_boundary",
+ )
+ cursor.execute(
+ f"""SELECT * FROM {self._schema}.run_events
+ WHERE tenant_id=%s AND run_id=%s AND kind='run.steer.superseded'
+ AND data->>'steering_event_id'=%s""",
+ (tenant_id, run_id, steering_event.id),
+ )
+ stored_row = cursor.fetchone()
+ if stored_row is not None:
+ stored.append(RunEvent.model_validate(stored_row))
+ return stored
+
+ async def list_pending_steering(self, tenant_id, run_id):
+ rows = await asyncio.to_thread(
+ self._fetch_all,
+ tenant_id,
+ f"""SELECT * FROM {self._schema}.run_steering_events
+ WHERE tenant_id=%s AND run_id=%s AND status IN ('pending','delivered')
+ ORDER BY sequence""",
+ (tenant_id, run_id),
+ )
+ return [RunSteeringEvent.model_validate(row) for row in rows]
+
+ async def list_steering(self, tenant_id, run_id):
+ rows = await asyncio.to_thread(
+ self._fetch_all,
+ tenant_id,
+ f"""SELECT * FROM {self._schema}.run_steering_events
+ WHERE tenant_id=%s AND run_id=%s ORDER BY sequence""",
+ (tenant_id, run_id),
+ )
+ return [RunSteeringEvent.model_validate(row) for row in rows]
+
+ async def mark_steering_consumed(self, tenant_id, run_id, event_ids):
+ if not event_ids:
+ return
+ await asyncio.to_thread(
+ self._mark_steering_consumed_sync, tenant_id, run_id, list(event_ids)
+ )
+
+ def _mark_steering_consumed_sync(
+ self, tenant_id, run_id, event_ids
+ ): # pragma: no cover -- Postgres-only, covered by the `postgres` CI job
+ with self._connect() as conn, conn.cursor() as cursor:
+ self._tenant(cursor, tenant_id)
+ cursor.execute(
+ f"""UPDATE {self._schema}.run_steering_events
+ SET status='consumed', consumed_at=NOW()
+ WHERE tenant_id=%s AND run_id=%s AND id=ANY(%s) AND status!='consumed'""",
+ (tenant_id, run_id, event_ids),
+ )
+
+ async def commit_steering_consumptions(self, tenant_id, run_id, consumptions):
+ if not consumptions:
+ return []
+ return await asyncio.to_thread(
+ self._commit_steering_consumptions_sync, tenant_id, run_id, list(consumptions)
+ )
+
+ def _commit_steering_consumptions_sync(
+ self, tenant_id: str, run_id: str, consumptions: list[SteeringConsumption]
+ ) -> list[RunEvent]: # pragma: no cover -- Postgres-only, covered by the `postgres` CI job
+ """SQL counterpart of
+ :meth:`InMemoryRepository.commit_steering_consumptions` -- see its
+ docstring (issue #16 PR #17 review point 2) for why the status
+ update and the ``run.steer.consumed`` event append must commit
+ together, in the same transaction, before the caller acks its
+ local consumption log.
+ """
+
+ with self._connect() as conn, conn.cursor() as cursor:
+ self._tenant(cursor, tenant_id)
+ # Same locking discipline as ``_append_event_sync``: lock the
+ # run row first so concurrent event appenders for this run
+ # serialize their sequence assignment instead of racing.
+ cursor.execute(
+ f"SELECT id FROM {self._schema}.runs WHERE tenant_id=%s AND id=%s FOR UPDATE",
+ (tenant_id, run_id),
+ )
+ return self._commit_steering_consumptions_body_sync(
+ cursor, tenant_id, run_id, consumptions
+ )
+
+ async def commit_steering_consumptions_if_owned(
+ self, tenant_id, run_id, worker_id, attempt, consumptions
+ ) -> list[RunEvent] | None:
+ if not consumptions:
+ return []
+ return await asyncio.to_thread(
+ self._commit_steering_consumptions_if_owned_sync,
+ tenant_id,
+ run_id,
+ worker_id,
+ attempt,
+ list(consumptions),
+ )
+
+ def _commit_steering_consumptions_if_owned_sync(
+ self,
+ tenant_id: str,
+ run_id: str,
+ worker_id: str,
+ attempt: int,
+ consumptions: list[SteeringConsumption],
+ ) -> (
+ list[RunEvent] | None
+ ): # pragma: no cover -- Postgres-only, covered by the `postgres` CI job
+ """Fenced counterpart of ``_commit_steering_consumptions_sync``.
+
+ Issue #16 PR #17 review round 7, point 2 (BLOCKER): see the
+ matching docstring on
+ :meth:`InMemoryRepository.commit_steering_consumptions_if_owned`.
+ The run row is locked ``FOR UPDATE`` first, filtered on
+ ``lease_owner``/``attempt``/active status -- exactly the same
+ pattern as ``_finish_run_if_owned_sync``/``_close_steering_sync`` --
+ *before* any steering row is touched, so a concurrent
+ ``claim_run()`` (which updates ``lease_owner``/``attempt`` under
+ its own ``FOR UPDATE`` on the same row) fully serializes against
+ this fencing check: a worker that has already lost its lease can
+ never observe a stale "still owned" result here.
+ """
+
+ with self._connect() as conn, conn.cursor() as cursor:
+ self._tenant(cursor, tenant_id)
+ cursor.execute(
+ f"""SELECT status FROM {self._schema}.runs
+ WHERE tenant_id=%s AND id=%s AND lease_owner=%s AND attempt=%s
+ AND status IN ('running','cancelling') FOR UPDATE""",
+ (tenant_id, run_id, worker_id, attempt),
+ )
+ if cursor.fetchone() is None:
+ return None
+ return self._commit_steering_consumptions_body_sync(
+ cursor, tenant_id, run_id, consumptions
+ )
+
+ def _commit_steering_consumptions_body_sync(
+ self,
+ cursor,
+ tenant_id: str,
+ run_id: str,
+ consumptions: list[SteeringConsumption],
+ ) -> list[RunEvent]: # pragma: no cover -- Postgres-only, covered by the `postgres` CI job
+ """Shared body; assumes the run row is already locked ``FOR UPDATE``
+ on ``cursor``'s transaction (by the caller, with or without lease
+ fencing applied first)."""
+
+ ids = [consumption.event.id for consumption in consumptions]
+ cursor.execute(
+ f"""UPDATE {self._schema}.run_steering_events
+ SET status='consumed', consumed_at=NOW()
+ WHERE tenant_id=%s AND run_id=%s AND id=ANY(%s) AND status!='consumed'
+ RETURNING id""",
+ (tenant_id, run_id, ids),
+ )
+ # Only ids that *actually* transitioned pending/delivered ->
+ # consumed in this call. A retried batch (e.g. a worker that
+ # re-sends consumptions after a commit whose ack it never saw)
+ # must not emit a second, semantically duplicate
+ # ``run.steer.consumed`` lifecycle event for a row some
+ # earlier call already consumed (issue #16 PR #17 review
+ # point 3).
+ transitioned = {row["id"] for row in cursor.fetchall()}
+ cursor.execute(
+ f"""SELECT COALESCE(MAX(sequence),0) AS next
+ FROM {self._schema}.run_events WHERE tenant_id=%s AND run_id=%s""",
+ (tenant_id, run_id),
+ )
+ next_sequence = int(cursor.fetchone()["next"])
+ stored: list[RunEvent] = []
+ for consumption in consumptions:
+ event = consumption.event
+ if event.id not in transitioned:
+ continue
+ next_sequence += 1
+ data: dict[str, Any] = {
+ "steering_event_id": event.id,
+ "sequence": event.sequence,
+ "kind": event.kind,
+ "queue_latency_seconds": consumption.queue_latency_seconds,
+ "node": consumption.node,
+ "namespace": list(consumption.namespace),
+ "task_id": consumption.task_id,
+ }
+ if event.source_event_id is not None:
+ data["source_event_id"] = event.source_event_id
+ run_event = RunEvent(
+ tenant_id=tenant_id,
+ run_id=run_id,
+ sequence=next_sequence,
+ kind="run.steer.consumed",
+ data=data,
+ )
+ cursor.execute(
+ f"""INSERT INTO {self._schema}.run_events
+ (id,tenant_id,run_id,sequence,kind,data,created_at)
+ VALUES (%s,%s,%s,%s,%s,%s,%s)""",
+ (
+ run_event.id,
+ tenant_id,
+ run_id,
+ run_event.sequence,
+ run_event.kind,
+ self._jsonb(run_event.data),
+ run_event.created_at,
+ ),
+ )
+ stored.append(run_event)
+ return stored
+
+ async def resume_run_with_pending_steering(
+ self, tenant_id, thread_id, assistant, request, old_run_id
+ ):
+ return await asyncio.to_thread(
+ self._resume_run_with_pending_steering_sync,
+ tenant_id,
+ thread_id,
+ assistant,
+ request,
+ old_run_id,
+ )
+
+ def _resume_run_with_pending_steering_sync(
+ self, tenant_id, thread_id, assistant, request, old_run_id
+ ): # pragma: no cover -- Postgres-only, covered by the `postgres` CI job
+ """SQL counterpart of
+ :meth:`InMemoryRepository.resume_run_with_pending_steering`.
+
+ Issue #16 PR #17 review point 1: the resumed Run's INSERT and the
+ old Run's steering migration must commit as one all-or-nothing unit
+ so no other transaction can ever see the new Run row without its
+ migrated steering already alongside it. Using a single connection
+ for both means they share one PostgreSQL transaction (autocommit is
+ off; the ``with self._connect()`` block only commits when it exits
+ normally) -- ``claim_run`` runs its own ``UPDATE ... FOR UPDATE
+ SKIP LOCKED`` against ``runs`` in a *different* connection/
+ transaction, so it cannot observe the freshly INSERTed new-Run row
+ at all until this transaction commits, by which point the
+ migration below has already happened.
+
+ This also closes the second race the review flagged: an ordinary
+ concurrent ``/steer`` against the new run (``_submit_steering_sync``)
+ takes ``SELECT ... FOR UPDATE`` on the **runs** row for
+ ``new_run.id`` before computing ``MAX(sequence)+1`` over
+ ``run_steering_events``. Because the new run's row was INSERTed --
+ and is therefore implicitly locked -- inside *this* transaction,
+ that concurrent ``FOR UPDATE`` blocks until this transaction
+ commits, so the two ``MAX(sequence)+1`` computations for the same
+ ``new_run_id`` can never race.
+ """
+
+ with self._connect() as conn, conn.cursor() as cursor:
+ self._tenant(cursor, tenant_id)
+ # Lock the paused run first so two concurrent resume attempts
+ # against the same paused run_id serialize instead of both
+ # migrating (and superseding) the same steering rows.
+ cursor.execute(
+ f"""SELECT status, superseded_by_run_id FROM {self._schema}.runs
+ WHERE tenant_id=%s AND id=%s FOR UPDATE""",
+ (tenant_id, old_run_id),
+ )
+ old_run_row = cursor.fetchone()
+ # Issue #16 PR #17 review round 4, point 1: the endpoint's own
+ # pre-check (``GET old run -> status == paused``) races with
+ # concurrent resumes and is not itself atomic with this call.
+ # Re-validate under the lock just acquired above -- still
+ # ``paused``, and not already superseded by a resume that won
+ # the race -- before creating a second descendant Run.
+ # Round 14 review (BLOCKER): typed column, not user-writable
+ # ``metadata`` -- see ``_request_cancel_sync`` above.
+ if old_run_row is None:
+ raise RunResumeConflictError(f"run {old_run_id!r} no longer exists")
+ if (
+ old_run_row.get("status") != "paused"
+ or old_run_row.get("superseded_by_run_id") is not None
+ ):
+ raise RunResumeConflictError(
+ f"run {old_run_id!r} is no longer resumable "
+ "(it is not paused, or has already been resumed by a concurrent request)"
+ )
+
+ cursor.execute(
+ f"""SELECT
+ COUNT(*) FILTER (WHERE status IN ('running','cancelling')) AS active,
+ COUNT(*) FILTER (WHERE status='pending') AS queued
+ FROM {self._schema}.runs WHERE tenant_id=%s""",
+ (tenant_id,),
+ )
+ counts = cursor.fetchone()
+ if counts["active"] >= self.limits.max_active_runs:
+ raise ConcurrentRunError("tenant active-run quota exceeded")
+ if counts["queued"] >= self.limits.max_queued_runs:
+ raise ConcurrentRunError("tenant queued-run quota exceeded")
+ if thread_id is not None:
+ cursor.execute(
+ f"""SELECT id FROM {self._schema}.runs WHERE tenant_id=%s
+ AND thread_id=%s AND status IN ('running','cancelling') FOR UPDATE""",
+ (tenant_id, thread_id),
+ )
+ active = cursor.fetchall()
+ strategy = MultitaskStrategy(request.multitask_strategy)
+ if active and strategy is MultitaskStrategy.REJECT:
+ raise ConcurrentRunError("thread already has an active run")
+ if active and strategy is MultitaskStrategy.CANCEL_PREVIOUS:
+ cursor.execute(
+ f"""UPDATE {self._schema}.runs SET status='cancelling'
+ WHERE tenant_id=%s AND thread_id=%s AND status='running'""",
+ (tenant_id, thread_id),
+ )
+
+ run = Run(
+ tenant_id=tenant_id,
+ thread_id=thread_id,
+ assistant_id=assistant.id,
+ graph_id=assistant.graph_id,
+ graph_version=assistant.graph_version,
+ input=request.input,
+ context={**assistant.context, **request.context},
+ config={**assistant.config, **request.config},
+ metadata=request.metadata,
+ resume=request.resume,
+ update=request.update,
+ goto=request.goto,
+ durability=request.durability,
+ )
+ if request.run_timeout is not None:
+ run.config["run_timeout"] = request.run_timeout
+ run.config.setdefault("max_state_bytes", self.limits.max_state_bytes)
+ for budget_name in ("max_model_calls", "max_tool_calls", "max_tokens", "max_cost"):
+ budget_value = getattr(request, budget_name)
+ if budget_value is not None:
+ run.config[budget_name] = budget_value
+ cursor.execute(
+ f"""INSERT INTO {self._schema}.runs
+ (id,tenant_id,thread_id,assistant_id,graph_id,graph_version,status,
+ idempotency_key,request_digest,input,context,config,metadata,resume,update,goto_node,durability,
+ attempt,created_at)
+ VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)""",
+ (
+ run.id,
+ tenant_id,
+ thread_id,
+ run.assistant_id,
+ run.graph_id,
+ run.graph_version,
+ enum_value(run.status),
+ run.idempotency_key,
+ run.request_digest,
+ self._jsonb(run.input) if run.input is not None else None,
+ self._jsonb(run.context),
+ self._jsonb(run.config),
+ self._jsonb(run.metadata),
+ self._jsonb(run.resume) if run.resume is not None else None,
+ self._jsonb(run.update) if run.update is not None else None,
+ run.goto,
+ enum_value(run.durability),
+ run.attempt,
+ run.created_at,
+ ),
+ )
+
+ # Round 14 review (BLOCKER): write the typed column
+ # atomically as part of this same resume transaction, not
+ # user-writable ``metadata``.
+ cursor.execute(
+ f"""UPDATE {self._schema}.runs SET superseded_by_run_id=%s
+ WHERE tenant_id=%s AND id=%s""",
+ (run.id, tenant_id, old_run_id),
+ )
+
+ cursor.execute(
+ f"""SELECT * FROM {self._schema}.run_steering_events
+ WHERE tenant_id=%s AND run_id=%s AND status IN ('pending','delivered')
+ ORDER BY sequence FOR UPDATE""",
+ (tenant_id, old_run_id),
+ )
+ pending_rows = cursor.fetchall()
+ transferred: list[RunSteeringEvent] = []
+ if pending_rows:
+ pending_ids = [row["id"] for row in pending_rows]
+ cursor.execute(
+ f"""UPDATE {self._schema}.run_steering_events
+ SET status='superseded' WHERE tenant_id=%s AND id=ANY(%s)""",
+ (tenant_id, pending_ids),
+ )
+ # Safe against a concurrent ordinary /steer on the new run:
+ # both this and ``_submit_steering_sync`` first take
+ # ``FOR UPDATE`` on the *new run's* ``runs`` row -- which
+ # this transaction already holds implicitly from the
+ # INSERT above -- before computing MAX(sequence)+1 here.
+ cursor.execute(
+ f"""SELECT COALESCE(MAX(sequence),0) AS next
+ FROM {self._schema}.run_steering_events WHERE tenant_id=%s AND run_id=%s""",
+ (tenant_id, run.id),
+ )
+ next_sequence = int(cursor.fetchone()["next"])
+ for row in pending_rows:
+ next_sequence += 1
+ event = RunSteeringEvent(
+ tenant_id=tenant_id,
+ run_id=run.id,
+ sequence=next_sequence,
+ kind=row["kind"],
+ payload=dict(row["payload"] or {}),
+ metadata=dict(row["metadata"] or {}),
+ idempotency_key=row["idempotency_key"],
+ status="pending",
+ source_event_id=row["source_event_id"] or row["id"],
+ created_at=row["created_at"],
+ )
+ cursor.execute(
+ f"""INSERT INTO {self._schema}.run_steering_events
+ (id,tenant_id,run_id,sequence,kind,payload,metadata,idempotency_key,
+ status,created_at,source_event_id)
+ VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)""",
+ (
+ event.id,
+ tenant_id,
+ run.id,
+ event.sequence,
+ event.kind,
+ self._jsonb(event.payload),
+ self._jsonb(event.metadata),
+ event.idempotency_key,
+ event.status,
+ event.created_at,
+ event.source_event_id,
+ ),
+ )
+ # Record the transferred event's ``run.steer.accepted``
+ # inside this same migration transaction (review round
+ # 4, point 2) instead of a separate ``append_event``
+ # call after this method returns.
+ self._ensure_steer_accepted_event_sync(
+ cursor,
+ tenant_id,
+ run.id,
+ event,
+ transferred_from_run_id=old_run_id,
+ )
+ # Issue #16 PR #17 review round 8, point 3 (BLOCKER):
+ # the old row's transition to ``superseded`` above must
+ # be paired with a durable ``run.steer.superseded``
+ # lifecycle event in this same migration transaction.
+ self._ensure_steer_superseded_event_sync(
+ cursor,
+ tenant_id,
+ old_run_id,
+ RunSteeringEvent.model_validate(row),
+ reason="resume_transfer",
+ superseded_by_run_id=run.id,
+ replacement_steering_event_id=event.id,
+ )
+ transferred.append(event)
+ return run, transferred
+
+ async def create_schedule(self, tenant_id, request):
+ value = Schedule(tenant_id=tenant_id, **request.model_dump())
+ await asyncio.to_thread(self._insert_schedule, value)
+ return value
+
+ def _insert_schedule(self, value):
+ with self._connect() as conn, conn.cursor() as cursor:
+ self._tenant(cursor, value.tenant_id)
+ cursor.execute(
+ f"""INSERT INTO {self._schema}.schedules
+ (id,tenant_id,assistant_id,cron,timezone,input,enabled,metadata,
+ created_at,updated_at) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)""",
+ (
+ value.id,
+ value.tenant_id,
+ value.assistant_id,
+ value.cron,
+ value.timezone,
+ self._jsonb(value.input),
+ value.enabled,
+ self._jsonb(value.metadata),
+ value.created_at,
+ value.updated_at,
+ ),
+ )
+
+ async def list_schedules(self, tenant_id):
+ rows = await asyncio.to_thread(
+ self._fetch_all,
+ tenant_id,
+ f"SELECT * FROM {self._schema}.schedules WHERE tenant_id=%s ORDER BY created_at DESC",
+ (tenant_id,),
+ )
return [Schedule.model_validate(row) for row in rows]
async def patch_schedule(self, tenant_id, schedule_id, request):
@@ -1152,8 +3384,7 @@ def _stats_sync(self, tenant_id: str) -> dict[str, Any]:
counts: dict[str, int] = {}
for name in ("run_events", "threads", "assistants", "schedules"):
cursor.execute(
- f"SELECT COUNT(*) AS count FROM {self._schema}.{name} "
- "WHERE tenant_id=%s",
+ f"SELECT COUNT(*) AS count FROM {self._schema}.{name} WHERE tenant_id=%s",
(tenant_id,),
)
counts[name] = int(cursor.fetchone()["count"])
@@ -1172,9 +3403,39 @@ def _claim_run_sync(self, worker_id: str, lease_seconds: int) -> Run | None:
with self._connect() as conn, conn.cursor() as cursor:
cursor.execute(
f"""UPDATE {self._schema}.runs SET status='pending',
- lease_owner=NULL, lease_expires_at=NULL
+ lease_owner=NULL, lease_expires_at=NULL, steering_closed=FALSE
WHERE status='running' AND lease_expires_at < NOW()"""
)
+ # Lease-expiry recovery rule (issue #16 PR #17 review round 11,
+ # BLOCKER): an expired ``cancelling`` lease must NOT be
+ # requeued to ``pending`` like an expired ``running`` lease
+ # above -- that would silently erase the fact that the caller
+ # already asked to stop and could let business execution
+ # resume after cancellation was requested. It instead resolves
+ # straight to the terminal ``cancelled`` state, mirroring the
+ # same ``run_cancelled`` disposition
+ # ``_finish_run_if_owned_sync``/``_retry_run_if_owned_sync``
+ # already use when cancellation wins during in-process
+ # finalization. This guarantees the run can never be stuck in
+ # ``cancelling`` forever and keeps its thread from being
+ # wedged for other queued runs on the same thread.
+ cursor.execute(
+ f"""UPDATE {self._schema}.runs SET status='cancelled',
+ lease_owner=NULL, lease_expires_at=NULL, finished_at=NOW(),
+ error=%s
+ WHERE status='cancelling' AND lease_expires_at < NOW()""",
+ (
+ self._jsonb(
+ {
+ "code": "run_cancelled",
+ "message": (
+ "run was cancelled but its worker's lease "
+ "expired before finalization completed"
+ ),
+ }
+ ),
+ ),
+ )
cursor.execute(
f"""
WITH candidate AS (
@@ -1207,11 +3468,14 @@ def _run_from_row(row: dict[str, Any]) -> Run:
value = dict(row)
value["goto"] = value.pop("goto_node", None)
for name in ("input", "context", "config", "metadata", "error", "output"):
- value[name] = value.get(name) or ({} if name in {"context", "config", "metadata"} else None)
+ value[name] = value.get(name) or (
+ {} if name in {"context", "config", "metadata"} else None
+ )
return Run.model_validate(value)
__all__ = [
+ "ACTIVE",
"InMemoryRepository",
"PostgresRepository",
"RepositoryLimits",
diff --git a/src/lingxigraph/server/worker.py b/src/lingxigraph/server/worker.py
index a92f989..262a574 100644
--- a/src/lingxigraph/server/worker.py
+++ b/src/lingxigraph/server/worker.py
@@ -24,10 +24,11 @@
from ..events import Event, EventKind
from ..runtime import CancellationToken
from ..serialization import JsonSerializer
+from ..steering import SteeringEvent
from ..store import BaseStore, InMemoryStore
from ..types import Command, RunStatus
from .eventbus import EventBus, InMemoryEventBus
-from .models import Run
+from .models import Run, RunEvent
from .registry import GraphRegistry
from .repository import InMemoryRepository
@@ -130,14 +131,49 @@ async def drain(self, *, timeout: float = 60.0) -> bool:
async def _execute(self, run: Run) -> None:
token = CancellationToken()
- heartbeat = asyncio.create_task(self._heartbeat(run, token))
+ # ``with_runtime`` returns a distinct bound graph instance -- build
+ # it first so the heartbeat loop shares the exact same steering
+ # channel the executor actually reads from, not a different
+ # instance's.
+ graph = self.registry.get(run.graph_id, run.graph_version).with_runtime(
+ checkpointer=self.checkpointer,
+ store=self.store_factory(run.tenant_id),
+ cache=self.cache,
+ )
+ # Register the steering channel *before* the graph ever executes a
+ # single task, even when there is nothing pending yet. This marks
+ # it server-owned (``owned_by_executor=False``, see
+ # SteeringChannel/CompiledStateGraph._run) so the executor's own
+ # embedded-run cleanup never releases it out from under us --
+ # otherwise a run claimed with no steering pending yet would get an
+ # executor-owned channel by default, and steering that arrived
+ # later plus the worker's final flush could race the executor
+ # popping the channel the moment the run finishes.
+ graph.get_steering_channel(run.id)
+ # Recover any steering events that were durably accepted while this
+ # run was queued (or before a prior worker crashed) -- the channel
+ # is the same live object the executor reads from at every safe
+ # point, so this is the "recover after restart" and "queued run"
+ # cases from issue #16 in one code path.
+ await self._sync_steering_in(run, graph)
+ heartbeat = asyncio.create_task(self._heartbeat(run, token, graph))
output: dict[str, Any] | None = None
+
+ # Issue #16 PR #17 review round 5, point 1: the *intended* outcome
+ # is computed here, purely in memory -- it is deliberately NOT
+ # written to the repository yet. Writing ``finish_run``/
+ # ``retry_run`` here (as round <=4 did) makes a terminal or paused
+ # status externally observable via GET/join/SSE *before* we know
+ # whether the run's final steering consumptions actually made it
+ # to durable storage below. If that flush then failed, overriding
+ # the already-visible status back to pending/retrying breaks the
+ # single most important Run state-machine invariant: once a
+ # terminal/paused status is externally observed, the run must
+ # never appear non-terminal again. So instead we settle on
+ # ``_Intent`` here, stop the heartbeat, durably flush steering, and
+ # only then -- once -- commit the outcome to the repository.
+ intent: Worker._Intent
try:
- graph = self.registry.get(run.graph_id, run.graph_version).with_runtime(
- checkpointer=self.checkpointer,
- store=self.store_factory(run.tenant_id),
- cache=self.cache,
- )
config = {
**run.config,
"configurable": {
@@ -174,36 +210,27 @@ async def _execute(self, run: Run) -> None:
**dict(snapshot.values),
"__interrupt__": [dataclasses.asdict(item) for item in snapshot.interrupts],
}
- await self.repository.finish_run(
- run.tenant_id, run.id, RunStatus.PAUSED, output=output
- )
+ intent = Worker._Intent(kind="finish", status=RunStatus.PAUSED, output=output)
else:
- await self.repository.finish_run(
- run.tenant_id, run.id, RunStatus.SUCCEEDED, output=output or {}
- )
- logger.info(
- "run succeeded",
- extra={"run_id": run.id, "tenant_id": run.tenant_id, "status": "succeeded"},
+ intent = Worker._Intent(
+ kind="finish", status=RunStatus.SUCCEEDED, output=output or {}
)
except GraphCancelledError as exc:
- await self.repository.finish_run(
- run.tenant_id,
- run.id,
- RunStatus.CANCELLED,
+ intent = Worker._Intent(
+ kind="finish",
+ status=RunStatus.CANCELLED,
error={"code": "run_cancelled", "message": str(exc)},
)
except GraphTimeoutError as exc:
- await self.repository.finish_run(
- run.tenant_id,
- run.id,
- RunStatus.TIMED_OUT,
+ intent = Worker._Intent(
+ kind="finish",
+ status=RunStatus.TIMED_OUT,
error={"code": "run_timed_out", "message": str(exc)},
)
except BudgetExceededError as exc:
- await self.repository.finish_run(
- run.tenant_id,
- run.id,
- RunStatus.FAILED,
+ intent = Worker._Intent(
+ kind="finish",
+ status=RunStatus.FAILED,
error={"code": "budget_exceeded", "message": str(exc)},
)
except Exception as exc:
@@ -211,54 +238,522 @@ async def _execute(self, run: Run) -> None:
retryable = self._is_retryable(exc)
if retryable and run.attempt < self.max_delivery_attempts:
error["code"] = "delivery_retry"
- await self.repository.retry_run(run.tenant_id, run.id, error=error)
- stored = await self.repository.append_event(
- run.tenant_id,
- run.id,
- "worker_retrying",
- {
- "attempt": run.attempt,
- "max_attempts": self.max_delivery_attempts,
- "error": error,
- },
- )
- await self.event_bus.publish(run.tenant_id, run.id, stored.sequence)
- logger.warning(
- "run delivery scheduled for retry",
- extra={
- "run_id": run.id,
- "tenant_id": run.tenant_id,
- "status": "pending",
- "error_type": type(exc).__name__,
- },
- )
+ intent = Worker._Intent(kind="retry", status=RunStatus.PENDING, error=error)
else:
status = RunStatus.DEAD_LETTER if retryable else RunStatus.FAILED
error["code"] = "dead_letter" if retryable else "run_failed"
- await self.repository.finish_run(
- run.tenant_id,
- run.id,
- status,
- error=error,
- )
- logger.error(
- "run delivery failed",
- extra={
- "run_id": run.id,
- "tenant_id": run.tenant_id,
- "status": status.value,
- "error_type": type(exc).__name__,
- },
- )
+ intent = Worker._Intent(kind="finish", status=status, error=error)
finally:
heartbeat.cancel()
await asyncio.gather(heartbeat, return_exceptions=True)
- async def _heartbeat(self, run: Run, token: CancellationToken) -> None:
+ # Issue #16 PR #17 review round 6, point 3 (BLOCKER): close new
+ # steering admission the instant graph execution ends and *before*
+ # the final flush/commit start -- there is no safe point left for
+ # the graph to ever consume a steering event accepted after this.
+ # This is fenced on our own (lease_owner, attempt): if we no
+ # longer hold the lease (e.g. it already expired and a new worker
+ # claimed the run), this fails and we must abandon the attempt
+ # without touching the run's status at all -- the new owner is now
+ # solely responsible for it (review round 6, point 1, BLOCKER).
+ #
+ # A ``paused`` outcome is deliberately excluded: pausing does not
+ # mean "no safe point left" the way a terminal outcome does --
+ # ``/steer`` against a paused run is an intentional, documented
+ # part of issue #16's design (the pending steering is transferred
+ # to the resumed run at ``/resume`` time), so it must keep
+ # accepting new steering input, not be permanently gated shut by
+ # a worker that is merely yielding control, not finishing.
+ #
+ # Issue #16 PR #17 review round 9, point 1 (BLOCKER): a ``retry``
+ # intent is likewise NOT terminal finalization -- the next delivery
+ # attempt is a legitimate future safe point, not the end of the
+ # line. Closing/superseding steering on retry meant any steering
+ # durably accepted but not yet drained by this attempt was marked
+ # ``superseded`` here and then, moments later, the Run was sent
+ # back to ``pending`` by ``retry_run_with_event_if_owned`` -- so the next
+ # attempt's ``_sync_steering_in`` (which only re-ingests
+ # ``pending``/``delivered`` rows) could never see it again,
+ # permanently losing steering on the very next delivery attempt.
+ # Only a genuinely terminal ``finish`` (i.e. not ``paused``) closes
+ # admission and supersedes leftovers; ``retry`` leaves steering
+ # fully open and recoverable, exactly like an ordinary mid-run
+ # failure that predates this finalization step entirely.
+ closing_steering = intent.kind == "finish" and intent.status is not RunStatus.PAUSED
+ if closing_steering:
+ gate_closed = await self.repository.close_steering(
+ run.tenant_id, run.id, self.worker_id, run.attempt
+ )
+ if not gate_closed:
+ logger.warning(
+ "run finalization abandoned: lease no longer owned before steering close",
+ extra={"run_id": run.id, "tenant_id": run.tenant_id},
+ )
+ return
+
+ # Issue #16 PR #17 review round 6, point 1 (BLOCKER): the lease
+ # lifecycle must stay alive through the *entire* finalization
+ # window, independent of the now-cancelled steering/event
+ # heartbeat -- the final flush can legitimately retry for a long
+ # time, and without this the lease can simply expire mid-flush,
+ # letting a new worker claim and even finish the run while we are
+ # still blocked, unaware our writes are no longer safe.
+ renewal = asyncio.create_task(self._lease_renewal_loop(run))
+ try:
+ # Final flush: anything drained between the last heartbeat tick
+ # and run completion must still be recorded as consumed,
+ # durably, before the run's outcome above is committed. See
+ # ``_final_steering_flush`` for why this retries for as long as
+ # it holds the lease rather than giving up after a handful of
+ # attempts (issue #16 PR #17 review round 5, point 2).
+ flushed = await self._final_steering_flush(run, graph)
+ if flushed:
+ # Issue #16 PR #17 review round 8, point 1 (BLOCKER):
+ # CHOSEN FIX -- "seal admission at the safe boundary +
+ # supersede late events". Round 7's fix forced an
+ # indefinite retry/dead-letter loop onto any run that ever
+ # had undrained steering, even when the graph legitimately
+ # never calls ``drain_steering()`` at all -- hardcoding an
+ # application decision ("does undrained steering mean
+ # rerun?") into the Runtime in violation of issue #16's
+ # Runtime/application boundary. The flush above commits
+ # everything this attempt's channel actually drained, but
+ # it cannot know about a steering row that was durably
+ # accepted (by a concurrent ``/steer`` racing
+ # ``close_steering``'s lock, or simply too late for any
+ # safe point this attempt reached) and therefore was never
+ # ingested into the channel at all. Any such row is now
+ # durably transitioned to a ``superseded`` disposition --
+ # never silently dropped, never forcing a rerun -- and the
+ # run finalizes with its originally computed intent. Only
+ # relevant when we actually closed admission -- a
+ # ``paused`` outcome intentionally leaves steering open for
+ # the eventual ``/resume`` transfer instead, and a
+ # ``retry`` intent never closed admission in the first
+ # place (review round 9, point 1, BLOCKER) so there is
+ # nothing to supersede.
+ if closing_steering:
+ # Issue #16 PR #17 review round 9, point 2 (BLOCKER):
+ # the leftover-steering disposition and the Run's
+ # terminal status commit must be the *same* durable
+ # write, not two separate fenced calls -- a crash or
+ # lease loss between them used to be able to leave
+ # steering permanently ``superseded`` while the Run
+ # itself never actually reached a terminal status (see
+ # ``finalize_run_with_steering_disposition_if_owned``'s
+ # docstring for the full failure mode). ``intent.kind``
+ # is always ``"finish"`` here -- ``closing_steering``
+ # is only true for a genuinely terminal finish.
+ result = await self.repository.finalize_run_with_steering_disposition_if_owned(
+ run.tenant_id,
+ run.id,
+ self.worker_id,
+ run.attempt,
+ intent.status,
+ output=intent.output,
+ error=intent.error,
+ )
+ if result is None:
+ # Lease was reclaimed by a new owner between the
+ # final flush and this call -- same as any other
+ # fenced write losing the race: abandon without
+ # touching the run further, the new owner is now
+ # solely responsible for it.
+ logger.warning(
+ "run finalization abandoned: lease no longer owned before "
+ "finalizing with steering disposition",
+ extra={"run_id": run.id, "tenant_id": run.tenant_id},
+ )
+ return
+ updated, superseded = result
+ for stored in superseded:
+ await self.event_bus.publish(run.tenant_id, run.id, stored.sequence)
+ self._log_finalize_outcome(run, updated, intent)
+ graph.forget_steering(run.id)
+ return
+ committed = await self._commit_intent(run, intent)
+ if committed:
+ graph.forget_steering(run.id)
+ return
+
+ # The flush never durably succeeded (the worker is draining, or
+ # this run's lease was lost to another worker) -- the intended
+ # outcome computed above must NEVER be written; doing so would
+ # either expose a false terminal/paused status or silently
+ # strand the drained-but-uncommitted steering consumptions.
+ # Route the run into the same retry/dead-letter path an
+ # ordinary delivery failure uses, so a later delivery attempt
+ # resyncs and retries the commit from the still-``pending`` DB
+ # row. Every write below is fenced on (lease_owner, attempt) --
+ # see ``finish_run_if_owned``/``retry_run_with_event_if_owned`` -- so a
+ # worker that has lost its lease mid-flush can never revert a
+ # new owner's run back to pending or overwrite its status
+ # (review round 6, point 1, BLOCKER).
+ flush_error = {
+ "code": "steering_flush_failed",
+ "message": (
+ "durable steering consumption commit could not complete before this "
+ "worker gave up its lease; run finalization was withheld to avoid "
+ "reporting a false status or losing the consumption"
+ ),
+ }
+ await self._retry_or_dead_letter(run, flush_error)
+ # Do NOT forget_steering here. Not because this channel's local
+ # consumption log will be reused later -- ``with_runtime()``
+ # builds a brand new graph/channel on every delivery attempt
+ # (issue #16 PR #17 review round 5, point 2), so this exact
+ # object is discarded regardless. Recovery is driven entirely
+ # by the still-``pending`` DB row a future attempt re-reads via
+ # ``list_pending_steering``, not by anything held in this
+ # channel across the attempt boundary -- forgetting here is
+ # simply unnecessary, not unsafe.
+ finally:
+ renewal.cancel()
+ await asyncio.gather(renewal, return_exceptions=True)
+
+ @dataclasses.dataclass
+ class _Intent:
+ """The finalization outcome ``_execute`` intends to commit.
+
+ Held purely in memory until the final steering flush durably
+ succeeds -- see the long comment in ``_execute`` for why.
+ """
+
+ kind: str # "finish" or "retry"
+ status: RunStatus
+ output: dict[str, Any] | None = None
+ error: dict[str, Any] | None = None
+
+ async def _commit_intent(self, run: Run, intent: "Worker._Intent") -> bool:
+ """Write the previously-computed intended outcome, exactly once.
+
+ Only ever called after the final steering flush has durably
+ succeeded, so this is the single point where a terminal/paused
+ status first becomes externally observable.
+
+ Issue #16 PR #17 review round 6, points 1 and 2 (BLOCKER): this
+ write is fenced on our own ``(lease_owner, attempt)`` via
+ ``finish_run_if_owned``/``retry_run_with_event_if_owned`` -- if we no longer
+ hold the lease (a new worker claimed the run while our flush was
+ still in flight), the write is refused entirely and this returns
+ ``False`` so the caller does not call ``forget_steering()`` on a
+ channel whose consumption record may still matter. The fenced
+ write also re-checks the run's current control state atomically:
+ if it is ``cancelling``, cancellation wins over a stale
+ ``succeeded``/``failed`` intent regardless of arrival order.
+ """
+
+ if intent.kind == "retry":
+ result = await self.repository.retry_run_with_event_if_owned(
+ run.tenant_id,
+ run.id,
+ self.worker_id,
+ run.attempt,
+ error=intent.error,
+ max_attempts=self.max_delivery_attempts,
+ )
+ if result is None:
+ logger.warning(
+ "run retry commit abandoned: lease no longer owned",
+ extra={"run_id": run.id, "tenant_id": run.tenant_id},
+ )
+ return False
+ retried_run, event = result
+ await self._log_retry_outcome_event(run, retried_run, event)
+ return True
+
+ updated = await self.repository.finish_run_if_owned(
+ run.tenant_id,
+ run.id,
+ self.worker_id,
+ run.attempt,
+ intent.status,
+ output=intent.output,
+ error=intent.error,
+ )
+ if updated is None:
+ logger.warning(
+ "run finalize commit abandoned: lease no longer owned",
+ extra={"run_id": run.id, "tenant_id": run.tenant_id},
+ )
+ return False
+ self._log_finalize_outcome(run, updated, intent)
+ return True
+
+ @staticmethod
+ def _log_finalize_outcome(run: Run, updated: Run, intent: "Worker._Intent") -> None:
+ """Log the actual resulting status of a terminal/paused finalize
+ write -- factored out so both ``_commit_intent`` (the ``paused``
+ and non-steering-closing paths) and the merged
+ ``finalize_run_with_steering_disposition_if_owned`` path (issue
+ #16 PR #17 review round 9, point 2, BLOCKER) log identically.
+ """
+
+ final_status = updated.status if isinstance(updated.status, str) else updated.status.value
+ if final_status == RunStatus.SUCCEEDED.value:
+ logger.info(
+ "run succeeded",
+ extra={"run_id": run.id, "tenant_id": run.tenant_id, "status": "succeeded"},
+ )
+ elif final_status == RunStatus.CANCELLED.value and intent.status != RunStatus.CANCELLED:
+ logger.info(
+ "run cancelled during finalization, overriding stale %s intent",
+ intent.status.value if hasattr(intent.status, "value") else intent.status,
+ extra={"run_id": run.id, "tenant_id": run.tenant_id, "status": "cancelled"},
+ )
+ elif final_status in (RunStatus.DEAD_LETTER.value, RunStatus.FAILED.value) and intent.error:
+ logger.error(
+ "run delivery failed",
+ extra={
+ "run_id": run.id,
+ "tenant_id": run.tenant_id,
+ "status": final_status,
+ },
+ )
+
+ async def _log_retry_outcome_event(
+ self, run: Run, updated: Run, event: RunEvent | None
+ ) -> None:
+ """Publish/log the outcome of a merged retry-with-event write.
+
+ Issue #16 PR #17 review round 10, point 1 (BLOCKER): the status
+ transition and the ``worker_retrying`` event are now written
+ together, atomically, by
+ ``repository.retry_run_with_event_if_owned`` -- so unlike the old
+ ``_emit_retry_outcome_event`` this method never decides *whether*
+ to append an event; it only publishes the event the repository
+ already durably created (when the transition actually landed on
+ ``pending``) or logs the cancel-wins outcome (when it didn't,
+ ``event`` is ``None``).
+ """
+
+ final_status = updated.status if isinstance(updated.status, str) else updated.status.value
+ if event is None:
+ if final_status == RunStatus.CANCELLED.value:
+ logger.info(
+ "run cancelled during finalization, retry request superseded by cancel",
+ extra={"run_id": run.id, "tenant_id": run.tenant_id, "status": "cancelled"},
+ )
+ return
+ await self.event_bus.publish(run.tenant_id, run.id, event.sequence)
+ logger.warning(
+ "run delivery scheduled for retry",
+ extra={"run_id": run.id, "tenant_id": run.tenant_id, "status": "pending"},
+ )
+
+ async def _retry_or_dead_letter(self, run: Run, error: dict[str, Any]) -> None:
+ """Route a run that could not safely finalize into the ordinary
+ retry/dead-letter path, fenced on ``(lease_owner, attempt)``.
+
+ The only remaining caller is a final steering flush that never
+ durably succeeded (the worker is draining, or the lease was lost
+ to another worker mid-flush): the intended outcome computed in
+ ``_execute`` must never be written, so this instead resolves the
+ attempt the same way an ordinary mid-run delivery failure would --
+ retry while attempts remain, dead-letter once they are exhausted.
+
+ Historical note: round 7 also routed "steering landed too late to
+ be safely consumed" through this same path, forcing an indefinite
+ retry/dead-letter loop onto any run that ever had undrained
+ steering -- even a graph that legitimately never calls
+ ``drain_steering()`` at all. Round 8 replaced that with "seal
+ admission at the safe boundary + supersede late events" (see
+ ``finalize_run_with_steering_disposition_if_owned``): a steering
+ row still ``pending``/``delivered`` once the run reaches its
+ terminal finalization boundary is durably superseded, not routed
+ through a retry loop, so it no longer reaches this method at all.
+ And a genuine delivery ``retry`` intent (as opposed to a terminal
+ finalization failure) never closes steering admission in the first
+ place (review round 9, point 1, BLOCKER) -- it goes straight to
+ ``_commit_intent``, leaving steering fully open and recoverable by
+ the next delivery attempt, without ever passing through here.
+ """
+
+ if run.attempt < self.max_delivery_attempts:
+ result = await self.repository.retry_run_with_event_if_owned(
+ run.tenant_id,
+ run.id,
+ self.worker_id,
+ run.attempt,
+ error=error,
+ max_attempts=self.max_delivery_attempts,
+ )
+ if result is None:
+ logger.warning(
+ "run finalization abandoned: lease lost during retry",
+ extra={"run_id": run.id, "tenant_id": run.tenant_id},
+ )
+ return
+ retried_run, event = result
+ await self._log_retry_outcome_event(run, retried_run, event)
+ else:
+ updated = await self.repository.finish_run_if_owned(
+ run.tenant_id,
+ run.id,
+ self.worker_id,
+ run.attempt,
+ RunStatus.DEAD_LETTER,
+ error=error,
+ )
+ if updated is None:
+ logger.warning(
+ "run finalization abandoned: lease lost before dead-lettering",
+ extra={"run_id": run.id, "tenant_id": run.tenant_id},
+ )
+ return
+ logger.error(
+ "run dead-lettered: could not safely finalize after max attempts",
+ extra={
+ "run_id": run.id,
+ "tenant_id": run.tenant_id,
+ "status": updated.status
+ if isinstance(updated.status, str)
+ else updated.status.value,
+ },
+ )
+
+ async def _lease_renewal_loop(self, run: Run) -> None:
+ """Keep the run's lease alive through the whole finalization window.
+
+ Issue #16 PR #17 review round 6, point 1 (BLOCKER): coupling lease
+ renewal solely to the ordinary steering/event heartbeat -- which
+ ``_execute`` cancels the moment the graph finishes -- lets the
+ lease simply expire while the final flush is still retrying,
+ letting a new worker claim (and even finish) the run out from
+ under us. This loop renews the lease on its own schedule for as
+ long as it runs; ``_execute`` cancels it once finalization (flush
+ + fenced commit) is fully done, one way or another.
+ """
+
+ try:
+ while True:
+ await asyncio.sleep(self.heartbeat_seconds)
+ alive = await self.repository.heartbeat(
+ run.tenant_id, run.id, self.worker_id, lease_seconds=self.lease_seconds
+ )
+ if not alive:
+ return
+ except asyncio.CancelledError:
+ raise
+
+ async def _final_steering_flush(
+ self, run: Run, graph: Any, *, base_delay: float = 0.5, max_delay: float = 10.0
+ ) -> bool:
+ """Durable flush of drained steering before finalizing.
+
+ Unlike :meth:`_sync_steering_out` (used at heartbeat ticks, where a
+ transient failure can simply be retried at the next tick), this is
+ called once the graph has already reached its terminal/paused
+ outcome and the heartbeat loop has stopped -- there is no later
+ safe point *within this delivery attempt*. Critically, there is
+ also no safe "next delivery attempt" to fall back on the way an
+ ordinary mid-run failure has: ``_execute`` returning discards this
+ locally-bound ``graph`` (and therefore its steering channel and
+ local consumption log) entirely, because ``with_runtime`` builds a
+ brand new ``CompiledStateGraph`` -- with a brand new, empty
+ ``_run_steering`` map -- on every call (issue #16 PR #17 review
+ round 5, point 2). A future redelivery would construct a fresh
+ channel with no memory of what this attempt already drained,
+ risking double-execution or a permanently stuck ``pending`` row.
+
+ So instead of a small bounded number of attempts, this retries
+ with capped exponential backoff for as long as it continues to
+ hold the run's lease and the worker is not draining, renewing the
+ lease between attempts so another worker does not reclaim the run
+ (and construct its own competing channel) out from under it mid
+ retry. It only gives up -- returning ``False`` -- if the worker
+ itself is shutting down, or the lease is confirmed lost to another
+ worker; both are cases where holding on any longer would be
+ pointless or actively harmful, and the caller falls back to the
+ ordinary retry/dead-letter path.
+ """
+
+ channel = graph.get_steering_channel(run.id)
+ attempt = 0
+ # Iterative capped growth rather than ``base_delay * 2 ** (attempt
+ # - 1)``: the latter computes a potentially enormous exponent
+ # before ``min()`` ever clamps it, which is wasted (and, given an
+ # unbounded retry count against a very long DB outage,
+ # eventually pathological) work. Doubling and clamping in place
+ # reaches the same steady-state ``max_delay`` without ever
+ # computing an intermediate value larger than that cap.
+ delay = base_delay
while True:
- await asyncio.sleep(self.heartbeat_seconds)
+ consumed = channel.peek_consumed()
+ if not consumed:
+ return True
+ attempt += 1
+ try:
+ # Issue #16 PR #17 review round 7, point 2 (BLOCKER): fenced
+ # on our own (lease_owner, attempt) -- see
+ # ``commit_steering_consumptions_if_owned``'s docstring. A
+ # ``None`` result means our lease has already been reclaimed
+ # by a new owner; nothing was written, so we must not ack
+ # locally either -- fall through to the "lost the lease"
+ # exit below, same as an exhausted retry.
+ stored_events = await self.repository.commit_steering_consumptions_if_owned(
+ run.tenant_id, run.id, self.worker_id, run.attempt, consumed
+ )
+ except Exception:
+ logger.warning(
+ "final steering flush attempt %d failed",
+ attempt,
+ extra={"run_id": run.id, "tenant_id": run.tenant_id},
+ exc_info=True,
+ )
+ if self._stop.is_set():
+ return False
+ await asyncio.sleep(delay)
+ delay = min(max_delay, delay * 2)
+ # Keep the lease alive while we keep retrying so another
+ # worker does not reclaim this run mid-flush and construct
+ # a competing graph/channel of its own.
+ alive = await self.repository.heartbeat(
+ run.tenant_id, run.id, self.worker_id, lease_seconds=self.lease_seconds
+ )
+ if not alive:
+ return False
+ continue
+ if stored_events is None:
+ logger.warning(
+ "final steering flush abandoned: lease no longer owned",
+ extra={"run_id": run.id, "tenant_id": run.tenant_id},
+ )
+ return False
+ channel.ack_consumed(consumption.event.id for consumption in consumed)
+ for stored in stored_events:
+ await self.event_bus.publish(run.tenant_id, run.id, stored.sequence)
+ return True
+
+ async def _heartbeat(self, run: Run, token: CancellationToken, graph: Any) -> None:
+ while True:
+ # Issue #16 PR #17 review round 4, point 5: wait on the
+ # EventBus (Redis, when configured) instead of a plain
+ # ``asyncio.sleep`` so a ``/steer`` call's ``publish()`` can
+ # wake this loop early -- low-latency steering discovery --
+ # while ``timeout=self.heartbeat_seconds`` still bounds the
+ # wait, so this remains a correctness-preserving fallback to
+ # the unconditional PostgreSQL poll below even if the publish
+ # is dropped, Redis is unavailable, or no EventBus wake ever
+ # arrives at all (``InMemoryEventBus``/``RedisEventBus.wait``
+ # both return on timeout rather than raising).
+ await self.event_bus.wait(run.tenant_id, run.id, timeout=self.heartbeat_seconds)
+ # Guarantee a real scheduler yield every iteration regardless of
+ # how quickly a particular ``EventBus.wait`` implementation
+ # returns (a degenerate/no-op bus could otherwise return without
+ # ever suspending, starving the run's own coroutine).
+ await asyncio.sleep(0)
if await self.repository.is_cancel_requested(run.tenant_id, run.id):
+ # Cancellation always takes priority; steering must never
+ # undo or delay it. We still sync steering below so an
+ # already-accepted event is not lost, but we do not let a
+ # steer block or reorder the cancel signal.
token.cancel()
+ # PostgreSQL is the source of truth for steering; this poll
+ # keeps working even if a Redis ``run.steer.available`` notify
+ # was dropped or Redis is unavailable entirely.
+ await self._sync_steering_in(run, graph)
+ await self._sync_steering_out(run, graph)
alive = await self.repository.heartbeat(
run.tenant_id,
run.id,
@@ -269,6 +764,79 @@ async def _heartbeat(self, run: Run, token: CancellationToken) -> None:
token.cancel()
return
+ async def _sync_steering_in(self, run: Run, graph: Any) -> None:
+ """Pull durably-accepted-but-not-yet-delivered events into the graph."""
+
+ pending = await self.repository.list_pending_steering(run.tenant_id, run.id)
+ if not pending:
+ return
+ channel = graph.get_steering_channel(run.id)
+ for row in pending:
+ channel.ingest(
+ SteeringEvent(
+ id=row.id,
+ run_id=run.id,
+ sequence=row.sequence,
+ kind=row.kind,
+ payload=row.payload,
+ metadata=row.metadata,
+ created_at=row.created_at,
+ source_event_id=row.source_event_id,
+ )
+ )
+
+ async def _sync_steering_out(self, run: Run, graph: Any) -> None:
+ """Flush events the graph has drained back to PostgreSQL + SSE.
+
+ Reads (peeks) the channel's local consumption log, durably commits
+ it -- steering status *and* the ``run.steer.consumed`` lifecycle
+ event, together, in one repository call -- and only then acks
+ those entries out of the local log. If the durable commit fails
+ (e.g. a transient DB error), nothing is acked: the entries stay in
+ the channel's consumption log and are retried on the next sync
+ (next heartbeat tick, or the final flush in ``_execute``). This is
+ the issue #16 PR #17 review point 2 fix -- the previous
+ pop-then-write ordering could destroy the local record before the
+ durable write was known to have succeeded, permanently losing a
+ consumption (and its observability event) on a transient failure.
+ """
+
+ channel = graph.get_steering_channel(run.id)
+ consumed = channel.peek_consumed()
+ if not consumed:
+ return
+ try:
+ # Issue #16 PR #17 review round 7, point 2 (BLOCKER): fenced on
+ # our own (lease_owner, attempt) -- a lease takeover mid-
+ # heartbeat has the identical hole the final flush did. A
+ # ``None`` result means a new worker already claimed this run
+ # (attempt+1); nothing was written, and we must not ack this
+ # batch locally -- it stays in the channel's consumption log,
+ # which is about to be discarded anyway once this delivery
+ # attempt's ``_execute`` notices the lost lease.
+ stored_events = await self.repository.commit_steering_consumptions_if_owned(
+ run.tenant_id, run.id, self.worker_id, run.attempt, consumed
+ )
+ except Exception:
+ logger.warning(
+ "steering consumption commit failed; will retry at the next safe point",
+ extra={"run_id": run.id, "tenant_id": run.tenant_id},
+ exc_info=True,
+ )
+ return
+ if stored_events is None:
+ logger.warning(
+ "steering consumption commit skipped: lease no longer owned",
+ extra={"run_id": run.id, "tenant_id": run.tenant_id},
+ )
+ return
+ # Only ack the ids we actually just committed -- a concurrent
+ # drain() may have appended more to the log between the peek above
+ # and here, and those must be left for the next cycle.
+ channel.ack_consumed(consumption.event.id for consumption in consumed)
+ for stored in stored_events:
+ await self.event_bus.publish(run.tenant_id, run.id, stored.sequence)
+
async def _append_event(self, run: Run, event: Event) -> None:
encoded = self._serializer.dumps(dataclasses.asdict(event))
if len(encoded) > self.repository.limits.max_event_bytes:
diff --git a/src/lingxigraph/steering.py b/src/lingxigraph/steering.py
new file mode 100644
index 0000000..e60abc1
--- /dev/null
+++ b/src/lingxigraph/steering.py
@@ -0,0 +1,331 @@
+"""Durable mid-run steering: provider-neutral data model + in-process channel.
+
+Steering lets an external caller inject new structured input into a Run
+while it is executing. LingxiGraph guarantees durable delivery, ordering,
+dedup and safe consumption; it never interprets what a steering event
+*means* -- that is entirely up to the application graph (see
+``runtime.drain_steering()`` in :mod:`lingxigraph.runtime`).
+
+This module is intentionally dependency-free so it works the same way in
+embedded/library usage (no Agent Server, no PostgreSQL) as it does behind
+the HTTP API -- the "degraded" implementation for SQLite/in-memory/embedded
+runtimes *is* this module, not a separate code path.
+"""
+
+from __future__ import annotations
+
+import threading
+from collections.abc import Iterable, Mapping
+from dataclasses import dataclass, field
+from datetime import UTC, datetime
+from typing import Any
+from uuid import uuid4
+
+#: Default cap on the serialized size of a steering event payload+metadata.
+MAX_STEERING_PAYLOAD_BYTES = 32_768
+
+
+class SteeringPayloadTooLarge(ValueError):
+ """Raised when a steering event payload exceeds the configured size cap."""
+
+
+@dataclass(frozen=True, slots=True)
+class SteeringEvent:
+ """A single durable, ordered piece of external input for a Run.
+
+ ``id`` doubles as the idempotency key used for dedup; ``sequence`` is a
+ monotonically increasing, per-run ordering position assigned at
+ acceptance time. ``payload``/``metadata`` must already be safe to
+ serialize through :mod:`lingxigraph.serialization`.
+ """
+
+ id: str
+ run_id: str
+ sequence: int
+ kind: str
+ payload: Mapping[str, Any]
+ metadata: Mapping[str, Any] = field(default_factory=dict)
+ created_at: datetime = field(default_factory=lambda: datetime.now(UTC))
+ #: Set when this event is a paused-run -> resumed-run transfer of an
+ #: originally-accepted event (see ``Repository.resume_run_with_pending_
+ #: steering`` / issue #16 PR #17 review point 3): the id of the
+ #: original event a client's ``/steer`` call actually received, so
+ #: downstream ``run.steer.consumed`` observability can be correlated
+ #: back to it even though ``id`` itself changed across the transfer.
+ source_event_id: str | None = None
+
+
+def validate_steering_payload(
+ payload: Mapping[str, Any],
+ metadata: Mapping[str, Any] | None = None,
+ *,
+ max_bytes: int = MAX_STEERING_PAYLOAD_BYTES,
+) -> None:
+ """Raise :class:`SteeringPayloadTooLarge` if the payload is oversized.
+
+ Also verifies the payload goes through the safe JSON serialization
+ boundary (no unsafe/unserializable values).
+ """
+
+ from .serialization import JsonSerializer, SerializationError
+
+ serializer = JsonSerializer()
+ try:
+ encoded = serializer.dumps(dict(payload))
+ encoded_metadata = serializer.dumps(dict(metadata or {}))
+ except SerializationError as exc:
+ raise SteeringPayloadTooLarge(f"steering payload is not safely serializable: {exc}") from exc
+ total = len(encoded) + len(encoded_metadata)
+ if total > max_bytes:
+ raise SteeringPayloadTooLarge(
+ f"steering payload size {total} exceeds max_bytes={max_bytes}"
+ )
+
+
+def new_steering_id() -> str:
+ return str(uuid4())
+
+
+@dataclass(frozen=True, slots=True)
+class SteeringConsumption:
+ """Where and how long an event waited before a safe point drained it.
+
+ Produced by :meth:`SteeringChannel.drain` for every event it returns,
+ and surfaced end-to-end as the ``run.steer.consumed`` observability
+ event's payload (see :mod:`lingxigraph.server.worker`) so a production
+ deployment can answer "where did this get consumed, and how long did it
+ queue" -- the two pieces of information issue #16's acceptance criteria
+ call out that a bare ``steering_event_id/sequence/kind`` record cannot
+ answer on its own.
+ """
+
+ event: SteeringEvent
+ consumed_at: datetime
+ #: The node (task) whose call to ``runtime.drain_steering()`` drained
+ #: this event -- ``None`` when drained outside node execution (e.g. a
+ #: direct ``channel.drain()`` call with no consumer context attached).
+ node: str | None = None
+ #: The subgraph namespace path of that node, e.g. ``("supervisor",)``.
+ namespace: tuple[str, ...] = ()
+ #: The specific task id (distinguishes parallel/``Send`` fan-out tasks
+ #: that share one node name within the same superstep).
+ task_id: str | None = None
+
+ @property
+ def queue_latency_seconds(self) -> float:
+ """Wall-clock time between durable acceptance and consumption."""
+
+ return max(0.0, (self.consumed_at - self.event.created_at).total_seconds())
+
+
+class SteeringChannel:
+ """Thread-safe, ordered, dedup'd inbox of :class:`SteeringEvent`.
+
+ This is the concrete "safe-point" primitive used by
+ :class:`lingxigraph.runtime.Runtime`. It is deliberately simple:
+
+ * ``submit`` is idempotent by ``event.id`` -- resubmitting the same id
+ is a no-op (supports at-least-once delivery from a durable inbox).
+ * ``drain`` atomically returns *and clears* all currently pending
+ events, in ascending ``sequence`` order. Once drained, an event is
+ never re-exposed by this channel again (though upstream at-least-once
+ delivery may still resubmit the same id from PostgreSQL after a
+ crash -- callers must dedup on ``event.id`` if that matters to them).
+ * ``peek`` is the read-only variant: it does not consume anything.
+
+ A ``on_drain`` callback (sync, best-effort) can be supplied so a server
+ integration (e.g. the durable-repository-backed worker) can mark events
+ ``consumed`` in PostgreSQL when the graph actually drains them.
+ """
+
+ def __init__(self, run_id: str = "", *, owned_by_executor: bool = True) -> None:
+ self.run_id = run_id
+ self._lock = threading.Lock()
+ self._pending: dict[str, SteeringEvent] = {}
+ self._seen_ids: set[str] = set()
+ self._next_sequence = 1
+ self._consumed_log: list[SteeringConsumption] = []
+ self.on_drain: Any = None
+ #: Whether the graph executor's own embedded run lifecycle may
+ #: release (``forget``) this channel automatically once its run
+ #: finishes. ``True`` for channels created implicitly by
+ #: ``CompiledStateGraph.steer()``/``_run()`` (plain embedded
+ #: ``invoke``/``ainvoke`` with no server involved). A server
+ #: integration explicitly registers a channel ahead of time via
+ #: ``get_steering_channel()`` and flips this to ``False`` -- it owns
+ #: the channel's lifecycle end to end (see ``Worker._execute``'s
+ #: explicit ``forget_steering`` call) since a queued run may receive
+ #: steering before any worker has claimed it.
+ self.owned_by_executor = owned_by_executor
+
+ def submit(
+ self,
+ *,
+ kind: str,
+ payload: Mapping[str, Any],
+ metadata: Mapping[str, Any] | None = None,
+ idempotency_key: str | None = None,
+ max_bytes: int = MAX_STEERING_PAYLOAD_BYTES,
+ ) -> tuple[SteeringEvent, bool]:
+ """Append a new event. Returns ``(event, created)``.
+
+ ``created`` is ``False`` when ``idempotency_key`` matches an event
+ already known to this channel (pending or already drained).
+ """
+
+ validate_steering_payload(payload, metadata, max_bytes=max_bytes)
+ event_id = idempotency_key or new_steering_id()
+ with self._lock:
+ existing = self._pending.get(event_id)
+ if existing is not None:
+ return existing, False
+ if event_id in self._seen_ids:
+ # Already drained/consumed previously -- durable dedup.
+ return SteeringEvent(
+ id=event_id,
+ run_id=self.run_id,
+ sequence=-1,
+ kind=kind,
+ payload=payload,
+ metadata=metadata or {},
+ ), False
+ event = SteeringEvent(
+ id=event_id,
+ run_id=self.run_id,
+ sequence=self._next_sequence,
+ kind=kind,
+ payload=dict(payload),
+ metadata=dict(metadata or {}),
+ )
+ self._next_sequence += 1
+ self._pending[event_id] = event
+ self._seen_ids.add(event_id)
+ return event, True
+
+ def ingest(self, event: SteeringEvent) -> bool:
+ """Accept an already-constructed durable event (server integration).
+
+ Preserves the caller-provided sequence number when it is larger
+ than anything seen so far, so ordering follows the durable source
+ of truth (PostgreSQL) rather than local submission order.
+ """
+
+ with self._lock:
+ if event.id in self._seen_ids:
+ return False
+ self._seen_ids.add(event.id)
+ self._pending[event.id] = event
+ self._next_sequence = max(self._next_sequence, event.sequence + 1)
+ return True
+
+ @property
+ def has_pending(self) -> bool:
+ with self._lock:
+ return bool(self._pending)
+
+ def peek(self) -> tuple[SteeringEvent, ...]:
+ with self._lock:
+ return tuple(sorted(self._pending.values(), key=lambda item: item.sequence))
+
+ def drain(
+ self,
+ *,
+ node: str | None = None,
+ namespace: tuple[str, ...] = (),
+ task_id: str | None = None,
+ ) -> tuple[SteeringEvent, ...]:
+ """Atomically consume and return pending events, in order.
+
+ ``node``/``namespace``/``task_id`` identify *where* the drain
+ happened -- the executor's safe point (see
+ :meth:`lingxigraph.runtime.Runtime.drain_steering`) passes its own
+ task coordinates through so the consumption record (see
+ :meth:`pop_consumed`) can answer "which node consumed this, in
+ which subgraph namespace" for observability. Callers outside node
+ execution (e.g. a bare ``channel.drain()``) may omit them.
+ """
+
+ now = datetime.now(UTC)
+ with self._lock:
+ drained = tuple(sorted(self._pending.values(), key=lambda item: item.sequence))
+ self._pending.clear()
+ self._consumed_log.extend(
+ SteeringConsumption(
+ event=event, consumed_at=now, node=node, namespace=namespace, task_id=task_id
+ )
+ for event in drained
+ )
+ if drained and self.on_drain is not None:
+ try:
+ self.on_drain(drained)
+ except Exception:
+ # Observability/persistence side-effects must never break
+ # graph execution -- the DB inbox remains the source of
+ # truth and a later safe point can retry marking consumed.
+ pass
+ return drained
+
+ def pop_consumed(self) -> tuple[SteeringConsumption, ...]:
+ """Return and clear the log of events drained since the last call.
+
+ Destructive: callers that must durably persist a consumption
+ before it is safe to forget it locally (any server-mode worker --
+ see :meth:`peek_consumed`/:meth:`ack_consumed`) must **not** use
+ this method, since a transient failure between the pop and the
+ durable write would silently lose the record (issue #16 PR #17
+ review point 2). This remains the right call for a caller that
+ either has nothing durable to write (plain embedded/library usage)
+ or has already durably committed and just wants to drain its own
+ bookkeeping. Each entry is a :class:`SteeringConsumption`,
+ carrying queue latency and the consuming node/namespace/task_id
+ alongside the raw event.
+ """
+
+ with self._lock:
+ consumed = tuple(self._consumed_log)
+ self._consumed_log.clear()
+ return consumed
+
+ def peek_consumed(self) -> tuple[SteeringConsumption, ...]:
+ """Read-only view of the not-yet-acked consumption log.
+
+ Use with :meth:`ack_consumed` for durable consumers: read the
+ entries, durably commit them (DB status update + lifecycle event),
+ and only then ack the specific ids that succeeded. Unlike
+ :meth:`pop_consumed` this never destroys data a failed durable
+ write could not yet account for.
+ """
+
+ with self._lock:
+ return tuple(self._consumed_log)
+
+ def ack_consumed(self, event_ids: Iterable[str]) -> None:
+ """Remove specific entries from the consumption log once durably committed.
+
+ Only entries whose ``event.id`` is in ``event_ids`` are removed --
+ anything appended by a concurrent :meth:`drain` between the
+ matching :meth:`peek_consumed` and this call is left alone so it
+ gets picked up on the next cycle instead of being silently
+ dropped.
+ """
+
+ ids = set(event_ids)
+ if not ids:
+ return
+ with self._lock:
+ if not self._consumed_log:
+ return
+ self._consumed_log[:] = [
+ entry for entry in self._consumed_log if entry.event.id not in ids
+ ]
+
+
+__all__ = [
+ "MAX_STEERING_PAYLOAD_BYTES",
+ "SteeringChannel",
+ "SteeringConsumption",
+ "SteeringEvent",
+ "SteeringPayloadTooLarge",
+ "new_steering_id",
+ "validate_steering_payload",
+]
diff --git a/tests/test_agent_server.py b/tests/test_agent_server.py
index 4dc30b8..d8608e6 100644
--- a/tests/test_agent_server.py
+++ b/tests/test_agent_server.py
@@ -286,6 +286,186 @@ def test_resource_store_schedule_and_cancel_lifecycle(self) -> None:
204,
)
+ def test_cancel_paused_run_terminates_immediately(self) -> None:
+ """Issue #16 PR #17 review round 12 (BLOCKER): cancelling a ``paused``
+ run must resolve to ``cancelled`` immediately instead of flipping to
+ a lease-less ``cancelling`` that no recovery mechanism (including
+ round 11's expired-lease recovery) can ever unstick, because a
+ paused run never held a lease in the first place.
+ """
+ app = create_app(
+ registry=make_registry(),
+ authenticator=Authenticator.insecure_dev(),
+ embedded_worker=True,
+ )
+ headers = {"x-tenant-id": "acme"}
+ with TestClient(app) as client:
+ assistant = client.post(
+ "/v1/assistants",
+ headers=headers,
+ json={"graph_id": "approval", "name": "approval"},
+ ).json()
+ thread = client.post("/v1/threads", headers=headers, json={}).json()
+ paused = client.post(
+ f"/v1/threads/{thread['id']}/runs",
+ headers=headers,
+ json={"assistant_id": assistant["id"], "input": {"value": 0}},
+ ).json()
+ paused = wait_for_status(client, paused["id"], headers, {"paused"}).json()
+ self.assertEqual(paused["status"], "paused")
+
+ # Requirement 4: a paused run with pending steering must keep
+ # that steering durable -- this fix must not implicitly
+ # consume/replay/supersede it.
+ steer = client.post(
+ f"/v1/runs/{paused['id']}/steer",
+ headers=headers,
+ json={"kind": "note", "payload": {"text": "still relevant"}},
+ )
+ self.assertEqual(steer.status_code, 202)
+
+ # Regression 1 (InMemory, direct repository call against a
+ # genuinely paused run produced by the Worker, not a
+ # hand-faked status).
+ repository = app.state.repository
+ self.assertIsNone(asyncio.run(repository.get_run("acme", paused["id"])).lease_owner)
+ cancel_result = asyncio.run(repository.request_cancel("acme", paused["id"]))
+ self.assertTrue(cancel_result)
+ after_cancel = asyncio.run(repository.get_run("acme", paused["id"]))
+ self.assertEqual(after_cancel.status, "cancelled")
+ self.assertIsNotNone(after_cancel.finished_at)
+ self.assertIsNone(after_cancel.lease_owner)
+ self.assertIsNone(after_cancel.lease_expires_at)
+ claimed = asyncio.run(repository.claim_run("worker", lease_seconds=30))
+ self.assertIsNone(claimed)
+
+ # Steering submitted while paused stays durable history -- not
+ # implicitly dropped by the cancel-while-paused fix.
+ events = asyncio.run(repository.list_events("acme", paused["id"]))
+ self.assertTrue(any(event.kind == "run.steer.accepted" for event in events))
+
+ def test_cancel_paused_run_http_e2e(self) -> None:
+ """Regression 2 (HTTP E2E): the full public control-API surface
+ must observe the paused-run cancel as an immediate terminal state,
+ with resume and a new-idempotency-key steer both rejected 409.
+ """
+ app = create_app(
+ registry=make_registry(),
+ authenticator=Authenticator.insecure_dev(),
+ embedded_worker=True,
+ )
+ headers = {"x-tenant-id": "acme"}
+ with TestClient(app) as client:
+ assistant = client.post(
+ "/v1/assistants",
+ headers=headers,
+ json={"graph_id": "approval", "name": "approval"},
+ ).json()
+ thread = client.post("/v1/threads", headers=headers, json={}).json()
+ paused = client.post(
+ f"/v1/threads/{thread['id']}/runs",
+ headers=headers,
+ json={"assistant_id": assistant["id"], "input": {"value": 0}},
+ ).json()
+ paused = wait_for_status(client, paused["id"], headers, {"paused"}).json()
+ self.assertEqual(paused["status"], "paused")
+
+ cancelled = client.post(f"/v1/runs/{paused['id']}/cancel", headers=headers)
+ self.assertEqual(cancelled.status_code, 200, cancelled.text)
+ self.assertEqual(cancelled.json()["status"], "cancelled")
+
+ joined = client.get(f"/v1/runs/{paused['id']}/join", headers=headers)
+ self.assertEqual(joined.status_code, 200, joined.text)
+ self.assertEqual(joined.json()["status"], "cancelled")
+
+ resume_after_cancel = client.post(
+ f"/v1/runs/{paused['id']}/resume",
+ headers=headers,
+ json={"resume": 7},
+ )
+ self.assertEqual(resume_after_cancel.status_code, 409)
+
+ steer_after_cancel = client.post(
+ f"/v1/runs/{paused['id']}/steer",
+ headers={**headers, "Idempotency-Key": "post-cancel-key"},
+ json={"kind": "note", "payload": {"text": "too late"}},
+ )
+ self.assertEqual(steer_after_cancel.status_code, 409)
+ self.assertEqual(steer_after_cancel.json()["code"], "run_terminal")
+
+ def test_cancel_of_a_resumed_run_returns_409_and_the_new_run_is_still_cancellable(
+ self,
+ ) -> None:
+ """Issue #16 PR #17 review round 13 (BLOCKER), HTTP E2E: resuming
+ ``A`` into ``B`` and then calling ``/cancel`` on the now-stale
+ ``A`` must return ``409 run_superseded`` -- not a false-success 200
+ -- and ``B`` (the run actually still executing) must remain
+ genuinely cancellable afterwards.
+ """
+ app = create_app(
+ registry=make_registry(),
+ authenticator=Authenticator.insecure_dev(),
+ embedded_worker=True,
+ )
+ headers = {"x-tenant-id": "acme"}
+ with TestClient(app) as client:
+ assistant = client.post(
+ "/v1/assistants",
+ headers=headers,
+ json={"graph_id": "approval", "name": "approval"},
+ ).json()
+ thread = client.post("/v1/threads", headers=headers, json={}).json()
+ paused = client.post(
+ f"/v1/threads/{thread['id']}/runs",
+ headers=headers,
+ json={"assistant_id": assistant["id"], "input": {"value": 0}},
+ ).json()
+ paused = wait_for_status(client, paused["id"], headers, {"paused"}).json()
+ self.assertEqual(paused["status"], "paused")
+
+ resumed = client.post(
+ f"/v1/runs/{paused['id']}/resume",
+ headers=headers,
+ json={"resume": 7},
+ )
+ self.assertEqual(resumed.status_code, 202, resumed.text)
+ new_run_id = resumed.json()["id"]
+ self.assertNotEqual(new_run_id, paused["id"])
+
+ # Cancelling the stale, resumed-away run must be rejected as
+ # superseded -- never a false-success 200 -- and the caller
+ # must be told exactly which run_id to target instead.
+ cancel_old = client.post(f"/v1/runs/{paused['id']}/cancel", headers=headers)
+ self.assertEqual(cancel_old.status_code, 409, cancel_old.text)
+ self.assertEqual(cancel_old.json()["code"], "run_superseded")
+ self.assertIn(new_run_id, cancel_old.json()["detail"])
+
+ # The old run must not have been mutated to a terminal state
+ # by the rejected cancel attempt.
+ old_after = client.get(f"/v1/runs/{paused['id']}", headers=headers).json()
+ self.assertEqual(old_after["status"], "paused")
+
+ # Cancelling the actual resumed descendant genuinely works
+ # (unless the tiny "approval" graph already raced to a
+ # terminal success before the cancel landed -- either way,
+ # it must never be the superseded 409 the old run gets).
+ new_run = client.get(f"/v1/runs/{new_run_id}", headers=headers).json()
+ self.assertNotEqual(new_run["status"], "cancelled")
+ cancel_new = client.post(f"/v1/runs/{new_run_id}/cancel", headers=headers)
+ # It must never be rejected as *superseded* -- that would mean
+ # this endpoint's own descendant is (wrongly) being treated as
+ # stale. A plain 409 "conflict" is acceptable if the tiny
+ # "approval" graph already raced to a terminal success before
+ # the cancel landed.
+ self.assertNotEqual(
+ cancel_new.json().get("code"), "run_superseded", cancel_new.text
+ )
+ if cancel_new.status_code == 200:
+ joined_new = wait_for_status(
+ client, new_run_id, headers, {"cancelled", "succeeded"}
+ ).json()
+ self.assertIn(joined_new["status"], {"cancelled", "succeeded"})
+
def test_interrupt_resume_and_protocol_gateways(self) -> None:
app = create_app(
registry=make_registry(),
diff --git a/tests/test_mvp_hardening.py b/tests/test_mvp_hardening.py
index 4a72620..8f2c2b4 100644
--- a/tests/test_mvp_hardening.py
+++ b/tests/test_mvp_hardening.py
@@ -552,6 +552,20 @@ def test_api_idempotency_request_limit_and_readiness(self) -> None:
content=b"x" * 513,
)
self.assertEqual(understated.status_code, 413)
+ invalid_length = client.post(
+ "/v1/runs",
+ headers={**headers, "content-length": "not-a-number"},
+ content=b"{}",
+ )
+ self.assertEqual(invalid_length.status_code, 400)
+ self.assertEqual(invalid_length.json()["code"], "invalid_request")
+ negative_length = client.post(
+ "/v1/runs",
+ headers={**headers, "content-length": "-1"},
+ content=b"{}",
+ )
+ self.assertEqual(negative_length.status_code, 400)
+ self.assertEqual(negative_length.json()["code"], "invalid_request")
class UnhealthyRepository(InMemoryRepository):
async def healthcheck(self) -> bool:
diff --git a/tests/test_platform_units.py b/tests/test_platform_units.py
index f34a34e..4376b39 100644
--- a/tests/test_platform_units.py
+++ b/tests/test_platform_units.py
@@ -223,6 +223,53 @@ async def aclose(self):
asyncio.run(bus.close())
redis.publish.assert_awaited_once()
+ def test_redis_wait_does_not_busy_loop_when_get_message_raises(self) -> None:
+ """Issue #16 PR #17 review round 7, point 4: if Redis disconnects
+ right after ``subscribe()`` succeeds, every subsequent
+ ``get_message()`` call can raise immediately. Before this fix,
+ ``_wait_cancellation_safe`` folded that exception into ``None`` --
+ indistinguishable from "no message yet" -- so ``RedisEventBus.
+ wait()``'s deadline loop just kept calling ``get_message()`` again
+ with no backoff for the rest of the timeout window. The exception
+ must now propagate out to ``wait()``'s own outer ``except
+ Exception: return``, so ``wait()`` returns promptly and
+ ``get_message`` is called only once."""
+
+ call_count = 0
+
+ class FlakyPubSub:
+ async def subscribe(self, _channel):
+ return None
+
+ async def get_message(self, **_kwargs):
+ nonlocal call_count
+ call_count += 1
+ raise ConnectionError("simulated redis disconnect")
+
+ async def aclose(self):
+ return None
+
+ redis = SimpleNamespace(
+ publish=AsyncMock(return_value=1),
+ pubsub=lambda: FlakyPubSub(),
+ aclose=AsyncMock(),
+ )
+ with patch("redis.asyncio.Redis.from_url", return_value=redis):
+ bus = RedisEventBus("redis://test")
+
+ async def scenario():
+ start = asyncio.get_event_loop().time()
+ await bus.wait("tenant", "run", timeout=5.0)
+ elapsed = asyncio.get_event_loop().time() - start
+ return elapsed
+
+ elapsed = asyncio.run(scenario())
+ # A busy-loop would burn through the whole 5s timeout window
+ # calling get_message() thousands of times; the fix returns almost
+ # immediately after the single raising call.
+ self.assertEqual(call_count, 1)
+ self.assertLess(elapsed, 1.0)
+
def test_redis_cache_roundtrip_clear_and_sync_wrappers(self) -> None:
serializer = JsonSerializer()
diff --git a/tests/test_postgres_integration.py b/tests/test_postgres_integration.py
index 4c13212..3261f67 100644
--- a/tests/test_postgres_integration.py
+++ b/tests/test_postgres_integration.py
@@ -108,7 +108,7 @@ class State(dict):
)
conn.execute(f'GRANT USAGE ON SCHEMA "{self.schema}" TO "{role}"')
conn.execute(
- f'GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA '
+ f"GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA "
f'"{self.schema}" TO "{role}"'
)
conn.execute(
@@ -132,6 +132,1388 @@ class State(dict):
conn.execute(f'DROP OWNED BY "{role}"')
conn.execute(f'DROP ROLE "{role}"')
+ def test_resume_run_with_pending_steering_is_atomic_and_race_free(self) -> None:
+ """Issue #16 PR #17 review point 1, PostgreSQL path.
+
+ Regresses both halves of the finding against a real PostgreSQL
+ transaction:
+
+ * A worker hammering ``claim_run`` concurrently with
+ ``resume_run_with_pending_steering`` must never observe the
+ resumed Run before its migrated steering is already present --
+ the previous two-transaction implementation (separate
+ ``create_run`` + ``transfer_pending_steering`` commits) allowed a
+ real window where the new Run was claimable (and even
+ finishable) with nothing migrated onto it.
+ * Sequence numbers assigned to the migrated events and to an
+ ordinary steer submitted against the resumed run afterwards must
+ never collide -- the previous ``_transfer_pending_steering_sync``
+ only locked the *old* run row, not the new one, so its
+ ``MAX(sequence)+1`` could race a concurrent ``_submit_steering_
+ sync`` doing the same computation for the new run.
+ """
+
+ from lingxigraph.server.models import AssistantCreate, RunCreate, ThreadCreate
+ from lingxigraph.server.repository import PostgresRepository
+
+ async def scenario() -> None:
+ repo = PostgresRepository(POSTGRES_URL, schema=self.schema)
+ await repo.setup()
+ assistant = await repo.create_assistant(
+ "acme", AssistantCreate(graph_id="graph", name="a"), "1.0.0"
+ )
+ thread = await repo.create_thread("acme", ThreadCreate())
+ old_run = await repo.create_run(
+ "acme", thread.id, assistant, RunCreate(assistant_id=assistant.id, input={})
+ )
+ await repo.submit_steering("acme", old_run.id, kind="user_input", payload={"m": 1})
+ await repo.submit_steering("acme", old_run.id, kind="user_input", payload={"m": 2})
+ await repo.finish_run("acme", old_run.id, "paused", output={})
+
+ claims: list[str] = []
+ stop = asyncio.Event()
+
+ async def hammer_claim() -> None:
+ while not stop.is_set():
+ claimed = await repo.claim_run("worker-a", lease_seconds=30)
+ if claimed is not None:
+ claims.append(claimed.id)
+ if claimed.id != old_run.id:
+ pending = await repo.list_pending_steering("acme", claimed.id)
+ self.assertEqual(
+ len(pending),
+ 2,
+ "resumed run was claimable before its steering migrated",
+ )
+ await asyncio.sleep(0)
+
+ hammer = asyncio.create_task(hammer_claim())
+ request = RunCreate(assistant_id=assistant.id, resume=1)
+ new_run, transferred = await repo.resume_run_with_pending_steering(
+ "acme", thread.id, assistant, request, old_run.id
+ )
+ self.assertEqual([event.sequence for event in transferred], [1, 2])
+ for _ in range(1000):
+ if new_run.id in claims:
+ break
+ await asyncio.sleep(0.002)
+ stop.set()
+ await hammer
+ self.assertIn(new_run.id, claims)
+
+ extra, _ = await repo.submit_steering(
+ "acme", new_run.id, kind="user_input", payload={"m": 3}
+ )
+ self.assertEqual(extra.sequence, 3)
+ all_events = await repo.list_steering("acme", new_run.id)
+ self.assertEqual(sorted(event.sequence for event in all_events), [1, 2, 3])
+
+ # Identity survives the transfer (review point 3).
+ self.assertTrue(
+ all(event.source_event_id for event in all_events if event.sequence in (1, 2))
+ )
+
+ asyncio.run(scenario())
+
+ def test_concurrent_resume_of_the_same_paused_run_creates_exactly_one_descendant(
+ self,
+ ) -> None:
+ """Issue #16 PR #17 review round 4, point 1, PostgreSQL path.
+
+ Two real concurrent transactions both calling
+ ``resume_run_with_pending_steering`` against the same paused run
+ must not both succeed: PostgreSQL's ``FOR UPDATE`` on the old run
+ row serializes them, but the loser must be rejected by the locked
+ revalidation (still ``paused``, no ``superseded_by_run_id`` yet)
+ rather than silently creating a second descendant Run and
+ overwriting the winner's ``superseded_by_run_id``.
+ """
+
+ from lingxigraph.errors import RunResumeConflictError
+ from lingxigraph.server.models import AssistantCreate, RunCreate, ThreadCreate
+ from lingxigraph.server.repository import PostgresRepository
+
+ async def scenario() -> None:
+ repo = PostgresRepository(POSTGRES_URL, schema=self.schema)
+ await repo.setup()
+ assistant = await repo.create_assistant(
+ "acme", AssistantCreate(graph_id="graph", name="a"), "1.0.0"
+ )
+ thread = await repo.create_thread("acme", ThreadCreate())
+ old_run = await repo.create_run(
+ "acme", thread.id, assistant, RunCreate(assistant_id=assistant.id, input={})
+ )
+ await repo.finish_run("acme", old_run.id, "paused", output={})
+
+ request = RunCreate(assistant_id=assistant.id, resume=1)
+ results = await asyncio.gather(
+ repo.resume_run_with_pending_steering(
+ "acme", thread.id, assistant, request, old_run.id
+ ),
+ repo.resume_run_with_pending_steering(
+ "acme", thread.id, assistant, request, old_run.id
+ ),
+ return_exceptions=True,
+ )
+ successes = [r for r in results if not isinstance(r, BaseException)]
+ failures = [r for r in results if isinstance(r, BaseException)]
+ self.assertEqual(len(successes), 1, results)
+ self.assertEqual(len(failures), 1, results)
+ self.assertIsInstance(failures[0], RunResumeConflictError)
+
+ winner_run, _ = successes[0]
+ all_runs = await repo.list_runs("acme", thread_id=thread.id)
+ descendants = [r for r in all_runs if r.id != old_run.id]
+ self.assertEqual(len(descendants), 1)
+ self.assertEqual(descendants[0].id, winner_run.id)
+
+ final_old = await repo.get_run("acme", old_run.id)
+ self.assertEqual(final_old.superseded_by_run_id, winner_run.id)
+
+ asyncio.run(scenario())
+
+ def test_concurrent_resume_vs_cancel_never_false_succeeds(self) -> None:
+ """Issue #16 PR #17 review round 13 (BLOCKER), PostgreSQL path.
+
+ Races a real ``resume_run_with_pending_steering`` transaction
+ against a real ``request_cancel`` transaction for the same paused
+ run's row lock. PostgreSQL's ``FOR UPDATE`` linearizes the two, but
+ under *either* winning order the caller must never be lied to:
+
+ * cancel wins the lock first -> old run cancelled, resume then
+ fails as a resume conflict (no descendant created).
+ * resume wins the lock first -> exactly one descendant created,
+ cancel is rejected as superseded (never a false-success True).
+
+ Under no interleaving may ``request_cancel`` return ``True`` while
+ a resumed descendant exists and keeps running untouched.
+ """
+
+ from lingxigraph.errors import RunResumeConflictError, RunSupersededError
+ from lingxigraph.server.models import AssistantCreate, RunCreate, ThreadCreate
+ from lingxigraph.server.repository import PostgresRepository
+
+ async def scenario() -> None:
+ repo = PostgresRepository(POSTGRES_URL, schema=self.schema)
+ await repo.setup()
+ assistant = await repo.create_assistant(
+ "acme", AssistantCreate(graph_id="graph", name="a"), "1.0.0"
+ )
+ thread = await repo.create_thread("acme", ThreadCreate())
+ old_run = await repo.create_run(
+ "acme", thread.id, assistant, RunCreate(assistant_id=assistant.id, input={})
+ )
+ await repo.finish_run("acme", old_run.id, "paused", output={})
+
+ request = RunCreate(assistant_id=assistant.id, resume=1)
+ resume_result, cancel_result = await asyncio.gather(
+ repo.resume_run_with_pending_steering(
+ "acme", thread.id, assistant, request, old_run.id
+ ),
+ repo.request_cancel("acme", old_run.id),
+ return_exceptions=True,
+ )
+
+ all_runs = await repo.list_runs("acme", thread_id=thread.id)
+ descendants = [r for r in all_runs if r.id != old_run.id]
+ final_old = await repo.get_run("acme", old_run.id)
+
+ if isinstance(cancel_result, bool) and cancel_result:
+ # Cancel linearized first: old run cancelled, resume must
+ # have failed as a conflict, and no descendant exists.
+ self.assertEqual(final_old.status, "cancelled")
+ self.assertIsInstance(resume_result, RunResumeConflictError)
+ self.assertEqual(descendants, [])
+ else:
+ # Resume linearized first: exactly one descendant exists,
+ # and cancel must have been rejected as superseded --
+ # never a silent False either, since that would also read
+ # to a caller as "nothing to cancel" without pointing at
+ # the real successor.
+ self.assertIsInstance(cancel_result, RunSupersededError)
+ self.assertEqual(len(descendants), 1)
+ new_run, _transferred = resume_result
+ self.assertEqual(descendants[0].id, new_run.id)
+ self.assertEqual(final_old.status, "paused")
+ self.assertEqual(final_old.superseded_by_run_id, new_run.id)
+
+ # The real descendant remains genuinely cancellable.
+ really_cancelled = await repo.request_cancel("acme", new_run.id)
+ self.assertTrue(really_cancelled)
+
+ asyncio.run(scenario())
+
+ def test_superseded_by_run_id_typed_column_against_live_postgres(self) -> None:
+ """Issue #16 PR #17 review round 14 (BLOCKER), PostgreSQL path.
+
+ Exercises the real ``runs.superseded_by_run_id`` column (added by
+ the ``0005_superseded_by_run_id`` migration) end to end against a
+ live database: forged ``metadata.superseded_by_run_id`` must be
+ inert for /steer, /cancel, and /resume, and a real resume must
+ write the typed column atomically in the same transaction that
+ creates the descendant run and migrates its steering.
+ """
+
+ from lingxigraph.errors import RunSupersededError
+ from lingxigraph.server.models import AssistantCreate, RunCreate, ThreadCreate
+ from lingxigraph.server.repository import PostgresRepository
+
+ async def scenario() -> None:
+ repo = PostgresRepository(POSTGRES_URL, schema=self.schema)
+ await repo.setup()
+ assistant = await repo.create_assistant(
+ "acme", AssistantCreate(graph_id="graph", name="a"), "1.0.0"
+ )
+ thread = await repo.create_thread("acme", ThreadCreate())
+
+ # A run created with a forged ``metadata.superseded_by_run_id``
+ # must never have that reflected in the typed column, and must
+ # never be spuriously superseded on /steer or /cancel.
+ forged = RunCreate(
+ assistant_id=assistant.id,
+ input={},
+ metadata={"superseded_by_run_id": "some-other-run-that-never-existed"},
+ )
+ forged_run = await repo.create_run("acme", thread.id, assistant, forged)
+ self.assertIsNone(forged_run.superseded_by_run_id)
+ self.assertEqual(
+ forged_run.metadata.get("superseded_by_run_id"),
+ "some-other-run-that-never-existed",
+ )
+ event, created = await repo.submit_steering(
+ "acme", forged_run.id, kind="user_input", payload={"m": 1}
+ )
+ self.assertTrue(created)
+ self.assertEqual(event.status, "pending")
+ cancelled = await repo.request_cancel("acme", forged_run.id)
+ self.assertTrue(cancelled)
+
+ # The same forged metadata on a genuinely ``paused`` run must
+ # not block a real /resume with a false RunResumeConflictError.
+ forged_thread = await repo.create_thread("acme", ThreadCreate())
+ forged_paused = await repo.create_run("acme", forged_thread.id, assistant, forged)
+ await repo.finish_run("acme", forged_paused.id, "paused", output={})
+ resume_request = RunCreate(assistant_id=assistant.id, resume=1)
+ resumed_run, _ = await repo.resume_run_with_pending_steering(
+ "acme", forged_thread.id, assistant, resume_request, forged_paused.id
+ )
+ self.assertIsNotNone(resumed_run.id)
+
+ # A REAL resume must write the typed column atomically and
+ # gate /steer and /cancel on it.
+ real_thread = await repo.create_thread("acme", ThreadCreate())
+ real_old = await repo.create_run(
+ "acme",
+ real_thread.id,
+ assistant,
+ RunCreate(assistant_id=assistant.id, input={}),
+ )
+ await repo.finish_run("acme", real_old.id, "paused", output={})
+ real_new, _ = await repo.resume_run_with_pending_steering(
+ "acme", real_thread.id, assistant, resume_request, real_old.id
+ )
+ real_old_after = await repo.get_run("acme", real_old.id)
+ self.assertEqual(real_old_after.superseded_by_run_id, real_new.id)
+ self.assertEqual(real_old_after.metadata, {})
+ with self.assertRaises(RunSupersededError):
+ await repo.submit_steering(
+ "acme", real_old.id, kind="user_input", payload={"m": 1}
+ )
+ with self.assertRaises(RunSupersededError):
+ await repo.request_cancel("acme", real_old.id)
+
+ asyncio.run(scenario())
+
+ def test_steer_accept_and_accepted_event_commit_atomically(self) -> None:
+ """Issue #16 PR #17 review round 4, point 2, PostgreSQL path.
+
+ Injects a failure between the steering row commit and its
+ ``run.steer.accepted`` append by directly deleting the accepted
+ event row after a normal ``submit_steering`` call (simulating the
+ pre-fix failure window where the row committed but the event append
+ transiently failed), then verifies an idempotency-key retry repairs
+ it -- exactly once, never duplicated -- and that consumption still
+ produces a complete accepted -> consumed pair.
+ """
+
+ from lingxigraph.server.models import AssistantCreate, RunCreate, ThreadCreate
+ from lingxigraph.server.repository import PostgresRepository
+ from lingxigraph.steering import SteeringConsumption, SteeringEvent
+
+ async def scenario() -> None:
+ repo = PostgresRepository(POSTGRES_URL, schema=self.schema)
+ await repo.setup()
+ assistant = await repo.create_assistant(
+ "acme", AssistantCreate(graph_id="graph", name="a"), "1.0.0"
+ )
+ thread = await repo.create_thread("acme", ThreadCreate())
+ run = await repo.create_run(
+ "acme", thread.id, assistant, RunCreate(assistant_id=assistant.id, input={})
+ )
+
+ event, created = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"m": 1}, idempotency_key="key-1"
+ )
+ self.assertTrue(created)
+ accepted = [
+ e
+ for e in await repo.list_events("acme", run.id)
+ if e.kind == "run.steer.accepted"
+ ]
+ self.assertEqual(len(accepted), 1)
+
+ # Simulate the pre-fix failure window: the steering row
+ # committed, but its ``run.steer.accepted`` event never made it
+ # (e.g. a transient failure right after the row's commit).
+ import psycopg
+
+ with psycopg.connect(POSTGRES_URL, autocommit=True) as conn:
+ conn.execute(
+ f'DELETE FROM "{self.schema}".run_events '
+ "WHERE tenant_id=%s AND run_id=%s AND kind='run.steer.accepted'",
+ ("acme", run.id),
+ )
+ gap = [
+ e
+ for e in await repo.list_events("acme", run.id)
+ if e.kind == "run.steer.accepted"
+ ]
+ self.assertEqual(gap, [])
+
+ # Retry with the same Idempotency-Key: the row already exists
+ # (``created`` is False) but the gap must be repaired.
+ retried_event, retried_created = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"m": 1}, idempotency_key="key-1"
+ )
+ self.assertFalse(retried_created)
+ self.assertEqual(retried_event.id, event.id)
+ repaired = [
+ e
+ for e in await repo.list_events("acme", run.id)
+ if e.kind == "run.steer.accepted"
+ ]
+ self.assertEqual(len(repaired), 1)
+ self.assertEqual(repaired[0].data["steering_event_id"], event.id)
+
+ # A further retry must not duplicate it.
+ await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"m": 1}, idempotency_key="key-1"
+ )
+ final_accepted = [
+ e
+ for e in await repo.list_events("acme", run.id)
+ if e.kind == "run.steer.accepted"
+ ]
+ self.assertEqual(len(final_accepted), 1)
+
+ # After consumption, a complete accepted -> consumed pair
+ # exists (never duplicated by the repair above).
+ consumption = SteeringConsumption(
+ event=SteeringEvent(
+ id=event.id,
+ run_id=run.id,
+ sequence=event.sequence,
+ kind=event.kind,
+ payload=event.payload,
+ metadata=event.metadata,
+ created_at=event.created_at,
+ ),
+ consumed_at=event.created_at,
+ node="n",
+ namespace=(),
+ task_id="t-0",
+ )
+ await repo.commit_steering_consumptions("acme", run.id, [consumption])
+ all_events = await repo.list_events("acme", run.id)
+ self.assertEqual(sum(1 for e in all_events if e.kind == "run.steer.accepted"), 1)
+ self.assertEqual(sum(1 for e in all_events if e.kind == "run.steer.consumed"), 1)
+
+ asyncio.run(scenario())
+
+ def test_commit_steering_consumptions_is_atomic(self) -> None:
+ """Issue #16 PR #17 review point 2, PostgreSQL path: status update
+ and the ``run.steer.consumed`` lifecycle event must land in the
+ same transaction."""
+
+ from lingxigraph.server.models import AssistantCreate, RunCreate, ThreadCreate
+ from lingxigraph.server.repository import PostgresRepository
+ from lingxigraph.steering import SteeringConsumption, SteeringEvent
+
+ async def scenario() -> None:
+ repo = PostgresRepository(POSTGRES_URL, schema=self.schema)
+ await repo.setup()
+ assistant = await repo.create_assistant(
+ "acme", AssistantCreate(graph_id="graph", name="a"), "1.0.0"
+ )
+ thread = await repo.create_thread("acme", ThreadCreate())
+ run = await repo.create_run(
+ "acme", thread.id, assistant, RunCreate(assistant_id=assistant.id, input={})
+ )
+ event, _ = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"m": 1}
+ )
+ steering_event = SteeringEvent(
+ id=event.id,
+ run_id=run.id,
+ sequence=event.sequence,
+ kind=event.kind,
+ payload=event.payload,
+ metadata=event.metadata,
+ created_at=event.created_at,
+ )
+ consumption = SteeringConsumption(
+ event=steering_event,
+ consumed_at=steering_event.created_at,
+ node="n",
+ namespace=(),
+ task_id="t",
+ )
+ stored = await repo.commit_steering_consumptions("acme", run.id, [consumption])
+ self.assertEqual(len(stored), 1)
+ self.assertEqual(stored[0].kind, "run.steer.consumed")
+ all_steering = await repo.list_steering("acme", run.id)
+ self.assertEqual(all_steering[0].status, "consumed")
+ all_events = await repo.list_events("acme", run.id)
+ self.assertEqual(
+ [e.kind for e in all_events if e.kind == "run.steer.consumed"],
+ ["run.steer.consumed"],
+ )
+
+ asyncio.run(scenario())
+
+ def test_alembic_upgrade_head_applies_source_event_id(self) -> None:
+ """Issue #16 PR #17 review round 3, point 1; round 15 (REQUIRED
+ VERIFICATION): explicitly asserts the *current* migration head
+ (0005_superseded_by_run_id), not just the round-3-era 0003 finding.
+
+ ``PostgresRepository._setup_sync()`` applies the SQL migration
+ files directly and is *not* a substitute for driving the real
+ Alembic revision chain that a production deployment uses
+ (``alembic upgrade head``). This regresses that ``0002_steering``
+ -> ``0003_steering_source_event`` -> ``0004_steering_closed`` ->
+ ``0005_superseded_by_run_id`` is reachable through Alembic itself
+ and actually creates every column/index the production SQL in
+ this PR hard-depends on (``run_steering_events.source_event_id``,
+ ``runs.steering_closed``, ``runs.superseded_by_run_id``) -- so a
+ future accidental break in the revision chain, a mis-registered
+ SQL resource, or a stale assertion left over from an earlier round
+ cannot silently drift the "tested head" away from the real one.
+ """
+
+ import psycopg
+ from alembic import command
+ from alembic.config import Config
+
+ repo_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+ cfg = Config(os.path.join(repo_root, "alembic.ini"))
+ cfg.set_main_option("script_location", os.path.join(repo_root, "migrations"))
+ cfg.set_main_option("sqlalchemy.url", POSTGRES_URL)
+ os.environ["LINGXIGRAPH_POSTGRES_SCHEMA"] = self.schema
+
+ with psycopg.connect(POSTGRES_URL, autocommit=True) as conn:
+ conn.execute(f'CREATE SCHEMA IF NOT EXISTS "{self.schema}"')
+ try:
+ command.upgrade(cfg, "0002_steering")
+ with psycopg.connect(POSTGRES_URL) as conn:
+ exists = conn.execute(
+ """SELECT 1 FROM information_schema.columns
+ WHERE table_schema=%s AND table_name='run_steering_events'
+ AND column_name='source_event_id'""",
+ (self.schema,),
+ ).fetchone()
+ self.assertIsNone(exists, "source_event_id should not exist before head")
+
+ command.upgrade(cfg, "head")
+ with psycopg.connect(POSTGRES_URL) as conn:
+ column = conn.execute(
+ """SELECT 1 FROM information_schema.columns
+ WHERE table_schema=%s AND table_name='run_steering_events'
+ AND column_name='source_event_id'""",
+ (self.schema,),
+ ).fetchone()
+ self.assertIsNotNone(column, "alembic head must create source_event_id")
+ index = conn.execute(
+ """SELECT 1 FROM pg_indexes
+ WHERE schemaname=%s AND indexname='run_steering_events_source'""",
+ (self.schema,),
+ ).fetchone()
+ self.assertIsNotNone(
+ index, "alembic head must create the source_event_id index"
+ )
+ steering_closed = conn.execute(
+ """SELECT 1 FROM information_schema.columns
+ WHERE table_schema=%s AND table_name='runs'
+ AND column_name='steering_closed'""",
+ (self.schema,),
+ ).fetchone()
+ self.assertIsNotNone(
+ steering_closed, "alembic head must create runs.steering_closed"
+ )
+ superseded_by_run_id = conn.execute(
+ """SELECT 1 FROM information_schema.columns
+ WHERE table_schema=%s AND table_name='runs'
+ AND column_name='superseded_by_run_id'""",
+ (self.schema,),
+ ).fetchone()
+ self.assertIsNotNone(
+ superseded_by_run_id,
+ "alembic head must create runs.superseded_by_run_id",
+ )
+ head_revision = conn.execute(
+ "SELECT version_num FROM alembic_version"
+ ).fetchone()
+ self.assertEqual(
+ head_revision[0],
+ "0005_superseded_by_run_id",
+ "alembic head must be 0005_superseded_by_run_id, not an earlier "
+ "revision -- if this fails, the revision chain or the head "
+ "assertion has drifted apart from the real production head",
+ )
+ finally:
+ os.environ.pop("LINGXIGRAPH_POSTGRES_SCHEMA", None)
+
+ def test_pause_resume_pause_resume_preserves_root_source_event_id(self) -> None:
+ """Issue #16 PR #17 review round 3, point 2.
+
+ A -> B (resume) -> C (resume): ``source_event_id`` on C's migrated
+ steering event must still point at A, not B, and the eventual
+ ``run.steer.consumed`` event's ``queue_latency_seconds`` must be
+ computed from A's original acceptance ``created_at``.
+ """
+
+ from lingxigraph.server.models import AssistantCreate, RunCreate, ThreadCreate
+ from lingxigraph.server.repository import PostgresRepository
+ from lingxigraph.steering import SteeringConsumption, SteeringEvent
+
+ async def scenario() -> None:
+ repo = PostgresRepository(POSTGRES_URL, schema=self.schema)
+ await repo.setup()
+ assistant = await repo.create_assistant(
+ "acme", AssistantCreate(graph_id="graph", name="a"), "1.0.0"
+ )
+ thread = await repo.create_thread("acme", ThreadCreate())
+
+ run_a = await repo.create_run(
+ "acme", thread.id, assistant, RunCreate(assistant_id=assistant.id, input={})
+ )
+ event_a, _ = await repo.submit_steering(
+ "acme", run_a.id, kind="user_input", payload={"m": 1}
+ )
+ self.assertIsNone(event_a.source_event_id)
+ await repo.finish_run("acme", run_a.id, "paused", output={})
+
+ run_b, transferred_b = await repo.resume_run_with_pending_steering(
+ "acme",
+ thread.id,
+ assistant,
+ RunCreate(assistant_id=assistant.id, resume=1),
+ run_a.id,
+ )
+ self.assertEqual(len(transferred_b), 1)
+ event_b = transferred_b[0]
+ self.assertEqual(event_b.source_event_id, event_a.id)
+ self.assertEqual(event_b.created_at, event_a.created_at)
+ await repo.finish_run("acme", run_b.id, "paused", output={})
+
+ run_c, transferred_c = await repo.resume_run_with_pending_steering(
+ "acme",
+ thread.id,
+ assistant,
+ RunCreate(assistant_id=assistant.id, resume=1),
+ run_b.id,
+ )
+ self.assertEqual(len(transferred_c), 1)
+ event_c = transferred_c[0]
+ # Root identity A survives, not the intermediate hop B.
+ self.assertEqual(event_c.source_event_id, event_a.id)
+ self.assertEqual(event_c.created_at, event_a.created_at)
+
+ steering_event = SteeringEvent(
+ id=event_c.id,
+ run_id=run_c.id,
+ sequence=event_c.sequence,
+ kind=event_c.kind,
+ payload=event_c.payload,
+ metadata=event_c.metadata,
+ created_at=event_c.created_at,
+ source_event_id=event_c.source_event_id,
+ )
+ consumption = SteeringConsumption(
+ event=steering_event,
+ consumed_at=steering_event.created_at,
+ node="n",
+ namespace=(),
+ task_id="t",
+ )
+ stored = await repo.commit_steering_consumptions("acme", run_c.id, [consumption])
+ self.assertEqual(len(stored), 1)
+ self.assertEqual(stored[0].data["source_event_id"], event_a.id)
+
+ asyncio.run(scenario())
+
+ def test_commit_steering_consumptions_is_idempotent_on_retry(self) -> None:
+ """Issue #16 PR #17 review round 3, point 3: a retried commit of
+ the same consumption batch (simulating a worker that resends after
+ an ack it never observed) must not produce a second
+ ``run.steer.consumed`` lifecycle event."""
+
+ from lingxigraph.server.models import AssistantCreate, RunCreate, ThreadCreate
+ from lingxigraph.server.repository import PostgresRepository
+ from lingxigraph.steering import SteeringConsumption, SteeringEvent
+
+ async def scenario() -> None:
+ repo = PostgresRepository(POSTGRES_URL, schema=self.schema)
+ await repo.setup()
+ assistant = await repo.create_assistant(
+ "acme", AssistantCreate(graph_id="graph", name="a"), "1.0.0"
+ )
+ thread = await repo.create_thread("acme", ThreadCreate())
+ run = await repo.create_run(
+ "acme", thread.id, assistant, RunCreate(assistant_id=assistant.id, input={})
+ )
+ event, _ = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"m": 1}
+ )
+ steering_event = SteeringEvent(
+ id=event.id,
+ run_id=run.id,
+ sequence=event.sequence,
+ kind=event.kind,
+ payload=event.payload,
+ metadata=event.metadata,
+ created_at=event.created_at,
+ )
+ consumption = SteeringConsumption(
+ event=steering_event,
+ consumed_at=steering_event.created_at,
+ node="n",
+ namespace=(),
+ task_id="t",
+ )
+ first = await repo.commit_steering_consumptions("acme", run.id, [consumption])
+ self.assertEqual(len(first), 1)
+ second = await repo.commit_steering_consumptions("acme", run.id, [consumption])
+ self.assertEqual(second, [])
+
+ all_events = await repo.list_events("acme", run.id)
+ consumed = [e for e in all_events if e.kind == "run.steer.consumed"]
+ self.assertEqual(len(consumed), 1)
+
+ asyncio.run(scenario())
+
+ def test_stale_worker_cannot_commit_steering_consumptions_after_lease_takeover(
+ self,
+ ) -> None:
+ """Issue #16 PR #17 review round 7, point 2 (BLOCKER), PostgreSQL
+ path: a worker whose lease has been reclaimed by a new owner must
+ never be able to durably commit a steering consumption -- its
+ write must be rejected atomically, touching nothing."""
+
+ from lingxigraph.server.models import AssistantCreate, RunCreate, ThreadCreate
+ from lingxigraph.server.repository import PostgresRepository
+ from lingxigraph.steering import SteeringConsumption, SteeringEvent
+
+ async def scenario() -> None:
+ repo = PostgresRepository(POSTGRES_URL, schema=self.schema)
+ await repo.setup()
+ assistant = await repo.create_assistant(
+ "acme", AssistantCreate(graph_id="graph", name="a"), "1.0.0"
+ )
+ thread = await repo.create_thread("acme", ThreadCreate())
+ run = await repo.create_run(
+ "acme", thread.id, assistant, RunCreate(assistant_id=assistant.id, input={})
+ )
+ event, _ = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"m": 1}
+ )
+
+ claimed_a = await repo.claim_run("worker-a", lease_seconds=1)
+ assert claimed_a is not None
+ self.assertEqual(claimed_a.attempt, 1)
+
+ steering_event = SteeringEvent(
+ id=event.id,
+ run_id=run.id,
+ sequence=event.sequence,
+ kind=event.kind,
+ payload=event.payload,
+ metadata=event.metadata,
+ created_at=event.created_at,
+ )
+ consumption = SteeringConsumption(
+ event=steering_event,
+ consumed_at=steering_event.created_at,
+ node="n",
+ namespace=(),
+ task_id="t",
+ )
+
+ # A's lease expires; B claims the same run at a new attempt.
+ await asyncio.sleep(1.2)
+ claimed_b = await repo.claim_run("worker-b", lease_seconds=30)
+ assert claimed_b is not None
+ self.assertEqual(claimed_b.attempt, 2)
+
+ # A's now-stale commit must be rejected outright.
+ result = await repo.commit_steering_consumptions_if_owned(
+ "acme", run.id, "worker-a", claimed_a.attempt, [consumption]
+ )
+ self.assertIsNone(result)
+
+ still_pending = await repo.list_pending_steering("acme", run.id)
+ self.assertEqual([row.id for row in still_pending], [event.id])
+ consumed_events = [
+ e
+ for e in await repo.list_events("acme", run.id)
+ if e.kind == "run.steer.consumed"
+ ]
+ self.assertEqual(consumed_events, [])
+
+ # B, the current owner, commits it normally.
+ stored = await repo.commit_steering_consumptions_if_owned(
+ "acme", run.id, "worker-b", claimed_b.attempt, [consumption]
+ )
+ self.assertEqual(len(stored), 1)
+ self.assertEqual(await repo.list_pending_steering("acme", run.id), [])
+
+ asyncio.run(scenario())
+
+ def test_claim_run_recovers_expired_cancelling_lease_to_cancelled_and_thread_unwedges(
+ self,
+ ) -> None:
+ """Issue #16 PR #17 review round 11 (BLOCKER), PostgreSQL path:
+ a run stuck in ``cancelling`` whose owning worker crashed before
+ finalizing (lease expired) must never be permanently stranded --
+ ``claim_run`` must resolve it to terminal ``cancelled`` on the
+ very next claim pass, clear its lease, and -- crucially -- stop
+ wedging its thread so a separate queued run on the SAME thread
+ can subsequently be claimed."""
+
+ from lingxigraph.server.models import AssistantCreate, RunCreate, ThreadCreate
+ from lingxigraph.server.repository import PostgresRepository
+
+ async def scenario() -> None:
+ repo = PostgresRepository(POSTGRES_URL, schema=self.schema)
+ await repo.setup()
+ assistant = await repo.create_assistant(
+ "acme", AssistantCreate(graph_id="graph", name="a"), "1.0.0"
+ )
+ thread = await repo.create_thread("acme", ThreadCreate())
+ run = await repo.create_run(
+ "acme", thread.id, assistant, RunCreate(assistant_id=assistant.id, input={})
+ )
+ # A second run queued on the same thread -- it must remain
+ # blocked while ``run`` is active, and become claimable once
+ # the stranded ``cancelling`` run is recovered.
+ queued = await repo.create_run(
+ "acme", thread.id, assistant, RunCreate(assistant_id=assistant.id, input={})
+ )
+
+ accepted, created = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"m": 1}
+ )
+ self.assertTrue(created)
+
+ claimed = await repo.claim_run("worker-a", lease_seconds=1)
+ assert claimed is not None
+ self.assertEqual(claimed.id, run.id)
+
+ cancel_requested = await repo.request_cancel("acme", run.id)
+ self.assertTrue(cancel_requested)
+ mid_flight = await repo.get_run("acme", run.id)
+ self.assertEqual(mid_flight.status, "cancelling")
+
+ closed = await repo.close_steering("acme", run.id, "worker-a", claimed.attempt)
+ self.assertTrue(closed)
+
+ # Worker A crashes here, never reaching the fenced terminal
+ # commit. Let the lease expire, then simulate worker B's next
+ # claim attempt.
+ await asyncio.sleep(1.2)
+
+ reclaimed = await repo.claim_run("worker-b", lease_seconds=30)
+
+ final = await repo.get_run("acme", run.id)
+ self.assertEqual(final.status, "cancelled")
+ self.assertIsNone(final.lease_owner)
+ self.assertIsNone(final.lease_expires_at)
+ self.assertIsNotNone(final.finished_at)
+ assert final.error is not None
+ self.assertEqual(final.error["code"], "run_cancelled")
+
+ # The queued run on the same thread is no longer wedged --
+ # either this very claim call already picked it up, or the
+ # next one does.
+ if reclaimed is not None and reclaimed.id == queued.id:
+ thread_unwedged_run = reclaimed
+ else:
+ self.assertIsNone(reclaimed)
+ thread_unwedged_run = await repo.claim_run("worker-c", lease_seconds=30)
+ assert thread_unwedged_run is not None
+ self.assertEqual(thread_unwedged_run.id, queued.id)
+ self.assertEqual(thread_unwedged_run.status, "running")
+
+ # Ordinary cancel already leaves pending steering as durable
+ # history (never implicitly consumed/replayed) -- the
+ # lease-recovery path must behave the same way here.
+ pending_steering = await repo.list_pending_steering("acme", run.id)
+ self.assertEqual([row.id for row in pending_steering], [accepted.id])
+
+ asyncio.run(scenario())
+
+ def test_cancel_paused_run_terminates_immediately_postgres_path(self) -> None:
+ """Issue #16 PR #17 review round 12 (BLOCKER), PostgreSQL path:
+ cancelling a genuinely ``paused`` run (produced by running a real
+ interrupt graph through the Worker against live Postgres, not a
+ hand-faked status) must resolve to ``cancelled`` immediately --
+ never a lease-less ``cancelling`` that no recovery mechanism can
+ ever unstick. Also confirms a same-thread queued run is not
+ permanently wedged by this fix."""
+
+ from typing import TypedDict
+
+ from fastapi.testclient import TestClient
+
+ from lingxigraph import END, START, StateGraph, interrupt
+ from lingxigraph.server import GraphRegistry, create_app
+ from lingxigraph.server.repository import PostgresRepository
+ from lingxigraph.server.security import Authenticator
+
+ class State(TypedDict):
+ value: int
+
+ def make_registry() -> GraphRegistry:
+ paused = StateGraph(State, name="interrupt-test", version="1.0.0")
+
+ def approval(_state):
+ return {"value": int(interrupt({"question": "new value?"}))}
+
+ paused.add_node("approval", approval)
+ paused.add_edge(START, "approval")
+ paused.add_edge("approval", END)
+ return GraphRegistry({"approval": paused.compile()})
+
+ def wait_for_status(client, run_id, headers, expected):
+ value = None
+ for _ in range(200):
+ value = client.get(f"/v1/runs/{run_id}", headers=headers)
+ if value.json()["status"] in expected:
+ return value
+ asyncio.run(asyncio.sleep(0.01))
+ return value
+
+ repository = PostgresRepository(POSTGRES_URL, schema=self.schema)
+ asyncio.run(repository.setup())
+ app = create_app(
+ registry=make_registry(),
+ repository=repository,
+ authenticator=Authenticator.insecure_dev(),
+ embedded_worker=True,
+ )
+ headers = {"x-tenant-id": "acme"}
+ with TestClient(app) as client:
+ assistant = client.post(
+ "/v1/assistants",
+ headers=headers,
+ json={"graph_id": "approval", "name": "approval"},
+ ).json()
+ thread = client.post("/v1/threads", headers=headers, json={}).json()
+ paused = client.post(
+ f"/v1/threads/{thread['id']}/runs",
+ headers=headers,
+ json={"assistant_id": assistant["id"], "input": {"value": 0}},
+ ).json()
+ paused = wait_for_status(client, paused["id"], headers, {"paused"}).json()
+ self.assertEqual(paused["status"], "paused")
+
+ # A second run queued on the same thread -- 'cancelling' counts
+ # as ACTIVE, so it must not stay permanently wedged by this fix.
+ queued = client.post(
+ f"/v1/threads/{thread['id']}/runs",
+ headers=headers,
+ json={"assistant_id": assistant["id"], "input": {"value": 1}},
+ ).json()
+
+ cancelled = client.post(f"/v1/runs/{paused['id']}/cancel", headers=headers)
+ self.assertEqual(cancelled.status_code, 200, cancelled.text)
+ self.assertEqual(cancelled.json()["status"], "cancelled")
+ self.assertIsNone(cancelled.json()["lease_owner"])
+
+ final = asyncio.run(repository.get_run("acme", paused["id"]))
+ self.assertEqual(final.status, "cancelled")
+ self.assertIsNotNone(final.finished_at)
+ self.assertIsNone(final.lease_owner)
+ self.assertIsNone(final.lease_expires_at)
+
+ resume_after_cancel = client.post(
+ f"/v1/runs/{paused['id']}/resume",
+ headers=headers,
+ json={"resume": 7},
+ )
+ self.assertEqual(resume_after_cancel.status_code, 409)
+
+ # The queued sibling run on the same thread eventually runs to
+ # completion -- it is not wedged by the now-terminal paused run.
+ queued_final = wait_for_status(
+ client, queued["id"], headers, {"paused", "succeeded", "failed"}
+ ).json()
+ self.assertIn(queued_final["status"], {"paused", "succeeded"})
+
+ def test_idempotency_key_replay_safe_across_finalizing_and_terminal_gates(self) -> None:
+ """Issue #16 PR #17 review round 7, point 3, PostgreSQL path: a
+ same-Idempotency-Key replay must return the existing event, never
+ a fresh 409, even after the run has gone finalizing or terminal --
+ only a genuinely new key against such a run still gets 409."""
+
+ from lingxigraph.errors import RunFinalizingError, RunTerminalError
+ from lingxigraph.server.models import AssistantCreate, RunCreate, ThreadCreate
+ from lingxigraph.server.repository import PostgresRepository
+
+ async def scenario() -> None:
+ repo = PostgresRepository(POSTGRES_URL, schema=self.schema)
+ await repo.setup()
+ assistant = await repo.create_assistant(
+ "acme", AssistantCreate(graph_id="graph", name="a"), "1.0.0"
+ )
+ thread = await repo.create_thread("acme", ThreadCreate())
+
+ # --- finalizing gate ---
+ run = await repo.create_run(
+ "acme", thread.id, assistant, RunCreate(assistant_id=assistant.id, input={})
+ )
+ claimed = await repo.claim_run("worker-a", lease_seconds=30)
+ assert claimed is not None
+ original, created = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"m": 1}, idempotency_key="key-1"
+ )
+ self.assertTrue(created)
+ gate_closed = await repo.close_steering("acme", run.id, "worker-a", claimed.attempt)
+ self.assertTrue(gate_closed)
+
+ replayed, replayed_created = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"m": 1}, idempotency_key="key-1"
+ )
+ self.assertFalse(replayed_created)
+ self.assertEqual(replayed.id, original.id)
+ self.assertEqual(replayed.sequence, original.sequence)
+
+ with self.assertRaises(RunFinalizingError):
+ await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"m": 2}, idempotency_key="key-2"
+ )
+
+ # --- terminal gate ---
+ terminal_run = await repo.create_run(
+ "acme", thread.id, assistant, RunCreate(assistant_id=assistant.id, input={})
+ )
+ terminal_original, terminal_created = await repo.submit_steering(
+ "acme",
+ terminal_run.id,
+ kind="user_input",
+ payload={"m": 1},
+ idempotency_key="key-3",
+ )
+ self.assertTrue(terminal_created)
+ await repo.finish_run("acme", terminal_run.id, "succeeded", output={})
+
+ terminal_replayed, terminal_replayed_created = await repo.submit_steering(
+ "acme",
+ terminal_run.id,
+ kind="user_input",
+ payload={"m": 1},
+ idempotency_key="key-3",
+ )
+ self.assertFalse(terminal_replayed_created)
+ self.assertEqual(terminal_replayed.id, terminal_original.id)
+ self.assertEqual(terminal_replayed.sequence, terminal_original.sequence)
+
+ with self.assertRaises(RunTerminalError):
+ await repo.submit_steering(
+ "acme",
+ terminal_run.id,
+ kind="user_input",
+ payload={"m": 2},
+ idempotency_key="key-4",
+ )
+
+ asyncio.run(scenario())
+
+ def test_supersede_pending_steering_if_owned_postgres_path(self) -> None:
+ """Issue #16 PR #17 review round 8, point 1 (BLOCKER), PostgreSQL
+ path: the fenced ``supersede_pending_steering_if_owned`` must
+ durably transition still-pending steering rows to ``superseded``
+ and record a matching ``run.steer.superseded`` lifecycle event, and
+ must be rejected outright for a worker whose lease has already been
+ reclaimed by a new owner."""
+
+ from lingxigraph.server.models import AssistantCreate, RunCreate, ThreadCreate
+ from lingxigraph.server.repository import PostgresRepository
+
+ async def scenario() -> None:
+ repo = PostgresRepository(POSTGRES_URL, schema=self.schema)
+ await repo.setup()
+ assistant = await repo.create_assistant(
+ "acme", AssistantCreate(graph_id="graph", name="a"), "1.0.0"
+ )
+ thread = await repo.create_thread("acme", ThreadCreate())
+ run = await repo.create_run(
+ "acme", thread.id, assistant, RunCreate(assistant_id=assistant.id, input={})
+ )
+ event, _ = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"m": 1}
+ )
+
+ claimed_a = await repo.claim_run("worker-a", lease_seconds=1)
+ assert claimed_a is not None
+
+ # A's lease expires; B claims the same run at a new attempt.
+ await asyncio.sleep(1.2)
+ claimed_b = await repo.claim_run("worker-b", lease_seconds=30)
+ assert claimed_b is not None
+ self.assertEqual(claimed_b.attempt, claimed_a.attempt + 1)
+
+ # A's now-stale attempt must be rejected outright, touching
+ # nothing.
+ stale_result = await repo.supersede_pending_steering_if_owned(
+ "acme", run.id, "worker-a", claimed_a.attempt
+ )
+ self.assertIsNone(stale_result)
+ self.assertEqual(
+ [row.id for row in await repo.list_pending_steering("acme", run.id)],
+ [event.id],
+ )
+
+ # B, the current owner, supersedes it normally.
+ stored = await repo.supersede_pending_steering_if_owned(
+ "acme", run.id, "worker-b", claimed_b.attempt
+ )
+ self.assertEqual(len(stored), 1)
+ self.assertEqual(stored[0].kind, "run.steer.superseded")
+ self.assertEqual(stored[0].data["steering_event_id"], event.id)
+ self.assertEqual(stored[0].data["reason"], "unconsumed_at_final_boundary")
+ self.assertIsNone(stored[0].data["superseded_by_run_id"])
+
+ self.assertEqual(await repo.list_pending_steering("acme", run.id), [])
+ all_steering = await repo.list_steering("acme", run.id)
+ self.assertEqual(all_steering[0].status, "superseded")
+
+ # Idempotent under retry: calling again must not duplicate the
+ # lifecycle event (nothing left pending to supersede either).
+ second = await repo.supersede_pending_steering_if_owned(
+ "acme", run.id, "worker-b", claimed_b.attempt
+ )
+ self.assertEqual(second, [])
+ superseded_events = [
+ e
+ for e in await repo.list_events("acme", run.id)
+ if e.kind == "run.steer.superseded"
+ ]
+ self.assertEqual(len(superseded_events), 1)
+
+ asyncio.run(scenario())
+
+ def test_finalize_run_with_steering_disposition_if_owned_postgres_path(self) -> None:
+ """Issue #16 PR #17 review round 9, point 2 (BLOCKER), PostgreSQL
+ path: the merged ``finalize_run_with_steering_disposition_if_owned``
+ must durably supersede leftover steering AND commit the run's
+ terminal status together, and must be rejected outright for a
+ worker whose lease has already been reclaimed by a new owner."""
+
+ from lingxigraph.server.models import (
+ AssistantCreate,
+ RunCreate,
+ RunStatus,
+ ThreadCreate,
+ )
+ from lingxigraph.server.repository import PostgresRepository
+
+ async def scenario() -> None:
+ repo = PostgresRepository(POSTGRES_URL, schema=self.schema)
+ await repo.setup()
+ assistant = await repo.create_assistant(
+ "acme", AssistantCreate(graph_id="graph", name="a"), "1.0.0"
+ )
+ thread = await repo.create_thread("acme", ThreadCreate())
+ run = await repo.create_run(
+ "acme", thread.id, assistant, RunCreate(assistant_id=assistant.id, input={})
+ )
+ event, _ = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"m": 1}
+ )
+
+ claimed_a = await repo.claim_run("worker-a", lease_seconds=1)
+ assert claimed_a is not None
+
+ # A's lease expires; B claims the same run at a new attempt.
+ await asyncio.sleep(1.2)
+ claimed_b = await repo.claim_run("worker-b", lease_seconds=30)
+ assert claimed_b is not None
+ self.assertEqual(claimed_b.attempt, claimed_a.attempt + 1)
+
+ # A's now-stale attempt must be rejected outright, touching
+ # neither the steering row nor the run's status.
+ stale_result = await repo.finalize_run_with_steering_disposition_if_owned(
+ "acme", run.id, "worker-a", claimed_a.attempt, RunStatus.SUCCEEDED, output={}
+ )
+ self.assertIsNone(stale_result)
+ self.assertEqual(
+ [row.id for row in await repo.list_pending_steering("acme", run.id)],
+ [event.id],
+ )
+ still_running = await repo.get_run("acme", run.id)
+ assert still_running is not None
+ self.assertEqual(still_running.status, "running")
+
+ # B, the current owner, finalizes normally: steering disposition
+ # and terminal status commit together.
+ result = await repo.finalize_run_with_steering_disposition_if_owned(
+ "acme", run.id, "worker-b", claimed_b.attempt, RunStatus.SUCCEEDED, output={}
+ )
+ assert result is not None
+ updated, superseded = result
+ self.assertEqual(updated.status, "succeeded")
+ self.assertEqual(len(superseded), 1)
+ self.assertEqual(superseded[0].data["steering_event_id"], event.id)
+ self.assertEqual(superseded[0].data["reason"], "unconsumed_at_final_boundary")
+ self.assertEqual(superseded[0].data["sequence"], event.sequence)
+ self.assertEqual(superseded[0].data["kind"], event.kind)
+
+ final = await repo.get_run("acme", run.id)
+ assert final is not None
+ self.assertEqual(final.status, "succeeded")
+ self.assertEqual(await repo.list_pending_steering("acme", run.id), [])
+ all_steering = await repo.list_steering("acme", run.id)
+ self.assertEqual(all_steering[0].status, "superseded")
+
+ asyncio.run(scenario())
+
+ def test_finalize_run_with_steering_disposition_rolls_back_atomically_on_failure(
+ self,
+ ) -> None:
+ """Issue #16 PR #17 review round 9, point 2 (BLOCKER) -- the real
+ Postgres rollback/takeover regression the review explicitly asked
+ for: deliberately fail partway through the merged transaction
+ (after the steering row has been superseded in-transaction, but
+ before the run's terminal UPDATE and the transaction's COMMIT) and
+ assert the WHOLE transaction rolls back -- the steering row is
+ still pending/delivered and recoverable by the next owner, and the
+ run's status is untouched. This must run against a real PostgreSQL
+ connection to prove actual transactional atomicity; an
+ InMemoryRepository ordering test cannot demonstrate this."""
+
+ from lingxigraph.server.models import (
+ AssistantCreate,
+ RunCreate,
+ RunStatus,
+ ThreadCreate,
+ )
+ from lingxigraph.server.repository import PostgresRepository
+
+ async def scenario() -> None:
+ repo = PostgresRepository(POSTGRES_URL, schema=self.schema)
+ await repo.setup()
+ assistant = await repo.create_assistant(
+ "acme", AssistantCreate(graph_id="graph", name="a"), "1.0.0"
+ )
+ thread = await repo.create_thread("acme", ThreadCreate())
+ run = await repo.create_run(
+ "acme", thread.id, assistant, RunCreate(assistant_id=assistant.id, input={})
+ )
+ event, _ = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"m": 1}
+ )
+ claimed = await repo.claim_run("worker-a", lease_seconds=30)
+ assert claimed is not None
+
+ class _InjectedFailure(Exception):
+ pass
+
+ real_ensure = repo._ensure_steer_superseded_event_sync
+
+ def failing_ensure(cursor, tenant_id, run_id, steering_event, **kwargs):
+ # Let the steering row's own UPDATE (to 'superseded') and
+ # the superseded-event insert happen first -- exactly what
+ # a real "fails right before the run's terminal UPDATE and
+ # COMMIT" crash would leave in-flight -- then blow up
+ # before the surrounding ``with self._connect()`` block
+ # ever reaches the run status UPDATE or exits normally.
+ real_ensure(cursor, tenant_id, run_id, steering_event, **kwargs)
+ raise _InjectedFailure("simulated crash before terminal status commit")
+
+ repo._ensure_steer_superseded_event_sync = failing_ensure # type: ignore[method-assign]
+
+ with self.assertRaises(_InjectedFailure):
+ await repo.finalize_run_with_steering_disposition_if_owned(
+ "acme", run.id, "worker-a", claimed.attempt, RunStatus.SUCCEEDED, output={}
+ )
+
+ repo._ensure_steer_superseded_event_sync = real_ensure # type: ignore[method-assign]
+
+ # Everything must have rolled back together: the run is still
+ # running (not succeeded), the steering row is still
+ # pending/delivered (never left stranded as superseded), and no
+ # run.steer.superseded event was recorded.
+ still_running = await repo.get_run("acme", run.id)
+ assert still_running is not None
+ self.assertEqual(still_running.status, "running")
+ self.assertEqual(
+ [row.id for row in await repo.list_pending_steering("acme", run.id)],
+ [event.id],
+ )
+ superseded_events = [
+ e
+ for e in await repo.list_events("acme", run.id)
+ if e.kind == "run.steer.superseded"
+ ]
+ self.assertEqual(superseded_events, [])
+
+ # A subsequent owner (the same worker, still holding its lease
+ # here) can still finalize normally and recover the row -- it
+ # was never lost.
+ result = await repo.finalize_run_with_steering_disposition_if_owned(
+ "acme", run.id, "worker-a", claimed.attempt, RunStatus.SUCCEEDED, output={}
+ )
+ assert result is not None
+ updated, superseded = result
+ self.assertEqual(updated.status, "succeeded")
+ self.assertEqual(len(superseded), 1)
+
+ asyncio.run(scenario())
+
+ def test_retry_with_event_if_owned_postgres_path(self) -> None:
+ """Issue #16 PR #17 review round 10, point 1 (BLOCKER), PostgreSQL
+ path: the merged ``retry_run_with_event_if_owned`` must return the
+ transaction's own resulting snapshot directly (no separate later
+ ``get_run()`` read), reject a stale attempt outright, and -- the
+ review's race B -- durably guarantee that ``worker_retrying``'s
+ RunEvent sequence is strictly earlier than any event a subsequent
+ owner's next attempt appends, because both writes are the same
+ real SQL transaction."""
+
+ from lingxigraph.server.models import (
+ AssistantCreate,
+ RunCreate,
+ ThreadCreate,
+ )
+ from lingxigraph.server.repository import PostgresRepository
+
+ async def scenario() -> None:
+ repo = PostgresRepository(POSTGRES_URL, schema=self.schema)
+ await repo.setup()
+ assistant = await repo.create_assistant(
+ "acme", AssistantCreate(graph_id="graph", name="a"), "1.0.0"
+ )
+ thread = await repo.create_thread("acme", ThreadCreate())
+ run = await repo.create_run(
+ "acme", thread.id, assistant, RunCreate(assistant_id=assistant.id, input={})
+ )
+ claimed_a = await repo.claim_run("worker-a", lease_seconds=1)
+ assert claimed_a is not None
+
+ # A's lease expires; B claims the same run at a new attempt.
+ await asyncio.sleep(1.2)
+ claimed_b = await repo.claim_run("worker-b", lease_seconds=30)
+ assert claimed_b is not None
+ self.assertEqual(claimed_b.attempt, claimed_a.attempt + 1)
+
+ # A's now-stale attempt must be rejected outright.
+ stale_result = await repo.retry_run_with_event_if_owned(
+ "acme",
+ run.id,
+ "worker-a",
+ claimed_a.attempt,
+ error={"code": "delivery_retry", "message": "transient"},
+ max_attempts=5,
+ )
+ self.assertIsNone(stale_result)
+ still_running = await repo.get_run("acme", run.id)
+ assert still_running is not None
+ self.assertEqual(still_running.status, "running")
+
+ # B retries normally: status transition and worker_retrying
+ # event commit together, and the returned Run is the
+ # transaction's own snapshot.
+ result = await repo.retry_run_with_event_if_owned(
+ "acme",
+ run.id,
+ "worker-b",
+ claimed_b.attempt,
+ error={"code": "delivery_retry", "message": "transient"},
+ max_attempts=5,
+ )
+ assert result is not None
+ updated, event = result
+ self.assertEqual(updated.status, "pending")
+ assert event is not None
+ self.assertEqual(event.kind, "worker_retrying")
+
+ final = await repo.get_run("acme", run.id)
+ assert final is not None
+ self.assertEqual(final.status, "pending")
+
+ # A third worker claims the retried run (attempt N+1 relative
+ # to B's retry) and appends its own execution event -- proving
+ # the earlier worker_retrying event's sequence is strictly
+ # earlier, i.e. the durable event stream's causal order is
+ # preserved even across a real Postgres transaction boundary.
+ claimed_c = await repo.claim_run("worker-c", lease_seconds=30)
+ assert claimed_c is not None
+ next_event = await repo.append_event(
+ "acme", run.id, "node_started", {"attempt": claimed_c.attempt}
+ )
+ self.assertLess(event.sequence, next_event.sequence)
+
+ asyncio.run(scenario())
+
+ def test_resume_transfer_emits_superseded_event_idempotently_postgres_path(self) -> None:
+ """Issue #16 PR #17 review round 8, point 3 (BLOCKER), PostgreSQL
+ path: the paused-run resume-transfer migration must record exactly
+ one ``run.steer.superseded`` event per transferred steering row, in
+ the same transaction as the status transition."""
+
+ from lingxigraph.server.models import AssistantCreate, RunCreate, ThreadCreate
+ from lingxigraph.server.repository import PostgresRepository
+
+ async def scenario() -> None:
+ repo = PostgresRepository(POSTGRES_URL, schema=self.schema)
+ await repo.setup()
+ assistant = await repo.create_assistant(
+ "acme", AssistantCreate(graph_id="graph", name="a"), "1.0.0"
+ )
+ thread = await repo.create_thread("acme", ThreadCreate())
+ old_run = await repo.create_run(
+ "acme", thread.id, assistant, RunCreate(assistant_id=assistant.id, input={})
+ )
+ original, _ = await repo.submit_steering(
+ "acme", old_run.id, kind="user_input", payload={"m": 1}
+ )
+ await repo.finish_run("acme", old_run.id, "paused", output={})
+
+ new_run, transferred = await repo.resume_run_with_pending_steering(
+ "acme",
+ thread.id,
+ assistant,
+ RunCreate(assistant_id=assistant.id),
+ old_run.id,
+ )
+ self.assertEqual(len(transferred), 1)
+
+ old_events = await repo.list_events("acme", old_run.id)
+ superseded_events = [e for e in old_events if e.kind == "run.steer.superseded"]
+ self.assertEqual(len(superseded_events), 1)
+ self.assertEqual(superseded_events[0].data["steering_event_id"], original.id)
+ self.assertEqual(superseded_events[0].data["reason"], "resume_transfer")
+ self.assertEqual(superseded_events[0].data["superseded_by_run_id"], new_run.id)
+ self.assertEqual(
+ superseded_events[0].data["replacement_steering_event_id"],
+ transferred[0].id,
+ )
+
+ old_steering = await repo.list_steering("acme", old_run.id)
+ self.assertEqual(old_steering[0].status, "superseded")
+
+ asyncio.run(scenario())
+
@unittest.skipUnless(REDIS_URL, "Redis integration URL not configured")
def test_redis_cache_pubsub_and_recovery_contract(self) -> None:
from lingxigraph.cache_redis import RedisCache
@@ -156,6 +1538,44 @@ async def scenario() -> None:
asyncio.run(scenario())
+ @unittest.skipUnless(REDIS_URL, "Redis integration URL not configured")
+ def test_redis_event_bus_wait_returns_promptly_when_heartbeat_is_cancelled(
+ self,
+ ) -> None:
+ """Issue #16 PR #17 review round 6, point 4 (REQUIRED consistency
+ fix): round 5 rewrote ``InMemoryEventBus.wait()`` away from
+ ``asyncio.wait_for(...)`` because cancelling the *caller* while it
+ is in-flight can hang forever on Python 3.11 -- but the production
+ ``RedisEventBus.wait()`` still used that exact pattern, and
+ ``Worker._heartbeat`` cancels a Redis-backed heartbeat exactly the
+ same way as an in-memory one. Reproduce that cancellation directly
+ against ``RedisEventBus.wait()`` (mirroring how ``Worker._execute``
+ does ``heartbeat.cancel(); await asyncio.gather(heartbeat, ...)``)
+ and assert it unblocks promptly instead of hanging.
+ """
+
+ from lingxigraph.server.eventbus import RedisEventBus
+
+ async def scenario() -> None:
+ prefix = "lingxigraph-test-" + uuid4().hex
+ bus = RedisEventBus(REDIS_URL, prefix=prefix)
+ try:
+ # A long timeout: if cancellation does not propagate
+ # promptly, this task hangs for (approximately) this long
+ # instead of returning almost immediately below.
+ waiter = asyncio.create_task(bus.wait("tenant", "run", timeout=30))
+ await asyncio.sleep(0.1)
+ waiter.cancel()
+ start = asyncio.get_event_loop().time()
+ with self.assertRaises(asyncio.CancelledError):
+ await asyncio.wait_for(waiter, timeout=2)
+ elapsed = asyncio.get_event_loop().time() - start
+ self.assertLess(elapsed, 2.0)
+ finally:
+ await bus.close()
+
+ asyncio.run(scenario())
+
if __name__ == "__main__":
unittest.main()
diff --git a/tests/test_runtime_edge_cases.py b/tests/test_runtime_edge_cases.py
new file mode 100644
index 0000000..393411e
--- /dev/null
+++ b/tests/test_runtime_edge_cases.py
@@ -0,0 +1,98 @@
+"""Direct unit tests for lingxigraph.runtime.Runtime edge cases.
+
+These exercise Runtime as a plain dataclass -- no graph execution needed --
+targeting branches that graph-execution tests don't happen to hit (deadline
+expiry, cancellation-token raise, empty steering channel, legacy stream
+writer misuse, budget-less consume_* no-ops, get_config outside a node).
+"""
+
+import unittest
+from datetime import UTC, datetime, timedelta
+
+from lingxigraph.errors import GraphCancelledError, GraphTimeoutError
+from lingxigraph.runtime import CancellationToken, Runtime, get_config, get_store
+
+
+class CancellationTokenTests(unittest.TestCase):
+ def test_raise_if_cancelled_raises_after_cancel(self) -> None:
+ token = CancellationToken()
+ self.assertFalse(token.cancelled)
+ token.cancel()
+ self.assertTrue(token.cancelled)
+ with self.assertRaises(GraphCancelledError):
+ token.raise_if_cancelled()
+
+
+class RuntimeEdgeCaseTests(unittest.TestCase):
+ def _runtime(self, **overrides: object) -> Runtime:
+ defaults: dict[str, object] = {"context": None, "config": {}}
+ defaults.update(overrides)
+ return Runtime(**defaults) # type: ignore[arg-type]
+
+ def test_peek_and_drain_steering_are_empty_without_a_channel(self) -> None:
+ runtime = self._runtime()
+ self.assertEqual(runtime.peek_steering(), ())
+ self.assertEqual(runtime.drain_steering(), ())
+ self.assertFalse(runtime.has_steering)
+
+ def test_raise_if_cancelled_raises_on_expired_deadline(self) -> None:
+ runtime = self._runtime(deadline=datetime.now(UTC) - timedelta(seconds=1))
+ with self.assertRaises(GraphTimeoutError):
+ runtime.raise_if_cancelled()
+
+ def test_raise_if_cancelled_delegates_to_cancellation_token(self) -> None:
+ token = CancellationToken()
+ token.cancel()
+ runtime = self._runtime(cancellation=token)
+ with self.assertRaises(GraphCancelledError):
+ runtime.raise_if_cancelled()
+
+ def test_raise_if_cancelled_is_a_no_op_when_nothing_set(self) -> None:
+ runtime = self._runtime()
+ runtime.raise_if_cancelled() # must not raise
+
+ def test_emit_rejects_empty_or_non_string_channel(self) -> None:
+ runtime = self._runtime()
+ with self.assertRaises(ValueError):
+ runtime.emit("", 1)
+ with self.assertRaises(ValueError):
+ runtime.emit(None, 1) # type: ignore[arg-type]
+
+ def test_emit_without_a_sink_is_a_silent_no_op(self) -> None:
+ runtime = self._runtime()
+ runtime.emit("progress", {"a": 1}) # must not raise, no emitter attached
+
+ def test_stream_writer_legacy_two_arg_form_requires_string_channel(self) -> None:
+ runtime = self._runtime()
+ writer = runtime.stream_writer
+ with self.assertRaises(TypeError):
+ writer(123, "extra", "too-many") # more than one legacy arg
+ with self.assertRaises(TypeError):
+ writer(123, "channel") # value must be the channel string in legacy form
+
+ def test_stream_writer_legacy_form_emits_when_sink_present(self) -> None:
+ captured: list[tuple[str, object]] = []
+ runtime = self._runtime(_emit=lambda channel, value: captured.append((channel, value)))
+ writer = runtime.stream_writer
+ writer("hello", "custom-channel")
+ self.assertEqual(captured, [("hello", "custom-channel")])
+
+ def test_consume_helpers_are_no_ops_without_a_budget(self) -> None:
+ runtime = self._runtime()
+ runtime.consume_tool_call("some_tool")
+ runtime.consume_model_usage({"tokens": 5})
+ runtime.consume_model_call() # none of these should raise
+
+
+class ContextAccessorTests(unittest.TestCase):
+ def test_get_config_outside_a_node_raises(self) -> None:
+ with self.assertRaises(RuntimeError):
+ get_config()
+
+ def test_get_store_outside_a_node_raises(self) -> None:
+ with self.assertRaises(RuntimeError):
+ get_store()
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/test_sdk.py b/tests/test_sdk.py
index 3afeb13..9c0ef1e 100644
--- a/tests/test_sdk.py
+++ b/tests/test_sdk.py
@@ -72,6 +72,7 @@ def test_all_resource_facades_and_sse(self) -> None:
client.runs.get("run")
client.runs.list("thread")
client.runs.cancel("run")
+ client.runs.steer("run", payload={"message": "hi"}, idempotency_key="msg-1")
client.runs.resume("run", True, goto="next")
self.assertEqual(client.runs.join("run", poll_interval=0)["status"], "succeeded")
self.assertEqual(list(client.runs.stream("run", after=2))[0]["sequence"], 3)
@@ -129,6 +130,7 @@ async def scenario() -> None:
await client.runs.get("run")
await client.runs.list("thread")
await client.runs.cancel("run")
+ await client.runs.steer("run", payload={"message": "hi"}, idempotency_key="msg-1")
await client.runs.resume("run", True)
self.assertEqual((await client.runs.join("run", poll_interval=0))["status"], "succeeded")
events = [event async for event in client.runs.stream("run")]
diff --git a/tests/test_steering.py b/tests/test_steering.py
new file mode 100644
index 0000000..c94a347
--- /dev/null
+++ b/tests/test_steering.py
@@ -0,0 +1,3674 @@
+"""Tests for durable mid-run steering (issue #16)."""
+
+from __future__ import annotations
+
+import asyncio
+import operator
+import time
+import unittest
+from typing import Annotated, Any, TypedDict
+
+from fastapi.testclient import TestClient
+
+from lingxigraph import END, START, RetryPolicy, Runtime, Send, StateGraph
+from lingxigraph.errors import RunResumeConflictError, RunSupersededError, RunTerminalError
+from lingxigraph.server import GraphRegistry, create_app
+from lingxigraph.server.models import ThreadCreate
+from lingxigraph.server.repository import InMemoryRepository
+from lingxigraph.server.security import Authenticator
+from lingxigraph.steering import (
+ MAX_STEERING_PAYLOAD_BYTES,
+ SteeringChannel,
+ SteeringConsumption,
+ SteeringPayloadTooLarge,
+ validate_steering_payload,
+)
+from lingxigraph.types import RunStatus
+
+
+class State(TypedDict, total=False):
+ ticks: int
+ drained: list[str]
+
+
+def wait_for_status(client, run_id, headers, expected, attempts: int = 200):
+ value = None
+ for _ in range(attempts):
+ value = client.get(f"/v1/runs/{run_id}", headers=headers)
+ if value.json()["status"] in expected:
+ return value
+ time.sleep(0.01)
+ return value
+
+
+class SteeringChannelTests(unittest.TestCase):
+ """Core steering primitive: ordering, dedup, drain-once semantics."""
+
+ def test_ordering_and_drain_once(self) -> None:
+ channel = SteeringChannel("run-1")
+ for index in range(3):
+ channel.submit(kind="user_input", payload={"i": index})
+ self.assertTrue(channel.has_pending)
+ drained = channel.drain()
+ self.assertEqual([event.payload["i"] for event in drained], [0, 1, 2])
+ self.assertEqual([event.sequence for event in drained], [1, 2, 3])
+ # Draining is one-shot: nothing left to drain again.
+ self.assertFalse(channel.has_pending)
+ self.assertEqual(channel.drain(), ())
+
+ def test_idempotency_key_dedup(self) -> None:
+ channel = SteeringChannel("run-1")
+ first, created_first = channel.submit(
+ kind="user_input", payload={"message": "a"}, idempotency_key="msg-1"
+ )
+ second, created_second = channel.submit(
+ kind="user_input",
+ payload={"message": "duplicate-should-be-ignored"},
+ idempotency_key="msg-1",
+ )
+ self.assertTrue(created_first)
+ self.assertFalse(created_second)
+ self.assertEqual(first.id, second.id)
+ self.assertEqual(len(channel.peek()), 1)
+
+ def test_dedup_survives_after_drain(self) -> None:
+ channel = SteeringChannel("run-1")
+ channel.submit(kind="user_input", payload={}, idempotency_key="msg-1")
+ channel.drain()
+ _, created = channel.submit(kind="user_input", payload={}, idempotency_key="msg-1")
+ self.assertFalse(created)
+
+ def test_payload_size_cap(self) -> None:
+ huge = {"message": "x" * (MAX_STEERING_PAYLOAD_BYTES + 1)}
+ with self.assertRaises(SteeringPayloadTooLarge):
+ validate_steering_payload(huge)
+
+ def test_ingest_preserves_durable_sequence(self) -> None:
+ channel = SteeringChannel("run-1")
+ from datetime import UTC, datetime
+
+ from lingxigraph.steering import SteeringEvent
+
+ channel.ingest(
+ SteeringEvent(
+ id="db-1",
+ run_id="run-1",
+ sequence=5,
+ kind="user_input",
+ payload={},
+ metadata={},
+ created_at=datetime.now(UTC),
+ )
+ )
+ self.assertTrue(channel.has_pending)
+ # A locally-submitted event after ingest gets a higher sequence.
+ event, _ = channel.submit(kind="user_input", payload={})
+ self.assertGreater(event.sequence, 5)
+
+
+class RuntimeDrainTests(unittest.TestCase):
+ """Runtime.has_steering / drain_steering / peek_steering (embedded mode)."""
+
+ def test_embedded_graph_steer_drain_and_ordering(self) -> None:
+ seen: list[list[str]] = []
+
+ def node(state: State, runtime: Runtime[Any]) -> dict[str, Any]:
+ events = runtime.drain_steering()
+ seen.append([event.payload["message"] for event in events])
+ return {"ticks": state.get("ticks", 0) + 1}
+
+ builder = StateGraph(State)
+ builder.add_node("node", node)
+ builder.add_edge(START, "node").add_edge("node", END)
+ graph = builder.compile()
+
+ # Submit three steers in a row *before* the run starts -- they
+ # should all be visible, in order, at the very first safe point.
+ run_id = "embedded-run-1"
+ graph.steer(run_id, kind="user_input", payload={"message": "a"})
+ graph.steer(run_id, kind="user_input", payload={"message": "b"})
+ graph.steer(run_id, kind="user_input", payload={"message": "c"})
+ self.assertTrue(graph.has_pending_steering(run_id))
+
+ graph.invoke({"ticks": 0}, {"configurable": {}}, run_id=run_id)
+ self.assertEqual(seen, [["a", "b", "c"]])
+ self.assertFalse(graph.has_pending_steering(run_id))
+
+ def test_no_channel_is_a_safe_no_op(self) -> None:
+ def node(state: State, runtime: Runtime[Any]) -> dict[str, Any]:
+ self_check = runtime.drain_steering()
+ assert self_check == ()
+ assert runtime.has_steering is False
+ assert runtime.steering_pending is False
+ return {"ticks": 1}
+
+ builder = StateGraph(State)
+ builder.add_node("node", node)
+ builder.add_edge(START, "node").add_edge("node", END)
+ result = builder.compile().invoke({"ticks": 0}, {"configurable": {}})
+ self.assertEqual(result["ticks"], 1)
+
+
+class SteeringConsumptionTests(unittest.TestCase):
+ """SteeringChannel.drain()'s observability record (issue #16 review point 4):
+ consuming node/namespace/task_id and queue latency, not just id/sequence/kind."""
+
+ def test_drain_records_consumer_location_and_latency(self) -> None:
+ channel = SteeringChannel("run-x")
+ channel.submit(kind="user_input", payload={"i": 1})
+ time.sleep(0.01)
+ drained = channel.drain(node="worker", namespace=("team",), task_id="worker#0")
+ self.assertEqual(len(drained), 1)
+ consumed = channel.pop_consumed()
+ self.assertEqual(len(consumed), 1)
+ record = consumed[0]
+ self.assertIsInstance(record, SteeringConsumption)
+ self.assertEqual(record.event.id, drained[0].id)
+ self.assertEqual(record.node, "worker")
+ self.assertEqual(record.namespace, ("team",))
+ self.assertEqual(record.task_id, "worker#0")
+ self.assertGreater(record.queue_latency_seconds, 0)
+
+ def test_drain_without_consumer_context_defaults_to_none(self) -> None:
+ channel = SteeringChannel("run-x")
+ channel.submit(kind="user_input", payload={})
+ channel.drain()
+ record = channel.pop_consumed()[0]
+ self.assertIsNone(record.node)
+ self.assertEqual(record.namespace, ())
+ self.assertIsNone(record.task_id)
+ self.assertGreaterEqual(record.queue_latency_seconds, 0)
+
+ def test_runtime_drain_steering_reports_the_calling_node(self) -> None:
+ """``Runtime.drain_steering()`` (the real node-facing API) must
+ thread the node/namespace/task_id through to the consumption
+ record automatically -- nodes never pass this by hand."""
+
+ captured: list[SteeringConsumption] = []
+
+ def node(state: State, runtime: Runtime[Any]) -> dict[str, Any]:
+ runtime.drain_steering()
+ return {"ticks": 1}
+
+ builder = StateGraph(State)
+ builder.add_node("node", node)
+ builder.add_edge(START, "node").add_edge("node", END)
+ graph = builder.compile()
+ run_id = "consumption-node-run"
+ graph.steer(run_id, kind="user_input", payload={})
+ channel = graph.get_steering_channel(run_id)
+ graph.invoke({"ticks": 0}, {"configurable": {}}, run_id=run_id)
+ captured.extend(channel.pop_consumed())
+ self.assertEqual(len(captured), 1)
+ self.assertEqual(captured[0].node, "node")
+ self.assertEqual(captured[0].namespace, ())
+
+
+class EmbeddedSteeringLifecycleTests(unittest.TestCase):
+ """Issue #16 review point 2: embedded/library-mode steering channels
+ must not accumulate one entry per run forever on a long-lived compiled
+ graph, while a paused run's channel must still survive until resume
+ actually completes."""
+
+ def test_plain_invoke_never_using_steering_does_not_leak(self) -> None:
+ builder = StateGraph(State)
+ builder.add_node("node", lambda state: {"ticks": state.get("ticks", 0) + 1})
+ builder.add_edge(START, "node").add_edge("node", END)
+ graph = builder.compile()
+ for _ in range(200):
+ graph.invoke({"ticks": 0}, {"configurable": {}})
+ self.assertEqual(len(graph._run_steering), 0)
+
+ def test_steer_before_invoke_still_releases_after_the_run_completes(self) -> None:
+ builder = StateGraph(State)
+
+ def node(state: State, runtime: Runtime[Any]) -> dict[str, Any]:
+ runtime.drain_steering()
+ return {"ticks": state.get("ticks", 0) + 1}
+
+ builder.add_node("node", node)
+ builder.add_edge(START, "node").add_edge("node", END)
+ graph = builder.compile()
+ for index in range(50):
+ run_id = f"leak-check-{index}"
+ graph.steer(run_id, kind="user_input", payload={"i": index})
+ graph.invoke({"ticks": 0}, {"configurable": {}}, run_id=run_id)
+ self.assertEqual(len(graph._run_steering), 0)
+
+ def test_paused_run_channel_survives_pause_and_releases_after_resume(self) -> None:
+ from lingxigraph import Command, interrupt
+ from lingxigraph.checkpoint import InMemorySaver
+
+ def node(state: State, runtime: Runtime[Any]) -> dict[str, Any]:
+ value = interrupt("pause")
+ drained = [event.payload["m"] for event in runtime.drain_steering()]
+ return {"ticks": int(value), "drained": drained}
+
+ builder = StateGraph(State)
+ builder.add_node("node", node)
+ builder.add_edge(START, "node").add_edge("node", END)
+ graph = builder.compile(checkpointer=InMemorySaver())
+ config = {"configurable": {"thread_id": "leak-thread"}}
+ run_id = "leak-pause-resume"
+
+ graph.invoke({"ticks": 0}, config, run_id=run_id)
+ # Paused: the channel must stay alive for a future resume/steer.
+ self.assertEqual(len(graph._run_steering), 1)
+
+ graph.steer(run_id, kind="user_input", payload={"m": "hi"})
+ result = graph.invoke(Command(resume="9"), config, run_id=run_id)
+ self.assertEqual(result["drained"], ["hi"])
+ # The run truly finished (no further interrupt) -- released.
+ self.assertEqual(len(graph._run_steering), 0)
+
+
+class ParallelAndSubgraphSteeringTests(unittest.TestCase):
+ """Issue #16 review point 3: node retry, parallel/superstep fan-out, and
+ subgraph namespace sharing were called out as required but missing."""
+
+ def test_retry_does_not_lose_steering_submitted_before_the_successful_attempt(
+ self,
+ ) -> None:
+ """A transient failure *before* draining must not swallow the event
+ -- it has to still be there for the attempt that actually succeeds."""
+
+ attempts: list[int] = []
+ seen: list[list[str]] = []
+
+ def flaky(state: State, runtime: Runtime[Any]) -> dict[str, Any]:
+ attempts.append(1)
+ if len(attempts) < 2:
+ raise ConnectionError("transient, before draining anything")
+ drained = [event.payload["m"] for event in runtime.drain_steering()]
+ seen.append(drained)
+ return {"ticks": state.get("ticks", 0) + 1}
+
+ builder = StateGraph(State)
+ builder.add_node(
+ "flaky", flaky, retry=RetryPolicy(max_attempts=3, initial_interval=0, jitter=False)
+ )
+ builder.add_edge(START, "flaky").add_edge("flaky", END)
+ graph = builder.compile()
+ run_id = "retry-not-swallowed"
+ graph.steer(run_id, kind="user_input", payload={"m": "hello"})
+ graph.invoke({"ticks": 0}, {"configurable": {}}, run_id=run_id)
+ self.assertEqual(len(attempts), 2)
+ self.assertEqual(seen, [["hello"]])
+
+ def test_retry_does_not_re_expose_an_event_already_drained_by_a_failed_attempt(
+ self,
+ ) -> None:
+ """Documented scope cut (see ``SteeringChannel.pop_consumed``):
+ drain-once semantics apply per *channel read*, not per node
+ *attempt*. If an attempt drains an event and then fails, that
+ event is gone from the channel for good -- a retry must not see it
+ a second time (no duplicate exposure), even though the attempt
+ that originally read it never got to finish using it."""
+
+ attempts: list[int] = []
+ seen: list[list[str]] = []
+
+ def flaky(state: State, runtime: Runtime[Any]) -> dict[str, Any]:
+ attempts.append(1)
+ drained = [event.payload["m"] for event in runtime.drain_steering()]
+ seen.append(drained)
+ if len(attempts) < 2:
+ raise ConnectionError("transient, after already draining")
+ return {"ticks": state.get("ticks", 0) + 1}
+
+ builder = StateGraph(State)
+ builder.add_node(
+ "flaky", flaky, retry=RetryPolicy(max_attempts=3, initial_interval=0, jitter=False)
+ )
+ builder.add_edge(START, "flaky").add_edge("flaky", END)
+ graph = builder.compile()
+ run_id = "retry-no-duplicate"
+ graph.steer(run_id, kind="user_input", payload={"m": "once"})
+ graph.invoke({"ticks": 0}, {"configurable": {}}, run_id=run_id)
+ self.assertEqual(len(attempts), 2)
+ self.assertEqual(seen, [["once"], []])
+
+ def test_parallel_send_fanout_drain_is_mutually_exclusive_and_ordered(self) -> None:
+ """Multiple nodes in the same superstep race to call
+ ``drain_steering()`` concurrently -- the channel's lock makes that
+ atomic: exactly one task observes the full, correctly ordered
+ batch and every other concurrent task observes nothing left."""
+
+ class ParallelState(TypedDict, total=False):
+ items: list[int]
+ drained_by_task: Annotated[list[tuple[int, tuple[str, ...]]], operator.add]
+
+ async def worker(payload: dict[str, Any], runtime: Runtime[Any]) -> dict[str, Any]:
+ events = runtime.drain_steering()
+ return {
+ "drained_by_task": [
+ (payload["idx"], tuple(event.payload["m"] for event in events))
+ ]
+ }
+
+ builder = StateGraph(ParallelState)
+ builder.add_node("plan", lambda state: {})
+ builder.add_node("worker", worker)
+ builder.add_edge(START, "plan")
+ builder.add_conditional_edges(
+ "plan",
+ lambda state: [Send("worker", {"idx": item}) for item in state["items"]],
+ )
+ builder.add_edge("worker", END)
+ graph = builder.compile()
+ run_id = "parallel-fanout-run"
+ graph.steer(run_id, kind="user_input", payload={"m": "a"})
+ graph.steer(run_id, kind="user_input", payload={"m": "b"})
+ result = graph.invoke({"items": [0, 1, 2, 3]}, {"configurable": {}}, run_id=run_id)
+
+ non_empty = [batch for _, batch in result["drained_by_task"] if batch]
+ self.assertEqual(len(non_empty), 1, result["drained_by_task"])
+ self.assertEqual(non_empty[0], ("a", "b"))
+ total_events = sum(len(batch) for _, batch in result["drained_by_task"])
+ self.assertEqual(total_events, 2)
+ self.assertFalse(graph.has_pending_steering(run_id))
+
+ def test_subgraph_node_shares_the_parent_runs_steering_channel(self) -> None:
+ """A subgraph node's ``runtime.drain_steering()`` must observe (and
+ durably consume) the exact same per-run channel as the parent --
+ steering is scoped per top-level run, not per namespace."""
+
+ class ChildState(TypedDict, total=False):
+ messages: Annotated[list[str], operator.add]
+
+ class ParentState(TypedDict, total=False):
+ messages: Annotated[list[str], operator.add]
+
+ def child_node(state: ChildState, runtime: Runtime[Any]) -> dict[str, Any]:
+ drained = [event.payload["m"] for event in runtime.drain_steering()]
+ self_namespace = runtime.namespace
+ return {"messages": [f"child-drained:{drained}:{self_namespace}"]}
+
+ child = StateGraph(ChildState)
+ child.add_node("work", child_node)
+ child.add_edge(START, "work").add_edge("work", END)
+
+ parent = StateGraph(ParentState)
+ parent.add_node("team", child.compile())
+ parent.add_edge(START, "team").add_edge("team", END)
+ graph = parent.compile()
+ run_id = "subgraph-shared-channel"
+ graph.steer(run_id, kind="user_input", payload={"m": "x"})
+ self.assertTrue(graph.has_pending_steering(run_id))
+
+ result = graph.invoke({"messages": []}, {"configurable": {}}, run_id=run_id)
+ self.assertEqual(result["messages"], ["child-drained:['x']:('team',)"])
+ self.assertFalse(graph.has_pending_steering(run_id))
+ # Top-level embedded run finished cleanly -- no leaked channel.
+ self.assertEqual(len(graph._run_steering), 0)
+
+
+class RepositorySteeringTests(unittest.TestCase):
+ """Durable-inbox semantics against InMemoryRepository (Postgres-shaped API)."""
+
+ def test_ordering_dedup_and_consumption(self) -> None:
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+
+ e1, c1 = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"i": 1}
+ )
+ e2, c2 = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"i": 2}
+ )
+ e3, c3 = await repo.submit_steering(
+ "acme",
+ run.id,
+ kind="user_input",
+ payload={"i": 2},
+ idempotency_key="dup",
+ )
+ e3b, c3b = await repo.submit_steering(
+ "acme",
+ run.id,
+ kind="user_input",
+ payload={"different": True},
+ idempotency_key="dup",
+ )
+ self.assertTrue(c1 and c2 and c3)
+ self.assertFalse(c3b)
+ self.assertEqual(e3.id, e3b.id)
+
+ pending = await repo.list_pending_steering("acme", run.id)
+ self.assertEqual([event.sequence for event in pending], [1, 2, 3])
+
+ await repo.mark_steering_consumed("acme", run.id, [e1.id, e2.id])
+ pending_after = await repo.list_pending_steering("acme", run.id)
+ self.assertEqual([event.id for event in pending_after], [e3.id])
+ all_events = await repo.list_steering("acme", run.id)
+ self.assertEqual(
+ {event.id: event.status for event in all_events},
+ {e1.id: "consumed", e2.id: "consumed", e3.id: "pending"},
+ )
+
+ asyncio.run(scenario())
+
+ def test_terminal_run_rejects_steer(self) -> None:
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+ await repo.finish_run("acme", run.id, "succeeded", output={})
+ with self.assertRaises(RunTerminalError):
+ await repo.submit_steering("acme", run.id, kind="user_input", payload={})
+
+ asyncio.run(scenario())
+
+ def test_tenant_isolation(self) -> None:
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+ await repo.submit_steering("acme", run.id, kind="user_input", payload={})
+ other_tenant_view = await repo.list_pending_steering("other-tenant", run.id)
+ self.assertEqual(other_tenant_view, [])
+
+ asyncio.run(scenario())
+
+
+class AtomicResumeMigrationTests(unittest.TestCase):
+ """Issue #16 PR #17 review point 1: resume-run creation and its steering
+ migration must be a single atomic operation -- a worker must never be
+ able to claim (let alone finish) the resumed Run before the steering
+ that was pending on the paused run has already been migrated onto it."""
+
+ def test_claim_run_never_observes_the_resumed_run_before_steering_migrated(
+ self,
+ ) -> None:
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ old_run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+ await repo.submit_steering("acme", old_run.id, kind="user_input", payload={"m": 1})
+ await repo.finish_run("acme", old_run.id, "paused", output={})
+
+ claims: list[str] = []
+ stop = asyncio.Event()
+
+ async def hammer_claim() -> None:
+ # Simulates a worker's poll loop racing the resume call --
+ # before the fix, ``create_run`` committed the new Run as
+ # ``pending`` (claimable) *before* the separate
+ # ``transfer_pending_steering`` call ran, so this could
+ # observe (and even finish) the new Run with nothing
+ # migrated onto it yet.
+ while not stop.is_set():
+ claimed = await repo.claim_run("worker-a", lease_seconds=30)
+ if claimed is not None:
+ claims.append(claimed.id)
+ if claimed.id != old_run.id:
+ pending = await repo.list_pending_steering("acme", claimed.id)
+ self.assertEqual(
+ len(pending),
+ 1,
+ "resumed run was claimable before its steering migrated",
+ )
+ await asyncio.sleep(0)
+
+ hammer = asyncio.create_task(hammer_claim())
+ from lingxigraph.server.models import RunCreate
+
+ request = RunCreate(
+ assistant_id=assistant.id,
+ resume=1,
+ metadata={"resumed_from_run_id": old_run.id},
+ )
+ new_run, transferred = await repo.resume_run_with_pending_steering(
+ "acme", old_run.thread_id, assistant, request, old_run.id
+ )
+ self.assertEqual(len(transferred), 1)
+ for _ in range(500):
+ if new_run.id in claims:
+ break
+ await asyncio.sleep(0.001)
+ stop.set()
+ await hammer
+ self.assertIn(new_run.id, claims)
+
+ asyncio.run(scenario())
+
+ def test_concurrent_resume_of_the_same_paused_run_only_succeeds_once(self) -> None:
+ """Issue #16 PR #17 review round 4, point 1: two concurrent
+ ``resume_run_with_pending_steering`` calls against the same paused
+ run must not both create a descendant Run -- exactly one must
+ succeed and ``superseded_by_run_id`` must be stable (never
+ overwritten by the loser)."""
+
+ from lingxigraph.server.models import RunCreate
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ old_run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+ await repo.finish_run("acme", old_run.id, "paused", output={})
+
+ request = RunCreate(assistant_id=assistant.id, resume=1)
+ results = await asyncio.gather(
+ repo.resume_run_with_pending_steering(
+ "acme", old_run.thread_id, assistant, request, old_run.id
+ ),
+ repo.resume_run_with_pending_steering(
+ "acme", old_run.thread_id, assistant, request, old_run.id
+ ),
+ return_exceptions=True,
+ )
+ successes = [r for r in results if not isinstance(r, BaseException)]
+ failures = [r for r in results if isinstance(r, BaseException)]
+ self.assertEqual(len(successes), 1)
+ self.assertEqual(len(failures), 1)
+ self.assertIsInstance(failures[0], RunResumeConflictError)
+
+ winner_run, _ = successes[0]
+ final_old = await repo.get_run("acme", old_run.id)
+ self.assertEqual(final_old.superseded_by_run_id, winner_run.id)
+
+ # The losing attempt must not have been silently allowed to
+ # retry and create a second Run either.
+ with self.assertRaises(RunResumeConflictError):
+ await repo.resume_run_with_pending_steering(
+ "acme", old_run.thread_id, assistant, request, old_run.id
+ )
+
+ asyncio.run(scenario())
+
+ def test_cancel_of_a_resumed_away_paused_run_is_rejected_as_superseded(self) -> None:
+ """Issue #16 PR #17 review round 13 (BLOCKER): once a paused run
+ ``A`` has been resumed into a descendant ``B``, ``A`` keeps
+ ``status='paused'`` by design (the typed ``superseded_by_run_id``
+ column marks it stale instead) so that ``/steer`` can point
+ callers at ``B``.
+ ``request_cancel(A)`` must honor that same signal atomically --
+ raising ``RunSupersededError`` -- instead of falsely succeeding by
+ flipping ``A`` to ``cancelled`` while ``B`` keeps executing
+ untouched.
+ """
+
+ from lingxigraph.server.models import RunCreate
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ old_run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+ await repo.finish_run("acme", old_run.id, "paused", output={})
+
+ request = RunCreate(assistant_id=assistant.id, resume=1)
+ new_run, _ = await repo.resume_run_with_pending_steering(
+ "acme", old_run.thread_id, assistant, request, old_run.id
+ )
+
+ with self.assertRaises(RunSupersededError):
+ await repo.request_cancel("acme", old_run.id)
+
+ # The old run must be untouched -- still paused, still pointing
+ # at the same descendant, not clobbered into a terminal state.
+ old_after = await repo.get_run("acme", old_run.id)
+ self.assertEqual(old_after.status, "paused")
+ self.assertEqual(old_after.superseded_by_run_id, new_run.id)
+
+ # The actual resumed descendant is unaffected and still
+ # cancellable normally.
+ new_before_cancel = await repo.get_run("acme", new_run.id)
+ self.assertNotEqual(new_before_cancel.status, "cancelled")
+ cancelled = await repo.request_cancel("acme", new_run.id)
+ self.assertTrue(cancelled)
+ new_after_cancel = await repo.get_run("acme", new_run.id)
+ self.assertEqual(new_after_cancel.status, "cancelled")
+
+ asyncio.run(scenario())
+
+ def test_transferred_and_new_steering_sequences_never_collide(self) -> None:
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ old_run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+ await repo.submit_steering("acme", old_run.id, kind="user_input", payload={"m": 1})
+ await repo.submit_steering("acme", old_run.id, kind="user_input", payload={"m": 2})
+ await repo.finish_run("acme", old_run.id, "paused", output={})
+
+ from lingxigraph.server.models import RunCreate
+
+ request = RunCreate(assistant_id=assistant.id, resume=1)
+ new_run, transferred = await repo.resume_run_with_pending_steering(
+ "acme", old_run.thread_id, assistant, request, old_run.id
+ )
+ self.assertEqual([event.sequence for event in transferred], [1, 2])
+ # A concurrent-in-spirit ordinary /steer against the resumed
+ # run must land on the next free sequence, never colliding
+ # with the migrated ones.
+ extra, _ = await repo.submit_steering(
+ "acme", new_run.id, kind="user_input", payload={"m": 3}
+ )
+ self.assertEqual(extra.sequence, 3)
+ all_events = await repo.list_steering("acme", new_run.id)
+ self.assertEqual(sorted(event.sequence for event in all_events), [1, 2, 3])
+
+ asyncio.run(scenario())
+
+ def test_forged_metadata_superseded_by_run_id_cannot_forge_control_state(self) -> None:
+ """Issue #16 PR #17 review round 14 (BLOCKER): ``superseded_by_run_id``
+ is authoritative Runtime control state -- it must live in a typed
+ ``Run`` field, never in user-writable ``metadata``. A caller who
+ submits ``metadata.superseded_by_run_id`` at run-creation time must
+ not be able to forge a superseded/resume-conflict signal against a
+ run that was never actually resumed.
+ """
+
+ from lingxigraph.server.models import RunCreate
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ forged = RunCreate(
+ assistant_id=assistant.id,
+ input={},
+ metadata={"superseded_by_run_id": "some-other-run-that-never-existed"},
+ )
+ run = await repo.create_run("acme", None, assistant, forged)
+
+ # The typed field must be inert to the forged metadata --
+ # this run was never actually resumed.
+ self.assertIsNone(run.superseded_by_run_id)
+ self.assertEqual(
+ run.metadata.get("superseded_by_run_id"),
+ "some-other-run-that-never-existed",
+ )
+
+ # /steer must work normally -- not spuriously superseded.
+ event, created = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"m": 1}
+ )
+ self.assertTrue(created)
+ self.assertEqual(event.status, "pending")
+
+ # /cancel must work normally -- not spuriously superseded.
+ cancelled = await repo.request_cancel("acme", run.id)
+ self.assertTrue(cancelled)
+ after = await repo.get_run("acme", run.id)
+ self.assertIn(after.status, ("cancelled", "cancelling"))
+
+ asyncio.run(scenario())
+
+ def test_forged_metadata_superseded_by_run_id_does_not_block_real_resume(self) -> None:
+ """Companion to the forging test above: a genuinely ``paused`` run
+ carrying the same forged ``metadata.superseded_by_run_id`` key must
+ still ``/resume`` normally -- the user metadata must never trigger
+ ``RunResumeConflictError``.
+ """
+
+ from lingxigraph.server.models import RunCreate
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ forged = RunCreate(
+ assistant_id=assistant.id,
+ input={},
+ metadata={"superseded_by_run_id": "some-other-run-that-never-existed"},
+ )
+ run = await repo.create_run("acme", None, assistant, forged)
+ await repo.finish_run("acme", run.id, "paused", output={})
+
+ resume_request = RunCreate(assistant_id=assistant.id, resume=1)
+ new_run, _transferred = await repo.resume_run_with_pending_steering(
+ "acme", run.thread_id, assistant, resume_request, run.id
+ )
+ self.assertIsNotNone(new_run.id)
+ old_after = await repo.get_run("acme", run.id)
+ self.assertEqual(old_after.superseded_by_run_id, new_run.id)
+
+ asyncio.run(scenario())
+
+
+class DurableAckOrderingTests(unittest.TestCase):
+ """Issue #16 PR #17 review point 2: the local consumption log must only
+ be acked *after* the durable commit (steering status + lifecycle event)
+ has actually succeeded -- never popped destructively beforehand."""
+
+ def test_channel_peek_and_ack_never_lose_entries_appended_concurrently(self) -> None:
+ channel = SteeringChannel("run-x")
+ channel.submit(kind="user_input", payload={"i": 1})
+ channel.drain(node="a")
+ first_batch = channel.peek_consumed()
+ self.assertEqual(len(first_batch), 1)
+ # Simulate a second drain happening on another task before the
+ # first batch is acked -- its entry must survive the ack below.
+ channel.submit(kind="user_input", payload={"i": 2})
+ channel.drain(node="b")
+ channel.ack_consumed(entry.event.id for entry in first_batch)
+ remaining = channel.peek_consumed()
+ self.assertEqual(len(remaining), 1)
+ self.assertEqual(remaining[0].node, "b")
+
+ def test_worker_retries_a_transient_durable_commit_failure_without_losing_it(
+ self,
+ ) -> None:
+ from lingxigraph.server.registry import GraphRegistry
+ from lingxigraph.server.worker import Worker
+ from lingxigraph.steering import SteeringEvent
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+ accepted, _ = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"m": 1}
+ )
+
+ channel = SteeringChannel(run.id, owned_by_executor=False)
+ channel.ingest(
+ SteeringEvent(
+ id=accepted.id,
+ run_id=run.id,
+ sequence=accepted.sequence,
+ kind=accepted.kind,
+ payload=accepted.payload,
+ metadata=accepted.metadata,
+ created_at=accepted.created_at,
+ )
+ )
+ # Simulate the graph safe point actually draining it -- this is
+ # what populates the channel's local consumption log that
+ # ``_sync_steering_out`` must not lose.
+ channel.drain(node="the_node", task_id="task-0")
+
+ class FakeGraph:
+ def get_steering_channel(self, run_id: str):
+ return channel
+
+ worker = Worker(GraphRegistry({}), repo)
+
+ calls = {"n": 0}
+ real_commit = repo.commit_steering_consumptions
+
+ async def flaky_commit(tenant_id, run_id, worker_id, attempt, consumptions):
+ calls["n"] += 1
+ if calls["n"] == 1:
+ raise ConnectionError("transient database hiccup")
+ return await real_commit(tenant_id, run_id, consumptions)
+
+ repo.commit_steering_consumptions_if_owned = flaky_commit # type: ignore[method-assign]
+
+ # First sync hits the injected transient failure: nothing may
+ # be lost -- the DB row must still read pending and the
+ # channel must still hold the unacked consumption record.
+ await worker._sync_steering_out(run, FakeGraph())
+ still_pending = await repo.list_pending_steering("acme", run.id)
+ self.assertEqual(len(still_pending), 1)
+ self.assertEqual(len(channel.peek_consumed()), 1)
+ consumed_events = [
+ event
+ for event in await repo.list_events("acme", run.id)
+ if event.kind == "run.steer.consumed"
+ ]
+ self.assertEqual(consumed_events, [])
+
+ # A later safe point (next heartbeat tick) retries and this
+ # time the durable commit succeeds -- the consumption is fully
+ # recorded exactly once, not lost and not duplicated.
+ await worker._sync_steering_out(run, FakeGraph())
+ self.assertEqual(calls["n"], 2)
+ pending_after = await repo.list_pending_steering("acme", run.id)
+ self.assertEqual(pending_after, [])
+ self.assertEqual(channel.peek_consumed(), ())
+ all_steering = await repo.list_steering("acme", run.id)
+ self.assertEqual(all_steering[0].status, "consumed")
+ consumed_events_after = [
+ event
+ for event in await repo.list_events("acme", run.id)
+ if event.kind == "run.steer.consumed"
+ ]
+ self.assertEqual(len(consumed_events_after), 1)
+
+ asyncio.run(scenario())
+
+ def test_final_flush_failure_never_reports_success_with_steering_lost(self) -> None:
+ """Issue #16 PR #17 review round 4, point 3 (round 5, point 1/2): a
+ durable-commit failure on the *final* flush -- with the heartbeat
+ already cancelled and no further safe point within this delivery
+ attempt -- must not leave the run reported succeeded/paused while
+ the DB steering row is silently stranded ``pending`` forever, and
+ must not discard the run's intended outcome before the flush is
+ known to have succeeded. This goes through the real
+ ``Worker._execute()`` finalization path (not an isolated call to
+ ``_sync_steering_out``).
+
+ ``_final_steering_flush`` now retries for as long as it holds the
+ lease rather than giving up after a fixed number of attempts (see
+ its docstring), so this simulates the one case it does give up in
+ -- the worker itself starting to drain mid-retry -- by setting
+ ``worker._stop`` as a side effect of the first failed commit.
+ """
+
+ from lingxigraph.server.worker import Worker
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("steerable"), graph_version="1"
+ )
+ run = await repo.create_run(
+ "acme", None, assistant, _run_create(assistant.id, {"ticks": 0, "drained": []})
+ )
+ await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"message": "stop"}
+ )
+
+ worker = Worker(make_registry(), repo)
+
+ # The heartbeat loop is irrelevant to what this test is
+ # checking (it only needs the steering that was already
+ # durably accepted *before* the run was claimed, which
+ # ``_execute`` syncs in up front, before the heartbeat task is
+ # even created) -- replace it with an inert coroutine so the
+ # test isn't also exercising unrelated heartbeat/event-bus
+ # concurrency while it deliberately holds the flush retry loop
+ # open.
+ async def inert_heartbeat(run, token, graph):
+ await asyncio.Event().wait()
+
+ worker._heartbeat = inert_heartbeat # type: ignore[method-assign]
+
+ async def always_fails(tenant_id, run_id, worker_id, attempt, consumptions):
+ # Simulate the worker starting to drain mid-retry so the
+ # flush's otherwise-unbounded retry loop bails immediately
+ # instead of looping (and sleeping) forever in this test.
+ worker._stop.set()
+ raise ConnectionError("transient database hiccup on final flush")
+
+ repo.commit_steering_consumptions_if_owned = always_fails # type: ignore[method-assign]
+
+ claimed = await worker.run_once()
+ self.assertTrue(claimed)
+
+ after_failed_flush = await repo.get_run("acme", run.id)
+ status = (
+ after_failed_flush.status.value
+ if hasattr(after_failed_flush.status, "value")
+ else after_failed_flush.status
+ )
+ # Must never be reported succeeded/paused while the durable
+ # steering commit never actually happened.
+ self.assertEqual(status, "pending")
+ still_pending = await repo.list_pending_steering("acme", run.id)
+ self.assertEqual(len(still_pending), 1)
+ consumed_events = [
+ event
+ for event in await repo.list_events("acme", run.id)
+ if event.kind == "run.steer.consumed"
+ ]
+ self.assertEqual(consumed_events, [])
+
+ # Recovery: restore the durable commit, un-drain the worker,
+ # and let the run be redelivered. It must eventually succeed
+ # with exactly one ``run.steer.consumed`` event -- not lost,
+ # not duplicated -- confirming the still-``pending`` DB row
+ # (not any in-process channel state) is what a fresh delivery
+ # attempt's brand new graph/channel actually recovers from.
+ del repo.commit_steering_consumptions_if_owned # type: ignore[attr-defined]
+ worker._stop = asyncio.Event()
+ claimed_again = await worker.run_once()
+ self.assertTrue(claimed_again)
+
+ finished = await repo.get_run("acme", run.id)
+ finished_status = (
+ finished.status.value if hasattr(finished.status, "value") else finished.status
+ )
+ self.assertEqual(finished_status, "succeeded")
+ self.assertIn("stop", finished.output["drained"])
+ self.assertEqual(await repo.list_pending_steering("acme", run.id), [])
+ consumed_events_after = [
+ event
+ for event in await repo.list_events("acme", run.id)
+ if event.kind == "run.steer.consumed"
+ ]
+ self.assertEqual(len(consumed_events_after), 1)
+
+ asyncio.run(scenario())
+
+ def test_concurrent_observers_never_see_a_transient_terminal_status_during_final_flush(
+ self,
+ ) -> None:
+ """Issue #16 PR #17 review round 5, point 1 (BLOCKER): a run's
+ terminal/paused status must never become externally observable
+ (via ``GET``/``join``/SSE) before the final steering flush has
+ durably succeeded -- and, symmetrically, once it *does* become
+ observable it must never move back to non-terminal.
+
+ This holds the final flush open with a gate the test controls,
+ continuously polls ``repository.get_run`` (the same read path
+ ``GET``/``join``/SSE all resolve to) throughout, and asserts:
+ (1) while the flush is blocked, the status is never
+ ``succeeded``/``paused``; (2) once ``succeeded`` is first
+ observed, every subsequent poll is also ``succeeded`` -- it never
+ reverts to ``pending``/``running``."""
+
+ from lingxigraph.server.worker import Worker
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("steerable"), graph_version="1"
+ )
+ run = await repo.create_run(
+ "acme", None, assistant, _run_create(assistant.id, {"ticks": 0, "drained": []})
+ )
+ await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"message": "stop"}
+ )
+
+ worker = Worker(make_registry(), repo)
+
+ async def inert_heartbeat(run, token, graph):
+ await asyncio.Event().wait()
+
+ worker._heartbeat = inert_heartbeat # type: ignore[method-assign]
+
+ gate = asyncio.Event()
+ flush_entered = asyncio.Event()
+ real_commit = repo.commit_steering_consumptions
+
+ async def gated_commit(tenant_id, run_id, worker_id, attempt, consumptions):
+ flush_entered.set()
+ await gate.wait()
+ return await real_commit(tenant_id, run_id, consumptions)
+
+ repo.commit_steering_consumptions_if_owned = gated_commit # type: ignore[method-assign]
+
+ observed_statuses: list[str] = []
+ stop_observing = asyncio.Event()
+
+ async def observer() -> None:
+ while not stop_observing.is_set():
+ current = await repo.get_run("acme", run.id)
+ status = (
+ current.status.value
+ if hasattr(current.status, "value")
+ else current.status
+ )
+ observed_statuses.append(status)
+ await asyncio.sleep(0.005)
+
+ observer_task = asyncio.ensure_future(observer())
+ execute_task = asyncio.ensure_future(worker.run_once())
+
+ # Let the run reach and block inside the final flush before
+ # releasing it -- this is the exact window the round 5 review
+ # identified as unsafe in the pre-fix code.
+ await asyncio.wait_for(flush_entered.wait(), timeout=5)
+ # A few more observer ticks while still gated shut.
+ await asyncio.sleep(0.05)
+ self.assertNotIn("succeeded", observed_statuses)
+ self.assertNotIn("paused", observed_statuses)
+
+ gate.set()
+ await asyncio.wait_for(execute_task, timeout=5)
+ # A few more polls after completion to catch any regression
+ # back out of the terminal state.
+ await asyncio.sleep(0.05)
+ stop_observing.set()
+ await observer_task
+
+ self.assertIn("succeeded", observed_statuses)
+ first_succeeded = observed_statuses.index("succeeded")
+ # Monotonic: nothing after the first "succeeded" observation
+ # is ever anything else.
+ self.assertTrue(all(s == "succeeded" for s in observed_statuses[first_succeeded:]))
+ self.assertNotIn("paused", observed_statuses)
+
+ final = await repo.get_run("acme", run.id)
+ final_status = (
+ final.status.value if hasattr(final.status, "value") else final.status
+ )
+ self.assertEqual(final_status, "succeeded")
+ self.assertIn("stop", final.output["drained"])
+
+ asyncio.run(scenario())
+
+ def test_redelivery_after_flush_failure_uses_a_genuinely_new_graph_instance(
+ self,
+ ) -> None:
+ """Issue #16 PR #17 review round 5, point 2: the previous claim that
+ "not calling ``forget_steering()`` leaves the consumption log
+ available for the next delivery" does not hold, because
+ ``with_runtime()`` builds a *brand new* ``CompiledStateGraph`` (and
+ therefore a brand new, empty ``_run_steering`` channel map) on
+ every call -- the locally-bound ``graph``/channel object from a
+ failed delivery attempt is never retained across attempts.
+
+ This test proves recovery does NOT depend on graph/channel object
+ identity surviving across attempts: it instruments
+ ``GraphRegistry.get`` to hand back a distinct, individually
+ identifiable ``CompiledStateGraph`` on each call, confirms the
+ second ``run_once()`` really did receive a different graph object
+ (not a hand-rolled second flush call on the first attempt's
+ object), and confirms the run still recovers correctly -- because
+ recovery is driven entirely by the still-``pending`` PostgreSQL
+ row, not by anything held in memory across the attempt boundary.
+ """
+
+ from lingxigraph.server.worker import Worker
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("steerable"), graph_version="1"
+ )
+ run = await repo.create_run(
+ "acme", None, assistant, _run_create(assistant.id, {"ticks": 0, "drained": []})
+ )
+ await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"message": "stop"}
+ )
+
+ registry = make_registry()
+ # Keep every bound graph alive for the rest of the test, not
+ # just its id(): CPython is free to reuse a freed object's
+ # memory address for the next allocation, so comparing ids of
+ # objects that were allowed to go out of scope is not a
+ # reliable "is this a genuinely distinct instance" check (this
+ # aliased in practice on 3.13's GC timing, id()s collided).
+ seen_graphs: list[object] = []
+ compiled = registry.get("steerable")
+ real_with_runtime = compiled.with_runtime
+
+ def tracking_with_runtime(*args: Any, **kwargs: Any):
+ bound = real_with_runtime(*args, **kwargs)
+ seen_graphs.append(bound)
+ return bound
+
+ compiled.with_runtime = tracking_with_runtime # type: ignore[method-assign]
+
+ worker = Worker(registry, repo)
+
+ async def inert_heartbeat(run, token, graph):
+ await asyncio.Event().wait()
+
+ worker._heartbeat = inert_heartbeat # type: ignore[method-assign]
+
+ real_commit = repo.commit_steering_consumptions
+ fail_once = {"done": False}
+
+ async def fail_first_attempt_only(
+ tenant_id, run_id, worker_id, attempt, consumptions
+ ):
+ if not fail_once["done"]:
+ fail_once["done"] = True
+ worker._stop.set()
+ raise ConnectionError("transient database hiccup on final flush")
+ return await real_commit(tenant_id, run_id, consumptions)
+
+ repo.commit_steering_consumptions_if_owned = fail_first_attempt_only # type: ignore[method-assign]
+
+ claimed = await worker.run_once()
+ self.assertTrue(claimed)
+ after_first_attempt = await repo.get_run("acme", run.id)
+ status = (
+ after_first_attempt.status.value
+ if hasattr(after_first_attempt.status, "value")
+ else after_first_attempt.status
+ )
+ self.assertEqual(status, "pending")
+ self.assertEqual(len(seen_graphs), 1)
+
+ worker._stop = asyncio.Event()
+ claimed_again = await worker.run_once()
+ self.assertTrue(claimed_again)
+
+ # The critical assertion: recovery went through a *second*,
+ # distinct ``with_runtime()`` call -- a genuinely new graph
+ # instance, exactly as a real redelivery (by this worker or a
+ # different one entirely) would -- and it still worked,
+ # because the recovery source of truth is the ``pending`` DB
+ # row, never anything held on the first attempt's discarded
+ # graph/channel.
+ self.assertEqual(len(seen_graphs), 2)
+ self.assertIsNot(seen_graphs[0], seen_graphs[1])
+
+ finished = await repo.get_run("acme", run.id)
+ finished_status = (
+ finished.status.value if hasattr(finished.status, "value") else finished.status
+ )
+ self.assertEqual(finished_status, "succeeded")
+ self.assertIn("stop", finished.output["drained"])
+ self.assertEqual(await repo.list_pending_steering("acme", run.id), [])
+ consumed_events = [
+ event
+ for event in await repo.list_events("acme", run.id)
+ if event.kind == "run.steer.consumed"
+ ]
+ self.assertEqual(len(consumed_events), 1)
+
+ asyncio.run(scenario())
+
+
+class FinalizationFencingTests(unittest.TestCase):
+ """Issue #16 PR #17 review round 6 (BLOCKERs 1-3): all writes a worker
+ makes during finalization must be fenced on its own
+ ``(lease_owner, attempt)``, cancellation must win over a stale
+ non-cancel intent inside that same fenced decision, and new steering
+ admission must close before the final flush starts."""
+
+ def test_stale_worker_cannot_revert_a_new_owners_lease_or_status(self) -> None:
+ """Round 6, point 1 (BLOCKER): worker A claims with a very short
+ lease and blocks inside the final flush; once the lease expires,
+ worker B claims the same run; releasing A's block must not let A's
+ now-stale finalization commit touch B's run at all."""
+
+ from lingxigraph.server.worker import Worker
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("steerable"), graph_version="1"
+ )
+ run = await repo.create_run(
+ "acme", None, assistant, _run_create(assistant.id, {"ticks": 0, "drained": []})
+ )
+ await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"message": "stop"}
+ )
+
+ worker_a = Worker(
+ make_registry(),
+ repo,
+ worker_id="worker-a",
+ lease_seconds=1,
+ heartbeat_seconds=10,
+ )
+
+ async def inert_heartbeat(run, token, graph):
+ await asyncio.Event().wait()
+
+ worker_a._heartbeat = inert_heartbeat # type: ignore[method-assign]
+
+ gate = asyncio.Event()
+ flush_entered = asyncio.Event()
+ real_commit = repo.commit_steering_consumptions
+
+ async def gated_commit(tenant_id, run_id, worker_id, attempt, consumptions):
+ flush_entered.set()
+ await gate.wait()
+ return await real_commit(tenant_id, run_id, consumptions)
+
+ repo.commit_steering_consumptions_if_owned = gated_commit # type: ignore[method-assign]
+
+ execute_task = asyncio.ensure_future(worker_a.run_once())
+ await asyncio.wait_for(flush_entered.wait(), timeout=5)
+
+ # Let A's 1-second lease actually expire without being renewed
+ # (its heartbeat_seconds=10 means the independent finalization
+ # renewal loop will not have ticked yet).
+ await asyncio.sleep(1.3)
+
+ claimed_b = await repo.claim_run("worker-b", lease_seconds=30)
+ self.assertIsNotNone(claimed_b)
+ self.assertEqual(claimed_b.id, run.id)
+ self.assertEqual(claimed_b.lease_owner, "worker-b")
+ self.assertEqual(claimed_b.attempt, 2)
+
+ # Now release A's blocked flush -- its commit succeeds, but its
+ # subsequent fenced finalize write must be refused because it
+ # no longer owns the lease/attempt.
+ gate.set()
+ await asyncio.wait_for(execute_task, timeout=5)
+
+ after = await repo.get_run("acme", run.id)
+ self.assertEqual(after.lease_owner, "worker-b")
+ self.assertEqual(after.attempt, 2)
+ status = after.status.value if hasattr(after.status, "value") else after.status
+ self.assertIn(status, ("running", "cancelling"))
+ self.assertNotEqual(status, "pending")
+ self.assertNotEqual(status, "succeeded")
+
+ asyncio.run(scenario())
+
+ def test_stale_worker_cannot_revert_new_owner_via_flush_failure_retry_path(self) -> None:
+ """Round 6, point 1 (BLOCKER), the mirror case: A's final flush
+ never succeeds at all (durable commit keeps failing) and A falls
+ back to its retry/dead-letter path -- that path's writes must also
+ be fenced, so A cannot reset B's actively-running attempt back to
+ ``pending``."""
+
+ from lingxigraph.server.worker import Worker
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("steerable"), graph_version="1"
+ )
+ run = await repo.create_run(
+ "acme", None, assistant, _run_create(assistant.id, {"ticks": 0, "drained": []})
+ )
+ await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"message": "stop"}
+ )
+
+ worker_a = Worker(
+ make_registry(),
+ repo,
+ worker_id="worker-a",
+ lease_seconds=1,
+ heartbeat_seconds=10,
+ )
+
+ async def inert_heartbeat(run, token, graph):
+ await asyncio.Event().wait()
+
+ worker_a._heartbeat = inert_heartbeat # type: ignore[method-assign]
+
+ entered = asyncio.Event()
+ release = asyncio.Event()
+
+ async def always_fails(tenant_id, run_id, worker_id, attempt, consumptions):
+ entered.set()
+ await release.wait()
+ raise ConnectionError("simulated persistent database outage")
+
+ repo.commit_steering_consumptions_if_owned = always_fails # type: ignore[method-assign]
+
+ execute_task = asyncio.ensure_future(worker_a.run_once())
+ await asyncio.wait_for(entered.wait(), timeout=5)
+ await asyncio.sleep(1.3)
+
+ claimed_b = await repo.claim_run("worker-b", lease_seconds=30)
+ self.assertIsNotNone(claimed_b)
+ self.assertEqual(claimed_b.attempt, 2)
+
+ release.set()
+ await asyncio.wait_for(execute_task, timeout=5)
+
+ after = await repo.get_run("acme", run.id)
+ self.assertEqual(after.lease_owner, "worker-b")
+ self.assertEqual(after.attempt, 2)
+ status = after.status.value if hasattr(after.status, "value") else after.status
+ self.assertIn(status, ("running", "cancelling"))
+
+ asyncio.run(scenario())
+
+ def test_cancel_always_wins_over_a_stale_success_intent_during_finalization(
+ self,
+ ) -> None:
+ """Round 6, point 2 (BLOCKER): the graph has already computed a
+ SUCCEEDED intent and the heartbeat is cancelled before a client's
+ cancel request lands; the fenced finalize write must observe the
+ run is ``cancelling`` and coerce the outcome to ``cancelled``,
+ never letting the stale ``succeeded`` intent win."""
+
+ from lingxigraph.server.worker import Worker
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("double"), graph_version="1"
+ )
+ run = await repo.create_run(
+ "acme", None, assistant, _run_create(assistant.id, {"ticks": 0})
+ )
+
+ worker = Worker(make_registry_double(), repo, worker_id="worker-a")
+
+ async def inert_heartbeat(run, token, graph):
+ await asyncio.Event().wait()
+
+ worker._heartbeat = inert_heartbeat # type: ignore[method-assign]
+
+ # The graph here has no steering to drain, so the final flush
+ # itself is a no-op -- gate the fenced finalize write instead,
+ # which is exactly where round 6 point 2 requires the
+ # cancelling re-check to happen.
+ gate = asyncio.Event()
+ commit_entered = asyncio.Event()
+ # Issue #16 PR #17 review round 9, point 2 (BLOCKER): a
+ # terminal ``finish`` intent now finalizes via the merged
+ # ``finalize_run_with_steering_disposition_if_owned`` -- gate
+ # that instead of the now-bypassed ``finish_run_if_owned``.
+ real_finalize = repo.finalize_run_with_steering_disposition_if_owned
+
+ async def gated_finalize(*args: Any, **kwargs: Any):
+ commit_entered.set()
+ await gate.wait()
+ return await real_finalize(*args, **kwargs)
+
+ repo.finalize_run_with_steering_disposition_if_owned = gated_finalize # type: ignore[method-assign]
+
+ execute_task = asyncio.ensure_future(worker.run_once())
+ await asyncio.wait_for(commit_entered.wait(), timeout=5)
+
+ # Cancellation arrives while the run still reads
+ # running/cancelling (finalization has not committed yet).
+ cancelled = await repo.request_cancel("acme", run.id)
+ self.assertTrue(cancelled)
+ mid_status = await repo.get_run("acme", run.id)
+ self.assertEqual(
+ mid_status.status.value
+ if hasattr(mid_status.status, "value")
+ else mid_status.status,
+ "cancelling",
+ )
+
+ gate.set()
+ await asyncio.wait_for(execute_task, timeout=5)
+
+ final = await repo.get_run("acme", run.id)
+ final_status = (
+ final.status.value if hasattr(final.status, "value") else final.status
+ )
+ self.assertEqual(final_status, "cancelled")
+ self.assertNotEqual(final_status, "succeeded")
+
+ asyncio.run(scenario())
+
+ def test_stale_worker_cannot_commit_steering_consumptions_after_lease_takeover(
+ self,
+ ) -> None:
+ """Issue #16 PR #17 review round 7, point 2 (BLOCKER).
+
+ Worker A drains steering event E during attempt N and blocks
+ inside the durable consumption commit. Its lease expires; worker B
+ claims the same run (attempt N+1). Releasing A's blocked commit
+ must be rejected outright -- A's commit must never touch E's
+ durable status or emit ``run.steer.consumed`` once it no longer
+ owns the lease/attempt, exactly like the already-fenced
+ ``finish_run_if_owned``/``retry_run_if_owned`` writes.
+ """
+
+ from lingxigraph.steering import SteeringConsumption, SteeringEvent
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("double"), graph_version="1"
+ )
+ run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+ accepted, _ = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"message": "stop"}
+ )
+
+ claimed_a = await repo.claim_run("worker-a", lease_seconds=1)
+ assert claimed_a is not None
+ self.assertEqual(claimed_a.attempt, 1)
+
+ steering_event = SteeringEvent(
+ id=accepted.id,
+ run_id=run.id,
+ sequence=accepted.sequence,
+ kind=accepted.kind,
+ payload=accepted.payload,
+ metadata=accepted.metadata,
+ created_at=accepted.created_at,
+ )
+ consumption = SteeringConsumption(
+ event=steering_event,
+ consumed_at=steering_event.created_at,
+ node="n",
+ namespace=(),
+ task_id="t-0",
+ )
+
+ # A's lease expires without renewal; B claims the same run at
+ # a new attempt while A is (conceptually) still "blocked" --
+ # this test drives the two calls sequentially, which exercises
+ # the exact same fencing check the concurrent version would.
+ await asyncio.sleep(1.2)
+ claimed_b = await repo.claim_run("worker-b", lease_seconds=30)
+ assert claimed_b is not None
+ self.assertEqual(claimed_b.attempt, 2)
+
+ # A's stale, now-recovering commit must be rejected entirely --
+ # nothing written, nothing acked.
+ result = await repo.commit_steering_consumptions_if_owned(
+ "acme", run.id, "worker-a", claimed_a.attempt, [consumption]
+ )
+ self.assertIsNone(result)
+
+ # E's durable status must be untouched -- still pending, no
+ # ``run.steer.consumed`` lifecycle event.
+ still_pending = await repo.list_pending_steering("acme", run.id)
+ self.assertEqual([event.id for event in still_pending], [accepted.id])
+ consumed_events = [
+ event
+ for event in await repo.list_events("acme", run.id)
+ if event.kind == "run.steer.consumed"
+ ]
+ self.assertEqual(consumed_events, [])
+
+ # B, the actual current owner, can commit it normally.
+ stored = await repo.commit_steering_consumptions_if_owned(
+ "acme", run.id, "worker-b", claimed_b.attempt, [consumption]
+ )
+ self.assertEqual(len(stored), 1)
+ self.assertEqual(await repo.list_pending_steering("acme", run.id), [])
+
+ asyncio.run(scenario())
+
+
+class SteeringClosedGateTests(unittest.TestCase):
+ """Issue #16 PR #17 review round 6, point 3 (BLOCKER): once graph
+ execution ends there is no safe point left for a newly accepted
+ steering event to ever be consumed, even though the Run row may still
+ read ``running``/``cancelling`` while the final flush is in flight."""
+
+ def test_steer_during_finalization_window_gets_a_stable_error_not_a_stuck_pending_row(
+ self,
+ ) -> None:
+ from lingxigraph.errors import RunFinalizingError
+ from lingxigraph.server.worker import Worker
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("double"), graph_version="1"
+ )
+ run = await repo.create_run(
+ "acme", None, assistant, _run_create(assistant.id, {"ticks": 0})
+ )
+
+ worker = Worker(make_registry_double(), repo, worker_id="worker-a")
+
+ async def inert_heartbeat(run, token, graph):
+ await asyncio.Event().wait()
+
+ worker._heartbeat = inert_heartbeat # type: ignore[method-assign]
+
+ gate = asyncio.Event()
+ commit_entered = asyncio.Event()
+ # Issue #16 PR #17 review round 9, point 2 (BLOCKER): a
+ # terminal ``finish`` intent now finalizes via the merged
+ # ``finalize_run_with_steering_disposition_if_owned`` -- gate
+ # that instead of the now-bypassed ``finish_run_if_owned``.
+ real_finalize = repo.finalize_run_with_steering_disposition_if_owned
+
+ async def gated_finalize(*args: Any, **kwargs: Any):
+ commit_entered.set()
+ await gate.wait()
+ return await real_finalize(*args, **kwargs)
+
+ repo.finalize_run_with_steering_disposition_if_owned = gated_finalize # type: ignore[method-assign]
+
+ execute_task = asyncio.ensure_future(worker.run_once())
+ await asyncio.wait_for(commit_entered.wait(), timeout=5)
+
+ # The graph has already reached END (no safe point left) but
+ # the run row still reads running/cancelling -- this is
+ # exactly the window round 6 point 3 is about.
+ still_active = await repo.get_run("acme", run.id)
+ status = (
+ still_active.status.value
+ if hasattr(still_active.status, "value")
+ else still_active.status
+ )
+ self.assertIn(status, ("running", "cancelling"))
+ self.assertTrue(still_active.steering_closed)
+
+ with self.assertRaises(RunFinalizingError):
+ await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"message": "too late"}
+ )
+
+ gate.set()
+ await asyncio.wait_for(execute_task, timeout=5)
+
+ # Never a terminal run with a pending steering inbox nobody
+ # will ever consume.
+ self.assertEqual(await repo.list_pending_steering("acme", run.id), [])
+ final = await repo.get_run("acme", run.id)
+ final_status = (
+ final.status.value if hasattr(final.status, "value") else final.status
+ )
+ self.assertEqual(final_status, "succeeded")
+
+ asyncio.run(scenario())
+
+ def test_steering_closed_gate_resets_on_a_fresh_delivery_attempt(self) -> None:
+ """A gate closed by an earlier, now-abandoned attempt must not
+ leak into the next delivery attempt's fresh claim -- otherwise a
+ perfectly ordinary retried run would spuriously reject all
+ ``/steer`` calls forever."""
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("double"), graph_version="1"
+ )
+ run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+ claimed = await repo.claim_run("worker-a", lease_seconds=1)
+ assert claimed is not None
+ closed = await repo.close_steering("acme", run.id, "worker-a", claimed.attempt)
+ self.assertTrue(closed)
+
+ await asyncio.sleep(1.2)
+ reclaimed = await repo.claim_run("worker-b", lease_seconds=30)
+ assert reclaimed is not None
+ self.assertEqual(reclaimed.attempt, 2)
+ self.assertFalse(reclaimed.steering_closed)
+
+ # Steering must be accepted normally under the new attempt.
+ accepted, created = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"message": "hello"}
+ )
+ self.assertTrue(created)
+ self.assertEqual(accepted.status, "pending")
+
+ asyncio.run(scenario())
+
+ def test_steer_that_wins_the_lock_race_before_gate_close_is_superseded_not_rerun(
+ self,
+ ) -> None:
+ """Issue #16 PR #17 review round 8, point 1 (BLOCKER) -- CHOSEN FIX:
+ "seal admission at the safe boundary + supersede late events".
+
+ A ``/steer`` call can win the run row's lock *before*
+ ``close_steering()`` does, even though the graph has already
+ finished and the heartbeat has already stopped -- "landed before
+ the gate closed" is not the same as "landed before the graph's
+ last safe point". This artificially blocks ``close_steering()``,
+ completes a ``submit_steering()`` while it is blocked (simulating
+ the race winner), then releases the block. Round 7's fix forced an
+ indefinite retry loop onto this run -- but the ``double`` graph
+ below never calls ``drain_steering()`` at all, so forcing a rerun
+ penalized a graph that is correctly, deliberately ignoring
+ steering. The run must instead finalize normally on this very
+ attempt, and the late-landing steering event must end up with a
+ durable ``superseded`` disposition and matching
+ ``run.steer.superseded`` lifecycle event -- never silently
+ dropped, never forcing a rerun.
+ """
+
+ from lingxigraph.server.worker import Worker
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("double"), graph_version="1"
+ )
+ run = await repo.create_run(
+ "acme", None, assistant, _run_create(assistant.id, {"ticks": 0})
+ )
+
+ worker = Worker(make_registry_double(), repo, worker_id="worker-a")
+
+ async def inert_heartbeat(run, token, graph):
+ await asyncio.Event().wait()
+
+ worker._heartbeat = inert_heartbeat # type: ignore[method-assign]
+
+ gate = asyncio.Event()
+ entered = asyncio.Event()
+ real_close = repo.close_steering
+
+ async def blocked_close(tenant_id, run_id, worker_id, attempt):
+ entered.set()
+ await gate.wait()
+ return await real_close(tenant_id, run_id, worker_id, attempt)
+
+ repo.close_steering = blocked_close # type: ignore[method-assign]
+
+ execute_task = asyncio.ensure_future(worker.run_once())
+ await asyncio.wait_for(entered.wait(), timeout=5)
+
+ # The graph has already reached END and the heartbeat is
+ # already cancelled -- there is no safe point left -- but the
+ # gate has not closed yet. A concurrent ``/steer`` wins the
+ # run lock first and durably commits.
+ accepted, created = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"message": "too late"}
+ )
+ self.assertTrue(created)
+
+ gate.set()
+ await asyncio.wait_for(execute_task, timeout=5)
+
+ final = await repo.get_run("acme", run.id)
+ final_status = (
+ final.status.value if hasattr(final.status, "value") else final.status
+ )
+ # The run finalizes normally on this attempt -- never forced
+ # into a retry loop just because a steering event existed that
+ # the graph never looked at.
+ self.assertEqual(final_status, "succeeded")
+
+ # The late-landing steering event is durably superseded, not
+ # left pending, not silently dropped.
+ still_pending = await repo.list_pending_steering("acme", run.id)
+ self.assertEqual(still_pending, [])
+ all_steering = await repo.list_steering("acme", run.id)
+ self.assertEqual(len(all_steering), 1)
+ self.assertEqual(all_steering[0].id, accepted.id)
+ self.assertEqual(all_steering[0].status, "superseded")
+
+ # A durable, observable run.steer.superseded lifecycle event
+ # was recorded for it -- not the resume-transfer reason, since
+ # this is the late-boundary-race disposition.
+ events = await repo.list_events("acme", run.id)
+ superseded_events = [e for e in events if e.kind == "run.steer.superseded"]
+ self.assertEqual(len(superseded_events), 1)
+ self.assertEqual(superseded_events[0].data["steering_event_id"], accepted.id)
+ self.assertEqual(
+ superseded_events[0].data["reason"], "unconsumed_at_final_boundary"
+ )
+ self.assertIsNone(superseded_events[0].data["superseded_by_run_id"])
+
+ asyncio.run(scenario())
+
+ def test_graph_that_never_drains_steering_still_finalizes_on_first_attempt(
+ self,
+ ) -> None:
+ """Issue #16 PR #17 review round 8, point 1 (BLOCKER) required
+ regression coverage (a): a graph that never calls
+ ``drain_steering()`` at all, receiving one ordinary steering
+ event, must still finalize successfully on its first attempt --
+ no forced rerun just because steering happened to exist. This is
+ the deliberate-ignore case explicitly allowed by issue #16 ("a
+ node may call drain_steering() once, in a loop, or not at all")."""
+
+ from lingxigraph.server.worker import Worker
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("double"), graph_version="1"
+ )
+ run = await repo.create_run(
+ "acme", None, assistant, _run_create(assistant.id, {"ticks": 0})
+ )
+ accepted, created = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"message": "ignored"}
+ )
+ self.assertTrue(created)
+
+ worker = Worker(make_registry_double(), repo, worker_id="worker-a")
+ claimed = await worker.run_once()
+ self.assertTrue(claimed)
+
+ final = await repo.get_run("acme", run.id)
+ final_status = (
+ final.status.value if hasattr(final.status, "value") else final.status
+ )
+ self.assertEqual(final_status, "succeeded")
+ self.assertEqual(final.attempt, 1)
+
+ all_steering = await repo.list_steering("acme", run.id)
+ self.assertEqual(len(all_steering), 1)
+ self.assertEqual(all_steering[0].status, "superseded")
+ events = await repo.list_events("acme", run.id)
+ superseded_events = [e for e in events if e.kind == "run.steer.superseded"]
+ self.assertEqual(len(superseded_events), 1)
+ self.assertEqual(superseded_events[0].data["steering_event_id"], accepted.id)
+
+ asyncio.run(scenario())
+
+
+class CancellingLeaseExpiryRecoveryTests(unittest.TestCase):
+ """Issue #16 PR #17 review round 11 (BLOCKER).
+
+ Both repositories only reclaimed an expired lease when the run was
+ exactly ``running``. A run that was moved to ``cancelling`` (via
+ ``request_cancel``) keeps its lease -- if the owning worker then
+ crashes before it can finalize the run to a terminal status, the
+ expired lease was never reclaimed because ``cancelling`` didn't match
+ either the expired-``running`` recovery query or the ordinary
+ ``pending`` claim selector. The run was stuck in ``cancelling``
+ forever, wedging its thread and leaving ``steering_closed`` (if it had
+ been set) permanently closed with no terminal status ever reached.
+
+ ``claim_run`` must now resolve an expired ``cancelling`` lease
+ straight to the terminal ``cancelled`` status (never back to
+ ``pending``, which would silently erase the cancellation signal and
+ could let business execution resume after the caller explicitly asked
+ to stop).
+ """
+
+ def test_claim_run_resolves_expired_cancelling_lease_to_cancelled(self) -> None:
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("double"), graph_version="1"
+ )
+ run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+
+ claimed = await repo.claim_run("worker-a", lease_seconds=1)
+ assert claimed is not None
+ self.assertEqual(claimed.status, "running")
+
+ # Reproduce the exact finalization-window scenario: cancel is
+ # requested while the worker still holds the lease, and the
+ # worker gets as far as closing steering admission before it
+ # crashes -- never reaching the fenced terminal commit.
+ cancelled_requested = await repo.request_cancel("acme", run.id)
+ self.assertTrue(cancelled_requested)
+ mid_flight = await repo.get_run("acme", run.id)
+ self.assertEqual(mid_flight.status, "cancelling")
+
+ # There is durable steering history pending when the crash
+ # happens.
+ accepted, created = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"message": "hi"}
+ )
+ self.assertTrue(created)
+ self.assertEqual(accepted.status, "pending")
+
+ closed = await repo.close_steering("acme", run.id, "worker-a", claimed.attempt)
+ self.assertTrue(closed)
+
+ # Worker A never comes back; let the lease expire, then
+ # simulate worker B's next claim attempt.
+ await asyncio.sleep(1.2)
+ reclaimed = await repo.claim_run("worker-b", lease_seconds=30)
+
+ # The stranded run must never be handed out as claimable work
+ # -- claim_run must return None (no other pending run exists).
+ self.assertIsNone(reclaimed)
+
+ final = await repo.get_run("acme", run.id)
+ self.assertEqual(final.status, "cancelled")
+ self.assertIsNone(final.lease_owner)
+ self.assertIsNone(final.lease_expires_at)
+ self.assertIsNotNone(final.finished_at)
+ assert final.error is not None
+ self.assertEqual(final.error["code"], "run_cancelled")
+
+ # A subsequent claim attempt must also never return this run.
+ self.assertIsNone(await repo.claim_run("worker-c", lease_seconds=30))
+
+ # Ordinary cancel already leaves pending steering as durable
+ # history (it is never implicitly consumed/replayed) -- the
+ # lease-recovery path must not behave differently.
+ pending_steering = await repo.list_pending_steering("acme", run.id)
+ self.assertEqual(len(pending_steering), 1)
+ self.assertEqual(pending_steering[0].id, accepted.id)
+
+ asyncio.run(scenario())
+
+ def test_expired_running_lease_still_recovers_to_pending(self) -> None:
+ """Confirm the pre-existing expired-``running`` recovery path is
+ unchanged by the new ``cancelling`` branch."""
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("double"), graph_version="1"
+ )
+ run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+ claimed = await repo.claim_run("worker-a", lease_seconds=1)
+ assert claimed is not None
+
+ await asyncio.sleep(1.2)
+ reclaimed = await repo.claim_run("worker-b", lease_seconds=30)
+ assert reclaimed is not None
+ self.assertEqual(reclaimed.id, run.id)
+ self.assertEqual(reclaimed.status, "running")
+ self.assertEqual(reclaimed.lease_owner, "worker-b")
+
+ asyncio.run(scenario())
+
+
+class RetryCommitCancelWinsEventTests(unittest.TestCase):
+ """Issue #16 PR #17 review round 8, point 2 (BLOCKER).
+
+ ``retry_run_if_owned`` already correctly coerces the transition to
+ ``cancelled`` when the run is ``cancelling`` at commit time -- it must
+ never be sent back to ``pending``. But the Worker used to only check
+ ``updated is not None`` before unconditionally appending a
+ ``worker_retrying`` event, so a durable "retrying" event could be
+ appended even though the actual write coerced the run to
+ ``cancelled``. These tests prove the Worker now branches on the
+ *actual* resulting status for both retry call sites.
+ """
+
+ def test_retry_or_dead_letter_does_not_emit_worker_retrying_when_cancel_wins(self) -> None:
+ from lingxigraph.server.worker import Worker
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("double"), graph_version="1"
+ )
+ run = await repo.create_run(
+ "acme", None, assistant, _run_create(assistant.id, {"ticks": 0})
+ )
+ claimed = await repo.claim_run("worker-a", lease_seconds=30)
+ assert claimed is not None
+ self.assertTrue(await repo.request_cancel("acme", run.id))
+ cancelling = await repo.get_run("acme", run.id)
+ self.assertEqual(cancelling.status, "cancelling")
+
+ worker = Worker(make_registry_double(), repo, worker_id="worker-a")
+ await worker._retry_or_dead_letter(
+ claimed, {"code": "delivery_retry", "message": "transient"}
+ )
+
+ final = await repo.get_run("acme", run.id)
+ self.assertEqual(final.status, "cancelled")
+ events = await repo.list_events("acme", run.id)
+ self.assertEqual([e for e in events if e.kind == "worker_retrying"], [])
+
+ asyncio.run(scenario())
+
+ def test_commit_intent_retry_does_not_emit_worker_retrying_when_cancel_wins(self) -> None:
+ from lingxigraph.server.worker import Worker
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("double"), graph_version="1"
+ )
+ run = await repo.create_run(
+ "acme", None, assistant, _run_create(assistant.id, {"ticks": 0})
+ )
+ claimed = await repo.claim_run("worker-a", lease_seconds=30)
+ assert claimed is not None
+ self.assertTrue(await repo.request_cancel("acme", run.id))
+
+ worker = Worker(make_registry_double(), repo, worker_id="worker-a")
+ intent = Worker._Intent(
+ kind="retry",
+ status=RunStatus.PENDING,
+ error={"code": "delivery_retry", "message": "transient"},
+ )
+ committed = await worker._commit_intent(claimed, intent)
+ self.assertTrue(committed)
+
+ final = await repo.get_run("acme", run.id)
+ self.assertEqual(final.status, "cancelled")
+ events = await repo.list_events("acme", run.id)
+ self.assertEqual([e for e in events if e.kind == "worker_retrying"], [])
+
+ asyncio.run(scenario())
+
+ def test_ordinary_retry_still_emits_worker_retrying_when_pending(self) -> None:
+ from lingxigraph.server.worker import Worker
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("double"), graph_version="1"
+ )
+ run = await repo.create_run(
+ "acme", None, assistant, _run_create(assistant.id, {"ticks": 0})
+ )
+ claimed = await repo.claim_run("worker-a", lease_seconds=30)
+ assert claimed is not None
+
+ worker = Worker(make_registry_double(), repo, worker_id="worker-a")
+ await worker._retry_or_dead_letter(
+ claimed, {"code": "delivery_retry", "message": "transient"}
+ )
+
+ final = await repo.get_run("acme", run.id)
+ self.assertEqual(final.status, "pending")
+ events = await repo.list_events("acme", run.id)
+ self.assertEqual(len([e for e in events if e.kind == "worker_retrying"]), 1)
+
+ asyncio.run(scenario())
+
+
+class RetryWithEventAtomicityTests(unittest.TestCase):
+ """Issue #16 PR #17 review round 10, point 1 (BLOCKER).
+
+ ``retry_run_if_owned()`` and the durable ``worker_retrying`` event used
+ to be two separate transactions, reopening a TOCTOU window between
+ "the run is truly pending" and "the event says it's retrying". These
+ tests prove the merged ``retry_run_with_event_if_owned`` closes both
+ races described in the review.
+ """
+
+ def test_race_a_cancel_in_the_gap_leaves_no_late_worker_retrying_event(self) -> None:
+ """A cancel that would have landed *between* the old two writes
+ must never let a stale worker append ``worker_retrying`` for a run
+ that is durably ``cancelled`` -- because there is no longer a gap
+ for it to land in: the status write and the event are the same
+ atomic operation."""
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("double"), graph_version="1"
+ )
+ run = await repo.create_run(
+ "acme", None, assistant, _run_create(assistant.id, {"ticks": 0})
+ )
+ claimed = await repo.claim_run("worker-a", lease_seconds=30)
+ assert claimed is not None
+
+ # Cancel arrives before the merged retry transaction runs --
+ # the run is already ``cancelling`` when the fenced write
+ # executes, so cancel-wins coercion applies inside the same
+ # critical section.
+ self.assertTrue(await repo.request_cancel("acme", run.id))
+
+ result = await repo.retry_run_with_event_if_owned(
+ "acme",
+ run.id,
+ "worker-a",
+ claimed.attempt,
+ error={"code": "delivery_retry", "message": "transient"},
+ max_attempts=5,
+ )
+ assert result is not None
+ updated, event = result
+ self.assertEqual(updated.status, "cancelled")
+ self.assertIsNone(event)
+
+ final = await repo.get_run("acme", run.id)
+ self.assertEqual(final.status, "cancelled")
+ events = await repo.list_events("acme", run.id)
+ self.assertEqual([e for e in events if e.kind == "worker_retrying"], [])
+
+ asyncio.run(scenario())
+
+ def test_race_b_worker_retrying_event_precedes_next_attempts_events(self) -> None:
+ """Worker A's ``worker_retrying(attempt N)`` event must always be
+ durable *before* the run becomes externally visible as
+ ``pending`` -- so a worker B that then claims it (attempt N+1) and
+ appends its own execution events can never produce an event with a
+ lower sequence than ``worker_retrying``. Because the merged method
+ makes both writes atomic, this is true by construction; this test
+ proves the observable sequencing."""
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("double"), graph_version="1"
+ )
+ run = await repo.create_run(
+ "acme", None, assistant, _run_create(assistant.id, {"ticks": 0})
+ )
+ claimed = await repo.claim_run("worker-a", lease_seconds=30)
+ assert claimed is not None
+
+ result = await repo.retry_run_with_event_if_owned(
+ "acme",
+ run.id,
+ "worker-a",
+ claimed.attempt,
+ error={"code": "delivery_retry", "message": "transient"},
+ max_attempts=5,
+ )
+ assert result is not None
+ updated, event = result
+ self.assertEqual(updated.status, "pending")
+ assert event is not None
+ self.assertEqual(event.kind, "worker_retrying")
+
+ # Worker B now claims the retried run (attempt N+1) and logs
+ # its own execution event.
+ claimed_b = await repo.claim_run("worker-b", lease_seconds=30)
+ assert claimed_b is not None
+ self.assertGreater(claimed_b.attempt, claimed.attempt)
+ next_event = await repo.append_event(
+ "acme", run.id, "node_started", {"attempt": claimed_b.attempt}
+ )
+
+ self.assertLess(event.sequence, next_event.sequence)
+
+ asyncio.run(scenario())
+
+
+class AtomicSteerAcceptedEventTests(unittest.TestCase):
+ """Issue #16 PR #17 review round 4, point 2: accepting a steering row
+ and recording its ``run.steer.accepted`` lifecycle event must be a
+ single atomic, idempotent unit -- a retried request (same
+ Idempotency-Key) that finds the steering row already durably created
+ must still repair a missing ``run.steer.accepted`` event rather than
+ permanently skip it."""
+
+ def test_idempotency_key_retry_repairs_a_missing_accepted_event(self) -> None:
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+
+ event, created = await repo.submit_steering(
+ "acme",
+ run.id,
+ kind="user_input",
+ payload={"m": 1},
+ idempotency_key="key-1",
+ )
+ self.assertTrue(created)
+ accepted = [
+ e
+ for e in await repo.list_events("acme", run.id)
+ if e.kind == "run.steer.accepted"
+ ]
+ self.assertEqual(len(accepted), 1)
+
+ # Simulate the pre-fix failure window: the steering row
+ # committed, but the ``run.steer.accepted`` append never
+ # happened (e.g. the process crashed / the DB call transiently
+ # failed right after the row committed).
+ repo._events[("acme", run.id)] = [
+ e for e in repo._events[("acme", run.id)] if e.kind != "run.steer.accepted"
+ ]
+ self.assertEqual(
+ [
+ e
+ for e in await repo.list_events("acme", run.id)
+ if e.kind == "run.steer.accepted"
+ ],
+ [],
+ )
+
+ # The client retries with the same Idempotency-Key: the
+ # steering row already exists (``created`` is False) but the
+ # gap must still be repaired, not permanently skipped.
+ retried_event, retried_created = await repo.submit_steering(
+ "acme",
+ run.id,
+ kind="user_input",
+ payload={"m": 1},
+ idempotency_key="key-1",
+ )
+ self.assertFalse(retried_created)
+ self.assertEqual(retried_event.id, event.id)
+ accepted_after = [
+ e
+ for e in await repo.list_events("acme", run.id)
+ if e.kind == "run.steer.accepted"
+ ]
+ self.assertEqual(len(accepted_after), 1)
+ self.assertEqual(accepted_after[0].data["steering_event_id"], event.id)
+
+ # A further retry must not duplicate it either.
+ await repo.submit_steering(
+ "acme",
+ run.id,
+ kind="user_input",
+ payload={"m": 1},
+ idempotency_key="key-1",
+ )
+ accepted_final = [
+ e
+ for e in await repo.list_events("acme", run.id)
+ if e.kind == "run.steer.accepted"
+ ]
+ self.assertEqual(len(accepted_final), 1)
+
+ asyncio.run(scenario())
+
+ def test_transfer_records_accepted_event_for_the_new_run_atomically(self) -> None:
+ """The paused-transfer path's ``run.steer.accepted`` must land on
+ the new run inside the same migration, correlated back to the
+ original event via ``source_event_id``/``transferred_from_run_id``."""
+
+ from lingxigraph.server.models import RunCreate
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ old_run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+ original, _ = await repo.submit_steering(
+ "acme", old_run.id, kind="user_input", payload={"m": 1}
+ )
+ await repo.finish_run("acme", old_run.id, "paused", output={})
+
+ new_run, transferred = await repo.resume_run_with_pending_steering(
+ "acme",
+ old_run.thread_id,
+ assistant,
+ RunCreate(assistant_id=assistant.id),
+ old_run.id,
+ )
+ self.assertEqual(len(transferred), 1)
+ new_events = await repo.list_events("acme", new_run.id)
+ accepted = [e for e in new_events if e.kind == "run.steer.accepted"]
+ self.assertEqual(len(accepted), 1)
+ self.assertEqual(accepted[0].data["steering_event_id"], transferred[0].id)
+ self.assertEqual(accepted[0].data["transferred_from_run_id"], old_run.id)
+ self.assertEqual(accepted[0].data["source_event_id"], original.id)
+
+ asyncio.run(scenario())
+
+
+class IdempotencyKeyReplaySafeAcrossAdmissionGatesTests(unittest.TestCase):
+ """Issue #16 PR #17 review round 7, point 3.
+
+ A same-Idempotency-Key replay must return the already-existing event
+ (id/sequence unchanged), never a fresh 409, regardless of whether the
+ run has since gone finalizing/terminal/superseded -- that is the
+ entire point of an idempotency key: safely retrying exactly this kind
+ of uncertain-outcome window. Only a genuinely *new* key against a
+ finalizing/terminal run should still get 409.
+ """
+
+ def test_replay_after_finalizing_returns_same_event_not_409(self) -> None:
+ from lingxigraph.errors import RunFinalizingError
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("double"), graph_version="1"
+ )
+ run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+ claimed = await repo.claim_run("worker-a", lease_seconds=30)
+ assert claimed is not None
+
+ original, created = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"m": 1}, idempotency_key="key-1"
+ )
+ self.assertTrue(created)
+
+ gate_closed = await repo.close_steering("acme", run.id, "worker-a", claimed.attempt)
+ self.assertTrue(gate_closed)
+
+ # Same key, replayed after the run started finalizing: must
+ # return the same event, not a 409.
+ replayed, replayed_created = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"m": 1}, idempotency_key="key-1"
+ )
+ self.assertFalse(replayed_created)
+ self.assertEqual(replayed.id, original.id)
+ self.assertEqual(replayed.sequence, original.sequence)
+
+ # A genuinely new key against the still-finalizing run must
+ # still be rejected -- this is a real new-admission decision.
+ with self.assertRaises(RunFinalizingError):
+ await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"m": 2}, idempotency_key="key-2"
+ )
+
+ asyncio.run(scenario())
+
+ def test_replay_after_terminal_returns_same_event_not_409(self) -> None:
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("double"), graph_version="1"
+ )
+ run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+
+ original, created = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"m": 1}, idempotency_key="key-1"
+ )
+ self.assertTrue(created)
+
+ await repo.finish_run("acme", run.id, "succeeded", output={})
+
+ replayed, replayed_created = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"m": 1}, idempotency_key="key-1"
+ )
+ self.assertFalse(replayed_created)
+ self.assertEqual(replayed.id, original.id)
+ self.assertEqual(replayed.sequence, original.sequence)
+
+ with self.assertRaises(RunTerminalError):
+ await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"m": 2}, idempotency_key="key-2"
+ )
+
+ asyncio.run(scenario())
+
+
+class InMemoryRepositorySteeringEdgeCaseTests(unittest.TestCase):
+ """Direct unit coverage of ``InMemoryRepository`` steering branches that
+ the HTTP/worker end-to-end tests above don't happen to exercise: unknown
+ runs, empty batches, mixed-status transfers, and idempotency-keyed
+ transfers."""
+
+ def test_submit_steering_on_unknown_run_raises_key_error(self) -> None:
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ with self.assertRaises(KeyError):
+ await repo.submit_steering("acme", "no-such-run", kind="user_input", payload={})
+
+ asyncio.run(scenario())
+
+ def test_mark_steering_consumed_with_empty_ids_is_a_no_op(self) -> None:
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+ await repo.submit_steering("acme", run.id, kind="user_input", payload={})
+ # Should return without touching anything.
+ await repo.mark_steering_consumed("acme", run.id, [])
+ pending = await repo.list_pending_steering("acme", run.id)
+ self.assertEqual(len(pending), 1)
+
+ asyncio.run(scenario())
+
+ def test_commit_steering_consumptions_with_empty_list_returns_empty(self) -> None:
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+ result = await repo.commit_steering_consumptions("acme", run.id, [])
+ self.assertEqual(result, [])
+
+ asyncio.run(scenario())
+
+ def test_commit_steering_consumptions_skips_non_matching_ids(self) -> None:
+ """The per-event update loop must keep scanning past events that
+ don't match any id in the batch, not just check the first one."""
+
+ from lingxigraph.steering import SteeringEvent
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+ e1, _ = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"i": 1}
+ )
+ e2, _ = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"i": 2}
+ )
+
+ consumption = SteeringConsumption(
+ event=SteeringEvent(
+ id=e2.id,
+ run_id=run.id,
+ sequence=e2.sequence,
+ kind=e2.kind,
+ payload=e2.payload,
+ metadata=e2.metadata,
+ created_at=e2.created_at,
+ ),
+ consumed_at=e2.created_at,
+ node="the_node",
+ )
+ stored = await repo.commit_steering_consumptions("acme", run.id, [consumption])
+ self.assertEqual(len(stored), 1)
+ self.assertEqual(stored[0].data["steering_event_id"], e2.id)
+
+ statuses = {
+ event.id: event.status for event in await repo.list_steering("acme", run.id)
+ }
+ self.assertEqual(statuses[e1.id], "pending")
+ self.assertEqual(statuses[e2.id], "consumed")
+
+ asyncio.run(scenario())
+
+ def test_resume_rejects_a_missing_old_run(self) -> None:
+ """Issue #16 PR #17 review round 4, point 1: the locked
+ revalidation inside ``resume_run_with_pending_steering`` must
+ reject (rather than silently proceed for) an old run that is no
+ longer present -- it can no longer prove the run was ``paused``."""
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+ # Simulate the old run row being gone by the time the resume
+ # migration runs.
+ del repo._runs[("acme", run.id)]
+ with self.assertRaises(RunResumeConflictError):
+ await repo.resume_run_with_pending_steering(
+ "acme", None, assistant, _run_create(assistant.id), run.id
+ )
+
+ asyncio.run(scenario())
+
+ def test_transfer_preserves_idempotency_key_and_skips_already_consumed(
+ self,
+ ) -> None:
+ """Only ``pending``/``delivered`` events transfer; an already
+ ``consumed`` event is left behind untouched, and a transferred
+ event's idempotency key must still dedupe under the new run id."""
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ old_run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+ consumed_event, _ = await repo.submit_steering(
+ "acme", old_run.id, kind="user_input", payload={"i": "done"}
+ )
+ await repo.mark_steering_consumed("acme", old_run.id, [consumed_event.id])
+ pending_event, _ = await repo.submit_steering(
+ "acme",
+ old_run.id,
+ kind="user_input",
+ payload={"i": "pending"},
+ idempotency_key="key-1",
+ )
+ await repo.finish_run("acme", old_run.id, "paused", output={})
+
+ new_run, transferred = await repo.resume_run_with_pending_steering(
+ "acme", None, assistant, _run_create(assistant.id), old_run.id
+ )
+
+ self.assertEqual(len(transferred), 1)
+ self.assertEqual(transferred[0].source_event_id, pending_event.id)
+ self.assertEqual(transferred[0].idempotency_key, "key-1")
+
+ old_events = {
+ event.id: event.status for event in await repo.list_steering("acme", old_run.id)
+ }
+ self.assertEqual(old_events[consumed_event.id], "consumed")
+ self.assertEqual(old_events[pending_event.id], "superseded")
+
+ # Resubmitting with the same idempotency key under the *new*
+ # run id must dedupe against the transferred row, proving the
+ # transfer registered it in ``_steering_keys`` for the new id.
+ dup_event, created = await repo.submit_steering(
+ "acme",
+ new_run.id,
+ kind="user_input",
+ payload={"different": True},
+ idempotency_key="key-1",
+ )
+ self.assertFalse(created)
+ self.assertEqual(dup_event.id, transferred[0].id)
+
+ asyncio.run(scenario())
+
+
+class InMemoryRepositoryRunLifecycleEdgeCaseTests(unittest.TestCase):
+ """Direct unit coverage of ``InMemoryRepository`` run-lifecycle branches
+ unrelated to steering payloads but exercised alongside them in the same
+ module (queue quotas, redrive, and claim blocking)."""
+
+ def test_redrive_run_returns_none_for_non_redrivable_status(self) -> None:
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+ # A freshly created run is "pending", not dead_letter/failed.
+ result = await repo.redrive_run("acme", run.id)
+ self.assertIsNone(result)
+
+ asyncio.run(scenario())
+
+ def test_create_run_respects_run_timeout_and_budget_config(self) -> None:
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ run = await repo.create_run(
+ "acme",
+ None,
+ assistant,
+ _run_create(assistant.id).model_copy(
+ update={"run_timeout": 42, "max_tool_calls": 3}
+ ),
+ )
+ self.assertEqual(run.config["run_timeout"], 42)
+ self.assertEqual(run.config["max_tool_calls"], 3)
+
+ asyncio.run(scenario())
+
+ def test_create_run_queued_quota_exceeded_raises(self) -> None:
+ async def scenario() -> None:
+ from lingxigraph.errors import ConcurrentRunError
+ from lingxigraph.server.repository import RepositoryLimits
+
+ repo = InMemoryRepository(
+ limits=RepositoryLimits(max_active_runs=10, max_queued_runs=1)
+ )
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ # First run stays "pending" (queued) since nothing claims it.
+ await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+ with self.assertRaises(ConcurrentRunError):
+ await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+
+ asyncio.run(scenario())
+
+ def test_claim_run_skips_thread_blocked_pending_run(self) -> None:
+ """A pending run whose thread already has an active run must be
+ skipped by ``claim_run`` (blocked branch), leaving it unclaimed."""
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ thread = await repo.create_thread("acme", ThreadCreate())
+ first = await repo.create_run(
+ "acme", thread.id, assistant, _run_create(assistant.id)
+ )
+ claimed_first = await repo.claim_run("worker-1")
+ self.assertEqual(claimed_first.id, first.id)
+
+ await repo.create_run("acme", thread.id, assistant, _run_create(assistant.id))
+ # `second` is pending on the same thread as the still-running
+ # `first`, so claim_run must skip it and return None.
+ claimed_second = await repo.claim_run("worker-2")
+ self.assertIsNone(claimed_second)
+
+ asyncio.run(scenario())
+
+
+def _assistant_create():
+ from lingxigraph.server.models import AssistantCreate
+
+ return AssistantCreate(graph_id="double")
+
+
+def _run_create(assistant_id: str, input: dict | None = None):
+ from lingxigraph.server.models import RunCreate
+
+ return RunCreate(
+ assistant_id=assistant_id, input=input if input is not None else {"value": 1}
+ )
+
+
+MAX_STEERABLE_TICKS = 15
+
+
+def make_registry() -> GraphRegistry:
+ """A graph that loops draining steering until told to stop.
+
+ Each tick sleeps briefly so a concurrently-running test has a real
+ wall-clock window to submit steering input over HTTP before the graph
+ reaches its tick cap -- mirrors a real "keep working until steered"
+ agent loop.
+ """
+
+ builder = StateGraph(State)
+
+ async def wait_for_stop(state: State, runtime: Runtime[Any]) -> dict[str, Any]:
+ await asyncio.sleep(0.05)
+ drained = [event.payload.get("message", "") for event in runtime.drain_steering()]
+ merged = list(state.get("drained", [])) + drained
+ return {"ticks": state.get("ticks", 0) + 1, "drained": merged}
+
+ def should_continue(state: State) -> str:
+ if state.get("drained") and "stop" in state["drained"]:
+ return END
+ if state.get("ticks", 0) >= MAX_STEERABLE_TICKS:
+ return END
+ return "wait_for_stop"
+
+ builder.add_node("wait_for_stop", wait_for_stop)
+ builder.add_edge(START, "wait_for_stop")
+ builder.add_conditional_edges("wait_for_stop", should_continue)
+ return GraphRegistry({"steerable": builder.compile()})
+
+
+def make_registry_double() -> GraphRegistry:
+ """A trivial, near-instant single-node graph registered as ``double``.
+
+ Used by the round 6 finalization-fencing/gate tests, which need the
+ graph to reach its terminal outcome (and the worker to enter its
+ finalization window) almost immediately, without any steering to
+ drain along the way.
+ """
+
+ builder = StateGraph(State)
+
+ async def double(state: State, runtime: Runtime[Any]) -> dict[str, Any]:
+ return {"ticks": state.get("ticks", 0) + 1}
+
+ builder.add_node("double", double)
+ builder.add_edge(START, "double")
+ builder.add_edge("double", END)
+ return GraphRegistry({"double": builder.compile()})
+
+
+class ServerSteeringTests(unittest.TestCase):
+ """End-to-end: HTTP accept -> durable inbox -> worker -> graph safe point."""
+
+ def test_running_run_receives_steer_and_consumes_it(self) -> None:
+ app = create_app(
+ registry=make_registry(),
+ authenticator=Authenticator.insecure_dev(),
+ embedded_worker=True,
+ )
+ # Fast heartbeat so mid-run steering syncs promptly for the test;
+ # production defaults to a longer interval.
+ app.state.worker.heartbeat_seconds = 0.02
+ headers = {"x-tenant-id": "acme"}
+ with TestClient(app) as client:
+ assistant = client.post(
+ "/v1/assistants", headers=headers, json={"graph_id": "steerable"}
+ ).json()
+ run = client.post(
+ "/v1/runs",
+ headers=headers,
+ json={"assistant_id": assistant["id"], "input": {"ticks": 0, "drained": []}},
+ ).json()
+ run_id = run["id"]
+
+ # Wait until the worker has actually claimed and started the run.
+ wait_for_status(client, run_id, headers, {"running"}, attempts=200)
+
+ accepted = client.post(
+ f"/v1/runs/{run_id}/steer",
+ headers=headers,
+ json={"kind": "user_input", "payload": {"message": "keep-going"}},
+ )
+ self.assertEqual(accepted.status_code, 202, accepted.text)
+ self.assertEqual(accepted.json()["sequence"], 1)
+ self.assertIn(accepted.json()["status"], {"pending", "delivered"})
+
+ stop = client.post(
+ f"/v1/runs/{run_id}/steer",
+ headers=headers,
+ json={"kind": "user_input", "payload": {"message": "stop"}},
+ )
+ self.assertEqual(stop.status_code, 202)
+
+ result = wait_for_status(client, run_id, headers, {"succeeded", "failed"})
+ self.assertEqual(result.json()["status"], "succeeded", result.text)
+ self.assertIn("keep-going", result.json()["output"]["drained"])
+ self.assertIn("stop", result.json()["output"]["drained"])
+
+ events = asyncio.run(app.state.repository.list_events("acme", run_id))
+ kinds = [event.kind for event in events]
+ self.assertIn("run.steer.accepted", kinds)
+ self.assertIn("run.steer.consumed", kinds)
+ # accepted must precede consumed for the lifecycle to make sense.
+ self.assertLess(
+ kinds.index("run.steer.accepted"), kinds.index("run.steer.consumed")
+ )
+
+ def test_duplicate_idempotency_key_does_not_duplicate(self) -> None:
+ app = create_app(registry=make_registry(), authenticator=Authenticator.insecure_dev())
+ headers = {"x-tenant-id": "acme"}
+ with TestClient(app) as client:
+ assistant = client.post(
+ "/v1/assistants", headers=headers, json={"graph_id": "steerable"}
+ ).json()
+ run = client.post(
+ "/v1/runs",
+ headers=headers,
+ json={"assistant_id": assistant["id"], "input": {"ticks": 0, "drained": []}},
+ ).json()
+ run_id = run["id"]
+ first = client.post(
+ f"/v1/runs/{run_id}/steer",
+ headers={**headers, "Idempotency-Key": "msg-123"},
+ json={"kind": "user_input", "payload": {"message": "hi"}},
+ ).json()
+ second = client.post(
+ f"/v1/runs/{run_id}/steer",
+ headers={**headers, "Idempotency-Key": "msg-123"},
+ json={"kind": "user_input", "payload": {"message": "hi-again"}},
+ ).json()
+ self.assertEqual(first["id"], second["id"])
+ self.assertEqual(first["sequence"], second["sequence"])
+ pending = asyncio.run(app.state.repository.list_pending_steering("acme", run_id))
+ self.assertEqual(len(pending), 1)
+
+ def test_terminal_run_returns_conflict_not_silent_event(self) -> None:
+ app = create_app(registry=make_registry(), authenticator=Authenticator.insecure_dev())
+ headers = {"x-tenant-id": "acme"}
+ with TestClient(app) as client:
+ assistant = client.post(
+ "/v1/assistants", headers=headers, json={"graph_id": "steerable"}
+ ).json()
+ run = client.post(
+ "/v1/runs",
+ headers=headers,
+ json={"assistant_id": assistant["id"], "input": {"ticks": 0, "drained": []}},
+ ).json()
+ run_id = run["id"]
+ cancelled = client.post(f"/v1/runs/{run_id}/cancel", headers=headers)
+ self.assertEqual(cancelled.json()["status"], "cancelled")
+ rejected = client.post(
+ f"/v1/runs/{run_id}/steer",
+ headers=headers,
+ json={"kind": "user_input", "payload": {}},
+ )
+ self.assertEqual(rejected.status_code, 409)
+ self.assertEqual(rejected.json()["code"], "run_terminal")
+
+ def test_missing_run_returns_404(self) -> None:
+ app = create_app(registry=make_registry(), authenticator=Authenticator.insecure_dev())
+ headers = {"x-tenant-id": "acme"}
+ with TestClient(app) as client:
+ response = client.post(
+ "/v1/runs/does-not-exist/steer",
+ headers=headers,
+ json={"kind": "user_input", "payload": {}},
+ )
+ self.assertEqual(response.status_code, 404)
+
+ def test_payload_too_large_rejected(self) -> None:
+ app = create_app(registry=make_registry(), authenticator=Authenticator.insecure_dev())
+ headers = {"x-tenant-id": "acme"}
+ with TestClient(app) as client:
+ assistant = client.post(
+ "/v1/assistants", headers=headers, json={"graph_id": "steerable"}
+ ).json()
+ run = client.post(
+ "/v1/runs",
+ headers=headers,
+ json={"assistant_id": assistant["id"], "input": {"ticks": 0, "drained": []}},
+ ).json()
+ response = client.post(
+ f"/v1/runs/{run['id']}/steer",
+ headers=headers,
+ json={"kind": "user_input", "payload": {"message": "x" * 500_000}},
+ )
+ self.assertEqual(response.status_code, 413)
+
+ def test_payload_between_32kb_and_generic_event_cap_still_rejected(self) -> None:
+ """Issue #16 PR #17 review round 5, point 3 (CONTRACT): the public
+ docs and ``steering.MAX_STEERING_PAYLOAD_BYTES`` both promise a
+ ~32KB cap, but the HTTP endpoint used to pass
+ ``repository.limits.max_event_bytes`` (262,144 bytes) as the
+ payload ceiling instead, silently accepting payloads up to 8x the
+ documented limit. A ~500KB payload (comfortably over *both*
+ numbers) couldn't have caught this ~40KB-sized contract mismatch;
+ this uses a payload sized specifically between the two limits."""
+
+ from lingxigraph.steering import MAX_STEERING_PAYLOAD_BYTES
+
+ app = create_app(registry=make_registry(), authenticator=Authenticator.insecure_dev())
+ headers = {"x-tenant-id": "acme"}
+ with TestClient(app) as client:
+ assistant = client.post(
+ "/v1/assistants", headers=headers, json={"graph_id": "steerable"}
+ ).json()
+ run = client.post(
+ "/v1/runs",
+ headers=headers,
+ json={"assistant_id": assistant["id"], "input": {"ticks": 0, "drained": []}},
+ ).json()
+ oversized_message = "x" * (MAX_STEERING_PAYLOAD_BYTES + 8_000)
+ self.assertLess(len(oversized_message), 262_144)
+ response = client.post(
+ f"/v1/runs/{run['id']}/steer",
+ headers=headers,
+ json={"kind": "user_input", "payload": {"message": oversized_message}},
+ )
+ self.assertEqual(response.status_code, 413)
+
+ def test_queued_run_can_receive_steer_before_worker_claims_it(self) -> None:
+ app = create_app(registry=make_registry(), authenticator=Authenticator.insecure_dev())
+ headers = {"x-tenant-id": "acme"}
+ with TestClient(app) as client:
+ assistant = client.post(
+ "/v1/assistants", headers=headers, json={"graph_id": "steerable"}
+ ).json()
+ run = client.post(
+ "/v1/runs",
+ headers=headers,
+ json={"assistant_id": assistant["id"], "input": {"ticks": 0, "drained": []}},
+ ).json()
+ self.assertEqual(
+ client.get(f"/v1/runs/{run['id']}", headers=headers).json()["status"], "pending"
+ )
+ accepted = client.post(
+ f"/v1/runs/{run['id']}/steer",
+ headers=headers,
+ json={"kind": "user_input", "payload": {"message": "stop"}},
+ )
+ self.assertEqual(accepted.status_code, 202)
+
+ worker = app.state.worker
+ asyncio.run(worker.run_once())
+ asyncio.run(worker.run_once())
+ result = wait_for_status(client, run["id"], headers, {"succeeded", "failed"})
+ self.assertEqual(result.json()["status"], "succeeded", result.text)
+ self.assertIn("stop", result.json()["output"]["drained"])
+
+ def test_worker_restart_recovers_pending_steering_from_postgres_shaped_repo(self) -> None:
+ """Simulates a worker crash: a fresh Worker resumes from the repository."""
+
+ from lingxigraph.server.worker import Worker
+
+ async def scenario() -> None:
+ registry = make_registry()
+ repository = InMemoryRepository()
+ assistant = await repository.create_assistant(
+ "acme", _assistant_create_named("steerable"), graph_version="1"
+ )
+ run = await repository.create_run(
+ "acme", None, assistant, _run_create(assistant.id, {"ticks": 0, "drained": []})
+ )
+ # Durable accept happens *before* any worker exists yet.
+ await repository.submit_steering(
+ "acme", run.id, kind="user_input", payload={"message": "stop"}
+ )
+
+ crashed_worker = Worker(registry, repository, heartbeat_seconds=0.05)
+ claimed = await crashed_worker.run_once()
+ self.assertTrue(claimed)
+ finished = await repository.get_run("acme", run.id)
+ self.assertEqual(
+ finished.status.value if hasattr(finished.status, "value") else finished.status,
+ "succeeded",
+ )
+ self.assertIn("stop", finished.output["drained"])
+
+ asyncio.run(scenario())
+
+ def test_redis_notify_loss_does_not_lose_steering(self) -> None:
+ """The worker heartbeat re-polls PostgreSQL regardless of eventbus."""
+
+ from lingxigraph.server.eventbus import EventBus
+ from lingxigraph.server.worker import Worker
+
+ class BlackHoleEventBus(EventBus):
+ async def publish(self, tenant_id: str, run_id: str, sequence: int) -> None:
+ return None
+
+ async def wait(self, tenant_id: str, run_id: str, *, timeout: float = 15.0) -> None:
+ return None
+
+ async def scenario() -> None:
+ registry = make_registry()
+ repository = InMemoryRepository()
+ assistant = await repository.create_assistant(
+ "acme", _assistant_create_named("steerable"), graph_version="1"
+ )
+ run = await repository.create_run(
+ "acme", None, assistant, _run_create(assistant.id, {"ticks": 0, "drained": []})
+ )
+ worker = Worker(
+ registry, repository, heartbeat_seconds=0.05, event_bus=BlackHoleEventBus()
+ )
+ task = asyncio.create_task(worker.run_once())
+ await asyncio.sleep(0.05)
+ await repository.submit_steering(
+ "acme", run.id, kind="user_input", payload={"message": "stop"}
+ )
+ await task
+ finished = await repository.get_run("acme", run.id)
+ status = (
+ finished.status.value if hasattr(finished.status, "value") else finished.status
+ )
+ self.assertEqual(status, "succeeded")
+ self.assertIn("stop", finished.output["drained"])
+
+ asyncio.run(scenario())
+
+ def test_cancel_takes_priority_over_steer(self) -> None:
+ """Steering must never undo or delay a cancel request.
+
+ Exercised on a still-queued run (no worker claims it): the durable
+ steer is accepted, and the subsequent cancel still takes effect
+ immediately and unconditionally -- a prior steer never blocks it.
+ """
+
+ app = create_app(registry=make_registry(), authenticator=Authenticator.insecure_dev())
+ headers = {"x-tenant-id": "acme"}
+ with TestClient(app) as client:
+ assistant = client.post(
+ "/v1/assistants", headers=headers, json={"graph_id": "steerable"}
+ ).json()
+ run = client.post(
+ "/v1/runs",
+ headers=headers,
+ json={"assistant_id": assistant["id"], "input": {"ticks": 0, "drained": []}},
+ ).json()
+ run_id = run["id"]
+ accepted = client.post(
+ f"/v1/runs/{run_id}/steer",
+ headers=headers,
+ json={"kind": "user_input", "payload": {"message": "keep-going"}},
+ )
+ self.assertEqual(accepted.status_code, 202)
+ cancel = client.post(f"/v1/runs/{run_id}/cancel", headers=headers)
+ self.assertEqual(cancel.status_code, 200)
+ self.assertEqual(cancel.json()["status"], "cancelled")
+ # The steer stays durably recorded (never rewound by cancel) but
+ # will simply never be delivered since the run never executes.
+ pending = asyncio.run(app.state.repository.list_pending_steering("acme", run_id))
+ self.assertEqual(len(pending), 1)
+
+ def test_paused_run_steer_is_transferred_and_consumed_after_resume(self) -> None:
+ """Documented choice (option B): steer during pause is durably
+ accepted under the *paused* run_id, but ``/resume`` creates a brand
+ new Run row -- so at resume time any still-pending steering is
+ atomically transferred onto the new run_id and delivered/consumed
+ by the graph there. This is the end-to-end regression for the
+ blocking review point: the old test only asserted resume succeeded,
+ never that the paused steer was actually drained/consumed."""
+
+ from lingxigraph import interrupt
+
+ paused_builder = StateGraph(State)
+
+ def approval(state: State, runtime: Runtime[Any]) -> dict[str, Any]:
+ drained_before = [event.payload for event in runtime.peek_steering()]
+ value = interrupt({"question": "approve?", "seen_before_resume": drained_before})
+ return {"ticks": int(value)}
+
+ def after_approval(state: State, runtime: Runtime[Any]) -> dict[str, Any]:
+ # Proves the transferred event is actually delivered to *this*
+ # (new) run's safe point, not merely accepted somewhere.
+ drained = [event.payload["message"] for event in runtime.drain_steering()]
+ return {"drained": drained}
+
+ paused_builder.add_node("approval", approval)
+ paused_builder.add_node("after_approval", after_approval)
+ paused_builder.add_edge(START, "approval").add_edge("approval", "after_approval")
+ paused_builder.add_edge("after_approval", END)
+ registry = GraphRegistry({"approval": paused_builder.compile()})
+
+ app = create_app(
+ registry=registry, authenticator=Authenticator.insecure_dev(), embedded_worker=True
+ )
+ headers = {"x-tenant-id": "acme"}
+ with TestClient(app) as client:
+ assistant = client.post(
+ "/v1/assistants", headers=headers, json={"graph_id": "approval"}
+ ).json()
+ thread = client.post("/v1/threads", headers=headers, json={}).json()
+ run = client.post(
+ f"/v1/threads/{thread['id']}/runs",
+ headers=headers,
+ json={"assistant_id": assistant["id"], "input": {"ticks": 0}},
+ ).json()
+ run_id = run["id"]
+ paused = wait_for_status(client, run_id, headers, {"paused"})
+ self.assertEqual(paused.json()["status"], "paused")
+
+ accepted = client.post(
+ f"/v1/runs/{run_id}/steer",
+ headers=headers,
+ json={"kind": "user_input", "payload": {"message": "while-paused"}},
+ )
+ self.assertEqual(accepted.status_code, 202)
+ pending = asyncio.run(app.state.repository.list_pending_steering("acme", run_id))
+ self.assertEqual(len(pending), 1)
+
+ resumed = client.post(
+ f"/v1/runs/{run_id}/resume", headers=headers, json={"resume": 9}
+ ).json()
+ new_run_id = resumed["id"]
+ self.assertNotEqual(new_run_id, run_id)
+ result = wait_for_status(client, new_run_id, headers, {"succeeded", "failed"})
+ self.assertEqual(result.json()["status"], "succeeded", result.text)
+
+ # The graph actually saw and drained the while-paused event
+ # under the *new* run.
+ self.assertEqual(result.json()["output"]["drained"], ["while-paused"])
+
+ # The old run's steering row is terminal ("superseded"), not
+ # stuck "pending" forever -- and the new run's inbox shows it
+ # went pending -> consumed there.
+ old_events = asyncio.run(app.state.repository.list_steering("acme", run_id))
+ self.assertEqual([event.status for event in old_events], ["superseded"])
+ new_events = asyncio.run(app.state.repository.list_steering("acme", new_run_id))
+ self.assertEqual(len(new_events), 1)
+ self.assertEqual(new_events[0].status, "consumed")
+ self.assertEqual(new_events[0].payload, {"message": "while-paused"})
+
+ new_run_events = asyncio.run(app.state.repository.list_events("acme", new_run_id))
+ kinds = [event.kind for event in new_run_events]
+ self.assertIn("run.steer.accepted", kinds)
+ self.assertIn("run.steer.consumed", kinds)
+ consumed_event = next(
+ event for event in new_run_events if event.kind == "run.steer.consumed"
+ )
+ self.assertEqual(consumed_event.data["node"], "after_approval")
+ self.assertIn("queue_latency_seconds", consumed_event.data)
+ self.assertGreaterEqual(consumed_event.data["queue_latency_seconds"], 0)
+
+ # Further steer attempts against the now-superseded old run_id
+ # are rejected loudly instead of silently pending again.
+ stale = client.post(
+ f"/v1/runs/{run_id}/steer",
+ headers=headers,
+ json={"kind": "user_input", "payload": {"message": "too-late"}},
+ )
+ self.assertEqual(stale.status_code, 409)
+ self.assertEqual(stale.json()["code"], "run_superseded")
+
+ def test_resume_with_no_pending_steering_still_marks_old_run_superseded(self) -> None:
+ """Even with nothing to transfer, resume closes the old run_id off
+ from future steering so it cannot silently pend forever a second
+ time (see RunSupersededError)."""
+
+ from lingxigraph import interrupt
+
+ builder = StateGraph(State)
+
+ def approval(state: State, runtime: Runtime[Any]) -> dict[str, Any]:
+ value = interrupt({"question": "approve?"})
+ return {"ticks": int(value)}
+
+ builder.add_node("approval", approval)
+ builder.add_edge(START, "approval").add_edge("approval", END)
+ registry = GraphRegistry({"approval": builder.compile()})
+
+ app = create_app(
+ registry=registry, authenticator=Authenticator.insecure_dev(), embedded_worker=True
+ )
+ headers = {"x-tenant-id": "acme"}
+ with TestClient(app) as client:
+ assistant = client.post(
+ "/v1/assistants", headers=headers, json={"graph_id": "approval"}
+ ).json()
+ thread = client.post("/v1/threads", headers=headers, json={}).json()
+ run = client.post(
+ f"/v1/threads/{thread['id']}/runs",
+ headers=headers,
+ json={"assistant_id": assistant["id"], "input": {"ticks": 0}},
+ ).json()
+ run_id = run["id"]
+ wait_for_status(client, run_id, headers, {"paused"})
+
+ resumed = client.post(
+ f"/v1/runs/{run_id}/resume", headers=headers, json={"resume": 1}
+ ).json()
+ wait_for_status(client, resumed["id"], headers, {"succeeded", "failed"})
+
+ stale = client.post(
+ f"/v1/runs/{run_id}/steer",
+ headers=headers,
+ json={"kind": "user_input", "payload": {}},
+ )
+ self.assertEqual(stale.status_code, 409)
+ self.assertEqual(stale.json()["code"], "run_superseded")
+
+ def test_paused_steer_identity_and_latency_survive_resume_transfer(self) -> None:
+ """Issue #16 PR #17 review point 3: a paused-run transfer must
+ preserve the *original* event's identity and acceptance time.
+
+ Proves both halves of the fix end to end:
+ * The id a client's paused ``/steer`` call got back
+ (``accepted_id``) is traceable, via ``source_event_id``, all the
+ way through the resumed run's ``run.steer.accepted`` and
+ ``run.steer.consumed`` events -- never silently replaced by an
+ unrelated new id.
+ * ``queue_latency_seconds`` on the eventual ``consumed`` event is
+ computed from the *original* acceptance time, so it includes the
+ time the event spent waiting while the run was paused, not just
+ the time since the resume/transfer moment.
+ """
+
+ from lingxigraph import interrupt
+
+ paused_builder = StateGraph(State)
+
+ def approval(state: State, runtime: Runtime[Any]) -> dict[str, Any]:
+ value = interrupt({"question": "approve?"})
+ return {"ticks": int(value)}
+
+ def after_approval(state: State, runtime: Runtime[Any]) -> dict[str, Any]:
+ drained = [event.payload["message"] for event in runtime.drain_steering()]
+ return {"drained": drained}
+
+ paused_builder.add_node("approval", approval)
+ paused_builder.add_node("after_approval", after_approval)
+ paused_builder.add_edge(START, "approval").add_edge("approval", "after_approval")
+ paused_builder.add_edge("after_approval", END)
+ registry = GraphRegistry({"approval": paused_builder.compile()})
+
+ app = create_app(
+ registry=registry, authenticator=Authenticator.insecure_dev(), embedded_worker=True
+ )
+ headers = {"x-tenant-id": "acme"}
+ with TestClient(app) as client:
+ assistant = client.post(
+ "/v1/assistants", headers=headers, json={"graph_id": "approval"}
+ ).json()
+ thread = client.post("/v1/threads", headers=headers, json={}).json()
+ run = client.post(
+ f"/v1/threads/{thread['id']}/runs",
+ headers=headers,
+ json={"assistant_id": assistant["id"], "input": {"ticks": 0}},
+ ).json()
+ run_id = run["id"]
+ wait_for_status(client, run_id, headers, {"paused"})
+
+ accepted = client.post(
+ f"/v1/runs/{run_id}/steer",
+ headers=headers,
+ json={"kind": "user_input", "payload": {"message": "while-paused"}},
+ ).json()
+ accepted_id = accepted["id"]
+
+ # A real pause-wait window: the queue latency computed after
+ # resume must include (at least) this much time.
+ pause_wait_seconds = 0.3
+ time.sleep(pause_wait_seconds)
+
+ resumed = client.post(
+ f"/v1/runs/{run_id}/resume", headers=headers, json={"resume": 9}
+ ).json()
+ new_run_id = resumed["id"]
+ result = wait_for_status(client, new_run_id, headers, {"succeeded", "failed"})
+ self.assertEqual(result.json()["status"], "succeeded", result.text)
+
+ new_events = asyncio.run(app.state.repository.list_events("acme", new_run_id))
+ accepted_transfer = next(
+ event for event in new_events if event.kind == "run.steer.accepted"
+ )
+ # The transferred event's id changed (it is a new durable row),
+ # but its source_event_id must point back at the exact id the
+ # client received from its paused-run /steer call.
+ self.assertNotEqual(accepted_transfer.data["steering_event_id"], accepted_id)
+ self.assertEqual(accepted_transfer.data["source_event_id"], accepted_id)
+
+ consumed = next(event for event in new_events if event.kind == "run.steer.consumed")
+ self.assertEqual(consumed.data["source_event_id"], accepted_id)
+ self.assertGreaterEqual(
+ consumed.data["queue_latency_seconds"], pause_wait_seconds * 0.9
+ )
+
+ steering_rows = asyncio.run(app.state.repository.list_steering("acme", new_run_id))
+ self.assertEqual(steering_rows[0].source_event_id, accepted_id)
+
+ def test_double_pause_resume_preserves_root_source_event_id(self) -> None:
+ """Issue #16 PR #17 review round 3, point 2: a *second* pause and
+ resume (A -> B -> C) must not overwrite the root ``source_event_id``
+ with the intermediate hop's id -- C's migrated event must still
+ correlate back to A, not B."""
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ from lingxigraph.server.models import RunCreate
+
+ run_a = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+ event_a, _ = await repo.submit_steering(
+ "acme", run_a.id, kind="user_input", payload={"m": 1}
+ )
+ self.assertIsNone(event_a.source_event_id)
+ await repo.finish_run("acme", run_a.id, "paused", output={})
+
+ run_b, transferred_b = await repo.resume_run_with_pending_steering(
+ "acme",
+ run_a.thread_id,
+ assistant,
+ RunCreate(assistant_id=assistant.id, resume=1),
+ run_a.id,
+ )
+ self.assertEqual(len(transferred_b), 1)
+ event_b = transferred_b[0]
+ self.assertEqual(event_b.source_event_id, event_a.id)
+ self.assertEqual(event_b.created_at, event_a.created_at)
+ await repo.finish_run("acme", run_b.id, "paused", output={})
+
+ run_c, transferred_c = await repo.resume_run_with_pending_steering(
+ "acme",
+ run_b.thread_id,
+ assistant,
+ RunCreate(assistant_id=assistant.id, resume=1),
+ run_b.id,
+ )
+ self.assertEqual(len(transferred_c), 1)
+ event_c = transferred_c[0]
+ # Root identity A must survive, never be replaced by B's id.
+ self.assertEqual(event_c.source_event_id, event_a.id)
+ self.assertEqual(event_c.created_at, event_a.created_at)
+
+ from lingxigraph.steering import SteeringConsumption, SteeringEvent
+
+ steering_event = SteeringEvent(
+ id=event_c.id,
+ run_id=run_c.id,
+ sequence=event_c.sequence,
+ kind=event_c.kind,
+ payload=event_c.payload,
+ metadata=event_c.metadata,
+ created_at=event_c.created_at,
+ source_event_id=event_c.source_event_id,
+ )
+ consumption = SteeringConsumption(
+ event=steering_event,
+ consumed_at=steering_event.created_at,
+ node="n",
+ namespace=(),
+ task_id="t",
+ )
+ stored = await repo.commit_steering_consumptions("acme", run_c.id, [consumption])
+ self.assertEqual(len(stored), 1)
+ self.assertEqual(stored[0].data["source_event_id"], event_a.id)
+
+ asyncio.run(scenario())
+
+ def test_commit_steering_consumptions_idempotent_on_retry(self) -> None:
+ """Issue #16 PR #17 review round 3, point 3: calling
+ ``commit_steering_consumptions`` twice with the same batch (a
+ worker retrying after an ack it never observed) must produce only
+ one ``run.steer.consumed`` lifecycle event, not two."""
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+ event, _ = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"m": 1}
+ )
+
+ from lingxigraph.steering import SteeringConsumption, SteeringEvent
+
+ steering_event = SteeringEvent(
+ id=event.id,
+ run_id=run.id,
+ sequence=event.sequence,
+ kind=event.kind,
+ payload=event.payload,
+ metadata=event.metadata,
+ created_at=event.created_at,
+ )
+ consumption = SteeringConsumption(
+ event=steering_event,
+ consumed_at=steering_event.created_at,
+ node="n",
+ namespace=(),
+ task_id="t",
+ )
+ first = await repo.commit_steering_consumptions("acme", run.id, [consumption])
+ self.assertEqual(len(first), 1)
+ second = await repo.commit_steering_consumptions("acme", run.id, [consumption])
+ self.assertEqual(second, [])
+
+ all_events = await repo.list_events("acme", run.id)
+ consumed = [e for e in all_events if e.kind == "run.steer.consumed"]
+ self.assertEqual(len(consumed), 1)
+
+ asyncio.run(scenario())
+
+
+class WorkerFencedWriteLeaseLostBranchTests(unittest.TestCase):
+ """Coverage for the "lease no longer owned" branches of every fenced
+ write the Worker performs during finalization -- these can only be
+ reached by simulating a lease loss between the Worker deciding what to
+ do and the repository actually committing it, so they're exercised
+ here by monkeypatching the fenced repository methods to return
+ ``None`` (exactly what a losing CAS/fence looks like from the
+ caller's perspective), rather than by orchestrating a real multi-
+ worker race (already covered elsewhere against real Postgres)."""
+
+ def test_retry_or_dead_letter_abandons_when_retry_lease_lost(self) -> None:
+ from lingxigraph.server.worker import Worker
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("double"), graph_version="1"
+ )
+ await repo.create_run(
+ "acme", None, assistant, _run_create(assistant.id, {"ticks": 0})
+ )
+ claimed = await repo.claim_run("worker-a", lease_seconds=30)
+ assert claimed is not None
+
+ worker = Worker(make_registry_double(), repo, worker_id="worker-a")
+
+ async def lost_lease(*args: Any, **kwargs: Any) -> None:
+ return None
+
+ repo.retry_run_with_event_if_owned = lost_lease # type: ignore[method-assign]
+ await worker._retry_or_dead_letter(
+ claimed, {"code": "delivery_retry", "message": "transient"}
+ )
+ # No exception, no crash, and (since the fenced write never
+ # touched anything) the run's real status is whatever it was
+ # before -- nothing durable to assert beyond "this returned".
+
+ asyncio.run(scenario())
+
+ def test_retry_or_dead_letter_abandons_when_dead_letter_lease_lost(self) -> None:
+ from lingxigraph.server.worker import Worker
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("double"), graph_version="1"
+ )
+ await repo.create_run(
+ "acme", None, assistant, _run_create(assistant.id, {"ticks": 0})
+ )
+ claimed = await repo.claim_run("worker-a", lease_seconds=30)
+ assert claimed is not None
+ claimed.attempt = 999 # force the max-attempts (dead-letter) branch
+
+ worker = Worker(
+ make_registry_double(), repo, worker_id="worker-a", max_delivery_attempts=1
+ )
+
+ async def lost_lease(*args: Any, **kwargs: Any) -> None:
+ return None
+
+ repo.finish_run_if_owned = lost_lease # type: ignore[method-assign]
+ await worker._retry_or_dead_letter(
+ claimed, {"code": "delivery_retry", "message": "transient"}
+ )
+
+ asyncio.run(scenario())
+
+ def test_commit_intent_finalize_abandons_when_lease_lost(self) -> None:
+ from lingxigraph.server.worker import Worker
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("double"), graph_version="1"
+ )
+ await repo.create_run(
+ "acme", None, assistant, _run_create(assistant.id, {"ticks": 0})
+ )
+ claimed = await repo.claim_run("worker-a", lease_seconds=30)
+ assert claimed is not None
+
+ worker = Worker(make_registry_double(), repo, worker_id="worker-a")
+
+ async def lost_lease(*args: Any, **kwargs: Any) -> None:
+ return None
+
+ repo.finish_run_if_owned = lost_lease # type: ignore[method-assign]
+ intent = Worker._Intent(kind="finish", status=RunStatus.SUCCEEDED, output={})
+ committed = await worker._commit_intent(claimed, intent)
+ self.assertFalse(committed)
+
+ asyncio.run(scenario())
+
+ def test_final_steering_flush_returns_false_when_heartbeat_confirms_lease_lost(
+ self,
+ ) -> None:
+ from lingxigraph.server.worker import Worker
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ registry = make_registry()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("steerable"), graph_version="1"
+ )
+ run = await repo.create_run(
+ "acme",
+ None,
+ assistant,
+ _run_create(assistant.id, {"ticks": 0, "drained": []}),
+ )
+ await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"message": "stop"}
+ )
+ claimed = await repo.claim_run("worker-a", lease_seconds=30)
+ assert claimed is not None
+
+ graph = registry.get("steerable").with_runtime()
+ channel = graph.get_steering_channel(run.id)
+ worker = Worker(make_registry(), repo, worker_id="worker-a")
+ await worker._sync_steering_in(claimed, graph)
+ channel.drain()
+
+ async def always_fails(*args: Any, **kwargs: Any) -> None:
+ raise ConnectionError("transient")
+
+ async def lease_gone(*args: Any, **kwargs: Any) -> bool:
+ return False
+
+ repo.commit_steering_consumptions_if_owned = always_fails # type: ignore[method-assign]
+ repo.heartbeat = lease_gone # type: ignore[method-assign]
+ result = await worker._final_steering_flush(
+ claimed, graph, base_delay=0.001, max_delay=0.001
+ )
+ self.assertFalse(result)
+
+ asyncio.run(scenario())
+
+ def test_final_steering_flush_returns_false_when_commit_reports_lease_lost(self) -> None:
+ from lingxigraph.server.worker import Worker
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ registry = make_registry()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("steerable"), graph_version="1"
+ )
+ run = await repo.create_run(
+ "acme",
+ None,
+ assistant,
+ _run_create(assistant.id, {"ticks": 0, "drained": []}),
+ )
+ await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"message": "stop"}
+ )
+ claimed = await repo.claim_run("worker-a", lease_seconds=30)
+ assert claimed is not None
+
+ graph = registry.get("steerable").with_runtime()
+ channel = graph.get_steering_channel(run.id)
+ worker = Worker(make_registry(), repo, worker_id="worker-a")
+ await worker._sync_steering_in(claimed, graph)
+ channel.drain()
+
+ async def lost_lease(*args: Any, **kwargs: Any) -> None:
+ return None
+
+ repo.commit_steering_consumptions_if_owned = lost_lease # type: ignore[method-assign]
+ result = await worker._final_steering_flush(claimed, graph)
+ self.assertFalse(result)
+
+ asyncio.run(scenario())
+
+ def test_sync_steering_out_skips_ack_when_commit_fails_transiently(self) -> None:
+ from lingxigraph.server.worker import Worker
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ registry = make_registry()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("steerable"), graph_version="1"
+ )
+ run = await repo.create_run(
+ "acme",
+ None,
+ assistant,
+ _run_create(assistant.id, {"ticks": 0, "drained": []}),
+ )
+ await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"message": "stop"}
+ )
+ claimed = await repo.claim_run("worker-a", lease_seconds=30)
+ assert claimed is not None
+
+ graph = registry.get("steerable").with_runtime()
+ channel = graph.get_steering_channel(run.id)
+ worker = Worker(make_registry(), repo, worker_id="worker-a")
+ await worker._sync_steering_in(claimed, graph)
+ channel.drain()
+
+ async def always_fails(*args: Any, **kwargs: Any) -> None:
+ raise ConnectionError("transient")
+
+ repo.commit_steering_consumptions_if_owned = always_fails # type: ignore[method-assign]
+ await worker._sync_steering_out(claimed, graph)
+ self.assertEqual(len(channel.peek_consumed()), 1)
+
+ asyncio.run(scenario())
+
+ def test_sync_steering_out_skips_ack_when_lease_lost(self) -> None:
+ from lingxigraph.server.worker import Worker
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ registry = make_registry()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("steerable"), graph_version="1"
+ )
+ run = await repo.create_run(
+ "acme",
+ None,
+ assistant,
+ _run_create(assistant.id, {"ticks": 0, "drained": []}),
+ )
+ await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"message": "stop"}
+ )
+ claimed = await repo.claim_run("worker-a", lease_seconds=30)
+ assert claimed is not None
+
+ graph = registry.get("steerable").with_runtime()
+ channel = graph.get_steering_channel(run.id)
+ worker = Worker(make_registry(), repo, worker_id="worker-a")
+ await worker._sync_steering_in(claimed, graph)
+ channel.drain()
+
+ async def lost_lease(*args: Any, **kwargs: Any) -> None:
+ return None
+
+ repo.commit_steering_consumptions_if_owned = lost_lease # type: ignore[method-assign]
+ await worker._sync_steering_out(claimed, graph)
+ self.assertEqual(len(channel.peek_consumed()), 1)
+
+ asyncio.run(scenario())
+
+ def test_delivery_retry_recovers_steering_not_yet_drained(self) -> None:
+ """Issue #16 PR #17 review round 9, point 1 (BLOCKER).
+
+ A REAL Worker-level delivery-retry regression -- not a graph-level
+ ``RetryPolicy`` test (a node-level retry never leaves the Worker's
+ ``_execute`` at all, so it can never exercise the
+ ``closing_steering``/``intent.kind == "retry"`` scoping bug this
+ covers). Steering E is durably pending; attempt 1's graph node
+ raises a retryable exception *before* calling ``drain_steering()``
+ at all, so the exception unwinds all the way out of
+ ``graph.astream`` and into ``Worker._execute``'s ``except
+ Exception`` branch, producing a ``kind="retry"`` intent. Before the
+ round 9 fix, ``closing_steering`` was true for *any* non-paused
+ intent (including retry), so this sequence would close steering
+ admission and supersede E -- permanently losing it -- the instant
+ before the Run was sent back to ``pending``. Assert E survives: it
+ is still ``pending``/``delivered`` (never ``superseded``) after
+ attempt 1, and attempt 2 (a fresh ``claim_run()``, mirroring a real
+ redelivery) re-ingests and drains it successfully.
+ """
+
+ from lingxigraph.server.worker import Worker
+
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create_named("flaky_before_drain"), graph_version="1"
+ )
+ run = await repo.create_run(
+ "acme", None, assistant, _run_create(assistant.id, {"ticks": 0, "drained": []})
+ )
+ accepted, created = await repo.submit_steering(
+ "acme", run.id, kind="user_input", payload={"message": "hello"}
+ )
+ self.assertTrue(created)
+
+ registry, attempts, drained_log = make_registry_flaky_before_drain()
+ worker = Worker(registry, repo, worker_id="worker-a", max_delivery_attempts=5)
+
+ # Attempt 1: the node raises a retryable exception before ever
+ # calling drain_steering() -- the run must go back to pending,
+ # and E must remain fully recoverable, never superseded.
+ claimed1 = await worker.run_once()
+ self.assertTrue(claimed1)
+ self.assertEqual(len(attempts), 1)
+
+ after_attempt_1 = await repo.get_run("acme", run.id)
+ assert after_attempt_1 is not None
+ status_1 = (
+ after_attempt_1.status.value
+ if hasattr(after_attempt_1.status, "value")
+ else after_attempt_1.status
+ )
+ self.assertEqual(status_1, "pending")
+
+ still_pending = await repo.list_pending_steering("acme", run.id)
+ self.assertEqual([event.id for event in still_pending], [accepted.id])
+ all_steering = await repo.list_steering("acme", run.id)
+ self.assertEqual(len(all_steering), 1)
+ self.assertIn(all_steering[0].status, ("pending", "delivered"))
+
+ events_after_1 = await repo.list_events("acme", run.id)
+ self.assertFalse(
+ any(event.kind == "run.steer.superseded" for event in events_after_1)
+ )
+
+ # Attempt 2: a fresh delivery attempt re-ingests E via the
+ # ordinary claim -> _sync_steering_in path and drains it.
+ claimed2 = await worker.run_once()
+ self.assertTrue(claimed2)
+ self.assertEqual(len(attempts), 2)
+ self.assertEqual(drained_log, [["hello"]])
+
+ final = await repo.get_run("acme", run.id)
+ assert final is not None
+ final_status = (
+ final.status.value if hasattr(final.status, "value") else final.status
+ )
+ self.assertEqual(final_status, "succeeded")
+
+ final_steering = await repo.list_steering("acme", run.id)
+ self.assertEqual(len(final_steering), 1)
+ self.assertEqual(final_steering[0].status, "consumed")
+
+ asyncio.run(scenario())
+
+
+def make_registry_flaky_before_drain() -> tuple[GraphRegistry, list[int], list[list[str]]]:
+ """A single-node graph whose node raises a retryable ``ConnectionError``
+ on its first *delivery* attempt, before ever calling
+ ``drain_steering()``, and succeeds (draining whatever is pending) on
+ the second. No node-level ``RetryPolicy`` is attached -- the exception
+ is meant to propagate all the way out of ``graph.astream`` into the
+ Worker's own delivery-retry handling, not be swallowed by a
+ node-internal retry.
+ """
+
+ attempts: list[int] = []
+ drained_log: list[list[str]] = []
+
+ builder = StateGraph(State)
+
+ async def flaky(state: State, runtime: Runtime[Any]) -> dict[str, Any]:
+ attempts.append(1)
+ if len(attempts) < 2:
+ raise ConnectionError("transient, before draining anything")
+ drained = [event.payload["message"] for event in runtime.drain_steering()]
+ drained_log.append(drained)
+ return {"ticks": state.get("ticks", 0) + 1}
+
+ builder.add_node("flaky", flaky)
+ builder.add_edge(START, "flaky")
+ builder.add_edge("flaky", END)
+ return GraphRegistry({"flaky_before_drain": builder.compile()}), attempts, drained_log
+
+
+class InMemoryRepositoryRound9BranchCoverageTests(unittest.TestCase):
+ """Direct unit coverage of the InMemory-reachable branches added/kept
+ across issue #16 PR #17 review round 9: the restored unfenced
+ ``retry_run()``, ``close_steering()``'s already-closed no-op, the
+ standalone ``supersede_pending_steering_if_owned()`` (both its
+ nothing-to-supersede and something-to-supersede paths), and
+ ``finalize_run_with_steering_disposition_if_owned(supersede_pending=False)``."""
+
+ def test_retry_run_unfenced_admin_path(self) -> None:
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+ await repo.claim_run("worker-a", lease_seconds=30)
+ updated = await repo.retry_run("acme", run.id, error={"code": "x"})
+ self.assertEqual(updated.status, "pending")
+ self.assertIsNone(updated.lease_owner)
+
+ asyncio.run(scenario())
+
+ def test_close_steering_is_a_no_op_when_already_closed(self) -> None:
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+ claimed = await repo.claim_run("worker-a", lease_seconds=30)
+ assert claimed is not None
+ first = await repo.close_steering("acme", run.id, "worker-a", claimed.attempt)
+ self.assertTrue(first)
+ second = await repo.close_steering("acme", run.id, "worker-a", claimed.attempt)
+ self.assertTrue(second)
+
+ asyncio.run(scenario())
+
+ def test_supersede_pending_steering_if_owned_standalone_no_pending(self) -> None:
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+ claimed = await repo.claim_run("worker-a", lease_seconds=30)
+ assert claimed is not None
+ result = await repo.supersede_pending_steering_if_owned(
+ "acme", run.id, "worker-a", claimed.attempt
+ )
+ self.assertEqual(result, [])
+
+ asyncio.run(scenario())
+
+ def test_supersede_pending_steering_if_owned_standalone_supersedes(self) -> None:
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+ claimed = await repo.claim_run("worker-a", lease_seconds=30)
+ assert claimed is not None
+ await repo.submit_steering("acme", run.id, kind="user_input", payload={"m": "x"})
+ result = await repo.supersede_pending_steering_if_owned(
+ "acme", run.id, "worker-a", claimed.attempt
+ )
+ self.assertEqual(len(result), 1)
+ self.assertEqual(result[0].kind, "run.steer.superseded")
+ steering = await repo.list_steering("acme", run.id)
+ self.assertEqual(steering[0].status, "superseded")
+
+ asyncio.run(scenario())
+
+ def test_finalize_run_with_steering_disposition_can_skip_supersede(self) -> None:
+ async def scenario() -> None:
+ repo = InMemoryRepository()
+ assistant = await repo.create_assistant(
+ "acme", _assistant_create(), graph_version="1"
+ )
+ run = await repo.create_run("acme", None, assistant, _run_create(assistant.id))
+ claimed = await repo.claim_run("worker-a", lease_seconds=30)
+ assert claimed is not None
+ await repo.submit_steering("acme", run.id, kind="user_input", payload={"m": "x"})
+ result = await repo.finalize_run_with_steering_disposition_if_owned(
+ "acme",
+ run.id,
+ "worker-a",
+ claimed.attempt,
+ RunStatus.PAUSED,
+ output=None,
+ error=None,
+ supersede_pending=False,
+ supersede_reason="n/a",
+ )
+ self.assertIsNotNone(result)
+ updated, superseded_events = result
+ self.assertEqual(updated.status, "paused")
+ self.assertEqual(superseded_events, [])
+ steering = await repo.list_steering("acme", run.id)
+ self.assertEqual(steering[0].status, "pending")
+
+ asyncio.run(scenario())
+
+
+def _assistant_create_named(graph_id: str):
+ from lingxigraph.server.models import AssistantCreate
+
+ return AssistantCreate(graph_id=graph_id)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/test_steering_module_edge_cases.py b/tests/test_steering_module_edge_cases.py
new file mode 100644
index 0000000..3e3a0fc
--- /dev/null
+++ b/tests/test_steering_module_edge_cases.py
@@ -0,0 +1,51 @@
+"""Direct unit tests for lingxigraph.steering module-level edge cases."""
+
+import unittest
+
+from lingxigraph.steering import (
+ SteeringChannel,
+ SteeringPayloadTooLarge,
+ validate_steering_payload,
+)
+
+
+class ValidateSteeringPayloadTests(unittest.TestCase):
+ def test_rejects_unserializable_payload(self) -> None:
+ class Unserializable:
+ pass
+
+ with self.assertRaises(SteeringPayloadTooLarge):
+ validate_steering_payload({"bad": Unserializable()})
+
+
+class SteeringChannelDrainCallbackTests(unittest.TestCase):
+ def test_on_drain_exception_is_swallowed(self) -> None:
+ channel = SteeringChannel(run_id="run-1")
+
+ def boom(_drained: object) -> None:
+ raise RuntimeError("observability sink is down")
+
+ channel.on_drain = boom
+ channel.submit(kind="note", payload={"a": 1})
+ # Must not raise even though on_drain blows up.
+ drained = channel.drain()
+ self.assertEqual(len(drained), 1)
+
+ def test_ack_consumed_with_empty_ids_is_a_no_op(self) -> None:
+ channel = SteeringChannel(run_id="run-1")
+ channel.submit(kind="note", payload={"a": 1})
+ channel.drain()
+ before = channel.peek_consumed()
+ channel.ack_consumed([])
+ after = channel.peek_consumed()
+ self.assertEqual(before, after)
+
+ def test_ack_consumed_is_a_no_op_when_log_already_empty(self) -> None:
+ channel = SteeringChannel(run_id="run-1")
+ # Nothing submitted/drained -- consumed log starts empty.
+ channel.ack_consumed(["nonexistent-id"]) # must not raise
+ self.assertEqual(channel.peek_consumed(), ())
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/test_tools_edge_cases.py b/tests/test_tools_edge_cases.py
new file mode 100644
index 0000000..154f21b
--- /dev/null
+++ b/tests/test_tools_edge_cases.py
@@ -0,0 +1,102 @@
+"""Direct unit tests for lingxigraph.tools pure-function edge branches."""
+
+import asyncio
+import unittest
+
+from lingxigraph import AIMessage, Command, InvalidUpdateError, ToolCall, tool
+from lingxigraph.tools import ToolNode, _make_spec, validate_json_schema
+
+
+def add(a: int, b: int = 1) -> int:
+ """Add two numbers."""
+ return a + b
+
+
+def var_args_tool(a: int, *args: int, **kwargs: int) -> int:
+ """A tool with *args/**kwargs, which must be skipped from the schema."""
+ return a
+
+
+class ToolDecoratorEdgeCaseTests(unittest.TestCase):
+ def test_tool_decorator_with_explicit_string_name_positional(self) -> None:
+ spec = tool("custom_name")(add)
+ self.assertEqual(spec.name, "custom_name")
+
+ def test_make_spec_skips_var_positional_and_var_keyword_parameters(self) -> None:
+ spec = _make_spec(
+ var_args_tool, name=None, return_direct=False
+ )
+ self.assertEqual(set(spec.parameters["properties"]), {"a"})
+
+
+class ValidateJsonSchemaEdgeCaseTests(unittest.TestCase):
+ def test_enum_violation_raises(self) -> None:
+ with self.assertRaises(ValueError):
+ validate_json_schema("z", {"enum": ["a", "b"]})
+
+ def test_array_items_are_validated_recursively(self) -> None:
+ schema = {"type": "array", "items": {"type": "integer"}}
+ with self.assertRaises(ValueError):
+ validate_json_schema([1, "not-an-int"], schema)
+ validate_json_schema([1, 2, 3], schema) # must not raise
+
+ def test_object_missing_required_argument_raises(self) -> None:
+ schema = {"type": "object", "properties": {"x": {"type": "integer"}}, "required": ["x"]}
+ with self.assertRaises(ValueError):
+ validate_json_schema({}, schema)
+
+ def test_object_rejects_unknown_argument_when_additional_properties_false(self) -> None:
+ schema = {
+ "type": "object",
+ "properties": {"x": {"type": "integer"}},
+ "additionalProperties": False,
+ }
+ with self.assertRaises(ValueError):
+ validate_json_schema({"x": 1, "y": 2}, schema)
+
+
+class ToolNodeConstructionEdgeCaseTests(unittest.TestCase):
+ def test_duplicate_tool_names_are_rejected(self) -> None:
+ spec_a = tool("dup")(add)
+ spec_b = tool("dup")(add)
+ with self.assertRaises(ValueError):
+ ToolNode([spec_a, spec_b])
+
+ def test_non_positive_read_only_settings_are_rejected(self) -> None:
+ spec = tool(add)
+ with self.assertRaises(ValueError):
+ ToolNode([spec], read_only_concurrency=0)
+ with self.assertRaises(ValueError):
+ ToolNode([spec], read_only_batch_size=0)
+
+ def test_call_without_a_trailing_ai_message_raises(self) -> None:
+ node = ToolNode([tool(add)])
+
+ async def scenario() -> None:
+ with self.assertRaises(InvalidUpdateError):
+ await node({"messages": []})
+
+ asyncio.run(scenario())
+
+ def test_command_returning_tool_must_be_the_only_call(self) -> None:
+ def make_command() -> Command:
+ return Command(goto="next")
+
+ node = ToolNode([tool(add), tool(make_command)])
+
+ async def scenario() -> None:
+ message = AIMessage(
+ content="",
+ tool_calls=[
+ ToolCall(id="1", name="add", args={"a": 1, "b": 2}),
+ ToolCall(id="2", name="make_command", args={}),
+ ],
+ )
+ with self.assertRaises(InvalidUpdateError):
+ await node({"messages": [message]})
+
+ asyncio.run(scenario())
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/test_worker_edge_cases.py b/tests/test_worker_edge_cases.py
new file mode 100644
index 0000000..e1ecc33
--- /dev/null
+++ b/tests/test_worker_edge_cases.py
@@ -0,0 +1,94 @@
+"""Direct unit tests for Worker lifecycle properties and small edge branches."""
+
+import asyncio
+import unittest
+
+from lingxigraph.server.registry import GraphRegistry
+from lingxigraph.server.repository import InMemoryRepository
+from lingxigraph.server.worker import Worker
+
+
+class WorkerLifecyclePropertyTests(unittest.TestCase):
+ def test_draining_ready_and_live_properties(self) -> None:
+ registry = GraphRegistry({})
+ repository = InMemoryRepository()
+ worker = Worker(registry, repository)
+
+ self.assertFalse(worker.draining)
+ self.assertTrue(worker.ready)
+ self.assertTrue(worker.live)
+
+ worker.stop()
+
+ self.assertTrue(worker.draining)
+ self.assertFalse(worker.ready)
+
+ def test_run_once_returns_false_immediately_while_draining(self) -> None:
+ async def scenario() -> None:
+ registry = GraphRegistry({})
+ repository = InMemoryRepository()
+ worker = Worker(registry, repository)
+ worker.stop()
+ claimed = await worker.run_once()
+ self.assertFalse(claimed)
+
+ asyncio.run(scenario())
+
+ def test_drain_returns_true_when_already_idle(self) -> None:
+ async def scenario() -> None:
+ registry = GraphRegistry({})
+ repository = InMemoryRepository()
+ worker = Worker(registry, repository)
+ # No active _execute in flight -- _idle is already set.
+ finished = await worker.drain(timeout=1.0)
+ self.assertTrue(finished)
+ self.assertTrue(worker.draining)
+
+ asyncio.run(scenario())
+
+
+class IsRetryableClassificationTests(unittest.TestCase):
+ """Direct unit coverage of ``Worker._is_retryable``'s branches: it
+ decides whether a node exception should route into the ordinary
+ delivery-retry path or fail the run outright."""
+
+ def test_validation_errors_are_never_retryable(self) -> None:
+ from lingxigraph.errors import GraphValidationError, InvalidUpdateError
+
+ self.assertFalse(Worker._is_retryable(GraphValidationError("bad graph")))
+ self.assertFalse(Worker._is_retryable(InvalidUpdateError("bad update")))
+ self.assertFalse(Worker._is_retryable(KeyError("missing")))
+ self.assertFalse(Worker._is_retryable(ValueError("bad value")))
+
+ def test_connection_and_persistence_errors_are_retryable(self) -> None:
+ from lingxigraph.errors import PersistenceError
+
+ self.assertTrue(Worker._is_retryable(ConnectionError("dropped")))
+ self.assertTrue(Worker._is_retryable(TimeoutError("timed out")))
+ self.assertTrue(Worker._is_retryable(PersistenceError("db hiccup")))
+
+ def test_plain_runtime_error_is_retryable(self) -> None:
+ self.assertTrue(Worker._is_retryable(RuntimeError("transient")))
+
+ def test_httpx_module_exceptions_are_retryable(self) -> None:
+ class FakeHttpxError(Exception):
+ pass
+
+ FakeHttpxError.__module__ = "httpx._exceptions"
+ self.assertTrue(Worker._is_retryable(FakeHttpxError("network blip")))
+
+ def test_name_based_transient_markers_are_retryable(self) -> None:
+ class UpstreamTemporaryFailure(Exception):
+ pass
+
+ self.assertTrue(Worker._is_retryable(UpstreamTemporaryFailure("try again")))
+
+ def test_unrelated_exception_is_not_retryable(self) -> None:
+ class SomeDomainError(Exception):
+ pass
+
+ self.assertFalse(Worker._is_retryable(SomeDomainError("permanent")))
+
+
+if __name__ == "__main__":
+ unittest.main()