Repository navigation
[BUG] A suspend with an unknown outcome locks the session forever: answers, cancel and delete are all refused #3312
Description
Bug description
If the controller's suspend of an idle or input-required session fails in a way whose outcome is unknown (for example a timeout), the session can no longer be used, cancelled or deleted, and nothing in kagent ever recovers it.
This is intentional in the code:
// go/core/internal/service/session/quiescence.go, quiesceIdleSession
// No timeout-based takeover: the Substrate request may still complete.
// Keep admission closed until its outcome can be safely reconciled.// go/core/internal/database/session_quiescence.go
// … Uncertain issued work is never reassigned after a timeout.But nothing does the reconciling. The claimed session_task_event row keeps quiescence_pending = TRUE with a quiescence_executor_id forever, and requireSettledRuntime rejects every operation on the session while that row exists.
Steps to reproduce
- Make the quiescence
SuspendActorcall fail with an unknown outcome. The easiest way is the 30s call timeout with a slow snapshot upload (see [BUG] Session suspend times out after 30s when the snapshot upload is slow (hard-coded Substrate CallTimeout) #3311). - Answer the pending ask-user question, from the UI or over A2A.
- Cancel the task.
- Delete the session.
Expected
The outcome gets reconciled. For example, a later pass reads the Actor's actual state from Substrate (GetActor) and then either records the snapshot, retries the suspend, or releases the claim. At the very least, delete should be able to override it, since deleting tears down the Actor whatever state it is in.
Actual
- Answering:
input was not accepted; retry after the session becomes available. It never becomes available. - Cancel and delete:
begin Session operation: runtime cleanup or lifecycle work is not yet settled: precondition failed. - On the Substrate side the Actor stays in
suspending(the atelet checkpoint was cancelled).
The only way out we found was editing the database by hand, marking the claimed event settled so that delete could proceed:
UPDATE <schema>.session_task_event
SET quiescence_pending = FALSE
WHERE sequence = <version from the "runtime boundary outcome unknown" log>
AND quiescence_executor_id = '<executor id>'
AND quiescence_pending;That is not something users should have to do. It also throws away the paused conversation, including the pending human-in-the-loop request.
Environment
kagent v1.0.0-alpha17, Substrate v0.6.0-alpha3, Claude harness, gVisor.
Activity
AlbertDoesProgramming commentedon Oct 23, 2026
+1, we hit the same lock on alpha16 (Substrate 0.6.0-alpha3) on kind, from a different trigger. The object store went away for about 40 s while a Session was suspending after its turn. The suspend failed with
CallAteletSuspend … context canceled. From then on the Session refused every turn and every delete with "lifecycle work is not yet settled". Recreating the worker pod didn't free it.So it isn't only the slow-upload case from #3311: any store outage during a suspend can do it. Idle Sessions that weren't mid-suspend recovered fine once the store was back. From a read of alpha17 the quiescence path looks unchanged, so I'd expect the same there.
Reconciling from
GetActor, and letting delete override, both sound right to us. Fwiw, a metric or condition for "Session held by an unsettled claim" would help operators alert on it before users notice. We're running kagent on AKS and happy to help test a fix.yyewolf commentedon Oct 23, 2026
Same here, I stumbled on it with a big upload to the S3 that took more than 30s and got locked the same way again
AlbertDoesProgramming commentedon Nov 3, 2026
Re-tested on v1.0.0-rc1 (Substrate v1.0.0-rc1) on kind. A ~50 s object-store outage still gets us here. Turns during the outage work off the node-local copy, then both post-turn suspends fail (
CallAteletSuspend … context canceled/DeadlineExceeded) and kagent logsruntime boundary outcome unknown.After that the Session refuses turns (
-32004 … retry after the session becomes available) and everyDeleteSession(runtime cleanup or lifecycle work is not yet settled), well past the store coming back. Deleting the worker pod moves the actors toCRASHED, but kagent still shows the SessionREADYand keeps refusing. With public suspend/resume gone in rc1, a database edit is the only way out for us. Would a reconcile from the actor's real state (CRASHED → release the claim) be the direction you'd want? Happy to help on it or review the #3255 stack if that's where it's heading.AlbertDoesProgramming commentedon Nov 9, 2026
Tried rc3's force delete (#3530) on this, but it doesn't reach it: after the same store outage,
DeleteSessionwith force comes backAborted("active checkpoint or idle lifecycle operation"), also after restarting the worker and the controller. Looks likeadmitForcedSessionDeletionrefuses whenever a claimed idle suspend is pending (go/core/internal/database/session_deletion.go:60-79), and nothing ever releases that claim. So force works on a Session mid-turn but not on this one.AlbertDoesProgramming commentedon Nov 9, 2026
On 1.0.0,
RevertSession(#3395) does get us out: once a worker restart crashes the actor, a revert brings the Session back to its last snapshot and the next turn and delete work. Before the restart the actor sits inSUSPENDINGand the revert is a no-op. So it's recoverable now, though it takes a worker restart and drops the failed turn. A revert that worked on aSUSPENDINGactor stuck past its call timeout would cover the rest.