[Turn Steal 5/N]: feat: rebuild, offload, and steal a paused turn by chiragjn · Pull Request #1070 · truefoundry/trueforge

chiragjn · 2026-10-15T04:27:45Z

Summary

Rebuild a paused turn from its own snapshot on the owning replica, replaying unconsumed inbox events before the new batch is queued. The drain runs in the background, the same way a non-streaming create does.
Offload a paused run after a keepalive (default 30s) without a terminal write. Shutdown offloads paused runs and still aborts running ones.
Steal only a paused turn whose owner heartbeat is missing, via the existing conditional executor claim. Publish-zero, timeout, peer 412, and transport errors retry instead of stealing. Redis log repair (backfill from the turn record) is not in this change; a new producer only reseeds its sequence from the existing stream tail.

Test plan

Pause a turn, wait out the keepalive, and confirm the row stays paused while the local run is gone.
Send an event to that paused turn on the same owner and confirm it rebuilds, replays the inbox, then applies the new event.
From another replica, send an event while the owner heartbeat exists and confirm the request is forwarded, not stolen.
Remove the owner heartbeat and confirm a paused turn is claimed once; a running turn is not stolen.
Subscribe to the turn after rebuild and confirm events continue from the existing stream tail.

What is still open, and should be called out in review rather than treated as surprise bugs:

P9 / R10. After a mid-wait heartbeat loss, the code reloads the turn and then claims. It does not read the heartbeat key again before claimTurnExecutor.
Section 2b. A missing Redis stream stays at sequence 1. Subscribe still 412s and does not repair the log.
Section 4. The cross-replica scenarios are not tested in this branch.
Keepalive send. A POST while the run is still in memory does not rebuild. New headers on that POST are not applied until offload.
Offloaded predecessor. createTurn freezes it in the database only, so an open subscribe does not see turn.done.
Shutdown offloads paused runs without the per-turn lock. The plan already records that.
Inbox insert is fenced on non-terminal status only, not on active_executor_id. The plan’s P5 text says the owner fence includes inbox insert. That gap is on ha/emit-pause, not introduced here.

Note

High Risk
Changes distributed turn ownership, event delivery, and resume/offload semantics for paused agent turns—core session execution paths with Redis peering and new steal/retry behavior.

Overview
Distributed paused turns can be rebuilt on the owning executor, offloaded from memory after a keepalive, and resumed or stolen when events arrive—without incorrectly terminating the turn in the store.

In trueforge-core, SessionHandle.rebuildTurn() reconstructs a paused turn from its snapshot (with inbox replay via TurnHandle.replayUnconsumedInbox()), TurnHandle gains a resume path that skips turn.created and treats turn-offload aborts as non-terminal, and AgentThreadOrchestrator.execute accepts resume so execution starts from the paused state.

On the server, ActiveTurnRegistry arms TURN_PAUSED_KEEPALIVE_SECONDS timers to offload paused runs; deliverTurnEvents (plus Redis turns/send-events) routes POST /events locally, forwards to the owner, rebuilds when this replica owns but has no in-memory run, or steals only when the owner heartbeat is missing. Peer publish/timeouts/412 map to 503 retry, not steal. Create-turn now cancels the previous turn locally or via peer sessions/cancel (abortAndWait). Cancel logic moves to runtime/cancelTurn.ts.

API/SDK: create turn events documents 503 (ServiceUnavailableError). Store wiring is refactored through resolvers.ts and direct sandboxProviderStore / webSearchProviderStore injection for HTTP and peer paths.

^{Reviewed by Cursor Bugbot for commit 6c35586. Bugbot is set up for automated code reviews on this repo. Configure here.}

Resume skips turn.created and replays the unconsumed inbox. A new producer reseeds its sequence from the existing stream tail. Co-authored-by: Cursor <cursoragent@cursor.com>

Keep a paused run for a keepalive, then stop it without a terminal write. A later send rebuilds on the owner, or claims the turn only when the owner's heartbeat is missing. Co-authored-by: Cursor <cursoragent@cursor.com>

changeset-bot · 2026-10-15T04:27:52Z

🦋 Changeset detected

Latest commit: 21ac55a

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
@truefoundry/trueforge	Patch
@truefoundry/trueforge-core	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

cursor

Stale Bugbot comment from a previous run.

cursor · 2026-10-15T04:33:35Z

+'@truefoundry/trueforge-core': patch
+---
+
+Rebuild a paused turn on its owner, offload it after a keepalive, and steal it only when the owner's heartbeat is gone.


Changeset added on stacked PR

Low Severity

This stacked 5/N change adds a .changeset file. Changesets belong only on a merge to main; stacked or non-main bases should omit them so version bumps stay on the release path.

^{Triggered by learned rule: Prereleases use rc tags only}

^{Reviewed by Cursor Bugbot for commit c3f9816. Configure here.}

A rebuilt turn starts paused so the first runnable batch persists running. Peer resume applies inline skills from the forwarded request. Creating the next turn aborts the previous run so its drain writes turn.done. Co-authored-by: Cursor <cursoragent@cursor.com>

cursor

Stale Bugbot comment from a previous run.

into cj-turn-resume

cursor

Stale Bugbot comment from a previous run.

A peer rebuild has no caller token, and the admin skill resolver is not a full skill store, so wrapping x-tfg-skills there needed a stub. Leave skills on that resolver until the header wrap is shared with models and MCP. Co-authored-by: Cursor <cursoragent@cursor.com>

Thirty seconds offloads a paused run before the next user event typically arrives. Co-authored-by: Cursor <cursoragent@cursor.com>

into cj-turn-resume Keep the closed-set inline skill store, and resolve peer and event-POST stores through the same header-aware resolvers. Co-authored-by: Cursor <cursoragent@cursor.com>

The owner was rebuilding model and MCP stores as the session creator with no credential. Carry the caller's context on the peer body, and let schedule dispatch pass an explicit null credential. Co-authored-by: Cursor <cursoragent@cursor.com>

cursor

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

There are 4 total unresolved issues (including 2 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 64e7d3d. Configure here.}

chiragjn · 2026-10-19T17:40:20Z

turns.ts starts the next turn and has to cancel the previous one. Importing that from sessions.ts was a cycle, because sessions.ts already imports startTurnInProcess from turns.ts. The cancel route, the peer body schema, cancelSessionTurn, cancelSessionTurnPeerHandler, and freezeTurnIgnoringMissing all live in the new file.

cancelTurnOnThisExecutor was not kept as its own function. The peer handler now calls activeTurns.abortAndWait, and the local cancel still calls activeTurns.cancelIfRunning.

TurnHandle.state is a prototype getter, so paused runs never offloaded and shutdown cancelled them. A rebuilt turn stays paused without a new pause event, and a live send must reset the timer so it cannot abort the run before running is persisted. Co-authored-by: Cursor <cursoragent@cursor.com>

heerambavi1998 · 2026-10-19T17:27:13Z

+export const NO_RESPONDER_PUBLISH_DETAIL = 'No executor received the published request';
+
 /** The target executor's heartbeat is missing or nobody is subscribed (map to HTTP 412). */
 export class NoResponderError extends Error {


wondering why we need to define these error types, doesnt redis have its error types?

heerambavi1998 · 2026-10-19T23:32:14Z

+}
+
+/** Freeze a non-terminal turn; missing turns are a no-op (already gone). */
+async function freezeTurnIgnoringMissing(


why is this needed? freezeTurn should itself ignore missing? was this not happening already?

remove util

Creating the next turn cancels a previous run that is not in this process, including a schedule run, so that path needs the session store. The turns router always registers turns/send-events. Co-authored-by: Cursor <cursoragent@cursor.com>

chiragjn · 2026-10-20T14:43:34Z

+        throw new TurnNotRunningError(input.turn_id, turn.state);
+      }
+
+      const tracing = input.resolver.createTracing();


bunch of logic here is same as createTurn, I will extract and make it common in later commits

chiragjn · 2026-10-20T14:45:45Z

+    for (const event of rows) {
+      try {
+        for await (const batch of this.orchestrator.send([event])) {
+          void batch;
+        }
+      } catch (error) {
+        if (error instanceof InvalidAgentSendInputError) {
+          continue;
+        }
+        throw error;
+      }
+    }


Do we need to validate before sending anything?
should it be send all or nothing?

chiragjn · 2026-10-20T14:51:08Z

+/**
+ * `state` is a getter on TurnHandle's prototype, so it is not an own property.
+ * A test double may omit it; that run is not treated as paused.
+ */
+function isPausedRun(run: ActiveTurnRun | undefined): run is ActiveTurnRun {
+  if (!run || !('state' in run.turn)) {


why do we need to check 'state' in run.turn ?

chiragjn · 2026-10-20T14:51:52Z

+   * under the turn lock: the generator stops without a terminal write.
+   * A later send on this owner rebuilds. `0` disables the timer.
+   */
+  armPausedKeepalive(input: { sessionId: string; turnId: string }): void {


nit: change name from arm to start?

chiragjn · 2026-10-20T14:56:36Z

+  requestHeaders?: Record<string, string> | undefined;
+  deps: BeginTurnExecutionDeps;
+}): Promise<{ turn: TurnHandle; startDrain: () => void }> {
+  assertGatewayMetadataRequestHeaders(input.requestHeaders);


check this once, why assert?

chiragjn · 2026-10-20T14:57:46Z

+    // Counter only. A missing stream stays at 1; log repair from the TurnRecord is later.
+    await turnEventStream.resumeProducer({ streamTTLSeconds: configuration.TURN_STREAM_TTL_SECONDS });


restoring the stream will be another PR

heerambavi1998

Server execution timeout needs to be increased
add a metric on # of client retries for peer failures

heerambavi1998 · 2026-10-20T15:05:18Z

+      });
+      if (!turn) {
+        throw new TurnNotFoundError(input.turn_id);
+      }


race condition possible here because executor ID changes.

might not need special handling because it can happen at any point during rebuildTurn / send / execute

heerambavi1998 · 2026-10-20T15:12:05Z

+      return handle;
+    } catch (error) {
+      await input.resolver.close().catch((closeError: unknown) => {
+        input.resolver.logger.warn('TurnResourceResolver.close() failed after rebuildTurn() error', {


logger inside resolver?

heerambavi1998 · 2026-10-20T15:14:08Z

  private readonly resolver: ITurnResourceResolver<TTurnCustom> | undefined;
  private readonly abortController: AbortController;
  private streamStarted = false;
+  /** Resume skips turn.created. Inbox replay happens before stream() is pulled. */


irrelevant here?

Whether to emit turn.created or not

heerambavi1998 · 2026-10-20T15:14:38Z

    orchestrator?: AgentThreadOrchestrator | undefined;
    resolver?: ITurnResourceResolver<TTurnCustom> | undefined;
    abortController?: AbortController | undefined;
+    resume?: boolean | undefined;


non optional?

might be better to separate out getTurn ?

heerambavi1998 · 2026-10-20T15:17:50Z

    if (!this.orchestrator) {
-      throw new Error(`TurnHandle.${method}() is only available on a live turn from SessionHandle.createTurn()`);
+      throw new Error(
+        `TurnHandle.send() is only available on a live turn. Perhaps this turn handle was created for read only actions?`,
+      );


lets separate it out and keep orchestrator mandatory?

heerambavi1998 · 2026-10-20T16:35:42Z

+  message: z.string(),
+});
+
+function isJsonValue(value: unknown): value is JSONValue {


same move to utils

heerambavi1998 · 2026-10-20T16:36:14Z

+
+const MAX_OWNERSHIP_ATTEMPTS = 4;
+
+const PeerRequestContextSchema = z.object({


we should keep all peer request schemas + API routes together

heerambavi1998 · 2026-10-20T16:38:01Z

+    if (!ownedByThisExecutor) {
+      await input.activeTurns.dropLocalTurn({ sessionId, turnId: input.turnId });
+    }


lets add the scenario where this can happen

heerambavi1998 · 2026-10-20T16:43:10Z

+   */
+  resume: () => Promise<{ turn: TurnHandle; startDrain: () => void }>;
+  /** Peer delivery is owner-local. Only the replica that received the client request may steal. */
+  allowSteal: boolean;


this should not be the responsibility of the caller?

sendEvents > check I am owner >
if yes, "rebuild optionally and run".
if not, peer. if no peer, steal with ownership

peer request > check I am owner >
if yes, "rebuild optionally and run".
if not, fail. if no peer, steal with ownership

heerambavi1998 · 2026-10-20T16:48:02Z

+
+type ForwardResult = TurnDeliveryResult | { type: 'peer'; peerOutcome: PeerOutcome };
+
+async function forwardTurnEvents(input: {


this can be in the actual peer request API

stealing can be one function, rebuild and run is another.

stealing is only called when actual user request is received. Peered request itself never steals

chiragjn and others added 2 commits October 15, 2026 09:32

feat: rebuild a paused turn from its own snapshot

00516af

Resume skips turn.created and replays the unconsumed inbox. A new producer reseeds its sequence from the existing stream tail. Co-authored-by: Cursor <cursoragent@cursor.com>

feat: offload a paused turn and steal it when the owner is gone

cd79652

Keep a paused run for a keepalive, then stop it without a terminal write. A later send rebuilds on the owner, or claims the turn only when the owner's heartbeat is missing. Co-authored-by: Cursor <cursoragent@cursor.com>

chiragjn requested review from debajyoti-truefoundry, heerambavi1998 and thesujai as code owners October 15, 2026 04:27

chiragjn changed the title ~~feat: rebuild, offload, and steal a paused turn~~ [Turn Steal 5/N]: feat: rebuild, offload, and steal a paused turn Oct 15, 2026

Regenerate OpenAPI document and SDKs

c3f9816

cursor Bot reviewed Oct 15, 2026

View reviewed changes

Comment thread packages/trueforge/src/apis/turns.ts

Comment thread packages/trueforge/src/main.ts Outdated

Merge branch 'ha/emit-pause' of https://github.com/truefoundry/trueforge

1839012

into cj-turn-resume

cursor Bot reviewed Oct 15, 2026

View reviewed changes

Comment thread packages/trueforge/src/runtime/activeTurns.ts

chiragjn and others added 4 commits October 16, 2026 09:49

feat: keep a paused turn in memory for five minutes

5a4ce8d

Thirty seconds offloads a paused run before the next user event typically arrives. Co-authored-by: Cursor <cursoragent@cursor.com>

Merge branch 'ha/emit-pause' of https://github.com/truefoundry/trueforge

a8183b2

into cj-turn-resume Keep the closed-set inline skill store, and resolve peer and event-POST stores through the same header-aware resolvers. Co-authored-by: Cursor <cursoragent@cursor.com>

cursor Bot reviewed Oct 19, 2026

View reviewed changes

Comment thread packages/trueforge/src/apis/turns.ts

Comment thread packages/trueforge/src/apis/turns.ts

chiragjn commented Oct 19, 2026

View reviewed changes

Comment thread packages/trueforge/src/main.ts Outdated

heerambavi1998 reviewed Oct 19, 2026

View reviewed changes

chiragjn and others added 2 commits October 20, 2026 09:52

Refactor resolvers

21ac55a

chiragjn commented Oct 20, 2026

View reviewed changes

Minor rename in inMemory store

6c35586

chiragjn commented Oct 20, 2026

View reviewed changes

heerambavi1998 reviewed Oct 20, 2026

View reviewed changes

		// Counter only. A missing stream stays at 1; log repair from the TurnRecord is later.
		await turnEventStream.resumeProducer({ streamTTLSeconds: configuration.TURN_STREAM_TTL_SECONDS });


		const MAX_OWNERSHIP_ATTEMPTS = 4;

		const PeerRequestContextSchema = z.object({


		type ForwardResult = TurnDeliveryResult \| { type: 'peer'; peerOutcome: PeerOutcome };

		async function forwardTurnEvents(input: {

Conversation

chiragjn commented Oct 15, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

changeset-bot Bot commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cursor Bot Oct 15, 2026

Choose a reason for hiding this comment

Changeset added on stacked PR

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chiragjn Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

heerambavi1998 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chiragjn commented Oct 15, 2026 •

edited by cursor Bot

Loading

changeset-bot Bot commented Oct 15, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

chiragjn Oct 20, 2026 •

edited

Loading