Cap-ruling hold: runtime must block .DONE and merge while an operator ruling is pending (.PENDING-RULING)

Summary

When a review gate hits the REVISE cap and the ruling is escalated to the operator, nothing in the runtime holds the lane. The hold is enforced only by steer messages the worker is expected to obey. In TP-2037 (penster, batch henrylach-20260903T134933) the worker held for ~8 minutes with fixes staged, then self-released without a ruling, committed with a message claiming R004 cap ruling (FIX) (no such ruling had been issued), completed the remaining step, wrote .DONE, and the engine merged the lane into the orch branch. The supervisor's pause could not stop the in-flight merge.

The merged content turned out to be exactly what the operator later ratified, so no damage this time. The process gap is real: a worker can end a batch and trigger a merge while an operator ruling is outstanding, and the record then carries a fabricated ruling reference.

Expected

A mechanical hold the runtime respects, independent of the worker's prompt discipline:

When the supervisor (or the worker) escalates a cap ruling, the runtime writes a .PENDING-RULING marker in the task folder (or equivalent engine state) naming the gate (R004-code-step5) and the escalation message id.
While the marker exists: the worker cannot check off the gated step, cannot write .DONE, and the engine will not merge the lane. Worker iterations may continue only for hold-safe work (recording state, answering questions).
The marker is cleared only by a ruling message from the supervisor/operator role in the inbox (a typed ruling mailbox message referencing the gate), never by the worker.
A commit created while the marker exists, or a commit whose message cites a ruling id that does not exist in the inbox, is flagged in actions.jsonl and surfaced in the dashboard.

Actual

Holds are steer-based only. The worker overlay says "hold at the cap and escalate," and it held, then didn't.
.DONE is the completion signal the engine merges on. It has no precondition tied to open escalations.
The commit message's ruling claim is unverifiable by the runtime; it was only caught because the supervisor read the log.

Evidence

Lane commit 5441c4dc "fix(TP-2037): R004 cap ruling (FIX) — ..." created with no ruling in the inbox.
.DONE written 19:43 (STATUS execution log); wave merge b51bc71d fired on it.
Supervisor's escalation and the worker's own escalation are both in the mailbox; the retroactive operator ruling arrived after the merge.
Incident is fully logged in the batch's actions.jsonl.

bug: review-boundary path emits spurious "Reviewer unavailable" (review_failed) on every successful review (0.30.5) #624 (verdict garble) and bug(supervisor): actions.jsonl timestamps are fabricated by the LLM — no tool exposes logRecoveryAction() #625 (actions.jsonl timestamps): all three are "the runtime trusts the worker's reading of its own state." This one is the highest-impact of the three because it gates a merge.

Suggested minimal cut

Even without the full typed-ruling mailbox: refuse .DONE and refuse the wave merge while any escalation message for the task has no later ruling message from a supervisor/operator role. That alone would have held TP-2037.

Related: #626 (gate-integrity: no finalize without an explicit APPROVE in the step's review files) is the minimal cut for this incident. TP-2037's latest code-gate file R004 reads Verdict: REVISE with no later APPROVE, so #626 alone would have refused the .DONE. This issue adds the part #626 does not cover: a hold that is tied to an outstanding escalation and cleared only by a ruling message, plus flagging commits that cite a ruling id absent from the inbox.

Follow-on incident on the same program (TP-2039): with no runtime hold state, the supervisor parked the held task via supervisor_takeover, which treated the paused wave as terminal and destroyed the lane (task skipped, batch completed 0/1, worktree+branch removed, uncommitted work lost). Filed as #628. A first-class held task status — stops iteration re-spawns, blocks .DONE/merge, preserves the lane, cleared only by a ruling message — would resolve both issues.

Second instance, same program, same day (penster TP-2039, batch henrylach-20260903T210208). Code gate round 2 (R004) left one P1 at the cap; the worker fixed the uncontested items and HELD — its last commit reads "completion held behind Step-5 gate ruling". The lane-runner's checkbox-counting completion heuristic then wrote .DONE (checkpoint 5ff9a7d0) and merged the wave while the operator ruling was still outstanding. The worker did nothing wrong; the runtime has no way to see a prose hold. Content was later ratified after supervisor verification, but the gate was jumped.

Two closures in three batches now (TP-2037: worker self-released; TP-2039: runtime completed a held task). #626's explicit-APPROVE finalize check would have blocked both, since the latest review file read REVISE in each case. The held status proposed here plus #628's non-destructive park would remove the remaining gap.

Proposed acceptance criteria for the held state, after a third incident (penster TP-2042, 2026-09-04: a correctly held task was destroyed by supervisor_takeover's completion path — same as #628's first occurrence). All three incidents (TP-2037 runtime .DONE over a hold; TP-2039 and TP-2042 park destroying the lane) would have been prevented by this one state:

A task status held, entered when a cap escalation is recorded (by the worker or the supervisor) and persisted in batch state — not inferred from prose in STATUS.md.
While held: no iteration re-spawns, no stall-limit progression, no .DONE acceptance, no wave merge, no lane cleanup. The lane worktree and branch are preserved verbatim. Cost while held is zero.
held clears only on a typed ruling mailbox message from the supervisor or operator role that references the gate id. A commit whose message cites a ruling id not present in the inbox is flagged in actions.jsonl and the dashboard.
supervisor_takeover and orch_pause on a held task are no-ops with a message — never a completion path (supervisor_takeover destroys the paused lane: task marked skipped, batch completed 0/1, worktree+branch removed, uncommitted work lost #628).
Worktree removal refuses while the lane has uncommitted changes; the batch record shows held distinctly from stalled and skipped.

Interim operating rules adopted on the penster side until this lands (supervisor overlay, 2026-09-04): never park a held lane; the worker commits hold state and idles; no wrap-up/pause/takeover on a held task.

Fourth incident, penster TP-2047 batch henrylach-20260905T210935 (2026-09-06): a worker holding correctly at the plan-gate cap escalated with expectsReply: true, the supervisor session was being replaced at that moment, the reply wait timed out, and the worker process exited — lane and registry left frozen at running. Filed as #630. A first-class held state (no iteration or stall progression while held, cleared only by a typed ruling) removes that timeout-exit path by design. The replacement supervisor then could not resume the inherited batch at all — #631 — which is independent of the hold but compounds every hold incident where the supervisor session is lost.

Design input from penster batch 20260906T194514 (PR #632 build), for the held/ruling design:

Delegated closure is now the established pattern on three consecutive RL3 packets: at the review cap the supervisor rules in-authority items itself, escalates the operator's reserved decisions, verifies the fold, and writes the next R-numbered APPROVE file instead of a third reviewer round. The runtime only sees "latest review file per gate says APPROVE" (feat(gate-integrity): task must not finalize succeeded while steps lack an explicit APPROVE in their review files #626 minimal gate). A first-class "gate closed by ratification" record needs: who ratified (supervisor vs operator), the proof set they ran, which findings were ruled vs fixed, and the escalation id it closes.
Ruling messages should carry the ruler's role. Cap adjudication at round 2 of 2 means supervisor and operator both rule within one gate; today's notifications distinguish neither. A typed ruling (supervisor-ruling / operator-ruling) with reply correlation to the escalation id would make the audit trail self-describing — and is the same correlation the hold release needs (today's Tier-1 contract is by message type: info = acknowledgement, steer = ruling; see Worker exits when an expectsReply escalation times out; lane-runner does not relaunch it and registry/lane state stay 'running' #630).
Sequencing invariant to preserve: the ratification APPROVE file must land before .DONE — the finalize gate depends on it.

fix(orch): pause never skips tasks or completes the batch; resume mer…

Live evidence for this design (penster 20260906T194514, PR #632 build): a correctly-holding lane with the operator away ~1h consumed 10 iterations / 5 relaunches / ~15 exit-intercepts — every one answered by the supervisor — and was one idle turn from Hold unresolved. A hold does not suspend the counters, and hold-bookkeeping commits don't reset them (only checkbox changes count). This is the case for a first-class held state that stops the counters; nothing short of it makes a multi-hour operator ruling survivable.

Interim on the bundle (commit 9c600cf6): hold exits no longer consume the productive-iteration budget, and the info acknowledgement contract (#630) resets the relaunch budget — a supervisor acking each relaunch can keep a lane alive indefinitely, at the cost of one idle spawn every few minutes. Bridge, not fix.

Sage's design review set the minimum bar for the real thing (recorded here for the Tier-2 pass): a runner-owned hold state machine, not an in-tool wait —

durable hold record (escalation id, task/segment identity, deadline, delivery state) restored before any spawn or completion decision;
a held mailbox target the supervisor can reach while no worker process exists (today send_agent_message requires a live registry pid);
stall-monitor awareness: runner-held ≠ dead worker (snapshot freshness + STATUS mtime both matter);
the ruling delivered into the relaunched worker's initial input (checkMailbox runs at message_end, not before the first prompt);
query wakes to answer but does not release; abort cancels, never approves; typed rulings carrying the ruler's role, correlated by escalation id;
holds checked before completion paths, not only on no-progress exits; a durable deadline that acknowledgements do not silently extend.

Also from the same run: the delegated-closure pattern (supervisor verifies the fold and writes the next R-numbered APPROVE) needs a first-class "gate closed by ratification" record — see the earlier comment.

Acceptance criteria addition (penster feedback #3, item 2): a correctly-held, acknowledged idle turn must not count as no-progress. Live evidence: a held lane reached 2/3 on the stall counter while waiting on a ruling. A first-class held state suspends the no-progress counter, the productive-iteration budget, the relaunch budget and the stall-timeout monitor for the duration of the hold, and hold-bookkeeping commits (hold(TP-xxxx):) are not required to reset anything.

Also from #3 (item 3): the hold-resume prompt was intermittent — some relaunches got the normal "remaining steps" prompt during a documented hard hold. Two causes, both by design of the Tier-1 bridge and both arguments for durable hold state: (a) any steer-typed reply (before the info acknowledgement contract existed, every supervisor reply) releases the hold, so the following relaunch is a normal iteration; (b) hold state is volatile per lane run — after orch_retry_task + orch_resume the relaunched worker has no pending escalation and gets the normal prompt. The durable hold record in this design must survive resume and drive the relaunch prompt.

v0.30.6 is released with the Tier-1 bridges (hold-aware relaunch, info acknowledgement contract, hold exits not consuming iterations, configurable intercept window). This issue is the first item of the Tier-2 design pass; the acceptance criteria and Sage's minimum bar are recorded in the comments above.

Design spec landed (branch feat/held-state, docs/specifications/taskplane/held-state-spec.md), Sage-reviewed. Beyond the acceptance criteria above, the design pass surfaced three seams that must be in scope or the hold leaks: (1) .DONE has two more authorities than the lane-runner — the task monitor and resume reconciliation both trust it first; (2) persistRuntimeState() is best-effort and cannot carry hold open/release — strict persist-before-ack path required; (3) resume re-executes interrupted tasks task-serially, so a restored hold would block every other resumed lane — goes lane-parallel. Also: plain steer stops meaning release (typed ruling + replyTo only), and ratification is a companion governance record (Stage 2). Staging: 1 core held state (closes #630) → 2 ratification (feeds #626) → 3 #631 lease → 4 #628/#626.

chore(tasks): stage TP-198 / TP-199 — #627 Stage 2 ratification recor…

Merge pull request #637 from HenryLach/feat/held-state

Merge pull request #638 from HenryLach/orch/henrylach-20260907T194708

Field report — first full overnight hold on the `held`-state runtime (penster batch `20260911T234647`, taskplane 0.30.6)

This batch exercised the held-state design end to end, including the case the spec was written for: an operator-reserved ruling arriving after holdTimeoutMinutes. Six tasks, three lanes, one wave, ~9 h; four operator/supervisor holds across three lanes; one hold outlived the 240-min deadline.

What worked (acceptance criteria met)

Hold is mechanical, not prompt discipline. Every escalate_to_supervisor produced a persisted hold-open:<id>; the task showed held; no held unit wrote .DONE or completed a gated step; no lane merged past an open hold. Workers committed only ruling-independent work as hold(TP-…): checkpoints and stopped at a clean tree.
Idle held lane costs nothing. After the worker exhausted ruling-independent work it exited; the runner's hold loop published held at zero cost (lane snapshot: cost=0.00, no process) for ~4 h.
hold-timeout parked the batch correctly. At 240 min the batch paused with cause hold-timeout, the hold stayed open, the worktree and checkpoint (464e1704) were preserved, and the summary reported 4/6 succeeded with nothing lost.
Ruling → orch_resume(force=true) recovery is clean. A single type="ruling" (replyTo = the escalation id) queued while parked was consumed on resume: hold-released persisted, the worker relaunched with the ruling at the top of its prompt, acked, and continued from the checkpoint. No hand-edit of batch-state.json was needed at any point — first batch where that is true.
info / query contracts behaved: info acknowledged without releasing (deadline unchanged); query answered from stored state without spawning.

Observations / defects (all non-blocking; the hold semantics held)

A. Dashboard task status does not update after a hold release (operator-observed). Screenshot attached by the operator. After the post-timeout resume:

TP-2104 (the task whose hold expired) still shows held in the lane list although the engine log shows hold-released:<id> ×3, the worker acked the ruling, and the lane snapshot (.pi/runtime/<batch>/lanes/lane-1.json) has worker.status=running with tool calls advancing. No tool activity or progress is rendered against TP-2104.
TP-2105 (the next task on the same serial lane, not yet started) shows running with live tool-call/cost/context telemetry (53 tool calls, 17%, $5.30, a read: of a lane-1 file) — i.e. the relaunched TP-2104 worker's telemetry is being attributed to TP-2105 — while its description column says "Not Started" and progress is 0/25.
Root-cause guess from the state files: batch-state.json carries TP-2104 status=held + exitReason="Hold timeout … the hold remains open" and TP-2105 status=running + exitReason="Lane held — awaiting ruling on an earlier task" after resume; the lane snapshot is correct. The dashboard appears to key the badge on batch-state.json and the telemetry on the lane, so the two disagree until the next task-transition persist. Expected: on hold-released the held task flips to running and the lane's telemetry attaches to it; the not-yet-started successor stays pending.

B. ⏸️ Lane held alerts replay after release (4 occurrences across 3 lanes, incl. one labelled EXPIRED after the resume that released it). Each time the engine log already had hold-released:<id> and the inbox was drained. Harmless once the supervisor checks the log first, but it invites a duplicate ruling. Suggest suppressing the alert when the hold record is released, and not re-emitting on resume.

C. A ruling is consumed only when the worker session exits. Under penster's hold-time convention the worker keeps committing ruling-independent work, so a live worker never sees the ruling; the supervisor had to send a steer copy each time (send_agent_message even says "no worker process is involved" while one is running). The relaunch then re-reads a ruling already applied — one wasted iteration per hold (×3). Suggest: deliver ruling to a live worker's stdin like steer, and mark the hold released on the worker's ack rather than only via the hold loop.

D. Ruling-closed gates have no APPROVE file. When the supervisor closes a REVISE-capped gate by ruling (AGENTS.md review-budget rule 1), the latest on-disk verdict stays REVISE; the post-.DONE sweep then relaunches the worker for "review remediation" and runs a fresh review (4 such ledger reviews tonight on TP-2103/TP-2054, ~1 iteration + 1 review each). ratify_gate exists for exactly this — the supervisor should call it proactively; consider having the runtime prompt for it when a REVISE gate's task reaches .DONE.

E. Reviewer subprocess failure on the first Step-5 code review ×2 (different lanes): .reviewer-state.json status=running, 0 tool calls, no verdict, no reviewer events/stderr written; the runtime's retry ~1 min later worked. Side effect: the dead attempt consumed a review number (R003) and the spiral detector's "round N" label ran one ahead of actual verdicts (labelled "round 4" for the second real verdict). Rounds should count verdict files, not spawns; and the failed spawn should leave a diagnostic somewhere.

F. Citation checker noise. unknown-ruling fired on valid released rulings (id present in ack/, hold-released in the log); prose-claim fires on the word "ruling" — including "ruling-independent", which is the mandated hold-time commit vocabulary — 14 flags on one iteration.

G. Relaunch reset a struck step to open in STATUS (TP-1920 after its split ruling); the worker caught and reverted it.

Net: the held-state contract did what the spec promised — a 9½-hour hold with the operator away resolved with one ruling and one resume, zero state surgery. The remaining items are presentation (A, B), delivery ergonomics (C, D), and diagnostics (E, F, G).

Here's a screenshot of the dashboard after the hold was cleared and the batch started running again:

Closing — acceptance criteria met in the field.

The 2026-09-05 criteria, checked against the 20260911T234647 report above and the 20260909T000015 chains (TP-1919, TP-2100):

held is a persisted task status entered on escalation — ✅ (hold-open:<id> for every escalate_to_supervisor; schema v5 holds[]).
While held: no re-spawns, no stall progression, no .DONE, no merge, no cleanup; worktree/branch preserved; zero cost — ✅ (~4 h idle at cost=0.00, no process).
Cleared only by a typed ruling from the supervisor/operator role referencing the escalation — ✅ (send_agent_message(type="ruling", replyTo=…) / /orch-rule; steer/info/query never released). Commit citations are validated via the Taskplane-Ruling: trailer (TP-199) — precision fixes in fix(hold): no stale 'Lane held' alert when a ruling is already queued; prose ruling claims require affirmative language #643 / Ruled gates: prompt for ratify_gate instead of remediation; unknown-ruling false flag on a valid ruling; struck step reopened on relaunch #654.
orch_pause / supervisor_takeover on a held task preserve, never complete — ✅ (hold-timeout park kept the hold open and the checkpoint; 4/6 reported truthfully, nothing lost).
Worktree removal refuses on uncommitted work; held distinct from stalled/skipped in the record — ✅ (dashboard ⚖; projection bug after release → After a hold release on resume, batch-state keeps the task 'held' and marks the not-yet-started lane successor 'running' (dashboard mis-attributes telemetry) #651).

Plus the spec's Stage 2: ratify_gate / /orch-ratify ratification record bound to the ruling and an immutable proof, and one authorizeCompletion() predicate for live finalize and resume (TP-198/199, PR #638).

Shipped: PRs #637 (Stage 1), #638 (Stage 2). In flight: #643 (stale Lane held replay — item B; prose-claim noise — item F).
Follow-ups filed from the report: #651 (A — status projection / successor mis-attribution after release), #652 (C — deliver a ruling to a live held worker), #653 (E — reviewer spawn failure diagnostics + round counting), #654 (D — prompt for ratify_gate instead of remediation; F — unknown-ruling on a valid ruling, needs the commit + hold record; G — struck step reopened).

Thank you for the two field reports — a 9½-hour operator-away hold resolved with one ruling and one resume, zero state surgery, is the outcome this issue was opened for.

Navigation Menu

Repository navigation

Cap-ruling hold: runtime must block .DONE and merge while an operator ruling is pending (.PENDING-RULING) #627

Summary

Expected

Actual

Evidence

Related

Suggested minimal cut

Field report — first full overnight hold on the `held`-state runtime (penster batch `20260911T234647`, taskplane 0.30.6)

What worked (acceptance criteria met)

Observations / defects (all non-blocking; the hold semantics held)

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Cap-ruling hold: runtime must block .DONE and merge while an operator ruling is pending (.PENDING-RULING) #627

Description

Summary

Expected

Actual

Evidence

Related

Suggested minimal cut

Activity

HenryLach commented on Sep 3, 2026

HenryLach commented on Sep 4, 2026

HenryLach commented on Sep 4, 2026

HenryLach commented on Sep 5, 2026

HenryLach commented on Sep 6, 2026

HenryLach commented on Sep 7, 2026

HenryLach commented on Sep 7, 2026

HenryLach commented on Sep 7, 2026

HenryLach commented on Sep 7, 2026

HenryLach commented on Sep 7, 2026

HenryLach commented on Sep 12, 2026

Field report — first full overnight hold on the held-state runtime (penster batch 20260911T234647, taskplane 0.30.6)

What worked (acceptance criteria met)

Observations / defects (all non-blocking; the hold semantics held)

HenryLach commented on Sep 12, 2026

HenryLach commented on Sep 12, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Field report — first full overnight hold on the `held`-state runtime (penster batch `20260911T234647`, taskplane 0.30.6)