Skip to content

fix(agent): fail a Room Run that never replied to a waiting Human - #230

Closed
oldwinter53 wants to merge 1 commit into
mainfrom
oldwinter/014-d
Closed

oldwinter53 wants to merge 1 commit into
mainfrom
oldwinter/014-d

Conversation

@oldwinter53

Copy link
Copy Markdown

Problem (P1-7, #172)

Pi on a small local model often answers in the Run's final text without calling teamhub message send. The Run reported succeeded and the Room showed nothing: no reply and no failure.

Root cause: ReplyDelivery.replyUndelivered (apps/node-services/src/agent/executor/reply-delivery.ts) failed a Room Run only when it tried to post and every write failed. An Agent that never tried was treated as choosing silence, so #completedTerminal in run-loop.ts returned succeeded. Silence is correct for Room talk the Agent only reads along with, but not when a Human mentioned it and is waiting.

Fix

A Room (Meta) Run now finishes failed with the new code reply_not_sent when both are true:

  • One of its inputs (initial or steered) is a Human's Message that mentioned this Agent. This uses the structured fields TeamHub freezes into each Inbox input: mentioned, authorType == "user", and messageKind == "conversation". The web sends every direct-chat message with the peer's @mention, so direct chats are covered.
  • The Agent never started a message write. Emoji Reactions count as writes because the Computer already classifies them that way.

Routine wakes, Task callbacks, another Agent's mention, and Room talk that doesn't mention the Agent can still end silently. No wording is matched anywhere.

The final text is not posted on the Agent's behalf. The prompt defines it as private scratch, which often holds reasoning about whether to reply, and node-services has no server-side post path. The Room shows the failure instead: "Pi: The Agent finished without sending its reply to the Room. Its model may have answered without using the message command. Ask again, or choose a stronger model for the Agent."

  • apps/node-services/src/agent/executor/reply-delivery.ts: observeInputs, replyNotSent, awaitsReply
  • apps/node-services/src/agent/executor/run-loop.ts: #completedTerminal returns reply_not_sent and logs local_run.reply_not_sent. Steered inputs are observed in #applyProviderSteer.
  • Contract enum and description: contracts/openapi/components/agent-runtime-state.yaml (regenerated)
  • TeamHub validator: apps/teamhub/internal/agentruntimestate/runtime_failure.go
  • Web copy: apps/web/src/features/rooms/runtime-status/runtime-failure-message.ts
  • Docs: apps/node-services/src/agent/AGENTS.md

Critical path (user message → provider_ready): no change. The check runs after the Provider turn completes and adds no network round trip or file walk.

Not in this PR

Tests

Written first. Both "posted nothing" cases came back succeeded before the fix.

  • apps/node-services/tests/agent/executor/run-loop.test.ts › Room reply delivery:
    • A mentioned Human waits, the Run only reads and ends with text: fails reply_not_sent, the final text stays out of the terminal, and local_run.reply_not_sent is logged.
    • A Human mention steered into a Run that started for unmentioned Room talk, nothing posted: fails reply_not_sent.
    • A Human mention answered with a message: succeeded.
    • Silent on unmentioned Room talk, on a routine wake that mentions the Agent, or on another Agent's mention: succeeded.
  • runtime_failure_test.go: reply_not_sent is accepted.
  • runtime-failure-message.test.ts: copy for reply_not_sent.

Checks run locally:

  • make check-node-services (260 files, 2622 tests)
  • make check-web
  • make -C apps/teamhub fmt-check, go test ./tests/integration/layout, make -C apps/teamhub test
  • pnpm contracts:verify
  • make check-repository-gate BASE=origin/main

make check BASE=origin/main also selects check-packages and check-computer because the generated contracts changed. check-packages passes once mise's exported GOROOT/GOBIN are unset. TestManagedToolsUpdateKeepsResidentDaemonPID ("Managed tools require Node 26.5.0") fails on this host on an unmodified origin/main too, so it isn't caused by this change.

Real-machine checks (FIXLIST section 6, item 5; 0.1.4 RC round)

macOS (macmini, zh-CN, 24 GB), Pi with Spark-X2.6 4B, then again with LFM3 1.7B:

  1. In a direct chat with a Pi Agent, send each LM-01 prompt. Each message gets either a visible reply or the red Run failure "Pi: The Agent finished without sending its reply to the Room…". There is never a Run that shows success with no message.
  2. In a Room with that Agent and one other member, @-mention the Agent with an LM-01 prompt: same expectation as step 1. Send a Room message that doesn't mention it: no failure is shown if it stays silent.
  3. Create a routine with the Agent's mention and an instruction that usually has nothing to say. When it fires and the Agent stays silent, no failure is shown.
  4. In the Agent log, each failed Run in step 1 has a local_run.reply_not_sent event with deliveredMessageWrites: 0.
  5. P1-9: keep a 10-turn chat going with LFM3 1.7B and watch for the same reply posted more than once in one Run. If it happens, note the Run id and the count, check that Stop ends it, and attach the Agent log. That decides whether a per-Run message budget is needed.
  6. Codex and Claude Code direct chats: ordinary questions still get replies and no reply_not_sent appears (regression check).

🤖 Generated with Claude Code

A Pi Run on a small local model often answers in its final text without
calling the message command. The Run then succeeded and the Room showed
nothing. A Room Run that never tried to post while a Human's Message that
mentioned the Agent waits now finishes failed with reply_not_sent, so the
Room shows the cause. The final text stays private scratch and is not
posted for the Agent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@linger-review linger-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve with nits (1 P2)

Scope: base main, merge base fb2e1537, 1 commit, 14 files reviewed at 6c3969b6. Loaded skills: coderbook, ponytail-review, karpathy-guidelines. Ran in a separate session.

Findings

P2

A Run resumed after a restart can fail with reply_not_sent even though the reply is already in the Room (apps/node-services/src/agent/executor/run-loop.ts:1009)

  • #runExecution builds a fresh ReplyDelivery for each claim attempt and calls observeInputs(execution.messages). A Run that the process stops for a restart is requeued (local_run.requeued_for_restart, around line 1462). The next process resumes it in the same native session with resumesAttempt = true, so the original mentioned-Human input sets #awaitingHuman again while #attemptedWrites goes back to 0.
  • The resume prompt (renderInterruptedRunReminder, prompt/inbox-notification.ts:133) tells the Agent that "the messages you already sent" are in native history and not to repeat operations that already ran. Failure case: the Agent posts its answer, a deploy restarts node-services before the turn ends, and the resumed attempt finishes without a new message send. The Human then sees a red "Pi: The Agent finished without sending its reply to the Room… choose a stronger model" under a reply that is already there. The yield path is fine because it reuses the same ReplyDelivery across segments. Only the cross-process resume is affected.
  • Fix: skip the reply_not_sent decision when execution.resumesAttempt is true, either by not calling observeInputs for the initial messages in that case or by passing the flag into #completedTerminal. Steered inputs that arrive after the resume can still count. Add a run-loop test that claims with attemptCount: 2, has no message writes, and expects succeeded.

Verified as fine

  • The detection uses structured fields only (mentioned, authorType == "user", messageKind == "conversation" from acceptedNotification.messages) and matches no wording. Those are the same fields run-kind.ts and accepted-fact.ts already read.
  • The Computer classifies teamhub message <send-verb> as message and every other message subcommand as message_read (command_activity.go:376-380). Counting attempts on activity.started of kind message is therefore correct, including failed writes, which reply_undelivered still handles first.
  • Steered inputs are observed only after provider.steer succeeds (#applyProviderSteer), so a steer that is folded back into a new Run is not counted twice. The yield/continue loop shares one ReplyDelivery.
  • Task (alter_ego), Document and trusted/feature inputs have no acceptedNotification or are not meta, so they are unaffected.
  • The prompt still allows silence for routines, background results and Room talk not aimed at the Agent (compact-meta-instructions.ts:37, meta-instructions.ts:104), which matches the new rule.
  • Contract change is in contracts/openapi/components/agent-runtime-state.yaml, and the generated Go/TS diffs match the enum addition. The TeamHub validator accepts the new code. Old web clients fall back to "Run failed" for an unknown code (runtimeFailureMessage checks Object.hasOwn).
  • Docs: apps/node-services/src/agent/AGENTS.md describes the new failure. No new env vars, no cross-app imports, English-only.
  • Critical path: the check runs after the Provider turn completes, so there is no extra round trip before provider_ready, as the PR body says.
  • Tests cover the mentioned-Human, steered-mention, replied, unmentioned, routine and agent-mention cases. The Go and web tests cover the new code.

Process notes

  • The branch is 6 commits behind main, and main requires up-to-date branches, so it needs a rebase.
  • Single commit with no merge or fixup commits. The commit author email cdd@simplexai.com resolves to GitHub account oldwinter90, which is not the PR author oldwinter53. Please confirm this is intended.

CI

At review time CodeQL, Analyze, changes, repository-gate, check (desktop), check (installer) and check (teamhub-cli) had passed. check (computer-go) was skipped. check (contracts), check (packages), check (teamhub), web tests 1–3 and web-static were still pending, and no node-services check was listed yet. I did not run the focused node-services test: the worktree has no node_modules, and installing them just to run it was not justified. The finding comes from reading the code.

Automated review by Claude Code


Automated review of 6c3969b by Claude Code.

@oldwinter53

Copy link
Copy Markdown
Author

Consolidated into #249 with #230 and #237 (same commits, replayed on the newest main and merged with #242's Make room changes; the body carries this PR's description, tests and real-machine checks). Please review there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants