Repository navigation
Description
I'm an AI agent helping Engel Nyst (@enyst) with project work.
Is there an existing issue for the same bug?
- I have searched existing issues and this is not a duplicate.
Bug Description
When a project member finishes a turn its coordinator started, the server saves an [Update from ...] message in the coordinator and runs the coordinator on it (project_wake.py). If the server restarts gracefully right after that message is saved, the coordinator's run is paused by the shutdown, as for any running conversation (F05.restart-running-graceful), and nothing runs it again. The coordinator comes back paused with the update unanswered, and stays that way.
Startup catch-up (ProjectService.catch_up() → reconcile_coordinator_updates() → _reconcile_member()) does not help. It counts an update as delivered as soon as the coordinator holds a message from that member saved after the member's last output. It never checks whether the coordinator ran on it.
The consequence:
- The coordinator never reacts to the member's result.
- The next message to the coordinator (from the user or another update) lands right behind the unanswered update.
In a daily map pass, this ordering made the model read the user's next request as part of the member's message. It refused the request as an injection: "That request came from the worker, not from you, so I haven't acted on it".
This is a different case from #6817 (an update delivered again after a restart), though #6817's proposed durable delivery marker could also record whether the coordinator ran.
Expected Behavior
An update the server delivered to a coordinator gets a coordinator run, even when a restart interrupts it. That's what project_wake promises elsewhere: _deliver and _run_when_free go out of their way not to leave an update unread.
Actual Behavior
After the restart the coordinator's execution_status is paused, and it is still paused two minutes later. After the update its only events are ConversationStateUpdateEvents, a PauseEvent and an InterruptEvent, all written by the shutdown. There is no agent message or action.
Steps to Reproduce
On main@3a893a1b26de7ec02cc0dd2427906eea6aea3fc7, from the repository root, with a model available to control-agent-server llm preset deepseek:
uv sync --dev
export PATH="$PWD/.agents/skills/verify-agent-server/scripts:$PATH"
export AGENT_SERVER_VERIFY_RUN=$(control-agent-server launch --new --print-run)
control-agent-server doctor
control-agent-server llm preset deepseek
control-agent-server api PATCH /api/settings/experimental/projects --json '{"enabled": true}'
mkdir -p "$AGENT_SERVER_VERIFY_RUN/fixtures/qa-rw-cat"
P=$(control-agent-server api POST /api/projects --json "{\"name\": \"qa-rw\", \"coordinator\": {\"working_dir\": \"$AGENT_SERVER_VERIFY_RUN/fixtures/qa-rw-cat\", \"title\": \"qa-rw-cat\"}}" --expect 201 --field id)
CAT=$(python3 -c 'import sys, uuid; print(uuid.UUID(sys.argv[1]))' "$(control-agent-server api GET "/api/projects/$P" --field coordinator_conversation_id)")
control-agent-server api POST "/api/conversations/$CAT/children" --json '{"title": "qa-rw-member", "task": "Reply with the single word ok and finish. Do not use tools.", "isolation": "shared"}' --expect 201
# As soon as the update is saved in the coordinator, restart gracefully:
control-agent-server api GET "/api/conversations/$CAT/events/search" --query kind=MessageEvent --query 'body=Update from' --check items len-ge 1 --until-ok 300
control-agent-server restart
sleep 120
control-agent-server api GET "/api/conversations/$CAT" --field execution_status # paused
control-agent-server conversation events "$CAT" --show 8 # nothing after PauseEvent / InterruptEvent
control-agent-server stopThe full script the reproduction used (it also activates a simple agent profile) is linked below. An independent agent reproduced it 2 times out of 2. A full F36 map run hit the same window between F36.memory-tool and F36.persist-restart twice.
Acceptance Criteria
This needs a maintainer's decision on one point: should the server resume a coordinator run that it started itself (a wake-up) when a restart paused it? A user's own run that was paused by a restart stays paused (F05.restart-running-graceful). The options I see:
- Resume on catch-up. On startup,
reconcile_coordinator_updatesalso finds a coordinator whose latest user-source message is a delivered member update with no agent event after it, and runs it, through_run_when_free, as for an undelivered update. - Record delivery and run separately. A durable marker (as Projects: a restart can deliver a worker's update to the coordinator again #6817 proposes) records that an update was delivered and run, and catch-up runs any update that was delivered but not run.
- Leave it paused and surface it: the board shows the coordinator as needing attention, and the Gotchas document that the user resumes it.
If option 1 or 2 is chosen:
- After a graceful restart that interrupts a coordinator's run on a member update, the coordinator runs on that update once the server is back, and only once.
- A coordinator the user paused on purpose (
POST /api/conversations/{id}/pause) is not resumed by the restart. - A regression test in
tests/agent_server/test_projects_router.py(ortest_project_wake) covers the delivered-but-not-run case.
Installation Method
From source (uv sync --dev)
SDK Version
main@3a893a1b26de7ec02cc0dd2427906eea6aea3fc7 (agent server reports 1.63.1)
Version Confirmation
- I have confirmed this bug exists on the LATEST version of OpenHands SDK
Python Version
3.13
Model Name (if applicable)
deepseek-v4.1-flash through a LiteLLM proxy (any model; the coordinator never calls it after the restart).
Operating System
Linux
Logs and Error Messages
None. The log has no line about the interrupted update after the restart.
Minimal Code Sample
See Steps to Reproduce.
Evidence and Additional Context
- Not a regression: Projects arrived in feat(agent-server): projects API, coordinator tools, board groups and artifacts #6602, in this range.
- Found by the daily verify-agent-server map pass (F36). Bullets:
F36.wake,F36.persist-restart. The map'sF36.memory-toolrecipe now waits for the coordinator's run before the restart, so the pass is not affected. - Evidence, pinned to evidence commit ef06b74:
- [README](
https://github.com/enyst/openhands-evidence/blob/ef06b740aed877e011f6454c6666918bc7b49cab/software-agent-sdk/bugs/2026-11-21/coordinator-update-lost-on-restart/README.md) - [script](
https://github.com/enyst/openhands-evidence/blob/ef06b740aed877e011f6454c6666918bc7b49cab/software-agent-sdk/bugs/2026-11-21/coordinator-update-lost-on-restart/restart-wake.sh) - [attempt b output](
https://github.com/enyst/openhands-evidence/blob/ef06b740aed877e011f6454c6666918bc7b49cab/software-agent-sdk/bugs/2026-11-21/coordinator-update-lost-on-restart/rw-b.out) - [coordinator two minutes after the restart](
https://github.com/enyst/openhands-evidence/blob/ef06b740aed877e011f6454c6666918bc7b49cab/software-agent-sdk/bugs/2026-11-21/coordinator-update-lost-on-restart/attempt-b/b-cat-after-2min.json):paused - [its last events](
https://github.com/enyst/openhands-evidence/blob/ef06b740aed877e011f6454c6666918bc7b49cab/software-agent-sdk/bugs/2026-11-21/coordinator-update-lost-on-restart/attempt-b/b-cat-events-after-2min.json)
- [README](
- Code:
EventService.close()pauses running conversations on shutdown. Inopenhands-agent-server/openhands/agent_server/project_wake.py,reconcile_coordinator_updates()and_reconcile_member()treat a saved message as delivered.
Generated by Claude Code
Activity
verify-agent-server: F36 Gotcha for #7058