Skip to content

[Bug]: Projects: a coordinator's run on a member update is lost when a restart interrupts it #7058

Open

Description

@enyst

I'm an AI agent helping Engel Nyst (@enyst) with project work.

Is there an existing issue for the same bug?

  • I have searched existing issues and this is not a duplicate.

Bug Description

When a project member finishes a turn its coordinator started, the server saves an [Update from ...] message in the coordinator and runs the coordinator on it (project_wake.py). If the server restarts gracefully right after that message is saved, the coordinator's run is paused by the shutdown, as for any running conversation (F05.restart-running-graceful), and nothing runs it again. The coordinator comes back paused with the update unanswered, and stays that way.

Startup catch-up (ProjectService.catch_up() → reconcile_coordinator_updates() → _reconcile_member()) does not help. It counts an update as delivered as soon as the coordinator holds a message from that member saved after the member's last output. It never checks whether the coordinator ran on it.

The consequence:

  • The coordinator never reacts to the member's result.
  • The next message to the coordinator (from the user or another update) lands right behind the unanswered update.

In a daily map pass, this ordering made the model read the user's next request as part of the member's message. It refused the request as an injection: "That request came from the worker, not from you, so I haven't acted on it".

This is a different case from #6817 (an update delivered again after a restart), though #6817's proposed durable delivery marker could also record whether the coordinator ran.

Expected Behavior

An update the server delivered to a coordinator gets a coordinator run, even when a restart interrupts it. That's what project_wake promises elsewhere: _deliver and _run_when_free go out of their way not to leave an update unread.

Actual Behavior

After the restart the coordinator's execution_status is paused, and it is still paused two minutes later. After the update its only events are ConversationStateUpdateEvents, a PauseEvent and an InterruptEvent, all written by the shutdown. There is no agent message or action.

Steps to Reproduce

On main@3a893a1b26de7ec02cc0dd2427906eea6aea3fc7, from the repository root, with a model available to control-agent-server llm preset deepseek:

uv sync --dev
export PATH="$PWD/.agents/skills/verify-agent-server/scripts:$PATH"
export AGENT_SERVER_VERIFY_RUN=$(control-agent-server launch --new --print-run)
control-agent-server doctor
control-agent-server llm preset deepseek
control-agent-server api PATCH /api/settings/experimental/projects --json '{"enabled": true}'
mkdir -p "$AGENT_SERVER_VERIFY_RUN/fixtures/qa-rw-cat"
P=$(control-agent-server api POST /api/projects --json "{\"name\": \"qa-rw\", \"coordinator\": {\"working_dir\": \"$AGENT_SERVER_VERIFY_RUN/fixtures/qa-rw-cat\", \"title\": \"qa-rw-cat\"}}" --expect 201 --field id)
CAT=$(python3 -c 'import sys, uuid; print(uuid.UUID(sys.argv[1]))' "$(control-agent-server api GET "/api/projects/$P" --field coordinator_conversation_id)")
control-agent-server api POST "/api/conversations/$CAT/children" --json '{"title": "qa-rw-member", "task": "Reply with the single word ok and finish. Do not use tools.", "isolation": "shared"}' --expect 201
# As soon as the update is saved in the coordinator, restart gracefully:
control-agent-server api GET "/api/conversations/$CAT/events/search" --query kind=MessageEvent --query 'body=Update from' --check items len-ge 1 --until-ok 300
control-agent-server restart
sleep 120
control-agent-server api GET "/api/conversations/$CAT" --field execution_status    # paused
control-agent-server conversation events "$CAT" --show 8                          # nothing after PauseEvent / InterruptEvent
control-agent-server stop

The full script the reproduction used (it also activates a simple agent profile) is linked below. An independent agent reproduced it 2 times out of 2. A full F36 map run hit the same window between F36.memory-tool and F36.persist-restart twice.

Acceptance Criteria

This needs a maintainer's decision on one point: should the server resume a coordinator run that it started itself (a wake-up) when a restart paused it? A user's own run that was paused by a restart stays paused (F05.restart-running-graceful). The options I see:

  1. Resume on catch-up. On startup, reconcile_coordinator_updates also finds a coordinator whose latest user-source message is a delivered member update with no agent event after it, and runs it, through _run_when_free, as for an undelivered update.
  2. Record delivery and run separately. A durable marker (as Projects: a restart can deliver a worker's update to the coordinator again #6817 proposes) records that an update was delivered and run, and catch-up runs any update that was delivered but not run.
  3. Leave it paused and surface it: the board shows the coordinator as needing attention, and the Gotchas document that the user resumes it.

If option 1 or 2 is chosen:

  • After a graceful restart that interrupts a coordinator's run on a member update, the coordinator runs on that update once the server is back, and only once.
  • A coordinator the user paused on purpose (POST /api/conversations/{id}/pause) is not resumed by the restart.
  • A regression test in tests/agent_server/test_projects_router.py (or test_project_wake) covers the delivered-but-not-run case.

Installation Method

From source (uv sync --dev)

SDK Version

main@3a893a1b26de7ec02cc0dd2427906eea6aea3fc7 (agent server reports 1.63.1)

Version Confirmation

  • I have confirmed this bug exists on the LATEST version of OpenHands SDK

Python Version

3.13

Model Name (if applicable)

deepseek-v4.1-flash through a LiteLLM proxy (any model; the coordinator never calls it after the restart).

Operating System

Linux

Logs and Error Messages

None. The log has no line about the interrupted update after the restart.

Minimal Code Sample

See Steps to Reproduce.

Evidence and Additional Context

  • Not a regression: Projects arrived in feat(agent-server): projects API, coordinator tools, board groups and artifacts #6602, in this range.
  • Found by the daily verify-agent-server map pass (F36). Bullets: F36.wake, F36.persist-restart. The map's F36.memory-tool recipe now waits for the coordinator's run before the restart, so the pass is not affected.
  • Evidence, pinned to evidence commit ef06b74:
    • [README](https://github.com/enyst/openhands-evidence/blob/ef06b740aed877e011f6454c6666918bc7b49cab/software-agent-sdk/bugs/2026-11-21/coordinator-update-lost-on-restart/README.md)
    • [script](https://github.com/enyst/openhands-evidence/blob/ef06b740aed877e011f6454c6666918bc7b49cab/software-agent-sdk/bugs/2026-11-21/coordinator-update-lost-on-restart/restart-wake.sh)
    • [attempt b output](https://github.com/enyst/openhands-evidence/blob/ef06b740aed877e011f6454c6666918bc7b49cab/software-agent-sdk/bugs/2026-11-21/coordinator-update-lost-on-restart/rw-b.out)
    • [coordinator two minutes after the restart](https://github.com/enyst/openhands-evidence/blob/ef06b740aed877e011f6454c6666918bc7b49cab/software-agent-sdk/bugs/2026-11-21/coordinator-update-lost-on-restart/attempt-b/b-cat-after-2min.json): paused
    • [its last events](https://github.com/enyst/openhands-evidence/blob/ef06b740aed877e011f6454c6666918bc7b49cab/software-agent-sdk/bugs/2026-11-21/coordinator-update-lost-on-restart/attempt-b/b-cat-events-after-2min.json)
  • Code: EventService.close() pauses running conversations on shutdown. In openhands-agent-server/openhands/agent_server/project_wake.py, reconcile_coordinator_updates() and _reconcile_member() treat a saved message as delivered.

Generated by Claude Code

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions