Skip to content

Projects board: failed judge asks the user, remove clears board tags, coordinator never auto-resolves - #7038

Merged
enyst merged 1 commit into
mainfrom
claude/project-thread-xwkk9b
Nov 21, 2026
Merged

enyst merged 1 commit into
mainfrom
claude/project-thread-xwkk9b

Conversation

@enyst

@enyst enyst commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

HUMAN:

This is a review of 6602 from another agent. Verify these are real or not or wdyt


AGENT:

Why

A review of PR 6602 found three board bugs. All three were confirmed against the merged code, and none of them were in it.

  1. A failed or timed-out judge hid a question. _judge returned None on an exception, so the turn stayed finished. A confident critic score could then auto-resolve a reply that was actually asking the user something.
  2. Removing a member left a stale chip. DELETE /api/projects/{id}/conversations/{cid} cleared only the project tag. turn, ask, resolved and resolvenote stayed on the conversation.
  3. The coordinator could resolve itself. ProjectService.watch attaches the board subscriber to every member, the coordinator included. That meant the judge could set resolved=auto on the coordinator, the conversation the user steers the project from.

Summary

  • Failed judge: project_board.py now tells "no judge available" (None) apart from "the judge failed" (it raises).
    • When the judge fails, the turn becomes asking (Needs you), with the reply's last paragraph as ask, and nothing resolves it.
    • The critic fallback applies only when there is no judge.
    • There are no text or question-mark heuristics.
  • Remove: the remove route calls a new ProjectService.release_member. It drops project and all board tags in one tag-locked update. release_members, used on project delete, now uses the same helper.
  • Coordinator: the subscriber gets is_coordinator. The coordinator's turns still get turn (and ask from a card), but the judge and the resolve step are skipped.

Issue Number

Part of #6637.

How to Test

uv run pytest tests/agent_server/test_project_board.py tests/agent_server/test_projects_router.py

There are three new or changed tests:

  • test_a_failed_judge_shows_the_turn_as_needs_you_and_never_resolves
  • test_the_coordinator_is_never_judged_or_resolved
  • test_remove_conversation_clears_the_project_and_board_tags

All three fail on main and pass with this change. The 135 board and router tests pass, along with 237 Projects-related agent-server tests. Ruff and pyright are clean on the changed files. I did not run the live end-to-end check.

Type

  • Bug fix

Notes

Projects is behind its experimental flag, so nothing here changes the app with the flag off. The F36 feature map is updated to match.


Generated by Claude Code


🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant Architectures Base Image Docs / Tags
java amd64, arm64 eclipse-temurin:17-jdk Link
python-slim amd64, arm64 python-node-runtime Link
python-minimal amd64, arm64 python-node-runtime Link
python amd64, arm64 python-node-runtime Link
golang amd64, arm64 golang:1.21-bookworm Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:9655e8d-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-9655e8d-python \
  ghcr.io/openhands/agent-server:9655e8d-python

All tags pushed for this build

ghcr.io/openhands/agent-server:9655e8d-golang-amd64
ghcr.io/openhands/agent-server:9655e8d4a9e3a9eced07424798cacbe2a428a9f6-golang-amd64
ghcr.io/openhands/agent-server:claude-project-thread-xwkk9b-golang-amd64
ghcr.io/openhands/agent-server:9655e8d-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:9655e8d-golang-arm64
ghcr.io/openhands/agent-server:9655e8d4a9e3a9eced07424798cacbe2a428a9f6-golang-arm64
ghcr.io/openhands/agent-server:claude-project-thread-xwkk9b-golang-arm64
ghcr.io/openhands/agent-server:9655e8d-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:9655e8d-java-amd64
ghcr.io/openhands/agent-server:9655e8d4a9e3a9eced07424798cacbe2a428a9f6-java-amd64
ghcr.io/openhands/agent-server:claude-project-thread-xwkk9b-java-amd64
ghcr.io/openhands/agent-server:9655e8d-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:9655e8d-java-arm64
ghcr.io/openhands/agent-server:9655e8d4a9e3a9eced07424798cacbe2a428a9f6-java-arm64
ghcr.io/openhands/agent-server:claude-project-thread-xwkk9b-java-arm64
ghcr.io/openhands/agent-server:9655e8d-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:9655e8d-python-amd64
ghcr.io/openhands/agent-server:9655e8d4a9e3a9eced07424798cacbe2a428a9f6-python-amd64
ghcr.io/openhands/agent-server:claude-project-thread-xwkk9b-python-amd64
ghcr.io/openhands/agent-server:9655e8d-python-node-runtime-amd64
ghcr.io/openhands/agent-server:9655e8d-python-arm64
ghcr.io/openhands/agent-server:9655e8d4a9e3a9eced07424798cacbe2a428a9f6-python-arm64
ghcr.io/openhands/agent-server:claude-project-thread-xwkk9b-python-arm64
ghcr.io/openhands/agent-server:9655e8d-python-node-runtime-arm64
ghcr.io/openhands/agent-server:9655e8d-python-minimal-amd64
ghcr.io/openhands/agent-server:9655e8d4a9e3a9eced07424798cacbe2a428a9f6-python-minimal-amd64
ghcr.io/openhands/agent-server:claude-project-thread-xwkk9b-python-minimal-amd64
ghcr.io/openhands/agent-server:9655e8d-python-node-runtime-minimal-amd64
ghcr.io/openhands/agent-server:9655e8d-python-minimal-arm64
ghcr.io/openhands/agent-server:9655e8d4a9e3a9eced07424798cacbe2a428a9f6-python-minimal-arm64
ghcr.io/openhands/agent-server:claude-project-thread-xwkk9b-python-minimal-arm64
ghcr.io/openhands/agent-server:9655e8d-python-node-runtime-minimal-arm64
ghcr.io/openhands/agent-server:9655e8d-python-slim-amd64
ghcr.io/openhands/agent-server:9655e8d4a9e3a9eced07424798cacbe2a428a9f6-python-slim-amd64
ghcr.io/openhands/agent-server:claude-project-thread-xwkk9b-python-slim-amd64
ghcr.io/openhands/agent-server:9655e8d-python-node-runtime-slim-amd64
ghcr.io/openhands/agent-server:9655e8d-python-slim-arm64
ghcr.io/openhands/agent-server:9655e8d4a9e3a9eced07424798cacbe2a428a9f6-python-slim-arm64
ghcr.io/openhands/agent-server:claude-project-thread-xwkk9b-python-slim-arm64
ghcr.io/openhands/agent-server:9655e8d-python-node-runtime-slim-arm64
ghcr.io/openhands/agent-server:9655e8d-golang
ghcr.io/openhands/agent-server:9655e8d4a9e3a9eced07424798cacbe2a428a9f6-golang
ghcr.io/openhands/agent-server:claude-project-thread-xwkk9b-golang
ghcr.io/openhands/agent-server:9655e8d-golang_tag_1.21-bookworm
ghcr.io/openhands/agent-server:9655e8d-java
ghcr.io/openhands/agent-server:9655e8d4a9e3a9eced07424798cacbe2a428a9f6-java
ghcr.io/openhands/agent-server:claude-project-thread-xwkk9b-java
ghcr.io/openhands/agent-server:9655e8d-eclipse-temurin_tag_17-jdk
ghcr.io/openhands/agent-server:9655e8d-python-minimal
ghcr.io/openhands/agent-server:9655e8d4a9e3a9eced07424798cacbe2a428a9f6-python-minimal
ghcr.io/openhands/agent-server:claude-project-thread-xwkk9b-python-minimal
ghcr.io/openhands/agent-server:9655e8d-python-node-runtime-minimal
ghcr.io/openhands/agent-server:9655e8d-python-slim
ghcr.io/openhands/agent-server:9655e8d4a9e3a9eced07424798cacbe2a428a9f6-python-slim
ghcr.io/openhands/agent-server:claude-project-thread-xwkk9b-python-slim
ghcr.io/openhands/agent-server:9655e8d-python-node-runtime-slim
ghcr.io/openhands/agent-server:9655e8d-python
ghcr.io/openhands/agent-server:9655e8d4a9e3a9eced07424798cacbe2a428a9f6-python
ghcr.io/openhands/agent-server:claude-project-thread-xwkk9b-python
ghcr.io/openhands/agent-server:9655e8d-python-node-runtime

About Multi-Architecture Support

  • Each variant tag (e.g., 9655e8d-python) is a multi-arch manifest supporting both amd64 and arm64
  • Docker automatically pulls the correct architecture for your platform
  • Individual architecture tags (e.g., 9655e8d-python-amd64) are also available if needed

… coordinator never auto-resolves

Three fixes from a review of #6602:

- A judge that fails or times out leaves the turn `asking` (Needs you),
  with the reply's last paragraph as `ask`, and never resolves it. The
  critic fallback applies only when no judge is available.
- Removing a member clears its board tags with the `project` tag, in one
  tag-locked update, so no stale chip stays.
- The coordinator's turns are recorded but never judged or resolved.

Co-authored-by: openhands <openhands@all-hands.dev>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwJH94xGSYwfoyccT4E
@github-actions

Copy link
Copy Markdown
Contributor

REST API breakage checks (OpenAPI) — ✅ PASSED

Result: ✅ PASSED

Action log

@github-actions

Copy link
Copy Markdown
Contributor

Agent-server conversation start benchmark

Base main@fde86c378397551fa3fc2113079d171e8afa9286 vs this PR merge e501134bb6, measured in the same job on the same runner, interleaved: 5 server starts per side and scenario. Medians with 95% bootstrap CIs. The LLM is a mock that answers instantly, so this is agent-server overhead only.

base this PR change
Create conversation (warm) 0.114 s [0.106, 0.130] 0.117 s [0.101, 0.127] +2% ≈ same
↳ first agent event 0.160 s [0.150, 0.180] 0.160 s [0.147, 0.179] +0% ≈ same
First create right after server start 0.260 s [0.145, 0.560] 0.321 s [0.248, 0.351] +24% ≈ same
↳ first agent event 0.405 s [0.332, 0.925] 0.590 s [0.398, 0.634] +46% ≈ same
First create 3 s after server start 0.147 s [0.101, 0.166] 0.157 s [0.108, 0.224] +7% ≈ same
↳ first agent event 0.393 s [0.255, 0.439] 0.420 s [0.259, 0.746] +7% ≈ same
Create in a new git worktree (warm, 0.5 s git fetch) 0.664 s [0.632, 0.692] 0.658 s [0.634, 0.694] -1% ≈ same
↳ first agent event 0.718 s [0.678, 0.747] 0.702 s [0.670, 0.758] -2% ≈ same
Worktree creates that stalled the server > 20 ms 0/20 (0%) 2/20 (10%) +10 pts ≈ same
Create with one stdio MCP server (warm, 3 s startup) 1.092 s [1.084, 1.109] 1.103 s [1.093, 1.114] +1% ≈ same
↳ first agent event 3.628 s [3.578, 3.677] 3.644 s [3.628, 3.695] +0% ≈ same
Creates that stalled the server > 20 ms 4/35 (11%) 1/35 (3%) -9 pts ≈ same
Server startup (/alive) 4.356 s [4.032, 5.398] 4.188 s [3.856, 4.910] -4% ≈ same
How this is measured

Each create is a Canvas-shaped POST /api/conversations (agent profile, worktree: true, first message included, so the request waits for agent startup). A side task polls /alive every 20 ms to catch event-loop stalls. Scenarios, each on its own server starts:

  • default: a fresh directory per create, 8 creates per start.
  • pause: the first create waits 3 s after startup, as a user opening the UI would.
  • worktree: every create branches a new worktree from a local repository whose origin answers fetches after 0.5 s.
  • mcp: the agent profile enables one stdio MCP server that starts after 3 s, like an npx/uvx launch.

A timing is flagged when the 95% CIs do not overlap and the medians differ by at least 50 ms, or when a row with at least 8 samples per side moves by at least 150 ms and 25%. A create stalled the server when its slowest /alive took over 20 ms; stall rows are flagged when the share of stalled creates moves by at least 20 points with a Fisher exact p below 0.01. The first-create rows have one sample per server start, so they are flagged only on a clear change. A "↳ first agent event" row times the create above it until the agent's first event arrives on a socket opened after the create returns, as Agent Canvas does, so it includes the connect and replay. Runner: blacksmith-2vcpu-ubuntu-2404-llqmqpi67jzp4fze, 178 environment variables. Run it locally with python scripts/agent_server_benchmarks/conversation_start.py --help.

@github-actions

Copy link
Copy Markdown
Contributor

Coverage

Coverage Report •
FileStmtsMissCoverMissing
openhands-agent-server/openhands/agent_server
   project_board.py212399%147–148, 319
   project_service.py3124386%147–148, 152–156, 211–213, 225–226, 289–290, 292, 313, 341–343, 349–350, 398, 401, 406, 439, 472, 505, 536–537, 573, 619–622, 699–700, 721–722, 727, 792–795
   projects_router.py4182095%223, 360–362, 387–389, 400, 421, 537–539, 596, 599, 612, 754, 846, 851, 997, 1079
TOTAL603782219263% 

@enyst

enyst commented Nov 20, 2026

Copy link
Copy Markdown
Member Author

The red "Validate PR description" check comes from the first run (44834372013), which read the original description. That description referenced #6602, a PR, where the check expects an issue. I've updated the description, and the run on the edited description (44834507562) passes. Re-running the old job just re-reads the old description.


Generated by Claude Code

@enyst
enyst requested a review from all-hands-bot November 21, 2026 00:11

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

Reviewed the full diff against main@fde86c37 and traced each of the three fixes through the surrounding code.

Verified:

  1. Failed judge — ConversationOutcomeSubscriber._record now distinguishes "judge failed" (raises, surfaced via judging.exception()) from "no judge" (None). On failure the turn becomes asking with ask_line(text) as the ask, _notify_finished delivers it as asking, and the early turn == "asking" return blocks both the critic fallback and the resolved=auto write. Cancellation by a new user message still keeps the turn finished, and exc_info=error on the warning is valid logging. The judge-failure path is reachable only inside the shown/judging block, so there is no un-shown failure state.
  2. Coordinator — _coordinator() is checked before judging and before the resolve step, so the coordinator is never judged and never auto-resolves; its turns still get turn/ask and coordinator delivery. _is_coordinator re-reads live tags and the project store per turn, so it stays correct if membership changes. The _question_open() branch before it preserves prior ask semantics for the coordinator too.
  3. Remove — ProjectService.release_member does one tag-locked update clearing all of RESERVED_TAG_KEYS (project, resolved, resolvenote, turn, ask) and preserves other tags; the router's 404 semantics (event_service is None or project-tag mismatch) and the coordinator 409 guard are unchanged from the old set_conversation_tag path. release_members (project delete) now delegates to the same helper, so both paths behave identically.

The F36 feature map is updated to match the new behavior on all three points. The REST contract and OpenAPI schema are unchanged (breakage check green), no SDK/client/persisted-settings surface is touched, and Projects stays behind its experimental flag.

Evidence: the three new/changed tests are absent on main and pass here; locally tests/agent_server/test_project_board.py + test_projects_router.py pass (135 tests), the wider Projects-related agent-server selection passes (228 tests), and ruff + pyright are clean on the changed files. CI on this head is green; the red "Validate PR description" check is from the pre-edit description and the re-run on this exact SHA (44834507562) passed, as the author's comment explains. No live end-to-end run was made, acceptable here since the failures are in-process logic fully covered by the real test paths.

No material correctness, security, or compatibility findings.

✅ APPROVED

@enyst
enyst merged commit 80c188e into main Nov 21, 2026
55 of 57 checks passed
@enyst
enyst deleted the claude/project-thread-xwkk9b branch November 21, 2026 00:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants