Skip to content

docs(verify): map maintenance 2026-11-21 - #7059

Open
enyst wants to merge 4 commits into
mainfrom
cloud/nifty-ride-xjugwj
Open

enyst wants to merge 4 commits into
mainfrom
cloud/nifty-ride-xjugwj

Conversation

@enyst

@enyst enyst commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

HUMAN:

Engel Nyst (@enyst) set up a daily automation that keeps the verify-agent-server and verify-sdk feature maps current. This PR is its run for main@3a893a1, after PR #7031.


AGENT:

I'm an AI agent helping Engel Nyst (@enyst) with project work.

Why

This is the daily delta pass over both feature maps, at TARGET main@3a893a1b:

  • Agent-server map: main@0275e714..main@3a893a1, 62 commits. Its baseline had not moved since 2026-11-17.
  • SDK map: main@722e5d40..main@3a893a1, 14 commits.

Every family that the owners commands listed was replayed live on fresh runs. Every edited family was then re-run by an independent agent. This pass looked closely at the new experimental flags: projects (#6602) and factories (#7016), plus the flag cache (#6936).

This machine has no DeepSeek key, only a LiteLLM proxy serving deepseek-v4.1-flash. Recipes that read $DEEPSEEK_API_KEY or assert the native deepseek/... models fail here for that reason alone. Every family with such fails was re-run through a local stand-in for the DeepSeek API that forwards to the proxy (shim runs below).

Summary

Agent-server map: delta pass. The baseline moves to 3a893a1.

  • The owners command listed 34 families: F01–F13, F16–F19, F21–F25, F27, F29–F34 and F36–F40.
  • First runs, by bullet: 815 pass, 115 fail, 29 xfail, 0 xpass, 11 blocked.
  • Every fail is one of these:
    • The missing DeepSeek key, and failures that cascade from it (110): F03 (1), F09 (27), F21 (9), F22 (9), F23 (17), F30 (14), F31 (2), F32 (3), F33 (22), F34 (7). Through the stand-in, F09, F21, F22, F23 and F30–F34 come to 289 pass, 2 fail, 9 xfail and 3 blocked (sub-feature rows). This includes every F09 live ACP bullet and every F23 provider-connection bullet, which kept the baseline from moving yesterday. The 2 fails:
      • F33.completion-logs: the stand-in reports the proxy's model name.
      • F34.endpoint-audit: the drift fixed here.
    • Model variance or timing, passed on re-run: F07.error-non-json ([Bug]: Session socket can send an error frame before sync while a run holds the state lock #6745, 2/2), F39.tool, F40.goal-asked (2/2).
    • Map drift: F34.endpoint-audit.
  • Blocked, with what they need:
    • Docker: F12.docker-runtime, F12.docker-startup-timeout, F31.storage-retention, F33.workspace-runtime-lifecycle.
    • openvscode-server: three F16 VS Code bullets.
    • A ChatGPT login: F09.acp-effort-codex, F24.login-complete.
    • Chromium installed only in a Playwright directory: F27.browser-playwright-path, F27.browser-probe.
  • Changed bullets, with their live results:

SDK map: delta pass. The baseline moves to 3a893a1.

Both maps

Issue Number

#5751

How to Test

uv sync --dev
export PATH="$PWD/.agents/skills/verify-agent-server/scripts:$PWD/.agents/skills/verify-sdk/scripts:$PATH"
control-agent-server map check && control-agent-server map coverage && sdk-map check
control-agent-server map run --file F36 --fresh --record --keep-going
control-agent-server map run --file F18 --fresh --record --keep-going
control-agent-server map run --file F34 --only F34.endpoint-audit --fresh --record
control-agent-server map run --file Fnn --fresh --record --keep-going    # each of the 34 families above
sdk-map run --file S03 && sdk-map run --file S06 && sdk-map run --file S09
sdk-map run --file Snn                                                   # each of the 12 families above

uv run pytest tests/cross/test_verify_agent_server_skill.py tests/cross/test_verify_sdk_skill.py → 96 passed.

Evidence (README):

  • [Agent-server report](https://github.com/enyst/openhands-evidence/blob/cfc9bdcefb92505deccdaa84bda7fcd68e093052/software-agent-sdk/map-maintenance/2026-11-21/agent-server-report-main.md): every replayed bullet of the 34 families, including re-runs.
  • [Stand-in runs](https://github.com/enyst/openhands-evidence/blob/cfc9bdcefb92505deccdaa84bda7fcd68e093052/software-agent-sdk/map-maintenance/2026-11-21/agent-server-report-shim.md): F09, F21–F23 and F30–F34 without the key gap.
  • [Independent verifier](https://github.com/enyst/openhands-evidence/blob/cfc9bdcefb92505deccdaa84bda7fcd68e093052/software-agent-sdk/map-maintenance/2026-11-21/agent-server-report-verifier.md): F18, F36, F34 and F19 from this branch. [F36 re-verification](https://github.com/enyst/openhands-evidence/blob/cfc9bdcefb92505deccdaa84bda7fcd68e093052/software-agent-sdk/map-maintenance/2026-11-21/agent-server-report-f36-reverify.md): after the memory-rest and memory-tool fixes.
  • [SDK report](https://github.com/enyst/openhands-evidence/blob/cfc9bdcefb92505deccdaa84bda7fcd68e093052/software-agent-sdk/map-maintenance/2026-11-21/sdk-report-main.md) and [SDK verifier](https://github.com/enyst/openhands-evidence/blob/cfc9bdcefb92505deccdaa84bda7fcd68e093052/software-agent-sdk/map-maintenance/2026-11-21/sdk-report-verifier.md).
  • F36.flag-off: [create while off](https://github.com/enyst/openhands-evidence/blob/cfc9bdcefb92505deccdaa84bda7fcd68e093052/software-agent-sdk/map-maintenance/2026-11-21/edited/F36/F36.flag-off/create.json) answers 404 Projects are turned off. [Back on](https://github.com/enyst/openhands-evidence/blob/cfc9bdcefb92505deccdaa84bda7fcd68e093052/software-agent-sdk/map-maintenance/2026-11-21/edited/F36/F36.flag-off/on-again.json), the same projects are served without a restart.
  • F36.runtime-env: [host switch off](https://github.com/enyst/openhands-evidence/blob/cfc9bdcefb92505deccdaa84bda7fcd68e093052/software-agent-sdk/map-maintenance/2026-11-21/edited/F36/F36.runtime-env/host-off.json) wins over the stored switch.
  • F36.pull-requests: [the board](https://github.com/enyst/openhands-evidence/blob/cfc9bdcefb92505deccdaa84bda7fcd68e093052/software-agent-sdk/map-maintenance/2026-11-21/edited/F36/F36.pull-requests/board.json) shows pull_requests from a gh pr create result.
  • F18.experimental-switch: [all six on after a restart](https://github.com/enyst/openhands-evidence/blob/cfc9bdcefb92505deccdaa84bda7fcd68e093052/software-agent-sdk/map-maintenance/2026-11-21/edited/F18/F18.experimental-switch/after-restart.json). F18.experimental-file-changes: [an edit on disk switches projects off](https://github.com/enyst/openhands-evidence/blob/cfc9bdcefb92505deccdaa84bda7fcd68e093052/software-agent-sdk/map-maintenance/2026-11-21/edited/F18/F18.experimental-file-changes/edited-off.json).
  • S03.route-task-to-model: [transcripts](https://github.com/enyst/openhands-evidence/blob/cfc9bdcefb92505deccdaa84bda7fcd68e093052/software-agent-sdk/map-maintenance/2026-11-21/edited/sdk/S-wt-S03-route/transcripts).

Video/Screenshots

The Agent Server changes here are the projects flag (#6602) and the routes it gates. The Canvas feature is F30 (Settings → Experimental), on Canvas 0f4cb6a against an Agent Server built from main@3a893a1b.

[F30.open: the Projects switch from #6602, off by default](https://raw.githubusercontent.com/enyst/openhands-evidence/cfc9bdcefb92505deccdaa84bda7fcd68e093052/software-agent-sdk/map-maintenance/2026-11-21/ui/projects-off.png)

[F30.toggle: Projects switched on; GET /api/projects answered 200 at once](https://raw.githubusercontent.com/enyst/openhands-evidence/cfc9bdcefb92505deccdaa84bda7fcd68e093052/software-agent-sdk/map-maintenance/2026-11-21/ui/projects-on.png)

[Canvas home with Projects on: no Projects UI yet (OpenHands/OpenHands#19495)](https://raw.githubusercontent.com/enyst/openhands-evidence/cfc9bdcefb92505deccdaa84bda7fcd68e093052/software-agent-sdk/map-maintenance/2026-11-21/ui/projects-on-home.png)

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes

The branch has the pass's commit plus three F36 recipe fixes, which came out of the independent re-runs. Squash on merge.


Generated by Claude Code


🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant Architectures Base Image Docs / Tags
java amd64, arm64 eclipse-temurin:17-jdk Link
python-slim amd64, arm64 python-node-runtime Link
python-minimal amd64, arm64 python-node-runtime Link
python amd64, arm64 python-node-runtime Link
golang amd64, arm64 golang:1.21-bookworm Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:4426d3f-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-4426d3f-python \
  ghcr.io/openhands/agent-server:4426d3f-python

All tags pushed for this build

ghcr.io/openhands/agent-server:4426d3f-golang-amd64
ghcr.io/openhands/agent-server:4426d3f1f9e6616db615bbaccfc75972d0138ecc-golang-amd64
ghcr.io/openhands/agent-server:cloud-nifty-ride-xjugwj-golang-amd64
ghcr.io/openhands/agent-server:4426d3f-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:4426d3f-golang-arm64
ghcr.io/openhands/agent-server:4426d3f1f9e6616db615bbaccfc75972d0138ecc-golang-arm64
ghcr.io/openhands/agent-server:cloud-nifty-ride-xjugwj-golang-arm64
ghcr.io/openhands/agent-server:4426d3f-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:4426d3f-java-amd64
ghcr.io/openhands/agent-server:4426d3f1f9e6616db615bbaccfc75972d0138ecc-java-amd64
ghcr.io/openhands/agent-server:cloud-nifty-ride-xjugwj-java-amd64
ghcr.io/openhands/agent-server:4426d3f-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:4426d3f-java-arm64
ghcr.io/openhands/agent-server:4426d3f1f9e6616db615bbaccfc75972d0138ecc-java-arm64
ghcr.io/openhands/agent-server:cloud-nifty-ride-xjugwj-java-arm64
ghcr.io/openhands/agent-server:4426d3f-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:4426d3f-python-amd64
ghcr.io/openhands/agent-server:4426d3f1f9e6616db615bbaccfc75972d0138ecc-python-amd64
ghcr.io/openhands/agent-server:cloud-nifty-ride-xjugwj-python-amd64
ghcr.io/openhands/agent-server:4426d3f-python-node-runtime-amd64
ghcr.io/openhands/agent-server:4426d3f-python-arm64
ghcr.io/openhands/agent-server:4426d3f1f9e6616db615bbaccfc75972d0138ecc-python-arm64
ghcr.io/openhands/agent-server:cloud-nifty-ride-xjugwj-python-arm64
ghcr.io/openhands/agent-server:4426d3f-python-node-runtime-arm64
ghcr.io/openhands/agent-server:4426d3f-python-minimal-amd64
ghcr.io/openhands/agent-server:4426d3f1f9e6616db615bbaccfc75972d0138ecc-python-minimal-amd64
ghcr.io/openhands/agent-server:cloud-nifty-ride-xjugwj-python-minimal-amd64
ghcr.io/openhands/agent-server:4426d3f-python-node-runtime-minimal-amd64
ghcr.io/openhands/agent-server:4426d3f-python-minimal-arm64
ghcr.io/openhands/agent-server:4426d3f1f9e6616db615bbaccfc75972d0138ecc-python-minimal-arm64
ghcr.io/openhands/agent-server:cloud-nifty-ride-xjugwj-python-minimal-arm64
ghcr.io/openhands/agent-server:4426d3f-python-node-runtime-minimal-arm64
ghcr.io/openhands/agent-server:4426d3f-python-slim-amd64
ghcr.io/openhands/agent-server:4426d3f1f9e6616db615bbaccfc75972d0138ecc-python-slim-amd64
ghcr.io/openhands/agent-server:cloud-nifty-ride-xjugwj-python-slim-amd64
ghcr.io/openhands/agent-server:4426d3f-python-node-runtime-slim-amd64
ghcr.io/openhands/agent-server:4426d3f-python-slim-arm64
ghcr.io/openhands/agent-server:4426d3f1f9e6616db615bbaccfc75972d0138ecc-python-slim-arm64
ghcr.io/openhands/agent-server:cloud-nifty-ride-xjugwj-python-slim-arm64
ghcr.io/openhands/agent-server:4426d3f-python-node-runtime-slim-arm64
ghcr.io/openhands/agent-server:4426d3f-golang
ghcr.io/openhands/agent-server:4426d3f1f9e6616db615bbaccfc75972d0138ecc-golang
ghcr.io/openhands/agent-server:cloud-nifty-ride-xjugwj-golang
ghcr.io/openhands/agent-server:4426d3f-golang_tag_1.21-bookworm
ghcr.io/openhands/agent-server:4426d3f-java
ghcr.io/openhands/agent-server:4426d3f1f9e6616db615bbaccfc75972d0138ecc-java
ghcr.io/openhands/agent-server:cloud-nifty-ride-xjugwj-java
ghcr.io/openhands/agent-server:4426d3f-eclipse-temurin_tag_17-jdk
ghcr.io/openhands/agent-server:4426d3f-python-minimal
ghcr.io/openhands/agent-server:4426d3f1f9e6616db615bbaccfc75972d0138ecc-python-minimal
ghcr.io/openhands/agent-server:cloud-nifty-ride-xjugwj-python-minimal
ghcr.io/openhands/agent-server:4426d3f-python-node-runtime-minimal
ghcr.io/openhands/agent-server:4426d3f-python-slim
ghcr.io/openhands/agent-server:4426d3f1f9e6616db615bbaccfc75972d0138ecc-python-slim
ghcr.io/openhands/agent-server:cloud-nifty-ride-xjugwj-python-slim
ghcr.io/openhands/agent-server:4426d3f-python-node-runtime-slim
ghcr.io/openhands/agent-server:4426d3f-python
ghcr.io/openhands/agent-server:4426d3f1f9e6616db615bbaccfc75972d0138ecc-python
ghcr.io/openhands/agent-server:cloud-nifty-ride-xjugwj-python
ghcr.io/openhands/agent-server:4426d3f-python-node-runtime

About Multi-Architecture Support

  • Each variant tag (e.g., 4426d3f-python) is a multi-arch manifest supporting both amd64 and arm64
  • Docker automatically pulls the correct architecture for your platform
  • Individual architecture tags (e.g., 4426d3f-python-amd64) are also available if needed

enyst and others added 4 commits November 21, 2026 14:34
Delta pass over both feature maps to main@3a893a1.

- F36: own the memory routes, drive reopen and artifacts/from-conversation,
  map the projects switch (flag-off, runtime env, tools while off), the
  failed resolution judge, pull requests (prs), board tags cleared on remove;
  Gotcha for #7054.
- F18: switch every experimental feature, edits to settings.json on disk.
- F34: endpoint audit allows the projects client routes.
- F19: drop a history reference.
- S03: route_task_to_model on a RouterLLM conversation.
- S06: the project coordinator and worker prompt presets.
- S09: drop a history reference; README maps the project tools and errors.py.
- Both baselines move to main@3a893a1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwN5qni9qsqRdGJfiMr
The coordinator can write to project memory on its own during earlier
model-backed bullets, so the recipe deletes existing files first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwN5qni9qsqRdGJfiMr
…on the updates

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwN5qni9qsqRdGJfiMr
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwN5qni9qsqRdGJfiMr
@github-actions

Copy link
Copy Markdown
Contributor

REST API breakage checks (OpenAPI) — ✅ PASSED

Result: ✅ PASSED

Action log

@enyst
enyst requested a review from all-hands-bot November 21, 2026 16:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant