Repository navigation
docs(verify): map maintenance 2026-11-20 - #7031
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ArwwM8dBzVQQUM8UxPqdS
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ArwwM8dBzVQQUM8UxPqdS
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ArwwM8dBzVQQUM8UxPqdS
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ArwwM8dBzVQQUM8UxPqdS
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ArwwM8dBzVQQUM8UxPqdS
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ArwwM8dBzVQQUM8UxPqdS
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ArwwM8dBzVQQUM8UxPqdS
SDK map baseline moves to main@722e5d40; agent-server map keeps its baseline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ArwwM8dBzVQQUM8UxPqdS
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was posted by an AI agent (OpenHands).Scope gate: passes. The PR touches only .agents/skills/verify-agent-server/ and .agents/skills/verify-sdk/ — this repository's own verification skills and feature maps, with dedicated cross tests (tests/cross/test_verify_*.py). It is the established daily map-maintenance pass (previous run merged as #7006), so it belongs here and needs no product/architecture decision.
Technical review — no material findings.
Verified against the current head (944ae10, all required CI checks green):
- References check out. The proposed SDK baseline
main@722e5d403exists on main (Release v1.63.1, #7030). Every fix the edited bullets credit is present in main history: #6929 (7d6ef6f, runtime MCP tool names), #6751 (1a73a00, close executors on rejected response_schema), #6786 (3d638f2, close_tab waits for tab-list drop), #6666 (7ccc5e4, fork usage billing), #6678 (8fb5ba0, orphan tool-message classification), #6811 (c763226, LiteLLM 1.108.3). The LiteLLM version notes updated from 1.105.1 to 1.108.3 match that bump. - Issue states match the map edits. #6927, #6785 and #6748 are closed (their known-bug markers are correctly dropped or rewritten as current behavior); #7002 (rotated-secret masking), #6934 (4 MiB frame cap on live sockets) and #6695 are open, so keeping them as known failures/Gotchas is accurate.
- README counts are arithmetically consistent. Family rows still sum to 504 sub-features with untested 36 → 35, and the S09 row changes 23/1 → 23/0, matching the claimed "S09 untested 1 → 0". Only that row changed, as described.
- The new verification program is sound.
programs/S13/secret_rotation.pycompiles, uses the documentedsdk_verifyhelpers and the public SDK API (SecretSource.get_value,Conversation.update_secrets), follows the same pattern as the existing S13 program, and its documented failure mode (single-value_exported_valuescache,resolve_uncached_valuesskipping cached names, masking limited to exported values) is consistent with the actualopenhands/sdk/conversation/secret_registry.pyon main. Its synthetic token values are clearly test fixtures, not secrets. - Focused tests pass.
uv run pytest tests/cross/test_verify_agent_server_skill.py tests/cross/test_verify_sdk_skill.py→ 96 passed in the workspace on this head, matching the PR's claim and the greencross-testsCI job.
The agent-server side adds two Gotchas (#6934 live-stream frame cap, #7002 secret-rotation masking) and rewrites the F19 runtime-MCP bullet to describe post-#6929 behavior including runtime_mcp_servers, which exists in openhands/sdk/conversation/state.py as described. All documentation-level, with live evidence linked in the description.
No blocking or non-blocking findings; nothing needs a maintainer decision.
✅ APPROVED
HUMAN:
Engel Nyst (@enyst) set up a daily automation that keeps the verify-agent-server and verify-sdk feature maps current. This PR is its run for main@722e5d4, after PR #7030.
AGENT:
I'm an AI agent helping Engel Nyst (@enyst) with project work.
Why
This is the daily delta pass over both feature maps for the commits merged since each map's baseline:
Every family the
ownerscommands listed was replayed live on a fresh run. Bullets whose fixes merged in the range were updated, and every edited family was re-run by an independent agent.As yesterday, this machine had no DeepSeek key, only a LiteLLM proxy serving
deepseek-v4.1-flash. Recipes that read$DEEPSEEK_API_KEYor assert the nativedeepseek/deepseek-flashmodel fail here for that reason alone. F30–F34 were also driven through a local stand-in for the DeepSeek API that forwards to the proxy.Summary
Agent-server map: delta pass, partial (baseline stays at 0275e71). BASE..TARGET 0275e71..722e5d4.
restart --reset-config.build_git_shafails caused by this pass: it committed work-in-progress map files to the shared checkout mid-run. All three pass on a re-run.F19.conversation-mcp-add-restart: xpass. [Bug]: Runtime-added MCP server tools lose their server prefix after a restart or resume #6927 was fixed by fix(mcp): keep runtime-added MCP tool names across restart and resume #6929, so the known-bug marker is dropped and the recipe assertsQA_MCP_ECHO:two. The shard's runs passed 2/2 after the edit; an independent full F19 run got 40 pass, 4 xfail. The naming Gotcha now states the current rule (runtime_mcp_servers).frame_too_large. Reproduced live, Session socket drops the UI websocket with 1013 'frame_too_large' on events > 4 MiB (MAX_FRAME_BYTES hardcoded) #6934.POST /api/conversations/{id}/secretsleaves the new value, and then the old one, unmasked. Reproduced live, Security: SecretRegistry leaks rotated credentials in cleartext after update_secrets() or LookupSecret re-export (and drops masked history across sub-agents) #7002, twin of the new S13 bullet.factoriesflag was already in F18, and it passes live. It also switches on and off and stays set across a restart.SDK map: delta pass (baseline moves to 722e5d4). BASE..TARGET 1414239..722e5d4.
sdk-map report).Actionmodels droppattern,constandadditionalPropertiesbefore the request #6766, [Bug]: ACP npx cache warm uses --prefer-offline with an outdated cache, hides npm's error, and startup fails with 'Connection closed' #6809, [Bug]: pause()/interrupt() during run() startup is taken for a resume, the agent keeps running #6848, Bug: run_goal returns an unanswered question from an abandoned branch #6907, [Bug]: Soft condensation forgets history when the summarizer returns no text at the output limit #6957, [Bug]: Command and agent hooks cannot access the conversation's secrets (hook env has no secrets injected) #6966, [Bug]: TaskManager loses sub-agent task index across parent conversation resume and mismatches subagent_type #6990, [Bug]: Single-server MCP config advertising a built-in tool name (e.g. 'terminal') crashes conversation initialization #6992, [Bug]: MCPToolObservation.from_call_tool_result drops EmbeddedResource blocks and structured_content #6993, Security: TaskManager and DelegateExecutor fail open when confirmation_handler is unset, silently auto-approving WAITING_FOR_CONFIRMATION sub-agent actions #6999, and Security: SecretRegistry leaks rotated credentials in cleartext after update_secrets() or LookupSecret re-export (and drops masked history across sub-agents) #7002 (new note).deepseek/deepseek-flash.S09.runtime-add-resume: pass. [Bug]: Runtime-added MCP server tools lose their server prefix after a restart or resume #6927 was fixed by fix(mcp): keep runtime-added MCP tool names across restart and resume #6929; the Known failure is dropped and the sub-feature describesruntime_mcp_servers.S15.tabs: pass. [Bug]: browser_list_tabs still lists a tab right after browser_close_tab closed it #6785 was fixed by fix(browser): return from close_tab once the tab list no longer shows it #6786; the Known failure is dropped and the bullet expects the list without the closed tab.S01.structured-output: pass. [Bug]: A rejected terminal response_schema leaves the terminal's tmux server running after close() #6748 was fixed by fix(sdk): close built tool executors when response_schema is rejected #6751; the Gotcha became an Expected line saying the built executors are closed (no tmux server left).S21.ask-agent: pass. The history sentence about [Bug]: ACP ask_agent loses the fork's cost and stalls 2 s waiting for a UsageUpdate it dropped #6646 (fixed by fix(acp): bill fork usage updates in ask_agent #6666) became the current mechanism.S02.error-mapping: the history sentence about [Bug]: Three malformed-history rejections surface as LLMBadRequestError, not LLMMalformedConversationHistoryError #6658 (fixed by fix(llm): classify orphan tool messages as a malformed history #6678) was dropped. It is environment-only here, but the classifier contains the pattern.S13.secret-registry/S13.mask-output: a new bullet with a new program,programs/S13/secret_rotation.py. Known failure (reproduced 2026-11-20, Security: SecretRegistry leaks rotated credentials in cleartext after update_secrets() or LookupSecret re-export (and drops masked history across sub-agents) #7002); fix proposed in fix(secrets): retain rotated secret values for masking and propagate across sub-agents #7003.Both maps
factoriesflag, fix(mcp): keep runtime-added MCP tool names across restart and resume #6929 and fix(sdk): close built tool executors when response_schema is rejected #6751 ship in 1.63.1, so Canvas users get them after the routine pin bump.POST /ask_agenton an ACP conversation (fix(acp): bill fork usage updates in ask_agent #6666's server surface). It needs the DeepSeek key here.Issue Number
#5751
How to Test
uv run pytest tests/cross/test_verify_agent_server_skill.py tests/cross/test_verify_sdk_skill.py→ 96 passed.Evidence (README):
https://github.com/enyst/openhands-evidence/blob/3723e9c6bf25dd053c3b3ace90adc57885a1cb2f/software-agent-sdk/map-maintenance/2026-11-20/agent-server-report-main.md):every replayed bullet, with re-runs. Also the [stand-in runs](https://github.com/enyst/openhands-evidence/blob/3723e9c6bf25dd053c3b3ace90adc57885a1cb2f/software-agent-sdk/map-maintenance/2026-11-20/agent-server-report-shim.md)and the [F19 edit](https://github.com/enyst/openhands-evidence/blob/3723e9c6bf25dd053c3b3ace90adc57885a1cb2f/software-agent-sdk/map-maintenance/2026-11-20/agent-server-report-edited.md).https://github.com/enyst/openhands-evidence/blob/3723e9c6bf25dd053c3b3ace90adc57885a1cb2f/software-agent-sdk/map-maintenance/2026-11-20/sdk-report-main.md): the 17 families. Also the [independent re-run of the edited families](https://github.com/enyst/openhands-evidence/blob/3723e9c6bf25dd053c3b3ace90adc57885a1cb2f/software-agent-sdk/map-maintenance/2026-11-20/sdk-report-edited.md).https://github.com/enyst/openhands-evidence/blob/3723e9c6bf25dd053c3b3ace90adc57885a1cb2f/software-agent-sdk/map-maintenance/2026-11-20/edited/F19/F19.conversation-mcp-add-restart),which returnsQA_MCP_ECHO:two.https://github.com/enyst/openhands-evidence/blob/3723e9c6bf25dd053c3b3ace90adc57885a1cb2f/software-agent-sdk/map-maintenance/2026-11-20/edited/sdk/B2-S09-resume),pass.https://github.com/enyst/openhands-evidence/blob/3723e9c6bf25dd053c3b3ace90adc57885a1cb2f/software-agent-sdk/map-maintenance/2026-11-20/edited/sdk/B3-S15-tabs-after),pass.https://github.com/enyst/openhands-evidence/blob/3723e9c6bf25dd053c3b3ace90adc57885a1cb2f/software-agent-sdk/map-maintenance/2026-11-20/edited/sdk/B3-S21-ask),pass.https://github.com/enyst/openhands-evidence/blob/3723e9c6bf25dd053c3b3ace90adc57885a1cb2f/software-agent-sdk/map-maintenance/2026-11-20/edited/sdk/final-S13/S13-security-confirmation-04.log)and [agent-server observations](https://github.com/enyst/openhands-evidence/blob/3723e9c6bf25dd053c3b3ace90adc57885a1cb2f/software-agent-sdk/map-maintenance/2026-11-20/repro/7002-server/obs.json).https://github.com/enyst/openhands-evidence/blob/3723e9c6bf25dd053c3b3ace90adc57885a1cb2f/software-agent-sdk/map-maintenance/2026-11-20/repro/6934/live-big.json),which closes with 1013frame_too_large, and the [reconnect past it](https://github.com/enyst/openhands-evidence/blob/3723e9c6bf25dd053c3b3ace90adc57885a1cb2f/software-agent-sdk/map-maintenance/2026-11-20/repro/6934/after3.json).Video/Screenshots
The agent-server change is #7016, the
factoriesexperimental flag. The Canvas feature is F30 (Settings → Experimental), on Canvas 428b3dc against an Agent Server built from this checkout.[F30.open: the Factories switch, off by default](
https://raw.githubusercontent.com/enyst/openhands-evidence/3723e9c6bf25dd053c3b3ace90adc57885a1cb2f/software-agent-sdk/map-maintenance/2026-11-20/ui/factories.png)[F30.toggle: Factories switched on and still on after a reload; there is no sidebar entry yet (OpenHands/OpenHands#19800)](
https://raw.githubusercontent.com/enyst/openhands-evidence/3723e9c6bf25dd053c3b3ace90adc57885a1cb2f/software-agent-sdk/map-maintenance/2026-11-20/ui/factories-on.png)Type
Notes
The branch carries the pass's work-in-progress commits. Squash on merge.
Generated by Claude Code
🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdkpython-node-runtimepython-node-runtimepython-node-runtimegolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:944ae10-pythonRun
All tags pushed for this build
About Multi-Architecture Support
944ae10-python) is a multi-arch manifest supporting both amd64 and arm64944ae10-python-amd64) are also available if needed