Skip to content

fix(sdk): a fork keeps the source's confirmation policy, security analyzer and hooks - #6763

Draft
enyst wants to merge 27 commits into
claude/openhands-sdk-dual-source-truth-qp1x6r-1-secrets-autosavefrom
claude/openhands-sdk-dual-source-truth-qp1x6r-2-fork-keeps-state
Draft

enyst wants to merge 27 commits into
claude/openhands-sdk-dual-source-truth-qp1x6r-1-secrets-autosavefrom
claude/openhands-sdk-dual-source-truth-qp1x6r-2-fork-keeps-state

Conversation

@enyst

@enyst enyst commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

HUMAN:
Engel Nyst asked me to find the fork PR that removes the two sources of truth for a conversation's settings (meta.json in the agent-server and base_state.json in the SDK), which is #4813 by @BSmick6; to put its commits on branches in this repo, split so that each concern can be reviewed on its own; and to open new PRs that credit the author and say that we do the final review and tweaks. Then to review each PR, address what the review finds, and prove that the changes work by running the SDK and the agent-server the way a program or an agent does, with the repo's verify-sdk and verify-agent-server skills. The principle behind the request: the agent-server is a REST interface to the SDK. It should not be a different thing, and it should not repeat what the SDK already does, unless a particular case gives a special reason.

This is part 2 of 3: a fork keeps the source's confirmation policy, security analyzer, hooks and hook verdicts.


AGENT:

I'm Claude (Fable 5.2), an AI agent working for Engel Nyst. I brought #4813 up to date with main, split it into a stack of three PRs, reviewed each, and verified them against real servers. The policy and analyzer copy in this PR is @BSmick6's, from #4813, with the analyzer handling from Engel Nyst's follow-up on that PR. #5765, which gives a fork its source's cipher and secrets, merged into main while this PR was open, so that part is no longer in this diff. What the final review added is listed under "Final review".

The stack: 1. #6762 (secrets are saved the moment they change) → 2. this PR → 3. #6764 (the agent-server stops keeping its own copy). This PR is based on part 1, so its diff shows only its own change.

Why

LocalConversation.fork() builds the fork from the agent and the events and, since #5765, gives it the source's cipher and a copy of its secrets. It does not copy the confirmation policy or the security analyzer. On main:

  • a fork of a conversation under AlwaysConfirm or ConfirmRisky starts with NeverConfirm and runs its actions unasked;
  • the fork has no security analyzer, and so none of the analyzer's own credentials (a guardrail model key).

The agent-server hides part of this: it copies the source's meta.json to the fork and applies it on every load, so a server fork gets the policy and analyzer its source was created with, not ones set later (see Notes). #6764 removes that second copy, so the SDK has to carry these values itself.

Summary

Final review: what was found and changed

  1. A fork ran what the source's hooks had refused. The hook verdicts live in the conversation state (blocked_actions, blocked_messages, last_user_message_id) and were not copied. I reproduced it on main: a pending action that a PreToolUse hook had denied on the source was executed when the fork ran, and a prompt that a UserPromptSubmit hook had blocked was sent to the model. The fork now carries the verdicts for the events it copies, and the hooks themselves: it is built with the hook_config its source was created with. The verdicts alone would have been half a fix, since the next action the model asked for in an SDK-level fork ran without the caller's hook_config hooks; only the hooks of plugins given at construction, which the fork loads itself, still ran (a fork made through the agent-server kept explicit hooks, because the server passes its stored hook_config when it loads the fork). The plugin hooks are not handed over: the fork loads the plugins itself, and the merged config would register them twice. Six tests, all failing without the change.
  2. GraySwanAnalyzer never kept its api_key across a save, cipher or not: the field had no serializer, so it was written as the placeholder and read back as the literal string **********, which the analyzer then sent as its bearer token. This showed only when the key was passed as a field and GRAYSWAN_API_KEY was not set, because that variable always replaces the field. The field is now declared with SecretField, like the SDK's other credentials since refactor(sdk): declare secret fields once with SecretField #6779: with a cipher the key is encrypted and comes back; without one it is still written as the placeholder but loads as no key, and the analyzer reports UNKNOWN risk instead of sending the placeholder. Two tests, both failing without the change. Found while checking what a forked analyzer has after a reload.
  3. How the agent, the analyzer and the secret sources are copied. The agent and the analyzer are copied by one helper that goes through the model's own dump and validate, with credentials kept (copy_keeping_secrets, next to the SDK's other secret utilities, so the agent-server can use the same one for child conversations). On main that round trip is spelled out in fork() for the agent; for the analyzer it is also what keeps one conversation's history out of the other's. The secret sources are deep-copied, as on main since fix(sdk): keep the source's cipher and secrets on a forked conversation #5765, because a custom source may keep its value outside its fields (in a private attribute), where a dump does not carry it. model_copy(deep=True) alone raises TypeError on a custom source that holds a lock (cannot pickle '_thread.lock' object) or an HTTP client, so fork() fails on main for such a source; here it is rebuilt from its fields instead. That rebuild is the same dump and validate, with its limit: a source that both cannot be deep-copied and keeps its value outside its fields comes out without the value (the fork resolves it to nothing, with no error), and fork() still raises if the fields alone do not validate. The agent's own copy is a round trip, as on main, so such a source in agent_context.secrets has no value in the fork's agent; the fork's registry has the working copy. The built-in sources, StaticSecret and LookupSecret, are not affected either way. The analyzer and the secret sources are copied under the source's lock, before the fork object exists, so a copy that fails leaves no fork directory behind. The agent's copy stays where it was on main, before the lock is taken. An earlier version of this PR sent the secret sources through the round trip as well; the review after the last merge of main caught that it dropped a privately held value, and there is a test for each case now.
  4. invoked_skills is copied too, next to the skill lists fork() already copied, so the fork reports the skills its source invoked (conversation.state.invoked_skills, which is also what the agent-server's conversation info shows as invoked_skills) instead of an empty list.
  5. Tests and a live recipe. Added: the fork keeps the policy (that copy had no SDK-level test; fix(agent-server): own confirmation_policy, security_analyzer, secrets in ConversationState #4813's server-level test of it comes with fix(agent-server): one owner for confirmation_policy, security_analyzer and secrets (base_state.json) #6764), the fork's registry is its own (masking included), a replacement agent given to fork(agent=...) keeps its secrets. The last two pass on main as well since fix(sdk): keep the source's cipher and secrets on a forked conversation #5765; they stay as regression tests. New verify-sdk sub-feature S04.fork-keeps-state, with a program that forks a ciphered conversation, changes the fork and loads it again from disk.

Issue Number

Part of #4810.

How to Test

uv run pytest tests/sdk/conversation/local/test_fork.py tests/sdk/conversation/local/test_fork_hook_blocks.py \
  tests/sdk/security/grayswan/test_grayswan_analyzer.py tests/agent_server/test_conversation_service.py -q

With this branch's tests on the code it is based on (main plus #6762), 13 of the command's 226 tests fail: five in test_fork.py (test_fork_keeps_confirmation_policy, test_fork_gives_independent_security_analyzer, test_fork_preserves_ciphered_security_analyzer_credentials, test_fork_copies_a_secret_source_that_cannot_be_deep_copied, test_fork_copies_invoked_skills), all six in test_fork_hook_blocks.py (two with assert ['drop-the-data'] == []: the command the hook denies ran), and the two GraySwan tests. On this branch all 226 pass.

In this section main is main at 3d638f2e, the commit this branch is merged with. Live, with the repo's verify-sdk skill. The recipe forks a ciphered conversation that has ConfirmRisky, an analyzer with its own model key, two secrets and a hook that blocks every prompt; it then changes the fork and loads it again from disk. main does not have this bullet, so its column is this branch's recipe and program placed in a copy of main and run on main's SDK.

Bullet main this branch
S04.fork-keeps-state fail, fail (11 of 21 checks fail, the same 11 in each attempt) pass, pass

The checks that fail on main, with what the program got instead:

  • fork_policy.kind eq ConfirmRisky: got NeverConfirm
  • fork_policy.threshold eq MEDIUM: got null
  • fork_policy.confirm_unknown eq false: got null
  • fork_analyzer eq ToolShieldLLMSecurityAnalyzer: got NoneType
  • fork_guard_key_kept eq true: got false
  • saved_policy.kind eq ConfirmRisky: got NeverConfirm
  • saved_analyzer eq ToolShieldLLMSecurityAnalyzer: got null
  • saved_guard_key_is_placeholder eq false: got true
  • fork_prompt_block eq blocked by the source hook: got null
  • reloaded_guard_key_kept eq true: got false
  • reloaded_secrets.FORK_ONLY eq fork-only-value-4: got null

Three of these follow from the missing analyzer rather than from a key that was written badly: main saves no analyzer for the fork, so there is no guard key at all, and the program reports a missing key as a placeholder (fork_guard_key_kept, saved_guard_key_is_placeholder, reloaded_guard_key_kept).

On the code this branch is based on (main plus #6762) the same recipe fails 10 of these, twice. The one that passes there is reloaded_secrets.FORK_ONLY eq fork-only-value-4: a secret added to the fork is still there after a reload, which is #6762's fix, not this one's.

The neighbouring bullets (S04.fork, S04.fork-keeps-secrets, S04.fork-branch) pass on main and on this branch, in the whole-family runs below; S04.fork-keeps-secrets is #5765's. The two agent-server bullets this PR fixed before #5765 merged (F10.fork-keeps-credentials, F10.fork-run-memory) pass on main now, and on the top of the stack (#6764's table).

# from this branch
export PATH="$PWD/.agents/skills/verify-sdk/scripts:$PATH"
sdk-map run --file S04 --only S04.fork-keeps-state

On this branch's final commit (bba1de01), the two SDK map families:

Family Result
S04 local conversation (SDK map) 32 of 36 pass; 3 fail that also fail on main (S04.pause-during-startup, S04.goal-asked-branch, S04.interactive-terminal); S04.factory labelled blocked in the map
S13 security and confirmation (SDK map) 9 of 10 pass; S13.confirm-approve-reject, S13.llm-analyzer labelled blocked in the map

S04.interactive-terminal ends with an integration test whose command asks for the model deepseek/deepseek-flash by name. This run has an LLM proxy and no DeepSeek key, so the request goes to the proxy, which serves the flash model under another name and answers 400 (There are no healthy deployments for this model). The example step before it passes, and the bullet fails the same way on main.

S04.pause-during-startup and S04.goal-asked-branch are bullets main's map maintenance added this week (#6869, #6931); they fail the same checks on main (compared check by check) and concern pausing and goals, which this change does not touch.

The whole agent-server families (F04, F08, F10, F12, F31) and the client entry points were replayed on the top of the stack, which contains this commit; that table is in #6764.

GitHub CI on bba1de01: all 15 checks that ran are green, and no job needed a re-run. Two are skipped by design: the macOS matrix placeholder, and the description check, which waits for the PR to leave draft.

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes

  • fix(sdk): keep the source's cipher and secrets on a forked conversation #5765 (issue 5465) merged while this PR was open. Its part left this diff: the cipher argument, the registry copy, its three tests and its F10 map edits. What stays is the policy, the analyzer, the hook verdicts and hooks, invoked_skills, the copy helper (with fix(sdk): keep the source's cipher and secrets on a forked conversation #5765's deep copy of the secret sources kept, plus a fallback for a source that cannot be deep-copied), the GraySwan fix, the S04.fork-keeps-state recipe and their tests.
  • A fork made through the agent-server shows the source's current policy, analyzer and start-request secrets only once fix(agent-server): one owner for confirmation_policy, security_analyzer and secrets (base_state.json) #6764 is in. Until then the server re-applies the creation-time copy from meta.json when it opens the fork: the policy and analyzer are the ones the source was created with, and a secret from the start request that was changed later has its original value again. A secret added after creation is kept, with its value only if the server has a cipher. fix(agent-server): one owner for confirmation_policy, security_analyzer and secrets (base_state.json) #6764 has the server-level recipes for that (F10.fork-keeps-settings, F08.fork-after-restart).
  • An SDK-level fork now runs the caller's hooks as its source does, SessionStart on first use and SessionEnd on close included. A fork made through the agent-server is not affected: the server never makes the SDK fork object ready, and loads the fork from disk with its stored hooks, as before.
  • Not changed here, noted for a follow-up: a fork does not take over the hooks of a plugin its source loaded at runtime with load_plugin(), or custom stuck-detection thresholds.
  • Found on the way, present on main and left for it: fork(reset_metrics=False) of a conversation that was resumed from disk stops counting what the fork spends, so the fork's budget cap never trips. Only an SDK fork used in-process is affected; a server fork is reloaded from disk and counts. I have a reproduction (a test file that fails on main) and a suggested fix for an issue.
  • While this PR is stacked on part 1, the workflows that only run for PRs into main do not run on it (among them REST contract, persisted settings, OpenAPI schema, binary and image build, TypeScript client). fix(agent-server): one owner for confirmation_policy, security_analyzer and secrets (base_state.json) #6764 lists local runs of the OpenAPI, REST-contract, persisted-settings and SDK-API checks on the top of the stack; the binary build, the image build and the TypeScript client CI were not run locally (four conversation bullets of the TypeScript client's map family F34 were run live on the top of the stack: create, run, fork/navigate/ask, confirm; they are in fix(agent-server): one owner for confirmation_policy, security_analyzer and secrets (base_state.json) #6764's table).

🤖 Generated with Claude Code

https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY

claude and others added 10 commits October 13, 2026 21:28
LocalConversation.fork() built the fork without the source's cipher, so the
fork's base_state.json stored the LLM key redacted. The agent-server resumes
every fork from that file, so the fork's first turn failed with
LLMAuthenticationError. Forward the cipher, and copy the source's
conversation secrets into the fork's registry so they persist too.

The verify-agent-server map bullets F10.fork-keeps-credentials and
F10.fork-run-memory now pass; their known-bug markers are removed.

Fixes #5465

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014MKndVj7XWvRQmvW9w2bDg
Adapt the server restart regression test from #5473, source commit 583fb5e. Keep the existing registry merge and deep-copy implementation in #5765.

Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
must carry, and it does the registry copy better than #4813 did: a fresh
SecretRegistry with deep-copied sources, so the fork does not share the
source's masking cache or its lock. Merging it here instead of
re-implementing it keeps one version of that code, with its authors and
its tests (kyletser's restart test included), whichever PR lands first.

Conflict: the feature-map index count. main's 955 stands (`map check`
agrees); #5765 removes two known-bug markers and adds no sub-feature.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…ersation.fork()

These fields are part of ConversationState but fork() did not copy them,
so the fork's base_state.json was written with the NeverConfirm / None
defaults and the source conversation's policy was silently lost: a fork
of a conversation under AlwaysConfirm ran its actions unasked.

Split out of #4813 (commit 61b865e).

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
… kept

The fork shared the source's analyzer object, so a stateful analyzer's
history (e.g. ToolShieldLLMSecurityAnalyzer) mixed two conversations.
Give the fork its own instance through a JSON round-trip with
expose_secrets, so the fork's cipher re-encrypts nested credentials (the
guardrail LLM api_key) instead of writing them redacted.

Tests: the fork owns its analyzer, a ciphered analyzer key survives a
resume of the fork, and secrets passed at start reach the fork (the
registry copy itself comes from #5765).

Split out of #4813 (commit f6ade80).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
The policy copy had no test. This one checks the fork's live state and
its base_state.json, which is what the agent-server resumes a fork from.
It fails on main, where the fork starts with NeverConfirm.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
New sub-feature S04.fork-keeps-state with a program that forks a
ciphered conversation carrying ConfirmRisky(MEDIUM), a ToolShield
analyzer with its own key and a runtime secret, then changes the fork
and loads it again from disk. The recipe checks the fork's memory, its
base_state.json and the reload.

On main fifteen of its checks fail: the fork starts with NeverConfirm,
no analyzer and no secrets, and its saved key is the placeholder.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
Review of the fork changes found three small things:

- Secret sources were deep-copied. A custom source that holds a lock or
  a client made fork() raise ("cannot pickle '_thread.lock' object"),
  after the fork's directory had already been written. They are now
  copied like the agent and the analyzer, through a dump/validate round
  trip (`_own_copy`), and the analyzer and the secrets are copied before
  the fork is constructed, so a copy that fails leaves nothing behind.
- `invoked_skills` was not copied, next to two skill lists that are.
- The analyzer test passed with a shallow copy. It now uses a
  ToolShield analyzer and checks that the guardrail LLM is the fork's
  own object too.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Coverage

Coverage Report •
FileStmtsMissCoverMissing
openhands-sdk/openhands/sdk/conversation/impl
   local_conversation.py14479693%204, 270, 478, 546, 582–585, 587, 847–848, 972, 1467, 1469, 1540, 1664, 1696, 1831–1832, 1838, 1847–1850, 1897, 1910, 1934–1935, 1941–1943, 1967, 1974–1977, 2030, 2051–2053, 2102, 2234, 2573–2574, 2578, 2583, 2586, 2594–2595, 2599–2601, 2623–2627, 2630, 2634–2635, 2645–2647, 2653, 2657–2658, 2758, 2955, 2958–2959, 3179, 3216, 3220, 3224, 3245, 3297, 3301, 3335, 3342–3343, 3351, 3355–3356, 3360, 3367, 3686–3687, 3705–3706, 3741, 3957, 3961, 3982, 4033, 4040–4041
openhands-sdk/openhands/sdk/security/grayswan
   analyzer.py1251290%130–131, 145, 226–228, 268, 279–280, 301–303
TOTAL57800932084% 

fork() copied the events but not the hook verdicts recorded against
them. A pending action that a PreToolUse hook had denied was executed
when the fork ran, and a user message that a UserPromptSubmit hook had
blocked was sent to the model, while the source did neither. This is on
main too; it is the same gap as the confirmation policy, a safety
decision that a fork silently dropped, so it is closed here.

The fork takes over `blocked_actions` and `blocked_messages` for the
events it copies, and `last_user_message_id` (the last user message it
actually has, for a fork of an earlier branch point).

Found in review of this stack. Each new test fails without the change:
the denied command runs, the blocked prompt reaches the model.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
`api_key` was a bare SecretStr with no serializer, so a saved
conversation state always held the redaction placeholder for it, even
with a cipher, and loading turned that placeholder into the key: the
analyzer sent `Authorization: Bearer **********`. Only the
GRAYSWAN_API_KEY environment variable hid this.

The field now goes through `serialize_secret` / `validate_secret`: with
a cipher the key is encrypted and comes back; without one it loads as
no key, and the analyzer reports UNKNOWN as it does when none is set.

This matters for this stack because a fork, and from the last PR every
reload, takes the analyzer from base_state.json alone.

Found in review of this stack. Both new tests fail without the change.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
Brings in the review fixes made on part 1: the shorter ACP masker
comment, the masked string form of state update events with its test,
and the feature-map note about changes made during a step.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…eps its secrets

Two properties of the registry copy that had no test:

- Replacing a secret on the fork must not change what the source masks.
  With a registry copied by `model_copy`, as #4813 first did, the two
  share one masking cache and the source stops masking its own value
  once the fork has resolved the new one. The test fails on that copy.
- `fork(agent=...)` with an agent that brings secrets in its context has
  them next to the ones copied from the source.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…-autosave' into claude/openhands-sdk-dual-source-truth-qp1x6r-2-fork-keeps-state
…-autosave' into claude/openhands-sdk-dual-source-truth-qp1x6r-2-fork-keeps-state
The helper that copies a model through its own dump and validate, with
credentials kept, moves from local_conversation.py to
utils/pydantic_secrets.py as copy_keeping_secrets. The agent-server makes
the same kind of copy for a child conversation and can import it instead
of spelling the round trip out again.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…-autosave' into claude/openhands-sdk-dual-source-truth-qp1x6r-2-fork-keeps-state

# Conflicts:
#	openhands-sdk/openhands/sdk/conversation/impl/local_conversation.py
…-autosave' into claude/openhands-sdk-dual-source-truth-qp1x6r-2-fork-keeps-state
@neubig

neubig commented Nov 13, 2026

Copy link
Copy Markdown
Member

Heads-up for the stack: the #5465 part of this PR (cipher + secret registry on fork) comes from #5765, which is being driven to merge on its own now (merged with current main, before/after Agent Canvas evidence added, re-review requested). Once #5765 lands, this PR's diff reduces to the confirmation-policy and security-analyzer copy, so it may be worth dropping Fixes #5465 from its description to avoid a double close-reference.

This comment/PR was created by an AI agent (OpenHands) on behalf of Graham Neubig.

…-autosave' into claude/openhands-sdk-dual-source-truth-qp1x6r-2-fork-keeps-state
…-autosave' into claude/openhands-sdk-dual-source-truth-qp1x6r-2-fork-keeps-state
Part of the fork's state was the hooks' verdicts on the events it
copies, but not the hooks. A pending action the source's PreToolUse
hook had denied stayed denied in the fork, and the next action the fork
asked for ran with no hook at all. The fork is now built with the
caller's own hook_config. The plugin hooks are not passed along: the
fork loads the plugins itself, and the merged config would register
them twice.

Hooks of a plugin the source loaded at runtime are still not carried.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
…-autosave' into claude/openhands-sdk-dual-source-truth-qp1x6r-2-fork-keeps-state
…-autosave' into claude/openhands-sdk-dual-source-truth-qp1x6r-2-fork-keeps-state

# Conflicts:
#	.agents/skills/verify-agent-server/references/feature-map/F10-fork-navigate-condense-ask.md
#	.agents/skills/verify-sdk/references/feature-map/README.md
…-autosave' into claude/openhands-sdk-dual-source-truth-qp1x6r-2-fork-keeps-state

main at 3d638f2 brings #5765 (the fork's cipher and secret registry) and
the SecretField refactor. What part 2 still adds to fork() stays on top of
main's version: the policy, the analyzer copy, the hooks and their
verdicts, invoked_skills, and one copy helper for the agent, the analyzer
and the secret sources. The helper builds its context with
client_context, and GraySwanAnalyzer.api_key is declared with SecretField
instead of the deprecated validator pair.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Agent-server conversation start benchmark

Base claude/openhands-sdk-dual-source-truth-qp1x6r-1-secrets-auto vs this PR merge c831065020, measured in the same job on the same runner, interleaved: 5 server starts per side and scenario. Medians with 95% bootstrap CIs. The LLM is a mock that answers instantly, so this is agent-server overhead only.

base this PR change
Create conversation (warm) 0.090 s [0.073, 0.111] 0.080 s [0.078, 0.095] -11% ≈ same
↳ first agent event 0.127 s [0.108, 0.148] 0.113 s [0.109, 0.134] -11% ≈ same
First create right after server start 0.210 s [0.085, 0.356] 0.214 s [0.084, 0.317] +2% ≈ same
↳ first agent event 0.461 s [0.243, 0.632] 0.363 s [0.243, 0.640] -21% ≈ same
First create 3 s after server start 0.096 s [0.087, 0.152] 0.122 s [0.098, 0.227] +27% ≈ same
↳ first agent event 0.266 s [0.220, 0.379] 0.317 s [0.250, 0.602] +19% ≈ same
Create in a new git worktree (warm, 0.5 s git fetch) 0.615 s [0.605, 0.631] 0.616 s [0.607, 0.635] +0% ≈ same
↳ first agent event 0.653 s [0.637, 0.666] 0.648 s [0.641, 0.674] -1% ≈ same
Worktree creates that stalled the server > 20 ms 0/20 (0%) 1/20 (5%) +5 pts ≈ same
Create with one stdio MCP server (warm, 3 s startup) 1.088 s [1.083, 1.113] 1.089 s [1.082, 1.096] +0% ≈ same
↳ first agent event 3.595 s [3.551, 3.665] 3.588 s [3.573, 3.663] -0% ≈ same
Creates that stalled the server > 20 ms 1/35 (3%) 0/35 (0%) -3 pts ≈ same
Server startup (/alive) 3.390 s [3.155, 3.687] 3.463 s [3.104, 4.147] +2% ≈ same
How this is measured

Each create is a Canvas-shaped POST /api/conversations (agent profile, worktree: true, first message included, so the request waits for agent startup). A side task polls /alive every 20 ms to catch event-loop stalls. Scenarios, each on its own server starts:

  • default: a fresh directory per create, 8 creates per start.
  • pause: the first create waits 3 s after startup, as a user opening the UI would.
  • worktree: every create branches a new worktree from a local repository whose origin answers fetches after 0.5 s.
  • mcp: the agent profile enables one stdio MCP server that starts after 3 s, like an npx/uvx launch.

A timing is flagged when the 95% CIs do not overlap and the medians differ by at least 50 ms, or when a row with at least 8 samples per side moves by at least 150 ms and 25%. A create stalled the server when its slowest /alive took over 20 ms; stall rows are flagged when the share of stalled creates moves by at least 20 points with a Fisher exact p below 0.01. The first-create rows have one sample per server start, so they are flagged only on a clear change. A "↳ first agent event" row times the create above it until the agent's first event arrives on a socket opened after the create returns, as Agent Canvas does, so it includes the connect and replay. Runner: blacksmith-2vcpu-ubuntu-2404-jijqqio6evvpzgak, 178 environment variables. Run it locally with python scripts/agent_server_benchmarks/conversation_start.py --help.

The fork copied every secret source through a dump and validate round
trip. A custom `SecretSource` that keeps its value in a private attribute
or in a field the dump leaves out came out of that with no value (and a
required field of that kind made `fork()` raise), where main's deep copy
(#5765) keeps it. The same held for such a source in the agent's context.

Secret sources are deep-copied again, as on main. A source the deep copy
cannot handle (one that holds a lock or an open client, which makes
`fork()` raise on main) is rebuilt from its fields instead. Tests for
each case; each fails with only the other half in place.

Found by the review of the merge with main.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
@enyst enyst changed the title fix(sdk): a fork keeps the source's confirmation policy, security analyzer and secrets fix(sdk): a fork keeps the source's confirmation policy, security analyzer and hooks Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants