Repository navigation
Description
I'm an AI agent helping Engel Nyst (@enyst) with project work.
Is there an existing issue for the same bug?
- I have searched existing issues and this is not a duplicate.
Related but different: #6487 (ask_agent usage double-counted after a resume, fixed by #6690), #6646 (ACP ask_agent loses the fork's cost), #5639 (LLM switched in before the first run is missing from the stats), #6432 (size of base_state.json writes), #6741 / #6823 (full_state frame size).
Bug Description
A side question (POST /api/conversations/{id}/ask_agent, LocalConversation.ask_agent, /btw in Agent Canvas) records its usage in the conversation's stats under the usage id ask-agent-llm, but only in memory. base_state.json is not rewritten until some later, unrelated state change (the next run, a pause, a tag edit).
Since #6690 the agent server also pushes a ConversationStateUpdateEvent with key stats on the WebSocket for that call, so clients now see two different answers:
- the socket event carries
ask-agent-llmat once; GET /api/conversations/{id}(answered frombase_state.json) does not;- a server restarted before the next state change has lost the usage for good.
In Agent Canvas the cost meter rises after /btw, drops back on page reload, and after a server restart the last side questions are missing from the total.
The cause is in the SDK: ask_agent changes state.stats in place and assigns no state field, and ConversationState saves only on a field assignment. The same program without a server shows it (see Minimal Code Sample).
Expected Behavior
When ask_agent returns, the usage it recorded is in base_state.json. GET /api/conversations/{id} then agrees with the stats event the socket already sent, and the usage is still there after a server restart or after the conversation is resumed with the SDK. ask_agent keeps answering at once while a run is in progress.
Actual Behavior
On main at 3c66af1 (prompt/completion tokens per usage id; the stub reports 1111/11, 2222/22 and 3333/33 for the three side questions):
== A. side question on a finished conversation ==
A1 after turn 1 (finished) REST GET stats: default=10/5
ask 1 -> SIDE1 (stub usage 1111/11)
A2 after ask 1 socket full_state on connect: default=10/5
A2 after ask 1 socket stats events: 1; last: ask-agent-llm=1111/11 default=10/5
A2 after ask 1 REST GET stats: default=10/5
A2 after ask 1 base_state.json: default=10/5
-- server restarted (graceful) --
A3 after restart REST GET stats: default=10/5
A3 after restart base_state.json: default=10/5
ask 2 -> SIDE2 (stub usage 2222/22)
A4 after restart + ask 2 socket full_state on connect: default=10/5
A4 after restart + ask 2 socket stats events: 1; last: ask-agent-llm=2222/22 default=10/5
A4 after restart + ask 2 REST GET stats: default=10/5
A4 after restart + ask 2 base_state.json: default=10/5
A5 after turn 2 (finished) REST GET stats: ask-agent-llm=2222/22 default=20/10
A5 after turn 2 (finished) base_state.json: ask-agent-llm=2222/22 default=20/10
ask 3 -> SIDE3 (stub usage 3333/33)
A6 after ask 3 socket full_state on connect: ask-agent-llm=2222/22 default=20/10
A6 after ask 3 socket stats events: 1; last: ask-agent-llm=5555/55 default=20/10
A6 after ask 3 REST GET stats: ask-agent-llm=2222/22 default=20/10
A6 after ask 3 base_state.json: ask-agent-llm=2222/22 default=20/10
-- server restarted (graceful) --
A7 after restart REST GET stats: ask-agent-llm=2222/22 default=20/10
- Ask 1 (1111 tokens) is on the socket, not on REST or disk, and is gone after the restart: ask 2 starts again from 0.
- Ask 2 reaches REST only after the next run saves the state.
- Ask 3 (cumulative 5555 on the socket) is lost by the second restart: REST stays at 2222.
While a run is blocked on its model:
== B. side question while a run is blocked on its model ==
ask B -> SIDEB in 406 ms (stub usage 4444/44); status: running
B1 after ask, still running socket full_state on connect: default=0/0
B1 after ask, still running socket stats events: 1; last: ask-agent-llm=4444/44 default=0/0
B1 after ask, still running REST GET stats: default=0/0
B1 after ask, still running base_state.json: default=0/0
B2 after interrupt (paused) REST GET stats: ask-agent-llm=4444/44 default=0/0
B2 after interrupt (paused) base_state.json: ask-agent-llm=4444/44 default=0/0
Steps to Reproduce
Checkout 3c66af1, uv sync --dev. No model key is needed: the provider is the verify skill's scripted llm-stub.
export PATH=$PWD/.agents/skills/verify-agent-server/scripts:$PATH
export AGENT_SERVER_VERIFY_RUN=$(control-agent-server launch --new --print-run)
control-agent-server doctor
STUB=$(control-agent-server fixture llm-stub --name qa-ask \
--step 'tool:finish:{"message":"DONE1"}' --step 'reply:SIDE1@usage=1111,11' --print-path)
AGENT="{\"llm\": {\"model\": \"openai/qa-stub\", \"api_key\": \"qa-stub-key\", \"base_url\": \"$STUB/v1\", \"num_retries\": 0}, \"tools\": []}"
A=$(control-agent-server conversation start --body-json "{\"agent\": $AGENT}" --no-autotitle --prompt QA_TURN1 --wait --print-id)
control-agent-server ws start /sockets/events/$A --name live --duration 120
control-agent-server api POST "/api/conversations/$A/ask_agent" --json '{"question": "QA_SIDE_Q1"}' --expect 200
# socket: one stats event that has ask-agent-llm
control-agent-server ws stop live --contains ask-agent-llm --expect-min 1 --wait 5
# REST and disk: fail on main (no ask-agent-llm)
control-agent-server api GET "/api/conversations/$A" --check stats.usage_to_metrics.ask-agent-llm.accumulated_token_usage.prompt_tokens eq 1111
control-agent-server state cat base_state.json --conversation "$A" --contains ask-agent-llm
# after a restart: still fails on main, and the usage cannot come back
control-agent-server restart
control-agent-server api GET "/api/conversations/$A" --check stats.usage_to_metrics.ask-agent-llm.accumulated_token_usage.prompt_tokens eq 1111
control-agent-server stopThe full sequence used for the tables above (three asks, two restarts, a run in between, an ask during a blocked run, and a condense for comparison) is https://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/bugs/2026-11-14/ask-agent-stats/repro.sh with its formatter https://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/bugs/2026-11-14/ask-agent-stats/summ.py.
Acceptance Criteria
- After
POST /api/conversations/{id}/ask_agentreturns on an idle or finished conversation,GET /api/conversations/{id}hasstats.usage_to_metrics["ask-agent-llm"]with the tokens of that call, equal to the socket'sstatsevent, with no run or other state change in between. -
base_state.jsonholds the same values, and they are returned after a server restart; a further ask after the restart adds to them. -
LocalConversation.ask_agentwith apersistence_dir: the usage is inbase_state.jsonwhen the call returns, and a conversation resumed from that directory counts it. - An ask during a run still returns without waiting for the step (the existing
F10.ask-agent-runningbehavior), and its usage is on REST while the run is blocked on its model. - The save goes through
ConversationState's own save and lock; there is no second writer ofbase_state.json. - Regression tests cover the SDK case (saved and resumed) and the case where another thread holds the state lock.
Installation Method
uv sync --dev in a checkout of the repository
If you selected "Other", please specify
No response
SDK Version
main at 3c66af1
Version Confirmation
- I have confirmed this bug exists on the LATEST version of OpenHands SDK
Python Version
3.13
Model Name (if applicable)
None (scripted provider stub; first seen with a real model in Agent Canvas)
Operating System
Linux
Logs and Error Messages
No response
Minimal Code Sample
SDK only, no server. The provider call is replaced by a scripted reply that reports 100 prompt tokens.
"""SDK twin: is ask_agent usage in base_state.json after the call, and after a reload?
Key-free: the provider call (litellm completion) is replaced by a scripted reply
that reports 100 prompt tokens per call.
"""
import json
import sys
import tempfile
from pathlib import Path
from unittest.mock import patch
from litellm.types.utils import Choices, Message as LiteLLMMessage, ModelResponse, Usage
from pydantic import SecretStr
from openhands.sdk import LLM, Agent
from openhands.sdk.conversation.impl.local_conversation import LocalConversation
def respond(*args, **kwargs):
return ModelResponse(
id="resp",
choices=[Choices(index=0, finish_reason="stop",
message=LiteLLMMessage(role="assistant", content="done"))],
model="gpt-4o",
usage=Usage(prompt_tokens=100, completion_tokens=1, total_tokens=101),
)
def tokens(stats: dict) -> dict:
return {k: v["accumulated_token_usage"]["prompt_tokens"]
for k, v in stats["usage_to_metrics"].items()}
def on_disk(conv) -> dict:
path = Path(conv.state.persistence_dir) / "base_state.json"
return tokens(json.loads(path.read_text())["stats"])
def in_memory(conv) -> dict:
return tokens(conv.conversation_stats.model_dump())
def make(root: Path, conversation_id=None):
return LocalConversation(
agent=Agent(llm=LLM(model="gpt-4o", api_key=SecretStr("k"), usage_id="agent"), tools=[]),
workspace=root / "ws", persistence_dir=root / "persist",
conversation_id=conversation_id, visualizer=None,
)
with patch("openhands.sdk.llm.llm.litellm_completion", side_effect=respond):
root = Path(tempfile.mkdtemp())
conv = make(root)
conv.send_message("hi")
conv.run()
print("after run memory", in_memory(conv), "disk", on_disk(conv))
conv.ask_agent("what are you doing?")
mem, disk = in_memory(conv), on_disk(conv)
print("after ask memory", mem, "disk", disk)
# Reload from what is on disk now (no close(), as after a crash or a second reader).
resumed = make(root, conv.id)
reloaded = in_memory(resumed)
print("after reload memory", reloaded)
ok = disk.get("ask-agent-llm") == 100 and reloaded.get("ask-agent-llm") == 100
print("VERDICT:", "ask usage is durable" if ok else "ask usage is NOT in base_state.json / lost on reload")
sys.exit(0 if ok else 1)Output on 3c66af1 (exit 1):
after run memory {'agent': 100} disk {'agent': 100}
after ask memory {'agent': 100, 'ask-agent-llm': 100} disk {'agent': 100}
after reload memory {'agent': 100}
VERDICT: ask usage is NOT in base_state.json / lost on reload
Evidence and Additional Context
Regression verdict: the missing save is not a regression. The same programs on fb75ae9 (before #6690) show the same REST, disk and restart values:
A2 after ask 1 socket full_state on connect: default=10/5
A2 after ask 1 socket stats events: 0
A2 after ask 1 REST GET stats: default=10/5
A2 after ask 1 base_state.json: default=10/5
A3 after restart REST GET stats: default=10/5
A3 after restart base_state.json: default=10/5
B1 after ask, still running socket full_state on connect: default=0/0
B1 after ask, still running socket stats events: 0
B1 after ask, still running REST GET stats: default=0/0
B1 after ask, still running base_state.json: default=0/0
What #6690 (4ca7853) changed is the socket: before it, an ask on a fresh conversation sent no stats event at all, so every client saw the same stale number; now the socket is right and REST is behind. (The doubled default/ask-agent-llm values that the older commit shows after a restart are #6487, which #6690 fixed.)
Other socket-pushed stats do not have this lag: POST .../condense on a finished conversation records a Condensation event, which moves the conversation head and saves the state, so REST and disk have the condenser usage at once on both commits (section C of the transcripts).
Transcripts:
ˋhttps://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/bugs/2026-11-14/ask-agent-stats/target-run1.txtˋ:agent server at 3c66af1https://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/bugs/2026-11-14/ask-agent-stats/base-run1.txt: agent server at fb75ae9ˋhttps://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/bugs/2026-11-14/ask-agent-stats/sdk-twin-target.txtˋ,ˋhttps://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/bugs/2026-11-14/ask-agent-stats/sdk-twin-base.txtˋ:the SDK program at both commitsˋhttps://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/bugs/2026-11-14/ask-agent-stats/steps-target-and-fix.txtˋ:the steps above, check by check, at 3c66af1 and with the proposed fixhttps://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/bugs/2026-11-14/ask-agent-stats/repro.sh,https://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/bugs/2026-11-14/ask-agent-stats/summ.py,https://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/bugs/2026-11-14/ask-agent-stats/steps.sh,https://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/bugs/2026-11-14/ask-agent-stats/sdk_twin.py: the programs
Generated by Claude Code
OpenHands AI triage
The following comments and acceptance criteria were added by the OpenHands AI agent.
Triage
Confirmed against main: LocalConversation.ask_agent runs its reply on the ask-agent-llm registry LLM, and Telemetry.on_response records usage by mutating the Metrics object stored in state.stats in place — no state field is assigned. ConversationState.__setattr__ autosaves base_state.json only on a public-field assignment, so the usage reaches only the agent-server's stats-update callback (the stats WebSocket event pushed since #6690) and stays out of base_state.json, which is what GET /api/conversations/{id} and a restart or resume read. Pre-#6690 code shows the same REST/disk lag; only the socket event was also missing then.
Bounded scope: make the usage recorded by ask_agent durable through ConversationState's existing save path (autosave/deferred save under its FIFOLock), so REST, base_state.json, and a restarted or resumed conversation agree with the stats socket event. No new persistence mechanism and no second writer of base_state.json.
Non-goals: no redesign of the stats-update callback, no change to the ConversationStateUpdateEvent shape, no backfill for usage already lost before the fix, no ACP fork cost accounting (#6646), no base_state.json size changes (#6432, #6741).
Acceptance Criteria
- After
POST /api/conversations/{id}/ask_agentreturns on an idle or finished conversation,GET /api/conversations/{id}returnsstats.usage_to_metrics["ask-agent-llm"]with the tokens of that call, matching thestatsConversationStateUpdateEventthe socket already pushed, with no run or other state change in between. -
base_state.jsonin the conversation's persistence directory contains the sameask-agent-llmusage when the call returns, and those values are returned after a graceful server restart; a further ask after the restart adds to them instead of restarting from 0. -
LocalConversation.ask_agentwith apersistence_dir: the usage is inbase_state.jsonwhen the call returns, and a conversation resumed from that directory (new instance, same conversation id) reports the recorded usage in its stats. - An ask issued while a run is blocked on its model still returns without waiting for the step, leaves execution status
running(existingF10.ask-agent-runningbehavior), and its usage is visible onGET /api/conversations/{id}while the run is still in progress. - The save goes through
ConversationState's own save and locking mechanism (base_state.jsonkeeps a single writer); no second persistence writer is introduced for stats. - Tests cover the SDK case (usage saved on call return and counted after resume) and the case where the state lock is held by the run thread while
ask_agentrecords usage, showing the usage is saved without deadlocking or corrupting the deferred save. - The docstring of
LocalConversation.ask_agent(currently "does not modify, persist, or become part of the conversation state") and any related SDK docs forask_agentno longer contradict the durable stats behavior.
Activity