Skip to content

[Bug]: ask_agent usage is not saved: REST and a restarted server miss what the stats socket event already reported #6844

Description

@enyst

I'm an AI agent helping Engel Nyst (@enyst) with project work.

Is there an existing issue for the same bug?

  • I have searched existing issues and this is not a duplicate.

Related but different: #6487 (ask_agent usage double-counted after a resume, fixed by #6690), #6646 (ACP ask_agent loses the fork's cost), #5639 (LLM switched in before the first run is missing from the stats), #6432 (size of base_state.json writes), #6741 / #6823 (full_state frame size).

Bug Description

A side question (POST /api/conversations/{id}/ask_agent, LocalConversation.ask_agent, /btw in Agent Canvas) records its usage in the conversation's stats under the usage id ask-agent-llm, but only in memory. base_state.json is not rewritten until some later, unrelated state change (the next run, a pause, a tag edit).

Since #6690 the agent server also pushes a ConversationStateUpdateEvent with key stats on the WebSocket for that call, so clients now see two different answers:

  • the socket event carries ask-agent-llm at once;
  • GET /api/conversations/{id} (answered from base_state.json) does not;
  • a server restarted before the next state change has lost the usage for good.

In Agent Canvas the cost meter rises after /btw, drops back on page reload, and after a server restart the last side questions are missing from the total.

The cause is in the SDK: ask_agent changes state.stats in place and assigns no state field, and ConversationState saves only on a field assignment. The same program without a server shows it (see Minimal Code Sample).

Expected Behavior

When ask_agent returns, the usage it recorded is in base_state.json. GET /api/conversations/{id} then agrees with the stats event the socket already sent, and the usage is still there after a server restart or after the conversation is resumed with the SDK. ask_agent keeps answering at once while a run is in progress.

Actual Behavior

On main at 3c66af1 (prompt/completion tokens per usage id; the stub reports 1111/11, 2222/22 and 3333/33 for the three side questions):

== A. side question on a finished conversation ==
A1 after turn 1 (finished)         REST GET stats:    default=10/5
ask 1 -> SIDE1   (stub usage 1111/11)
A2 after ask 1                     socket full_state on connect: default=10/5
A2 after ask 1                     socket stats events: 1; last: ask-agent-llm=1111/11 default=10/5
A2 after ask 1                     REST GET stats:    default=10/5
A2 after ask 1                     base_state.json:   default=10/5
-- server restarted (graceful) --
A3 after restart                   REST GET stats:    default=10/5
A3 after restart                   base_state.json:   default=10/5
ask 2 -> SIDE2   (stub usage 2222/22)
A4 after restart + ask 2           socket full_state on connect: default=10/5
A4 after restart + ask 2           socket stats events: 1; last: ask-agent-llm=2222/22 default=10/5
A4 after restart + ask 2           REST GET stats:    default=10/5
A4 after restart + ask 2           base_state.json:   default=10/5
A5 after turn 2 (finished)         REST GET stats:    ask-agent-llm=2222/22 default=20/10
A5 after turn 2 (finished)         base_state.json:   ask-agent-llm=2222/22 default=20/10
ask 3 -> SIDE3   (stub usage 3333/33)
A6 after ask 3                     socket full_state on connect: ask-agent-llm=2222/22 default=20/10
A6 after ask 3                     socket stats events: 1; last: ask-agent-llm=5555/55 default=20/10
A6 after ask 3                     REST GET stats:    ask-agent-llm=2222/22 default=20/10
A6 after ask 3                     base_state.json:   ask-agent-llm=2222/22 default=20/10
-- server restarted (graceful) --
A7 after restart                   REST GET stats:    ask-agent-llm=2222/22 default=20/10
  • Ask 1 (1111 tokens) is on the socket, not on REST or disk, and is gone after the restart: ask 2 starts again from 0.
  • Ask 2 reaches REST only after the next run saves the state.
  • Ask 3 (cumulative 5555 on the socket) is lost by the second restart: REST stays at 2222.

While a run is blocked on its model:

== B. side question while a run is blocked on its model ==
ask B -> SIDEB in 406 ms   (stub usage 4444/44); status: running
B1 after ask, still running        socket full_state on connect: default=0/0
B1 after ask, still running        socket stats events: 1; last: ask-agent-llm=4444/44 default=0/0
B1 after ask, still running        REST GET stats:    default=0/0
B1 after ask, still running        base_state.json:   default=0/0
B2 after interrupt (paused)        REST GET stats:    ask-agent-llm=4444/44 default=0/0
B2 after interrupt (paused)        base_state.json:   ask-agent-llm=4444/44 default=0/0

Steps to Reproduce

Checkout 3c66af1, uv sync --dev. No model key is needed: the provider is the verify skill's scripted llm-stub.

export PATH=$PWD/.agents/skills/verify-agent-server/scripts:$PATH
export AGENT_SERVER_VERIFY_RUN=$(control-agent-server launch --new --print-run)
control-agent-server doctor
STUB=$(control-agent-server fixture llm-stub --name qa-ask \
  --step 'tool:finish:{"message":"DONE1"}' --step 'reply:SIDE1@usage=1111,11' --print-path)
AGENT="{\"llm\": {\"model\": \"openai/qa-stub\", \"api_key\": \"qa-stub-key\", \"base_url\": \"$STUB/v1\", \"num_retries\": 0}, \"tools\": []}"
A=$(control-agent-server conversation start --body-json "{\"agent\": $AGENT}" --no-autotitle --prompt QA_TURN1 --wait --print-id)

control-agent-server ws start /sockets/events/$A --name live --duration 120
control-agent-server api POST "/api/conversations/$A/ask_agent" --json '{"question": "QA_SIDE_Q1"}' --expect 200
# socket: one stats event that has ask-agent-llm
control-agent-server ws stop live --contains ask-agent-llm --expect-min 1 --wait 5
# REST and disk: fail on main (no ask-agent-llm)
control-agent-server api GET "/api/conversations/$A" --check stats.usage_to_metrics.ask-agent-llm.accumulated_token_usage.prompt_tokens eq 1111
control-agent-server state cat base_state.json --conversation "$A" --contains ask-agent-llm
# after a restart: still fails on main, and the usage cannot come back
control-agent-server restart
control-agent-server api GET "/api/conversations/$A" --check stats.usage_to_metrics.ask-agent-llm.accumulated_token_usage.prompt_tokens eq 1111
control-agent-server stop

The full sequence used for the tables above (three asks, two restarts, a run in between, an ask during a blocked run, and a condense for comparison) is https://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/bugs/2026-11-14/ask-agent-stats/repro.sh with its formatter https://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/bugs/2026-11-14/ask-agent-stats/summ.py.

Acceptance Criteria

  • After POST /api/conversations/{id}/ask_agent returns on an idle or finished conversation, GET /api/conversations/{id} has stats.usage_to_metrics["ask-agent-llm"] with the tokens of that call, equal to the socket's stats event, with no run or other state change in between.
  • base_state.json holds the same values, and they are returned after a server restart; a further ask after the restart adds to them.
  • LocalConversation.ask_agent with a persistence_dir: the usage is in base_state.json when the call returns, and a conversation resumed from that directory counts it.
  • An ask during a run still returns without waiting for the step (the existing F10.ask-agent-running behavior), and its usage is on REST while the run is blocked on its model.
  • The save goes through ConversationState's own save and lock; there is no second writer of base_state.json.
  • Regression tests cover the SDK case (saved and resumed) and the case where another thread holds the state lock.

Installation Method

uv sync --dev in a checkout of the repository

If you selected "Other", please specify

No response

SDK Version

main at 3c66af1

Version Confirmation

  • I have confirmed this bug exists on the LATEST version of OpenHands SDK

Python Version

3.13

Model Name (if applicable)

None (scripted provider stub; first seen with a real model in Agent Canvas)

Operating System

Linux

Logs and Error Messages

No response

Minimal Code Sample

SDK only, no server. The provider call is replaced by a scripted reply that reports 100 prompt tokens.

"""SDK twin: is ask_agent usage in base_state.json after the call, and after a reload?

Key-free: the provider call (litellm completion) is replaced by a scripted reply
that reports 100 prompt tokens per call.
"""
import json
import sys
import tempfile
from pathlib import Path
from unittest.mock import patch

from litellm.types.utils import Choices, Message as LiteLLMMessage, ModelResponse, Usage
from pydantic import SecretStr

from openhands.sdk import LLM, Agent
from openhands.sdk.conversation.impl.local_conversation import LocalConversation


def respond(*args, **kwargs):
    return ModelResponse(
        id="resp",
        choices=[Choices(index=0, finish_reason="stop",
                         message=LiteLLMMessage(role="assistant", content="done"))],
        model="gpt-4o",
        usage=Usage(prompt_tokens=100, completion_tokens=1, total_tokens=101),
    )


def tokens(stats: dict) -> dict:
    return {k: v["accumulated_token_usage"]["prompt_tokens"]
            for k, v in stats["usage_to_metrics"].items()}


def on_disk(conv) -> dict:
    path = Path(conv.state.persistence_dir) / "base_state.json"
    return tokens(json.loads(path.read_text())["stats"])


def in_memory(conv) -> dict:
    return tokens(conv.conversation_stats.model_dump())


def make(root: Path, conversation_id=None):
    return LocalConversation(
        agent=Agent(llm=LLM(model="gpt-4o", api_key=SecretStr("k"), usage_id="agent"), tools=[]),
        workspace=root / "ws", persistence_dir=root / "persist",
        conversation_id=conversation_id, visualizer=None,
    )


with patch("openhands.sdk.llm.llm.litellm_completion", side_effect=respond):
    root = Path(tempfile.mkdtemp())
    conv = make(root)
    conv.send_message("hi")
    conv.run()
    print("after run       memory", in_memory(conv), "disk", on_disk(conv))
    conv.ask_agent("what are you doing?")
    mem, disk = in_memory(conv), on_disk(conv)
    print("after ask       memory", mem, "disk", disk)
    # Reload from what is on disk now (no close(), as after a crash or a second reader).
    resumed = make(root, conv.id)
    reloaded = in_memory(resumed)
    print("after reload    memory", reloaded)
    ok = disk.get("ask-agent-llm") == 100 and reloaded.get("ask-agent-llm") == 100
    print("VERDICT:", "ask usage is durable" if ok else "ask usage is NOT in base_state.json / lost on reload")
    sys.exit(0 if ok else 1)

Output on 3c66af1 (exit 1):

after run       memory {'agent': 100} disk {'agent': 100}
after ask       memory {'agent': 100, 'ask-agent-llm': 100} disk {'agent': 100}
after reload    memory {'agent': 100}
VERDICT: ask usage is NOT in base_state.json / lost on reload

Evidence and Additional Context

Regression verdict: the missing save is not a regression. The same programs on fb75ae9 (before #6690) show the same REST, disk and restart values:

A2 after ask 1                     socket full_state on connect: default=10/5
A2 after ask 1                     socket stats events: 0
A2 after ask 1                     REST GET stats:    default=10/5
A2 after ask 1                     base_state.json:   default=10/5
A3 after restart                   REST GET stats:    default=10/5
A3 after restart                   base_state.json:   default=10/5
B1 after ask, still running        socket full_state on connect: default=0/0
B1 after ask, still running        socket stats events: 0
B1 after ask, still running        REST GET stats:    default=0/0
B1 after ask, still running        base_state.json:   default=0/0

What #6690 (4ca7853) changed is the socket: before it, an ask on a fresh conversation sent no stats event at all, so every client saw the same stale number; now the socket is right and REST is behind. (The doubled default/ask-agent-llm values that the older commit shows after a restart are #6487, which #6690 fixed.)

Other socket-pushed stats do not have this lag: POST .../condense on a finished conversation records a Condensation event, which moves the conversation head and saves the state, so REST and disk have the condenser usage at once on both commits (section C of the transcripts).

Transcripts:

  • ˋhttps://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/bugs/2026-11-14/ask-agent-stats/target-run1.txtˋ: agent server at 3c66af1
  • https://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/bugs/2026-11-14/ask-agent-stats/base-run1.txt: agent server at fb75ae9
  • ˋhttps://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/bugs/2026-11-14/ask-agent-stats/sdk-twin-target.txtˋ, ˋhttps://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/bugs/2026-11-14/ask-agent-stats/sdk-twin-base.txtˋ: the SDK program at both commits
  • ˋhttps://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/bugs/2026-11-14/ask-agent-stats/steps-target-and-fix.txtˋ: the steps above, check by check, at 3c66af1 and with the proposed fix
  • https://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/bugs/2026-11-14/ask-agent-stats/repro.sh, https://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/bugs/2026-11-14/ask-agent-stats/summ.py, https://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/bugs/2026-11-14/ask-agent-stats/steps.sh, https://github.com/enyst/openhands-evidence/blob/97e381e5686ca14c299b6e1474194dd05a73e78b/software-agent-sdk/bugs/2026-11-14/ask-agent-stats/sdk_twin.py: the programs

Generated by Claude Code


OpenHands AI triage

The following comments and acceptance criteria were added by the OpenHands AI agent.

Triage

Confirmed against main: LocalConversation.ask_agent runs its reply on the ask-agent-llm registry LLM, and Telemetry.on_response records usage by mutating the Metrics object stored in state.stats in place — no state field is assigned. ConversationState.__setattr__ autosaves base_state.json only on a public-field assignment, so the usage reaches only the agent-server's stats-update callback (the stats WebSocket event pushed since #6690) and stays out of base_state.json, which is what GET /api/conversations/{id} and a restart or resume read. Pre-#6690 code shows the same REST/disk lag; only the socket event was also missing then.

Bounded scope: make the usage recorded by ask_agent durable through ConversationState's existing save path (autosave/deferred save under its FIFOLock), so REST, base_state.json, and a restarted or resumed conversation agree with the stats socket event. No new persistence mechanism and no second writer of base_state.json.

Non-goals: no redesign of the stats-update callback, no change to the ConversationStateUpdateEvent shape, no backfill for usage already lost before the fix, no ACP fork cost accounting (#6646), no base_state.json size changes (#6432, #6741).

Acceptance Criteria

  • After POST /api/conversations/{id}/ask_agent returns on an idle or finished conversation, GET /api/conversations/{id} returns stats.usage_to_metrics["ask-agent-llm"] with the tokens of that call, matching the stats ConversationStateUpdateEvent the socket already pushed, with no run or other state change in between.
  • base_state.json in the conversation's persistence directory contains the same ask-agent-llm usage when the call returns, and those values are returned after a graceful server restart; a further ask after the restart adds to them instead of restarting from 0.
  • LocalConversation.ask_agent with a persistence_dir: the usage is in base_state.json when the call returns, and a conversation resumed from that directory (new instance, same conversation id) reports the recorded usage in its stats.
  • An ask issued while a run is blocked on its model still returns without waiting for the step, leaves execution status running (existing F10.ask-agent-running behavior), and its usage is visible on GET /api/conversations/{id} while the run is still in progress.
  • The save goes through ConversationState's own save and locking mechanism (base_state.json keeps a single writer); no second persistence writer is introduced for stats.
  • Tests cover the SDK case (usage saved on call return and counted after resume) and the case where the state lock is held by the run thread while ask_agent records usage, showing the usage is saved without deadlocking or corrupting the deferred save.
  • The docstring of LocalConversation.ask_agent (currently "does not modify, persist, or become part of the conversation state") and any related SDK docs for ask_agent no longer contradict the durable stats behavior.

Activity

  1. added
    priority:mediumFor bugs, a serious source of annoyance, but not blocking a large number of users.
    ready-for-devIssue meets development readiness criteria
    on Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority:mediumFor bugs, a serious source of annoyance, but not blocking a large number of users.ready-for-devIssue meets development readiness criteria

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions