Skip to content

Aggregate cache-restored dict metadata with live models after a resume - #103

Open
xh123637 wants to merge 1 commit into
ArtificialAnalysis:mainfrom
xh123637:fix/92-aggregate-mixed-metadata
Open

xh123637 wants to merge 1 commit into
ArtificialAnalysis:mainfrom
xh123637:fix/92-aggregate-mixed-metadata

Conversation

@xh123637

Copy link
Copy Markdown

Summary

Fixes #92. A session resumed with session(resume=True, cache_on_interrupt=True) crashed at run end: the per-tool metadata restored from the JSON cache is plain dicts, the new run appends live metadata models for the same tools, and aggregating the mixed list raised TypeError: unsupported operand type(s) for +: 'dict' and 'ToolUseCountMetadata' — after the agent had already finished its work.

Root cause

  • CacheState.to_dict serializes metadata items with model_dump(mode="json") (core/cache.py), and CacheState.from_dict puts the resulting dicts back into run_metadata_by_turn unchanged — by design: the cache layer is type-agnostic, and the existing tests (test_cache_state_preserves_run_metadata_by_turn, test_v01_cache_state_preserves_aggregate_run_metadata) pin that restored metadata stays as plain dicts. The serialized form carries no type information, and there is no registry mapping a tool name to its metadata class, so restore-time rehydration cannot know which model each dict belongs to (ToolUseCountMetadata, SubAgentMetadata, the browser tools' SearchMetadata/NavigateMetadata/InputTextMetadata, or a user's own Addable model).
  • At run end, _merge_run_metadata flattens the per-turn lists and the logger calls aggregate_metadata, whose _aggregate_list (core/models.py) only special-cases dict + dict; a mixed pair fell through to aggregated + m and raised.

Changes

_aggregate_list now tolerates mixed lists: when one side is a dict and the other is a live model, the dict is validated into the live item's type (type(m).model_validate(...)) and the pair is combined with the model's own __add__, in either order. Lists are per-tool, so the live sibling is always the right schema. This:

  • works for every Addable metadata model, including custom ones and SubAgentMetadata (whose nested run_metadata lists aggregate through the same function);
  • preserves the cached counts instead of dropping the dict entries;
  • needs no cache-format change, and also fixes caches already written by released versions, since those dicts are exactly what this handles;
  • leaves dict-only and model-only lists behaving exactly as before.

If a cached dict genuinely cannot validate into the sibling's type (e.g. the metadata schema changed across versions), model_validate raises a ValidationError naming the field, instead of the opaque TypeError.

Tests

I wrote the tests first and checked that they fail without the fix. With src/stirrup/core/models.py reverted, all four new tests in tests/test_agent.py fail with the reported TypeError (including a dict + SubAgentMetadata variant); with the fix, all pass:

  • test_resumed_run_metadata_aggregates_cached_dicts_with_live_models — the reported flow end to end: a CacheState round trip through JSON, a resumed turn appending a live ToolUseCountMetadata, and the run-end aggregation returning {"my_tool": {"num_uses": 5}} (cached 2 + live 3).
  • test_aggregate_metadata_combines_dicts_and_models_in_either_order — dict-first, model-first, and two dicts followed by a model.
  • test_aggregate_metadata_rehydrates_dicts_into_the_live_model_type — a custom metadata model (stand-in for tool-specific types like SearchMetadata), proving dicts take the live sibling's type rather than a hard-coded one.
  • test_aggregate_metadata_combines_cached_sub_agent_dicts_with_live_models — a cache-restored SubAgentMetadata dict aggregated with a live one, including the nested per-tool lists.

Local runs (Python 3.13, uv sync --locked --dev --extra mcp; CI syncs --all-extras):

  • tests/test_agent.py + tests/test_logging.py: 63 passed.
  • Full suite (pytest --ignore=tests/test_browser_use.py -m "not docker and not e2b and not browser"): 322 passed / 32 failed with the fix, against 34 failed on unmodified main in the same sandbox. The with-fix failure list is a strict subset of the baseline list — the 32 shared failures are the chat-completions/open-responses client tests failing inside httpx URL parsing in this sandbox (proxy environment), pre-existing and unrelated; the 2 baseline-only failures are flaky environment tests (local cross-environment write, web-fetch IPv6 literal) that passed in the with-fix run.
  • ruff format --check . and ruff check . pass; ty check reports no diagnostics in the changed files (the 22 whole-repo diagnostics are identical on unmodified main and sit in optional-extra modules whose dependencies were not installed in this sync).

run_metadata restored from the JSON cache holds plain dicts, while a resumed
run appends live metadata models for the same tools. _aggregate_list only
handled dict+dict, so the mixed list raised TypeError at run end. Rehydrate
dict entries into the type of the live model in the same list so __add__
applies, in either order.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Resumed session crashes at run end: aggregate_metadata mixes cached JSON dicts with live ToolUseCountMetadata

1 participant