Skip to content

feat(snapshot): report a save's file count and phase durations, and log each save - #697

Draft
andrewpark-scaleai wants to merge 2 commits into
mainfrom
andrew-park/rel-snapshot-save-telemetry
Draft

andrewpark-scaleai wants to merge 2 commits into
mainfrom
andrew-park/rel-snapshot-save-telemetry

Conversation

@andrewpark-scaleai

Copy link
Copy Markdown
Contributor

Why

The goal is that every failure of an agent run leaves enough in the logs to debug it, and that code heals what it can. When a snapshot_agent_state save is slow, agent-env's logs cannot say why. The agent logs its archive's file count and size, but that line stays in the sandbox. agent-env logs nothing about the save on success, and only the exception on a timeout. So a slow save cannot be told apart as "the workspace is big", "archiving is slow" or "the upload is slow" without reaching into a live sandbox.

What changed

  • ObjectSnapshotSaveResponse gains an optional stats field: file_count, archive_seconds and upload_seconds (SnapshotSaveStats). The claude-code agent fills it.
  • capture_workspace logs event=snapshot_save_completed with agent_name, context_id, elapsed_seconds, timeout_seconds, workspace_archive_bytes (from the uploaded object) and the three stats.
  • On a timeout it logs event=snapshot_save_timeout with the elapsed time, the budget and the workspace grant's cap, then re-raises.

Design decisions

  • Stats ride on the response. The worker already parses the save response, and the archive's size is already in it. A separate size call would add protocol surface for a number the agent has at hand.
  • Optional and ignorable. A worker parses the response with unknown fields ignored, so an older worker accepts a newer agent's answer. An agent that sends no stats logs them as null. The card now lists stats under the response's optional fields.
  • The claude-code floor moves to the protocol that ships the field (>=0.1.551). The agent's own handler validates its answer against the installed protocol, which rejects unknown fields. A release publishes the protocol before it builds the image, so a released image gets the field. If a release lands before this merges, raise the floor to that next release.
  • A timeout carries no stats. The agent has not answered yet. Seeing progress during a save needs a streamed or polled save, a larger protocol change left for later.
  • Fields are log-only. Ids and sizes are unbounded, so none become metric tags.

Scope

  • Only claude-code reports stats. Other agents keep their current answers and log null stats.
  • Does not change any timeout or retry.

Blast Radius

Adds two log lines per save and one optional response field. It changes the advertised snapshot/v1 card (an added optional field) and the claude-code image's protocol floor. No control flow changes.

Verification

  • PYTHONPATH=src:packages/agentenv-protocol/src:. .venv/bin/python -m pytest tst/unit/task_step/test_agent_state_capture.py tst/unit/agents/claude_code/test_workspace_snapshot.py -q: 42 passed. Without the source commit, 5 fail: no snapshot_save_completed or snapshot_save_timeout record, and no stats in the agent's answer or card.
  • tst/unit/agents, tst/unit/a2a_agent and the snapshot step tests: 1336 passed and 2 card assertions failed, which now expect stats (44 passed in their files). Protocol package tests (packages/agentenv-protocol/tests): 1626 passed, 1 skipped.
  • After deploy: search agent-env-worker logs for @event:snapshot_save_completed and @event:snapshot_save_timeout. Completed lines from claude-code agents should carry non-null file_count, archive_seconds and upload_seconds.

Ubuntu added 2 commits November 15, 2026 07:57
…og each save

The save response gains optional stats. claude-code fills them, and
capture_workspace logs snapshot_save_completed with them and the
workspace archive's size, or snapshot_save_timeout with the wait.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant