Skip to content

fix(computer): a frame an agent gets carries no capture time, which was the driver's real time - #703

Merged
earakely-scale merged 3 commits into
mainfrom
edgararakelyan/frames-carry-no-real-time
Nov 16, 2026
Merged

earakely-scale merged 3 commits into
mainfrom
edgararakelyan/frames-carry-no-real-time

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

What changes

A frame an agent gets, from computer_screenshot or any tool that captures one, no longer carries captured_at in what the agent reads. The capture time moves to the tool answer's _meta (agentenv.io/capturedAt), which MCP clients don't show their model.

The driver deliberately keeps real time (its waits and recordings do), so captured_at was the real UTC time. On a computer that follows the universe clock, every screenshot result told the agent the real date and a UTC clock that disagreed with the desktop's own. In the FD-5483 pilot, a run whose universe was Saturday 6 February 2027 in Denver got "captured_at": "2026-11-15T23:18:43Z" on each frame.

No agent needs the field. The recording test does: it lines each screenshot up with the video by the driver's capture time. So:

  • FrameData, the structured content, drops it;
  • send() puts the frame's capture time in the answer's _meta, for screenshots and for inputs and waits that capture one;
  • the recording test's drive_desk verifier reads it from there;
  • on standalone FastMCP, the result adapter now keeps _meta too.

Recordings are unchanged: they stamp their own frames (RecordedFrame). cua-driver's forwarded tools answer with its own daemon's results, which this doesn't touch.

Testing

  • test_computer_frames.py: a screenshot's structured content has no captured_at, and its _meta carries the capture time, a crop's being its frame's. test_computer_spaces.py checks the same through standalone FastMCP; it fails without the adapter fix.
  • The protocol's tests: 1636 passed. With tst/unit/drivers and the recorder test: 1776 passed. ruff check is clean.
  • End to end on an x86 host (local Docker):

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge.

Summary

Moves screenshot capture times out of agent-visible frame data and into tool-result _meta.

  • Frame capture times stay out of agent-visible frame data.
  • Standalone FastMCP replies keep tool answer metadata.

Both earlier findings are addressed. No new actionable issues were found.

Reviews (7) · Last reviewed commit: "fix(protocol): a driver on standalone Fa..." · Reviewed by Greptile

…as the driver's real time

The driver keeps real time, so every screenshot result's captured_at gave an agent on a computer following the universe
clock the real date and a UTC clock beside the universe's (2026-11-15T23:18Z, in a Saturday 6 February 2027 run). No
agent needs it and nothing read it: FrameData drops it, and Frame, which only copied it to the result, too. Recordings
keep stamping their own frames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner November 16, 2026 00:21
…ch recordings line up with and agents aren't shown

The recording test lines each screenshot up with the video by when the driver captured it, so that time stays, as
`agentenv.io/capturedAt` in the tool answer's _meta (MCP clients don't show _meta to their model), not in the
structured content an agent reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…a, where a frame's capture time is

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale merged commit f9aa197 into main Nov 16, 2026
68 of 70 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/frames-carry-no-real-time branch November 16, 2026 04:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant