Skip to content

feat(browser): browser-recording/v1 records one context at a time, with its video, log and network.har (M8.22) - #674

Merged
earakely-scale merged 7 commits into
mainfrom
edgararakelyan/fd-4970-m822-browser-recordingv1-one
Nov 14, 2026
Merged

earakely-scale merged 7 commits into
mainfrom
edgararakelyan/fd-4970-m822-browser-recordingv1-one

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

M8.22 (FD-4970): one browser recording per context.

Since M8.15 and M8.16 each agent can have a context of its own, but browser-recording/v1 recorded the whole browser: frames of whichever tab an agent last acted in, any context, and the log, with no video and no HAR. The only video was the page driver's computer-recording/v1, the page tab across contexts. Now a recording is of one context, a lease's or the default one's: a video of the tab its agent is on, its own log, the browser's network log as HAR, opt-in frames, and the contexts its agents make. Two leases recorded at once each show only their own.

Design note: M8.22: One browser recording per context, decisions 1a to 5a, Nov 13.

What changes

  • start_browser_recording {name, context?, fps?, network?, frames?} records the lease context names, else the default context.
    • fps is 10 by default, 15 at most.
    • network is on by default where the browser keeps a log, as a browser alone does and one in a desktop computer doesn't; asking for it where there's none refuses. frames is now off by default, as the design doc said (no stored prod task used the step).
    • to_dict writes context, fps and network only when set, so existing tasks get no new version.
    • A browser whose card takes none of them refuses them up front.
  • The video (decision 1) follows the tab the agent last acted on in the context, a popup that tab opens, and the tab before once one closes; it's blank while the context has no tab.
    • The browser's own connection screencasts the followed tab and acknowledges each frame. A tab that sends nothing for a second is captured instead, at most twice a second.
    • Frames are encoded by computer-recording/v1's recording class, now Video, which takes a stream of frames instead of the driver. Computers behave as before.
    • video.followed in the manifest says which tab showed when, and how.
  • The log: a recording keeps its context's lines. CDP action entries from a scoped connection now carry the lease's label as context, so a browser-level command lands in the right recording.
  • The network log (decision 3): the browser's whole log at the end as network.har, start-tab loads included, marked browser-wide, with the lines written during the recording and the leases open meanwhile.
  • Children (decision 2): a context an agent makes over an address scoped to a recorded context is recorded with it under contexts/<label>/ until its context or the parent ends.
  • A lease's recording ends with its lease, context_ended.
  • The end step fails the run as an infrastructure failure when the video, a segment, the HAR or a line of it, a frame, or any of a child's is missing, or the video stopped early. A HAR with no entries is no failure: an agent may load nothing. A recording's objects may be as big as the store's single upload, up to 5 GiB, as a long run's HAR can be. Registration lists every object under the recording's folder, so children's files are in the universe.
  • chromium installs av and Pillow through its own requirements.txt: a computer app gets agentenv-framework-protocol[app], which has neither. A browser app without them refuses a start.
  • The page driver's computer-recording/v1 is unchanged (decision 5). The design doc's line now says its display is the page tab.

Decided while building, for your review

The design note lists 13, in its last section. The ones worth a look:

  1. av and Pillow come from chromium's own requirements, not the app extra.
  2. CDP action entries from a scoped connection gain a context field.
  3. A silent tab is captured at most twice a second, and only while it's hidden: a visible one that sends nothing is a still page.
  4. With frames on, no frame is taken while the context has no tab, and a recording that ends with its lease takes no last frames: neither counts as missing.
  5. At most 8 recordings open at once, 4 children each.
  6. The video moves only for what shows an agent at work on a tab: input, navigation, a tab brought to the front, a dialog or file chooser answered, MCP tool calls. Not the scripts, bindings and emulation a client sends every page it attaches to, which the log counts as acting: CI caught the default context's video jumping to its blank start tab for a frame as Playwright connected. Frames still follow every acting entry.
  7. A tab switch drops the old tab's frame and asks the new tab for a capture at once: the video is blank for a frame or so until the new tab's first frame arrives, rather than showing the old tab under the new one's name. A frame that doesn't decode shows the tab's last good frame.

How it was tested

  • Unit:

    • test_browser_recording.py (26), on a real BrowserControl and Contexts over a fake pipe that paints each tab its own colour. Every video is encoded, then decoded with PyAV, and each frame's colour checked.
      • Covered: two leases at once each filming only their own tabs; a popup followed and the tab before after it closes; screenshots of a hidden silent tab and none of a visible still one; blank until a context's first tab opens.
      • Also: the log's lines of a context and none of another's; the whole network log as HAR with during; refusals with no log, an unleased context, a bad fps or no encoder.
      • Also: a child recorded with the recording scoped to its maker and ended with its context; a lease's release ending its recording, frames on, none missing; frames keyed to bursts on the context's tabs only; and one screencast serving everything that follows a tab.
    • The step tests (16): the start asking for context, fps and network only when set; the grant's max_objects from the card; registration of video, HAR and children; the end failing for each missing part; an older browser refused before a start; the round trip writing nothing new for an existing step.
    • The whole protocol package: 1,614 passed, 1 skipped; the computer recording tests unchanged (46).
  • Integration, local Docker on an agent-environments-admin c8i.4xlarge (amd64, KVM), with real Playwright and Puppeteer:

    • A new test records buyer, seller and the default context at once while each one's agent acts through its scoped address.

      • Every frame of each video is its own context's colour: buyer 405 red and 41 orange (its popup), seller 442 blue, the default 443 green.
      • Buyer's followed is red, orange, red. The context seller's agent made is recorded under contexts/agent-bob-1/ in its colour.
      • The HAR has every context's loads (11 entries). Buyer's agent screencast its own tab meanwhile and kept getting frames. Buyer's recording ended with its lease, context_ended.
    • Inside a desktop computer, a browser recording ran beside the driver's computer-recording/v1, each with its video.

    • Regressions, all passing:

      Tier Result
      browser_computer_test.py, M8.23's recording tests among them, now with video 30 passed, 1 skipped
      browser_multi_env_test.py and browser_gateway_test.py 13 passed
      website_browser_test.py 4 passed
      chromium_app_test.py 15 passed, 1 skipped
  • Chaos, on the same VM, not committed:

    1. The browser's server killed with a lease and the default context recorded: both ends registered what their folders held, the segment each had uploaded.
    2. Six leases recorded at once for 31 s: 304 to 311 frames each, about 10 fps, none dropped; load average 0.4 to 0.8 on 16 vCPUs.
    3. A lease's every tab closed, then one opened again: its video went screencast, blank, screencast.
  • arm64, on an agent-environments-admin m7g.4xlarge: the unit tests (42), the desktop test, the browser recording, lease and scoped-address tests including the new one (7), and the three chaos scenarios (304 to 314 frames each with six leases at once) all passed.

  • CI, first run: everything passed but test_release_protocol_pins, which wants a protocol line in each shipped requirements file for the release to pin. Chromium's now names agentenv-framework-protocol[app], and the new end-to-end test passed again with it.

  • CI, second run: the new end-to-end test caught the follow bug in item 6 above on the runner. Fixed; the end-to-end test and M8.23's recording tests passed again on the VM.

  • Review fixes (1257019), from your review: the network default above, an empty HAR no failure, the 5 GiB object cap. On a fresh VM: the unit tests (66), the desktop test on the defaults, the end-to-end test and M8.23's recording tests passed. Follow-ups filed: FD-5846 (real providers and small sandboxes), FD-5847 (the unexplained blank frame), FD-5848 (ops on a lease in the default log), FD-5849 (JavaScript-only agents and the video), FD-5850 (children past the cap), FD-5851 (image size and HAR memory).

  • Greptile, round 1: 4 P1 and 2 P2, all fixed in 77ae37b with tests: frames of the tab acted on, a stopped video ending its recording and failing the end, unreadable HAR lines counted, a child's manifest upload in what's missing, the tab switch above, and a recording ended once. CI then saw one blank frame in buyer's video, which the end-to-end test now allows at a switch (at most 3, never another context's colour); three VM runs in a row passed with at most one.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; the numbered finding is fixed and no new actionable issues remain.

Summary

The PR adds per-context browser recordings with video, filtered action logs, browser-wide HAR files, optional frames, and child recordings.

  • Browser recordings follow one context and collect its activity.
  • Browser recording steps choose what to save and check the finished files.

Reviews (7) · Last reviewed commit: "refactor(browser): name the line number ..." · Reviewed by Greptile

…th its video, log and network.har (M8.22)

A recording is of a lease or the default context: a video of the tab its agent is on, screencast by the browser's own
connection and captured while hidden, encoded by computer-recording/v1's recording class (now Video, taking a stream
of frames); the context's log lines, a scoped connection's actions now naming its lease; the browser's whole network
log as HAR, marked browser-wide; opt-in frames; and the contexts an agent makes over an address scoped to it,
recorded with it. A lease's recording ends with the lease. start_browser_recording takes context, fps and network,
frames now off by default; the end fails the run when any part is missing. chromium installs av and Pillow through its
own requirements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@socket-security

socket-security Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedpypi/​pillow@​13.0.08710010010090

View full report

@socket-security

socket-security Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Warning

Review the following alerts detected in dependencies.

According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.

Priority Alert  (click "▶" to expand/collapse) Action
Low priority
Potential vulnerability: pypi pillow with risk level "medium"

Location: Package overview

From: src/agent_env/browsers/envs/chromium/requirements.txt → pypi/pillow@13.0.0

ℹ Read more on: This package | This alert | Navigating potential vulnerabilities

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: It is advisable to proceed with caution. Engage in a review of the package's security aspects and consider reaching out to the package maintainer for the latest information or patches.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore pypi/pillow@13.0.0. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn

View full report

earakely-scale and others added 2 commits November 13, 2026 13:41
…which a release pins

Each shipped env's requirements file holds one protocol line for the release to pin; chromium's new one had none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…at work on a tab

Playwright sends every page it attaches to its scripts, bindings and emulation, which the log counts as acting, so
the default context's video jumped to its blank start tab for a frame as an agent connected. The video now moves for
input, navigation, a tab brought to the front, a dialog or file chooser answered and MCP tool calls; frames still
follow every acting entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale marked this pull request as ready for review November 13, 2026 22:20
@earakely-scale
earakely-scale requested a review from a team as a code owner November 13, 2026 22:20
earakely-scale and others added 3 commits November 13, 2026 14:35
- A frame is of the tab its context's agent last acted on, apart from the tab the video follows.
- A video that stops at its limit or after failing ends its recording, and the end step fails it as stopped early.
- Lines the network log couldn't read are counted, and fail the end.
- A child's manifest that didn't upload is what its parent says is missing of it.
- A tab switch drops the old tab's frame and asks the new tab for one at once; a late screencast frame of the tab
  left is ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…shows the tab's last that did

A child ended by its context's end and by its parent's at once ran its end twice; an end while one is under way now
waits for it. A screencast frame that doesn't decode shows the followed tab's last good frame rather than a blank one.
The end-to-end test allows the blank frame a tab switch shows until the new tab's first frame, never another
context's colour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…one, and an empty one is no failure

- network defaults to on only where the browser keeps a log, as a browser alone does and one in a desktop computer
  doesn't, so a recording on the defaults starts there; asking for it where there's none still refuses.
- A network log with no entries no longer fails the end: an agent may load nothing. One that's missing, unreadable
  or didn't upload still does.
- A recording's objects may be as big as the store's single upload, up to 5 GiB, as a long run's network.har can be.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…twork log's last_seq reads after

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale merged commit 048df43 into main Nov 14, 2026
38 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/fd-4970-m822-browser-recordingv1-one branch November 14, 2026 23:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant