Skip to content

test(evals): exercise the workspace V4 CLI - #2838

Draft
shrey150 wants to merge 1 commit into
agent/browse-v4-5-remove-xpathfrom
agent/browse-v4-6-release
Draft

shrey150 wants to merge 1 commit into
agent/browse-v4-5-remove-xpathfrom
agent/browse-v4-6-release

Conversation

@shrey150

@shrey150 shrey150 commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Summary

Make workspace evals and packaging consume the V4 Browse CLI proven in #2835.

  • Point Stagehand evals at the workspace Browse package instead of published Browse 0.9.5.
  • Update native and external-agent adapters to the current Browse command grammar.
  • Keep eval session names within Unix socket limits, restrict the wrapper to driver commands, and clean daemons up on abort/signals.
  • Add command-mapping and real wrapper-process coverage.

The Browse migration changeset now belongs to #2835, the first complete supported-parity head; this PR has no changeset in its own diff. The unrelated eval CLI --verbose changes from the old draft are intentionally excluded.

Stack (#2872)

  1. chore(cli): import Browse V3 baseline #2833 — exact Browse V3 baseline import
  2. refactor(cli): migrate Browse runtime and commands to Stagehand V4 #2834 — Stagehand V4 runtime and standard command parity
  3. feat(cli): restore cursor overlay through page.evaluate #2869 — CLI-owned cursor overlay
  4. fix(cli): restore V3 network capture through a CDP sidecar #2849 — CLI-private CDP sidecar; V3 network parity
  5. refactor(cli): remove coordinate XPath return flag #2835 — remove --return-xpath; supported V3 parity/release checkpoint
  6. test(evals): exercise the workspace V4 CLI #2838 — eval and packaging integration
  7. fix(cli): persist context names in Browserbase #2839 — managed Context names (fast-follow)
  8. refactor(cli): consume shared Functions core #2701 — shared Functions core consumer (fast-follow)

Review boundary

This is a consumer/integration layer, not another runtime-parity layer. Its diff is limited to eval adapters, their tests, workspace dependency wiring, and the lockfile.

E2E Test Matrix

The current remote PR head is f13b62d4c17afc55673b82a34ea03622e6eec570. The complete matrix below ran at f8e2579571b5c059753881f14b80abb718066a94; the only inherited change since then is #2849's test-helper timeout diagnostic, whose focused network tests and Browse lint/typecheck passed at sidecar head 9887732b6. The product diff is unchanged from proof commit 1dd7d0c8ef7ac65697f91780620a417a8ceb6bdd, where the packed-consumer and browser-backed wrapper flows below ran. Browser-backed runs used an isolated daemon directory; the pre-existing default daemon was not touched.

Command / flow Observed output Confidence / sufficiency
Fully verified head f8e257957: pnpm install --frozen-lockfile; build browse; build @browserbasehq/stagehand-evals Frozen install passed; Browse generated its real Oclif manifest; eval ESM and CLI bundles both built. Proves the regenerated lockfile and exact current workspace dependency graph build.
Fully verified head f8e257957: full @browserbasehq/stagehand-evals test suite 81 test files / 633 tests passed in 102.97 seconds. Proves the propagated parent/lock changes did not regress the complete eval suite, including Browse integration coverage.
Product proof head 1dd7d0c8: pnpm --filter browse pack; install the tarball in an empty npm consumer; run node_modules/.bin/browse --version and browse open --help Tarball installed; publish rewrite changed workspace:* to Stagehand 4.1.0; installed binary reported browse/0.9.6; V4 --remote and --cdp flags were present. Proves the publish artifact is consumable outside the monorepo and does not leak an invalid workspace range. Product code is unchanged at the current head.
Product proof head 1dd7d0c8: built ESM prepareBrowseCliHarnessAdapter twice Metadata resolved the workspace packages/cli/bin/run.js at version 0.9.6; generated sessions and workdirs were distinct. Proves evals consume the workspace CLI and isolate concurrent/per-row state.
Product proof head 1dd7d0c8: execute the generated pinned wrapper as a real subprocess: browse open https://example.com, then browse get title Wrapper launched a real Browserbase V4 session, returned selected-target/session identity fields, and read Example Domain. Proves the wrapper executes the pinned entrypoint and appends its environment/session flags correctly.
Product proof head 1dd7d0c8: invoke browse cloud contexts list through that wrapper Rejected locally with exit 64. Proves non-driver commands cannot escape the eval harness boundary.
Product proof head 1dd7d0c8: register both live adapter cleanups; cooperative abort; aggressive abort Cooperative abort preserved both workdirs; aggressive abort stopped cleanup paths and removed both workdirs. Proves the two-stage abort semantics and real adapter cleanup are wired together.

@browserbasehq/stagehand-evals is private and is intentionally tested as a built workspace consumer, while the public browse package is additionally tested as an isolated packed consumer.

@changeset-bot

changeset-bot Bot commented Aug 27, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: d57f6ff

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@shrey150
shrey150 force-pushed the agent/browse-v4-6-release branch from 503061a to 5ec53cb Compare August 28, 2026 02:52
@shrey150
shrey150 force-pushed the agent/browse-v4-6-release branch from b3a3b41 to 1dd7d0c Compare September 11, 2026 17:49
@shrey150
shrey150 force-pushed the agent/browse-v4-6-release branch from 1dd7d0c to f8e2579 Compare September 11, 2026 18:23
@shrey150
shrey150 force-pushed the agent/browse-v4-6-release branch from f8e2579 to f13b62d Compare September 11, 2026 20:18
@shrey150
shrey150 force-pushed the agent/browse-v4-6-release branch from f13b62d to 1cef69e Compare September 15, 2026 23:10
shrey150 added a commit that referenced this pull request Sep 15, 2026
## Summary

Import `packages/cli/**` exactly from the published `browse@0.9.6` V3
release, without changing its source or runtime behavior.

This is intentionally a provenance checkpoint, not a line-by-line
feature review. The imported source is kept runnable by a root,
version-scoped pnpm override that resolves its unchanged Stagehand
dependency to `3.7.1`. #2834 removes that override and starts the V4
migration.

## Exact-source provenance

- Annotated tag: `browse@0.9.6`
(`548c56407431db27823a212f53475443c7e8358d`)
- Release commit: `1d49a95c0c230c346f8d50647e10303d6310fcd2`
- Authoritative CLI tree: `b4048badce921cf54f199f96033d9a014ef977ec`
- This PR's `HEAD:packages/cli` tree:
`b4048badce921cf54f199f96033d9a014ef977ec`

The tag's ignored README whitespace is retained too; formatting the
import would invalidate the tree proof.

## Verification

- Current remote head: `a77e1507b85e3c02553f36ead6ebd0237b0cccc6`, based
on current `main`.
- `HEAD:packages/cli` exactly equals the published V3 tree hash above.
- pnpm 11 frozen install and the repository supply-chain release-age
policy pass.
- Browse lint, typecheck, and build pass; the full baseline suite
passes: 25 files / 366 tests.
- A fresh extension build still exactly matches the Go-embedded archive:
SHA-256
`8efc7d171a625cca95c02d02d369b59435fae776cae6c7dd2f6fe72eb19785c0` on
both files. This specifically verifies that adding the V3 dependency
graph does not perturb the current V4 extension artifact.
- This layer intentionally exercises V3 through the scoped Stagehand
3.7.1 override. V4 behavior starts in #2834.

## Stack (#2872)

1. **#2833 — exact Browse V3 baseline import**
2. #2834 — Stagehand V4 runtime and standard command parity
3. #2869 — CLI-owned cursor overlay
4. #2849 — CLI-private CDP sidecar; V3 network parity
5. #2835 — remove `--return-xpath`; supported V3 parity/release
checkpoint
6. #2838 — eval and packaging integration
7. #2839 — managed Context names (fast-follow)
8. #2701 — shared Functions core consumer (fast-follow)

## Review and landing boundary

Review this PR by verifying the tree hashes, dependency pin, root
package wiring, and changeset—not by treating the imported V3 source as
newly authored code. This head deliberately imports V3 code into the V4
repository and is not independently publishable. It lands only as the
base of the complete stack.

The framework network-event schema proposal in #2832 is intentionally
outside this landing stack.
@shrey150
shrey150 force-pushed the agent/browse-v4-6-release branch from 1cef69e to 25220d1 Compare September 15, 2026 23:31
shrey150 added a commit that referenced this pull request Sep 16, 2026
…2834)

## Summary

Migrate Browse's browser lifecycle and standard command surface together
from Stagehand V3 to V4.

- Replace the V3 constructor/init lifecycle with V4 browser factories
and `Stagehand.create()`.
- Support managed local, Browserbase remote, and attached CDP connection
targets.
- Preserve owned-versus-attached cleanup, daemon persistence,
Browserbase session identity, and timeout handling.
- Restore navigation, page information, deterministic locator actions,
keyboard/mouse input, viewport/screenshot, snapshot, eval, and tab
commands on V4 APIs.
- Keep click/fill/select deterministic; this does not add a model-free
structured `act()` path.
- Make the remaining cursor, network, and coordinate-XPath gaps fail
explicitly for the stack layers that restore or remove them.

## Stack (#2872)

1. #2833 — exact Browse V3 baseline import
2. **#2834 — Stagehand V4 runtime and standard command parity**
3. #2869 — CLI-owned cursor overlay
4. #2849 — CLI-private CDP sidecar; V3 network parity
5. #2835 — remove `--return-xpath`; supported V3 parity/release
checkpoint
6. #2838 — eval and packaging integration
7. #2839 — managed Context names (fast-follow)
8. #2701 — shared Functions core consumer (fast-follow)

## Review shape

The lifecycle and command migration remain two ordered implementation
commits:

1. `389e2dae6` — V4 browser/session foundation and lifecycle ownership.
2. `b45167462` — standard command translation on that foundation.

They are one PR because both commits rewrite the same nine command/test
files. Reviewing their combined final diff avoids temporary
deletion/stubbing followed by reimplementation, while the commits still
provide useful lifecycle-versus-command checkpoints. Review follow-up
`24178275f` adds narrowly scoped ownership, error-sanitization, and
timeout guards. The resulting PR diff is 23 files, +988/−436.

Cursor DOM injection and private CDP network transport remain separate
because they are independently reviewable mechanisms and cleanly
additive diffs. The legacy coordinate `returnXPath` request is still
accepted here but fails explicitly until #2835 removes the option. This
remains an intentionally non-publishable intermediate head.
## E2E Test Matrix

Fresh post-flatten verification used the actual built CLI at final head
`6f7e9c209`. Every daemon command used an isolated owner-only runtime
directory.

| Command / flow | Observed output | Confidence / sufficiency |
| --- | --- | --- |
| `pnpm install --frozen-lockfile` | Lockfile passed supply-chain
policy, was already up to date, and installation completed | Proves the
flattened stack resolves exactly from the committed lockfile |
| `pnpm exec turbo run build --filter=browse` | Protocol, extension,
Stagehand SDK, and Browse CLI built successfully (4/4 tasks) | Proves
the CLI was tested against this head's protocol/extension/SDK artifacts,
not stale workspace `dist` files |
| Compare the rebuilt extension with
`packages/sdk-go/internal/extensionassets/stagehand-extension.zip` |
Exact byte match; both SHA-256
`8efc7d171a625cca95c02d02d369b59435fae776cae6c7dd2f6fe72eb19785c0`;
archive manifest and package version both `1.0.2` | Confirms the
TypeScript/CLI build and Go-embedded extension are synchronized |
| Built CLI: `browse open <synthetic-data-url> --remote`; `browse
status` | Remote browser connected and initialized; deterministic
fixture loaded | Exercises production Browserbase provisioning plus the
V4 daemon/session lifecycle on the exact final head |
| `browse get text //h1`; `fill`; `select`; `click`; `is checked`; `wait
selector`; `highlight`; `viewport`; `screenshot`; `snapshot --full` |
XPath returned `Ready`; input became `Ada`; select became `b`; click
produced `Clicked`; checkbox was true; PNG was 17,761 bytes; snapshot
contained the fixture | Covers deterministic V4 reads, actions, waits,
state, and rendering without an LLM |
| Set a page marker; `tab new`; `tab list`; `tab close`; read the marker
from a new CLI process | Tab count changed to 2 and the original page
returned marker `yes` | Proves daemon persistence, active-tab handling,
and state reuse across invocations |
| Inspect the isolated runtime directory/PID; `browse stop`; poll the
Browserbase session | Modes were `0700` / `0600`; the owned remote
session reached `COMPLETED` | Proves owner-only daemon files and owned
Browserbase resource cleanup |
| `browse cursor`; `browse network on`; `browse mouse hover ...
--return-xpath` | Each exited 1 with the intended explicit
layer-boundary error | Confirms this intermediate layer fails honestly
until the cursor, network, and flag-removal layers land |
| `pnpm --filter browse test` | 25 files / 385 tests passed | Full
Browse unit/integration suite on the exact final head |

The runner has no Chrome/Chromium installation, so a fresh attached-CDP
ownership smoke was not possible. Attached-browser non-ownership remains
covered by the focused suite and is not claimed as a fresh live result
here.
Pin eval agents to the V4 workspace CLI, isolate per-run sessions, restrict the wrapper to driver commands, and guarantee cleanup on abort and process shutdown.
@shrey150
shrey150 force-pushed the agent/browse-v4-6-release branch from 25220d1 to d57f6ff Compare September 16, 2026 00:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant