feat: Playwright-batch facade surface, HardBench audit, and harness fixes from the smoke gates - #2867

Closed

miguelg719 wants to merge 49 commits into

bench/hardbenchmark-portfrom

feat/facade-batch-surface

Commits on Aug 31, 2026

feat(integrations): facade run executes the Playwright batch surface
miguelg719
committed
79aa866
feat(evals): stagehand_facade is the Playwright batch surface; add stagehand_facade_legacy
miguelg719
committed
3341bd6
fix(evals): default the rubric verifier to a live judge model
miguelg719
committed
8cb2314
bench(hardbenchmark): validity audit — quarantine 1 bot-walled task, flag 20 for verdict review
miguelg719
committed
3cfd185
fix(evals): fx harness never reached the Stagehand MCP tools
miguelg719
committed
876f940
fix(integrations): stop mastra/pi sessions retaining request bodies and duplicate screenshots
miguelg719
committed
994e9bc
fix(deepagents): keep provider tool-call blocks out of assistant text
miguelg719
committed
4fb8a0f
fix(evals): keep the run summary alive when a Braintrust row has no output
miguelg719
committed
e0bc551
style(pi-sdk): format session test
miguelg719
committed
9ffb9c3
fix(evals): codex harness reaches runner-mounted MCP tools; count facade tool calls
miguelg719
committed
d0d1be9
feat(integrations): facade page.frameLocator; clearer strict-mode and layout errors; lenient snapshot ids
miguelg719
committed
16e34f3
feat(evals): flag passes that never called the mounted browser surface
miguelg719
committed
be44128
test(evals): pin default_tools_approval_mode on runner-mounted codex MCP servers
miguelg719
committed
e123685
style(evals): oxfmt audit script and artifacts
miguelg719
committed
f8456cb
fix(integrations): frame-scoped getAttribute fails with guidance
miguelg719
committed
9b3d571
feat(evals): experiment metadata always carries tool_surface, model, provider, dataset
miguelg719
committed
68c5f45
feat(integrations): demote raw SDK event logs to debug, drop stream deltas
miguelg719
committed
f0cffe3
feat(evals): resolve the facade Browserbase session before the agent starts
miguelg719
committed
2695dd6
feat(evals): shared readable step trace for every external harness
miguelg719
committed
ff8a73a
feat(evals): log the Browserbase session URL as the first line of every harness task
miguelg719
committed
d307b1a
feat(evals): trace the agent's summary and final answer
miguelg719
committed
e37d961
fix(pi): keep pi's stock system prompt and default reasoning on
miguelg719
committed
8d5470b
bench(hardbenchmark): quarantine Macy's (bot-wall), imgur (session death), Best Buy trade-in (date rot) — n 45 → 42
miguelg719
committed
35b7a52
fix(fx): final answer is the agent's conclusion, not its narration
miguelg719
committed
21c647e
fix(evals): deterministic outcome gates, strict process score, answer grounding
miguelg719
committed
2c22cc8
fix(sdk-ts): client-side deadline for callback batches
miguelg719
committed
1567a31
fix(facade): terminal browser-session-lost error instead of dead-session loops
miguelg719
committed
6beccf6
fix(evals): propagate browser_session_lost stop reason
miguelg719
committed
6f63372
fix(facade): explicit 1 h Browserbase session timeout, no keep-alive
miguelg719
committed
813111b
fix(evals): only a trailing report parses without a marker
miguelg719
committed
ab063f5
feat(harnesses): request and capture model reasoning summaries
miguelg719
committed
62c2928
style(evals): oxfmt AUDIT-REVIEW.md
miguelg719
committed
e264b0d
feat(eve): route non-first-party creators through the Vercel AI Gateway
miguelg719
committed
977e7b4
fix(eve): pin a context window for gateway models missing from eve's catalog
miguelg719
committed
22a0a87
fix(evals): gated rows name the gate in the row error
miguelg719
committed
bc85928
fix(evals): answer grounding is advisory by default
miguelg719
committed
98e1812
feat(evals): dataset-aware step budgets (HardBench defaults to 75)
miguelg719
committed
0a3ce86
fix(eve): final answer and reasoning pairing
miguelg719
committed
247e051
feat(evals): record terminationReason on persisted trajectories
miguelg719
committed
1d3f6ec
fix(evals): persist gated outcome in scores/result.json
miguelg719
committed
4410556
feat(evals): split agent vs verifier wall-clock
miguelg719
committed
a692ce5
feat(evals): normalized token usage across harnesses
miguelg719
committed
71dd3c9
feat(evals): estimated cost from normalized usage and a versioned price map
miguelg719
committed
63ce610
test(facade): deterministic layout-error retry window
miguelg719
committed
4c5c977
bench(hardbenchmark): require on-site evidence for constraint criteria on FP-prone tasks
miguelg719
committed
e27975b
fix(facade): proxied + verified Browserbase sessions by default in evals (native-path parity)
miguelg719
committed
2acb9ab
test(codex-sdk): usage recovered from the CODEX_HOME rollout on budget abort
miguelg719
committed
c337309
fix(codex-sdk): keep token usage when the step budget aborts the turn
miguelg719
committed
3304ad4
feat(evals): billed cost per row (reported, else computed at provider rate)
miguelg719
committed
745e229