From 643d88d12296ef45a3638920e4deaee3b27ce8e3 Mon Sep 17 00:00:00 2001 From: Joel Goldfoot Date: Sun, 27 Sep 2026 09:17:17 -0700 Subject: [PATCH] docs: weekly research scan [2026-09-27] [STALE] Operator no longer leads OSWorld; add OSWorld-Verified figures from the maintainers' results file. [NEW] Affora (arXiv:2609.19125), GUI-agent nudge susceptibility (arXiv:2609.19843), MCP Skills Extension SEP-2640 Final, WCAG 3.0 Working Draft 10 Sept 2026. [CITATION] WebMCP: debugging annotation, toolactivated/toolcancel events, Permissions-Policy tools=() mitigation, executeTool inputObject retyped to any. Co-Authored-By: Claude Opus 5.5 --- README.md | 102 +++++++++++++++++++++++++++++++++++---------- docs/whitepaper.md | 92 +++++++++++++++++++++++++++++++++------- 2 files changed, 159 insertions(+), 35 deletions(-) diff --git a/README.md b/README.md index d7bd5c3..2f9cc18 100644 --- a/README.md +++ b/README.md @@ -232,6 +232,30 @@ merely possible. cheaper non-GUI route, so adoption is not solely the interface's problem; and 28.69% absolute on OSWorld-MCP says the remaining ceiling sits somewhere other than tool availability. +- **In the one study that tested it head-to-head, semantic markup alone did not + move agents — making the needed meaning consumable did.** **Affora** (2026) is + a design system for interfaces read by both people and computer-use agents. On + its primary comparison set, adding ARIA and structured data to the existing + interface left completion flat — **46/60 (77%) at baseline, 45/60 (75%) with + ARIA and structured data** — while Affora's rewrites of the shared interface + and an `agent.md` instruction file each reached **60/60 (100%)**. A + WebMCP-style tool condition reached 55/55 on a separately scoped action set + that the author states is not a matched comparison (arXiv:2609.19125). Read it + narrowly: the author evaluated small and mid-tier models, not frontier ones; + the interfaces were administrative and commerce apps; and gains appeared + "where Affora addresses existing deficits, but limited effects where those + deficits are absent or outside its coverage." For BiModal Design it is a + caution against treating Layer 2 annotations as sufficient on their own, and + support for the framework's premise that one interface can serve two readers + without a separate agent-only surface. It is not a test of BiModal Design. +- **Interfaces steer agents the way they steer people.** A randomized + online-shopping experiment ran **3,600 agents (21,600 simulations) across six + models from three providers** and found them susceptible to both a default + nudge and a social-influence nudge. Turning up reasoning cut susceptibility to + the default nudge but **raised** it to the social-influence nudge — reasoning + redirected, rather than removed, the effect of choice architecture + (arXiv:2609.19843). Deceptive patterns are therefore not a human-only concern: + an interface built for both modes steers both. - **The source benchmarks have themselves been shown to be gameable.** **BenchJack** (2026) audited 10 popular agent benchmarks spanning software engineering, web navigation, desktop computing, and terminal operations, and @@ -524,7 +548,8 @@ npm test - **OpAgent** — "Operator Agent for Web Navigation" achieving 71.6% on WebArena (arXiv:2602.13559) - **Operator** — OpenAI's Computer-Using Agent (87% on WebVoyager, 58.1% on - WebArena, 38.1% on OSWorld) + WebArena, 38.1% on OSWorld, as reported by OpenAI). No longer the OSWorld + leader; see OSWorld-Verified below. - **Project Mariner** — Google's agent featuring "Teach & Repeat" capabilities, achieving 84.0% on ScreenSpot and 83.5% on WebVoyager (Google, 2025) - **ScreenSpot** — Benchmark for spatial and visual understanding in GUIs @@ -533,27 +558,37 @@ npm test - **WebMCP** — Browser API letting a page register tools for in-browser agents via `document.modelContext`. W3C Web Machine Learning Community Group, Draft Community Group Report — a continuously updated editor's draft, so it is cited - by retrieval date rather than by the date in its header (retrieved 7 September - 2026; header then read 4 September 2026). Not a W3C Standard; not on the - Standards Track. The `ModelContext` interface gained a specified + by retrieval date rather than by the date in its header (retrieved 27 + September 2026; header then read 26 September 2026). Not a W3C Standard; not + on the Standards Track. The `ModelContext` interface gained a specified `executeTool()` method on 14 August 2026, revised on 17 August 2026 to take a - structured object rather than a JSON string; `RegisteredTool.inputSchema` was - retyped from `DOMString` to `object` in the same window. On 19 August 2026 the - draft specified `AbortSignal` integration for tool execution and preserved - in-flight executions after unregistration (PRs #247, #248) — the cancellation - and teardown semantics a page needs before exposing a long-running tool to an - agent. On **3 September 2026** `ToolAnnotations` gained - **`consequentialHint`** — "If true, indicates that executing the tool will - result in consequential actions that are significant, real-world, or - non-reversible, ex: booking a flight, transferring money" — joining - `readOnlyHint` and `untrustedContentHint`. **Implementation status as - re-checked on 7 September 2026** (per the spec repository's own - `implementation-status.md`, which is the source for these): origin trials live - in **Chrome 149** and **Edge 150**; experimental support in Brave's Leo AI - chat; **ChatGPT Desktop** listed as supporting WebMCP — the first non-browser - client on the page, added 26 August 2026 in a pull request from an - OpenAI-affiliated author and merged by a spec editor. Firefox and Safari have - standards-positions entries only. + structured object rather than a JSON string, and retyped again on 10 September + 2026 so that `inputObject` is `any` rather than `object` (PR #251); + `RegisteredTool.inputSchema` was retyped from `DOMString` to `object` in the + August window. On 19 August 2026 the draft specified `AbortSignal` integration + for tool execution and preserved in-flight executions after unregistration + (PRs #247, #248) — the cancellation and teardown semantics a page needs before + exposing a long-running tool to an agent. On **3 September 2026** + `ToolAnnotations` gained **`consequentialHint`** — "If true, indicates that + executing the tool will result in consequential actions that are significant, + real-world, or non-reversible, ex: booking a flight, transferring money" — + joining `readOnlyHint` and `untrustedContentHint`. On **17 September 2026** + `ToolAnnotations` gained a fourth member, **`debugging`** — "If true, + indicates that the tool is intended for debugging and developer tooling rather + than end-user interactions" — and `ModelContext` gained **`toolactivated`** + and **`toolcancel`** events, fired when a tool's execution begins and when it + is cancelled (PRs #253, #245). On **15 September 2026** the spec's first + listed security mitigation became **`Permissions-Policy: tools=()`**: the API + is gated behind the policy-controlled feature `tools` (default allowlist + `'self'`), and an empty allowlist disables WebMCP for the document and every + descendant frame, enforced "before script runs on the page" (PR #275). + **Implementation status as re-checked on 27 September 2026** (unchanged since + 7 September) (per the spec repository's own `implementation-status.md`, which + is the source for these): origin trials live in **Chrome 149** and **Edge + 150**; experimental support in Brave's Leo AI chat; **ChatGPT Desktop** listed + as supporting WebMCP — the first non-browser client on the page, added 26 + August 2026 in a pull request from an OpenAI-affiliated author and merged by a + spec editor. Firefox and Safari have standards-positions entries only. - **ComponentBench** — "Diagnosing Component-Level Failures in Computer-Use Agents" — Guan, Lin, Cheng-Yue, Wang, Zhou (arXiv:2608.18307, 18 August 2026). 97 canonical UI components as 2,910 programmatically verified tasks with human @@ -581,6 +616,31 @@ npm test desktop applications; a post-trained 9B model reaches 40.2% on OSWorld against its own GUI-only 23.4%, and 28.69% on OSWorld-MCP against the untuned base model's 20.90%. An agent-training result; makes no interface-design claim. +- **Affora** — "Affora: A Design System for Agent-Friendly Interfaces" — Jin Gao + (arXiv:2609.19125, 16 September 2026). Three controlled studies plus + evaluation on WebArena's Magento storefront, WebShop, and four independently + authored applications. Primary set: baseline 46/60, ARIA and structured data + 45/60, `agent.md` 60/60, Affora 60/60. Small and mid-tier models only. +- **Nudge susceptibility in GUI agents** — "A Dual-Process Perspective on Nudge + Susceptibility in LLM-Based GUI Agents" — Halimeh, Kaltenpoth, Bösch, Müller + (arXiv:2609.19843, 17 September 2026). 3,600 agents, 21,600 simulations, six + models (GPT-5.4 and -mini, Gemini 3.5 Flash and 3.1 Flash Lite, Claude Sonnet + 4.6 and Haiku 4.5). Reasoning lowered susceptibility to default nudges and + raised it to social-influence nudges. +- **OSWorld-Verified results** — the OSWorld maintainers' results file behind + the leaderboard at osworld-v1.xlang.ai + (`static/data/osworld_verified_results.xlsx`, last modified 7 August 2026). + Highest entry 90.19% (Intelligence-Indeed Agent, an agentic framework, 25 July + 2026); highest general model 85.96% (`claude-fable-5[1m]`, 1 August 2026), + then `claude-opus-5[1m]` at 83.39%. None of the fifteen highest entries lists + the accessibility tree as an additional input. +- **MCP Skills Extension (SEP-2640)** — Final, merged 13 September 2026; + extension identifier `io.modelcontextprotocol/skills`. Serves Agent Skills + over the existing Resources primitive under a `skill://` URI convention, with + `skills/list` and `skills/get` methods; the skill format and its progressive + disclosure model are delegated to the Agent Skills specification. SDK + implementation PRs remained open at merge. + https://modelcontextprotocol.io/seps/2640-skills-extension - **Code execution with MCP** — Building more efficient agents (Anthropic, Nov 2025) diff --git a/docs/whitepaper.md b/docs/whitepaper.md index 998f85b..9f1b73d 100644 --- a/docs/whitepaper.md +++ b/docs/whitepaper.md @@ -648,6 +648,22 @@ it's measurable: the interface's problem alone; and 28.69% in absolute terms says that once adoption is trained in, the binding constraint has moved somewhere other than tool availability. +- **Affora (2026)**: The nearest published analogue to this framework's own + thesis — a design system for one interface read by two kinds of reader — and + the first head-to-head test of semantic annotation against alternatives. On + its primary comparison set, ARIA and structured-data augmentation left + completion flat (46/60 at baseline, 45/60 augmented; 28/33 and 29/33 on the + harder set), while Affora's rewrites of the shared interface and an `agent.md` + instruction file each reached 60/60. A WebMCP-style condition reached 55/55 on + a separately scoped action set the author does not treat as a matched + comparison. On independent interfaces, gains tracked the presence of the + deficit a rule addresses: rewriting already-sound components produced no + improvement (arXiv:2609.19125). The author evaluated small and mid-tier models + only. The lesson for Layer 2 is a limit rather than a refutation: attributes + that restate what a control already declares add little; what moved agents was + making required information and actions present in a representation they + consume. That is consistent with treating Layer 2 as necessary but not + sufficient, and it is not a measurement of BiModal Design. - **BenchJack (2026)**: Turned the evaluation lens on the benchmarks themselves. Applied to 10 popular agent benchmarks spanning software engineering, web navigation, desktop computing, and terminal operations, its automated @@ -1400,7 +1416,13 @@ The OSWorld benchmark has tracked the rapid advancement of Computing User Agents (CUAs). At the original OSWorld 1.0 release, the best model completed 12.24% of tasks against a human baseline of 72.36%. Frontier scores climbed steeply from there — steeply enough that OSWorld 1.0 is now superseded: on 2026-06-26 the -maintainers directed users to **OSWorld 2.0** as the current version. +maintainers directed users to **OSWorld 2.0** as the current version. The +maintainers' OSWorld-Verified results file (last modified 7 August 2026) shows +how far: the highest entry scores 90.19% (an agentic framework) and the highest +general model 85.96%, both above the 72.36% human baseline reported for the +original task set — a comparison to read loosely, since that baseline predates +the Verified revision. None of the fifteen highest entries lists the +accessibility tree as an additional input. That successor complicates any straight-line projection toward "solved" UI execution. OSWorld 2.0 comprises 108 long-horizon workflows that take human @@ -1459,8 +1481,9 @@ fallback, not the primary design target. Recent developments with systems like OpenAI Operator and Google Project Mariner have demonstrated highly capable, hybrid vision-and-semantic agents. Operator -achieves an 87% success rate on WebVoyager and leads in benchmarks with scores -of 58.1% on WebArena and 38.1% on OSWorld. +was reported at an 87% success rate on WebVoyager, 58.1% on WebArena and 38.1% +on OSWorld. Those were leading figures when published; they no longer are. On +OSWorld-Verified the highest entries now exceed 80% (see §9.1). Project Mariner introduces advanced capabilities, achieving high scores of 84.0% on ScreenSpot and 83.5% on WebVoyager. This highlights the necessity of @@ -1689,6 +1712,12 @@ explicit content policies: content without bias or manipulation - **Deceptive markup**: Never use semantic structure to mislead agents (e.g., marking promotional content as editorial) +- **Choice architecture**: Nudges act on agents too. In a randomized + online-shopping experiment with 3,600 agents across six models, agents were + susceptible to both default and social-influence nudges, and more reasoning + lowered the first while raising the second (arXiv:2609.19843). Defaults, + scarcity cues and social proof in a BiModal interface steer both readers and + should be held to the same standard for both ### **11.5 Regulatory Landscape** @@ -1707,6 +1736,10 @@ explicit content policies: WCAG-EM audit can treat it as the procedural companion to a BiModal Design Layer 2 assessment, though it evaluates human accessibility conformance and does not assess Layers 1 or 3-5 +- **WCAG 3.0**: A new Working Draft was published on 10 September 2026, the + first since 3 March 2026; it "includes all requirements that have reached the + developing status." It remains a draft, not a Recommendation. Layer 2 + continues to target WCAG 2.2 ### **11.6 Human-in-the-Loop (HITL) for Tool-Agent-User Interaction (τ-bench)** @@ -2346,7 +2379,8 @@ resilient, semantic, structured, and protocol-aware. 9. **OpAgent**: "Operator Agent for Web Navigation" — Guo, Yang, Yang et al., 71.6% on WebArena (arXiv:2602.13559, 14 February 2026) 10. **Operator**: OpenAI's Computer-Using Agent (87% on WebVoyager, 58.1% on - WebArena, 38.1% on OSWorld) + WebArena, 38.1% on OSWorld, as reported by OpenAI; no longer the OSWorld + leader — see OSWorld-Verified below) 11. **Project Mariner**: Google's agent featuring "Teach & Repeat" capabilities, achieving 84.0% on ScreenSpot and 83.5% on WebVoyager (Google, 2025) 12. **ScreenSpot**: Benchmark for spatial and visual understanding in GUIs @@ -2401,6 +2435,18 @@ resilient, semantic, structured, and protocol-aware. post-trained 9B model reaches 40.2% on OSWorld against its own GUI-only 23.4%, and 28.69% on OSWorld-MCP against the untuned base model's 20.90%. An agent-training result; the paper makes no interface-design claim. +26. **Affora**: "Affora: A Design System for Agent-Friendly Interfaces" — Jin + Gao; arXiv:2609.19125 (v1, 16 September 2026). Primary set: baseline 46/60, + ARIA and structured data 45/60, `agent.md` 60/60, Affora 60/60. Small and + mid-tier models only. +27. **Nudge susceptibility in GUI agents**: "A Dual-Process Perspective on Nudge + Susceptibility in LLM-Based GUI Agents" — Halimeh, Kaltenpoth, Bösch, + Müller; arXiv:2609.19843 (v1, 17 September 2026). 3,600 agents, 21,600 + simulations, six models from three providers. +28. **OSWorld-Verified results**: OSWorld maintainers' results file, + https://osworld-v1.xlang.ai/static/data/osworld_verified_results.xlsx (last + modified 7 August 2026). Highest entry 90.19% (agentic framework); highest + general model 85.96% (`claude-fable-5[1m]`); `claude-opus-5[1m]` 83.39%. ### **Agent Protocols** @@ -2410,17 +2456,24 @@ resilient, semantic, structured, and protocol-aware. Round-Trip Requests (MRTR), cacheable list results, header-based routing, Extensions framework, Tasks and MCP Apps extensions, authorization hardening, formal feature lifecycle and deprecation policy. Changelog: - https://modelcontextprotocol.io/specification/2026-07-28/changelog + https://modelcontextprotocol.io/specification/2026-07-28/changelog. The + Skills Extension (SEP-2640, `io.modelcontextprotocol/skills`) reached Final + on 13 September 2026: Agent Skills served over the Resources primitive under + a `skill://` convention, with `skills/list` and `skills/get`, and the skill + format and its progressive-disclosure model delegated to the Agent Skills + specification. It is the protocol-level form of the progressive discovery the + MCP roadmap describes. SDK implementations were still open at merge. 2. **WebMCP**: https://webmachinelearning.github.io/webmcp/ — W3C Web Machine Learning Community Group. Continuously updated editor's draft published as a Draft Community Group Report; the document header is regenerated on each commit, so it carries no stable publication date and should be cited by retrieval date rather than by the date shown. Exposes `document.modelContext`. Not a W3C Standard; not on the Standards Track. - (Retrieved 7 September 2026; the header then read 4 September 2026.) The + (Retrieved 27 September 2026; the header then read 26 September 2026.) The `ModelContext` interface now specifies `executeTool()` alongside `registerTool()` and `getTools()`, added 14 August 2026 and revised 17 August - 2026 to accept a structured object rather than a JSON string; + 2026 to accept a structured object rather than a JSON string, and retyped on + 10 September 2026 so that `inputObject` is `any` (PR #251); `RegisteredTool.inputSchema` was retyped from `DOMString` to `object` on 14 August 2026. On 19 August 2026 the draft specified `AbortSignal` integration for tool execution and preserved in-flight executions after unregistration @@ -2430,13 +2483,22 @@ resilient, semantic, structured, and protocol-aware. `consequentialHint`, defined as "If true, indicates that executing the tool will result in consequential actions that are significant, real-world, or non-reversible, ex: booking a flight, transferring money", joining - `readOnlyHint` and `untrustedContentHint`. Implementation status per the - repository's `implementation-status.md`, re-checked 7 September 2026: Chrome - 149 and Edge 150 origin trials, Brave Leo AI chat experimental, ChatGPT - Desktop listed as supported (added 26 August 2026, the first non-browser - client), Firefox and Safari standards-positions entries only. The - repository's README lists headless task completion as a goal and fully - autonomous browser-less workflows as a non-goal. + `readOnlyHint` and `untrustedContentHint`. On 17 September 2026 + `ToolAnnotations` gained a `debugging` member ("If true, indicates that the + tool is intended for debugging and developer tooling rather than end-user + interactions"), and `ModelContext` gained `toolactivated` and `toolcancel` + events (PRs #253, #245). On 15 September 2026 the Security and Privacy + Considerations listed `Permissions-Policy: tools=()` as the first mitigation: + the API is gated behind the policy-controlled feature `tools` (default + allowlist `'self'`), and an empty allowlist disables it for the document and + every descendant frame before script runs (PR #275). Implementation status + per the repository's `implementation-status.md`, re-checked 27 September 2026 + and unchanged since 7 September: Chrome 149 and Edge 150 origin trials, Brave + Leo AI chat experimental, ChatGPT Desktop listed as supported (added 26 + August 2026, the first non-browser client), Firefox and Safari + standards-positions entries only. The repository's README lists headless task + completion as a goal and fully autonomous browser-less workflows as a + non-goal. 3. **Agent-to-Agent Protocol (A2A)**: https://a2a-protocol.org — launched by Google April 2025; the Linux Foundation announced A2A as the project's new home on 23 June 2025. Specification v1.0.0 was released on 12 March 2026 @@ -2467,6 +2529,8 @@ resilient, semantic, structured, and protocol-aware. 4. **WCAG 2.2**: https://www.w3.org/TR/WCAG22/ 5. **WCAG-EM 2.0**: https://www.w3.org/TR/wcag-em-2/ — W3C Group Note, 23 July 2026 +6. **WCAG 3.0**: https://www.w3.org/TR/2026/WD-wcag-3.0-20260910/ — W3C Working + Draft, 10 September 2026 (not a Recommendation) ### **BiModal Design Resources**