From 46f61ad0c0b480c3751dc92058c1a179e71d2fd3 Mon Sep 17 00:00:00 2001 From: Joel Goldfoot Date: Mon, 7 Sep 2026 14:06:02 -0700 Subject: [PATCH] docs: weekly research scan [2026-09-07] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two verified findings and one citation update, all confirmed against primary sources with a refutation pass. [NEW] arXiv:2609.00524 — three-week deployment study of a screen-reader accessible computer-use agent (8 blind users, 1,258 commands, 12 desktop applications, screenshots + UI trees supplied). GPT-5 best at 52.5%. Bounds the framework's AOM claim without overstating it: the study makes no with/without accessibility-metadata comparison, and the text says so. [NEW] arXiv:2609.05374 — CUA-Universe. A post-trained 9B model reaches 40.2% on OSWorld against its own GUI-only 23.4%, and 28.69% on OSWorld-MCP against the untuned base model's 20.90%. Recorded as an agent-training result that qualifies the adoption gap, not as an interface-design claim. [CITATION] WebMCP — `consequentialHint` added to `ToolAnnotations` on 3 Sept 2026; draft header 4 Sept 2026; implementation status now lists ChatGPT Desktop. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 78 ++++++++++++++++++++++++++++++++++++---------- docs/whitepaper.md | 77 ++++++++++++++++++++++++++++++++++++++------- 2 files changed, 126 insertions(+), 29 deletions(-) diff --git a/README.md b/README.md index 0a9391d..d7bd5c3 100644 --- a/README.md +++ b/README.md @@ -207,6 +207,31 @@ merely possible. because it is the same move the framework argues for at Layer 5 — make the tool route cheap and findable — implemented at enterprise scale rather than proposed. +- **A populated accessibility tree is the floor, not the finish line.** A + three-week deployment study put a screen-reader-accessible computer-use agent + in front of **8 blind users**, collecting **1,258 real commands across 12 + desktop applications**; every model received both screenshots **and** UI trees + carrying control roles and hierarchy. The strongest, GPT-5, succeeded on + **52.5%** (95% CI 49.8-55.3), ahead of Claude Sonnet 48.5%, Gemini 2.5 CU + 43.9%, UI-TARS 39.8% and Qwen3-VL 37.9%. The failure classes the authors name + are grounding, planning, constraint-tracking and termination + (arXiv:2609.00524). **The study does not compare agents with and without + accessibility metadata**, so it is not a measurement of Layer 2's benefit. + What it does establish is the boundary of this framework's own claim: + supplying a well-formed AOM removes a class of failure, and about half of real + user commands still fail for reasons no structural layer reaches. +- **The adoption gap is partly trainable, which bounds what interface design + alone can fix.** **CUA-Universe** (2026) turned 16 real desktop applications + into hybrid GUI+CLI environments and post-trained a 9B model on the harvested + trajectories. Against its own GUI-only operating point that model rises from + **23.4% to 40.2%** success on OSWorld (+16.8 pts); against the untuned base + model it lifts OSWorld-MCP from **20.90% to 28.69%** (+7.79 pts) while cutting + steps 27% and tokens 30% (arXiv:2609.05374). **The paper makes no claim about + interface or website design** — it is an agent-training result, recorded here + because it cuts both ways for Layer 5. An agent can be taught to prefer the + cheaper non-GUI route, so adoption is not solely the interface's problem; and + 28.69% absolute on OSWorld-MCP says the remaining ceiling sits somewhere other + than tool availability. - **The source benchmarks have themselves been shown to be gameable.** **BenchJack** (2026) audited 10 popular agent benchmarks spanning software engineering, web navigation, desktop computing, and terminal operations, and @@ -507,23 +532,28 @@ npm test Windows UI Automation with vision (Microsoft, 2025) - **WebMCP** — Browser API letting a page register tools for in-browser agents via `document.modelContext`. W3C Web Machine Learning Community Group, Draft - Community Group Report — a continuously updated editor's draft with no stable - publication date; the header read **26 August 2026** when retrieved on 31 - August 2026. Not a W3C Standard; not on the Standards Track. The - `ModelContext` interface gained a specified `executeTool()` method on 14 - August 2026, revised on 17 August 2026 to take a structured object rather than - a JSON string; `RegisteredTool.inputSchema` was retyped from `DOMString` to - `object` in the same window. On 19 August 2026 the draft specified - `AbortSignal` integration for tool execution and preserved in-flight - executions after unregistration (PRs #247, #248) — the cancellation and - teardown semantics a page needs before exposing a long-running tool to an - agent. **Implementation status as of 26 August 2026** (per the spec - repository's own `implementation-status.md`, which is the source for these): - origin trials live in **Chrome 149** and **Edge 150**; experimental support in - Brave's Leo AI chat; **ChatGPT Desktop** listed as supporting WebMCP — the - first non-browser client on the page, added 26 August 2026 in a pull request - from an OpenAI-affiliated author and merged by a spec editor. Firefox and - Safari have standards-positions entries only. + Community Group Report — a continuously updated editor's draft, so it is cited + by retrieval date rather than by the date in its header (retrieved 7 September + 2026; header then read 4 September 2026). Not a W3C Standard; not on the + Standards Track. The `ModelContext` interface gained a specified + `executeTool()` method on 14 August 2026, revised on 17 August 2026 to take a + structured object rather than a JSON string; `RegisteredTool.inputSchema` was + retyped from `DOMString` to `object` in the same window. On 19 August 2026 the + draft specified `AbortSignal` integration for tool execution and preserved + in-flight executions after unregistration (PRs #247, #248) — the cancellation + and teardown semantics a page needs before exposing a long-running tool to an + agent. On **3 September 2026** `ToolAnnotations` gained + **`consequentialHint`** — "If true, indicates that executing the tool will + result in consequential actions that are significant, real-world, or + non-reversible, ex: booking a flight, transferring money" — joining + `readOnlyHint` and `untrustedContentHint`. **Implementation status as + re-checked on 7 September 2026** (per the spec repository's own + `implementation-status.md`, which is the source for these): origin trials live + in **Chrome 149** and **Edge 150**; experimental support in Brave's Leo AI + chat; **ChatGPT Desktop** listed as supporting WebMCP — the first non-browser + client on the page, added 26 August 2026 in a pull request from an + OpenAI-affiliated author and merged by a spec editor. Firefox and Safari have + standards-positions entries only. - **ComponentBench** — "Diagnosing Component-Level Failures in Computer-Use Agents" — Guan, Lin, Cheng-Yue, Wang, Zhou (arXiv:2608.18307, 18 August 2026). 97 canonical UI components as 2,910 programmatically verified tasks with human @@ -537,6 +567,20 @@ npm test standardized contract for `tools/call` result forms. Both target the discoverability half of the adoption gap. Tasks moved into an official extension (SEP-2663). https://blog.modelcontextprotocol.io/posts/mcp-roadmap/ +- **Computer-Use Agents for Blind Users** — "Are We There Yet? Assessing + Computer-Use Agents for Blind Users' Accessible Interaction with Desktop + Applications" — Kodandaram, Padma Reddy, Bi, Zhou, Ramakrishnan, Ashok + (arXiv:2609.00524, 1 September 2026). Three-week diary study, 8 blind + screen-reader users, 1,258 commands across 12 desktop applications, all models + given screenshots and UI trees. GPT-5 52.5% (95% CI 49.8-55.3), Claude Sonnet + 48.5%, Gemini 2.5 CU 43.9%, UI-TARS 39.8%, Qwen3-VL 37.9%. No with/without + accessibility-metadata comparison is made. +- **CUA-Universe** — "A Scalable and Dynamic Environment for Hybrid GUI+CLI + Agents" — Shi, Wang, Fang, Liang, Jin, Zhao, Liu, Chen, Wang + (arXiv:2609.05374, 4 September 2026). Environment-to-data pipeline over 16 + desktop applications; a post-trained 9B model reaches 40.2% on OSWorld against + its own GUI-only 23.4%, and 28.69% on OSWorld-MCP against the untuned base + model's 20.90%. An agent-training result; makes no interface-design claim. - **Code execution with MCP** — Building more efficient agents (Anthropic, Nov 2025) diff --git a/docs/whitepaper.md b/docs/whitepaper.md index f0a40bd..998f85b 100644 --- a/docs/whitepaper.md +++ b/docs/whitepaper.md @@ -318,6 +318,23 @@ Furthermore, insights from **VisualWebArena** demonstrate that pure visual reasoning frequently fails in complex interfaces without strong textual and structural grounding. Layer 2 provides this essential grounding. +**A populated AOM is the floor, not the finish line.** A three-week deployment +study of a screen-reader-accessible computer-use agent — 8 blind users, 1,258 +real commands across 12 desktop applications, every model supplied with both +screenshots and UI trees carrying control roles and hierarchy — found the +strongest model, GPT-5, succeeding on only 52.5% of commands (95% CI 49.8-55.3), +followed by Claude Sonnet at 48.5%, Gemini 2.5 CU at 43.9%, UI-TARS at 39.8% and +Qwen3-VL at 37.9%. The failure classes the authors identify are grounding, +planning, constraint-tracking and termination (arXiv:2609.00524). Two things +follow, and the second constrains the first. The study makes **no** comparison +between agents with and without accessibility metadata, so it is not a +measurement of what Layer 2 buys; nothing here should be read as quantifying +this layer's contribution. What it does bound is the claim above: a well-formed +AOM supplies reliable interaction targets, and roughly half of real user +commands still fail for reasons that live in the agent's planning and +termination behaviour rather than in the interface's semantics. Layer 2 removes +a class of failure. It does not remove the rest. + As modern interfaces increasingly rely on Web Components, the encapsulation provided by the Shadow DOM creates a new challenge: it hides critical state and semantic information from the AOM. To solve this, developers must use the @@ -616,6 +633,21 @@ it's measurable: this framework names — making the tool route cheap to find and cheap to carry — and it converges with the progressive-discovery direction the MCP roadmap set out in August 2026. +- **CUA-Universe (2026)**: The complement to the previous entry, and a check on + reading the adoption gap as purely an interface problem. Its App-Forge + pipeline turns 16 real desktop applications into reproducible hybrid GUI+CLI + environments, Task-Weave synthesizes hybrid tasks over them, and Path-Steer + harvests verified efficient trajectories for post-training. A 9B model trained + on that data reaches 40.2% success on OSWorld against its own GUI-only + operating point of 23.4% (+16.8 pts), and 28.69% on OSWorld-MCP against the + untuned base model's 20.90% (+7.79 pts), with 27% fewer steps and 30% fewer + tokens (arXiv:2609.05374). The paper makes **no claim about interface or + website design** — it is an agent-training result, and is cited here only for + what it implies about Layer 5's boundary. An agent can be taught to route + around the GUI when a cheaper surface exists, which means tool adoption is not + the interface's problem alone; and 28.69% in absolute terms says that once + adoption is trained in, the binding constraint has moved somewhere other than + tool availability. - **BenchJack (2026)**: Turned the evaluation lens on the benchmarks themselves. Applied to 10 popular agent benchmarks spanning software engineering, web navigation, desktop computing, and terminal operations, its automated @@ -2355,6 +2387,20 @@ resilient, semantic, structured, and protocol-aware. full-catalog `tools/list` exposure; in production at PayPal over 2,000+ indexed tools, tool-token consumption falls from 140.2k (70.1% of context) to 1.3k (0.8%). +24. **Computer-Use Agents for Blind Users**: "Are We There Yet? Assessing + Computer-Use Agents for Blind Users' Accessible Interaction with Desktop + Applications" — Kodandaram, Padma Reddy, Bi, Zhou, Ramakrishnan, Ashok; + arXiv:2609.00524 (v1, 1 September 2026). Three-week IRB-approved diary study + with 8 blind screen-reader users; 1,258 commands across 12 desktop + applications; all models supplied with screenshots and UI trees. GPT-5 52.5% + (95% CI 49.8-55.3), Claude Sonnet 48.5%, Gemini 2.5 CU 43.9%, UI-TARS 39.8%, + Qwen3-VL 37.9%. No with/without accessibility-metadata comparison is made. +25. **CUA-Universe**: "A Scalable and Dynamic Environment for Hybrid GUI+CLI + Agents" — Shi, Wang, Fang, Liang, Jin, Zhao, Liu, Chen, Wang; + arXiv:2609.05374 (v1, 4 September 2026). 16 desktop applications; a + post-trained 9B model reaches 40.2% on OSWorld against its own GUI-only + 23.4%, and 28.69% on OSWorld-MCP against the untuned base model's 20.90%. An + agent-training result; the paper makes no interface-design claim. ### **Agent Protocols** @@ -2371,19 +2417,26 @@ resilient, semantic, structured, and protocol-aware. commit, so it carries no stable publication date and should be cited by retrieval date rather than by the date shown. Exposes `document.modelContext`. Not a W3C Standard; not on the Standards Track. - (Header date 26 August 2026; retrieved 31 August 2026.) The `ModelContext` - interface now specifies `executeTool()` alongside `registerTool()` and - `getTools()`, added 14 August 2026 and revised 17 August 2026 to accept a - structured object rather than a JSON string; `RegisteredTool.inputSchema` was - retyped from `DOMString` to `object` on 14 August 2026. On 19 August 2026 the - draft specified `AbortSignal` integration for tool execution and preserved - in-flight executions after unregistration (PRs #247, #248) — cancellation and - teardown semantics, which are the properties a page needs before it can - safely expose a long-running Layer 5 tool to an agent. Implementation status - per the repository's `implementation-status.md` as of 26 August 2026: Chrome + (Retrieved 7 September 2026; the header then read 4 September 2026.) The + `ModelContext` interface now specifies `executeTool()` alongside + `registerTool()` and `getTools()`, added 14 August 2026 and revised 17 August + 2026 to accept a structured object rather than a JSON string; + `RegisteredTool.inputSchema` was retyped from `DOMString` to `object` on 14 + August 2026. On 19 August 2026 the draft specified `AbortSignal` integration + for tool execution and preserved in-flight executions after unregistration + (PRs #247, #248) — cancellation and teardown semantics, which are the + properties a page needs before it can safely expose a long-running Layer 5 + tool to an agent. On 3 September 2026 `ToolAnnotations` gained + `consequentialHint`, defined as "If true, indicates that executing the tool + will result in consequential actions that are significant, real-world, or + non-reversible, ex: booking a flight, transferring money", joining + `readOnlyHint` and `untrustedContentHint`. Implementation status per the + repository's `implementation-status.md`, re-checked 7 September 2026: Chrome 149 and Edge 150 origin trials, Brave Leo AI chat experimental, ChatGPT - Desktop listed as supported (added 26 August 2026), Firefox and Safari - standards-positions entries only. + Desktop listed as supported (added 26 August 2026, the first non-browser + client), Firefox and Safari standards-positions entries only. The + repository's README lists headless task completion as a goal and fully + autonomous browser-less workflows as a non-goal. 3. **Agent-to-Agent Protocol (A2A)**: https://a2a-protocol.org — launched by Google April 2025; the Linux Foundation announced A2A as the project's new home on 23 June 2025. Specification v1.0.0 was released on 12 March 2026