Skip to content

Weekly research scan [2026-09-07] #171

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jgoldfoot merged 1 commit into from
Sep 7, 2026
Merged

Weekly research scan [2026-09-07] #171

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 61 additions & 17 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,31 @@ merely possible.
because it is the same move the framework argues for at Layer 5 — make the
tool route cheap and findable — implemented at enterprise scale rather than
proposed.
- **A populated accessibility tree is the floor, not the finish line.** A
three-week deployment study put a screen-reader-accessible computer-use agent
in front of **8 blind users**, collecting **1,258 real commands across 12
desktop applications**; every model received both screenshots **and** UI trees
carrying control roles and hierarchy. The strongest, GPT-5, succeeded on
**52.5%** (95% CI 49.8-55.3), ahead of Claude Sonnet 48.5%, Gemini 2.5 CU
43.9%, UI-TARS 39.8% and Qwen3-VL 37.9%. The failure classes the authors name
are grounding, planning, constraint-tracking and termination
(arXiv:2609.00524). **The study does not compare agents with and without
accessibility metadata**, so it is not a measurement of Layer 2's benefit.
What it does establish is the boundary of this framework's own claim:
supplying a well-formed AOM removes a class of failure, and about half of real
user commands still fail for reasons no structural layer reaches.
- **The adoption gap is partly trainable, which bounds what interface design
alone can fix.** **CUA-Universe** (2026) turned 16 real desktop applications
into hybrid GUI+CLI environments and post-trained a 9B model on the harvested
trajectories. Against its own GUI-only operating point that model rises from
**23.4% to 40.2%** success on OSWorld (+16.8 pts); against the untuned base
model it lifts OSWorld-MCP from **20.90% to 28.69%** (+7.79 pts) while cutting
steps 27% and tokens 30% (arXiv:2609.05374). **The paper makes no claim about
interface or website design** — it is an agent-training result, recorded here
because it cuts both ways for Layer 5. An agent can be taught to prefer the
cheaper non-GUI route, so adoption is not solely the interface's problem; and
28.69% absolute on OSWorld-MCP says the remaining ceiling sits somewhere other
than tool availability.
- **The source benchmarks have themselves been shown to be gameable.**
**BenchJack** (2026) audited 10 popular agent benchmarks spanning software
engineering, web navigation, desktop computing, and terminal operations, and
Expand Down Expand Up @@ -507,23 +532,28 @@ npm test
Windows UI Automation with vision (Microsoft, 2025)
- **WebMCP** — Browser API letting a page register tools for in-browser agents
via `document.modelContext`. W3C Web Machine Learning Community Group, Draft
Community Group Report — a continuously updated editor's draft with no stable
publication date; the header read **26 August 2026** when retrieved on 31
August 2026. Not a W3C Standard; not on the Standards Track. The
`ModelContext` interface gained a specified `executeTool()` method on 14
August 2026, revised on 17 August 2026 to take a structured object rather than
a JSON string; `RegisteredTool.inputSchema` was retyped from `DOMString` to
`object` in the same window. On 19 August 2026 the draft specified
`AbortSignal` integration for tool execution and preserved in-flight
executions after unregistration (PRs #247, #248) — the cancellation and
teardown semantics a page needs before exposing a long-running tool to an
agent. **Implementation status as of 26 August 2026** (per the spec
repository's own `implementation-status.md`, which is the source for these):
origin trials live in **Chrome 149** and **Edge 150**; experimental support in
Brave's Leo AI chat; **ChatGPT Desktop** listed as supporting WebMCP — the
first non-browser client on the page, added 26 August 2026 in a pull request
from an OpenAI-affiliated author and merged by a spec editor. Firefox and
Safari have standards-positions entries only.
Community Group Report — a continuously updated editor's draft, so it is cited
by retrieval date rather than by the date in its header (retrieved 7 September
2026; header then read 4 September 2026). Not a W3C Standard; not on the
Standards Track. The `ModelContext` interface gained a specified
`executeTool()` method on 14 August 2026, revised on 17 August 2026 to take a
structured object rather than a JSON string; `RegisteredTool.inputSchema` was
retyped from `DOMString` to `object` in the same window. On 19 August 2026 the
draft specified `AbortSignal` integration for tool execution and preserved
in-flight executions after unregistration (PRs #247, #248) — the cancellation
and teardown semantics a page needs before exposing a long-running tool to an
agent. On **3 September 2026** `ToolAnnotations` gained
**`consequentialHint`** — "If true, indicates that executing the tool will
result in consequential actions that are significant, real-world, or
non-reversible, ex: booking a flight, transferring money" — joining
`readOnlyHint` and `untrustedContentHint`. **Implementation status as
re-checked on 7 September 2026** (per the spec repository's own
`implementation-status.md`, which is the source for these): origin trials live
in **Chrome 149** and **Edge 150**; experimental support in Brave's Leo AI
chat; **ChatGPT Desktop** listed as supporting WebMCP — the first non-browser
client on the page, added 26 August 2026 in a pull request from an
OpenAI-affiliated author and merged by a spec editor. Firefox and Safari have
standards-positions entries only.
- **ComponentBench** — "Diagnosing Component-Level Failures in Computer-Use
Agents" — Guan, Lin, Cheng-Yue, Wang, Zhou (arXiv:2608.18307, 18 August 2026).
97 canonical UI components as 2,910 programmatically verified tasks with human
Expand All @@ -537,6 +567,20 @@ npm test
standardized contract for `tools/call` result forms. Both target the
discoverability half of the adoption gap. Tasks moved into an official
extension (SEP-2663). https://blog.modelcontextprotocol.io/posts/mcp-roadmap/
- **Computer-Use Agents for Blind Users** — "Are We There Yet? Assessing
Computer-Use Agents for Blind Users' Accessible Interaction with Desktop
Applications" — Kodandaram, Padma Reddy, Bi, Zhou, Ramakrishnan, Ashok
(arXiv:2609.00524, 1 September 2026). Three-week diary study, 8 blind
screen-reader users, 1,258 commands across 12 desktop applications, all models
given screenshots and UI trees. GPT-5 52.5% (95% CI 49.8-55.3), Claude Sonnet
48.5%, Gemini 2.5 CU 43.9%, UI-TARS 39.8%, Qwen3-VL 37.9%. No with/without
accessibility-metadata comparison is made.
- **CUA-Universe** — "A Scalable and Dynamic Environment for Hybrid GUI+CLI
Agents" — Shi, Wang, Fang, Liang, Jin, Zhao, Liu, Chen, Wang
(arXiv:2609.05374, 4 September 2026). Environment-to-data pipeline over 16
desktop applications; a post-trained 9B model reaches 40.2% on OSWorld against
its own GUI-only 23.4%, and 28.69% on OSWorld-MCP against the untuned base
model's 20.90%. An agent-training result; makes no interface-design claim.
- **Code execution with MCP** — Building more efficient agents (Anthropic,
Nov 2025)

Expand Down
77 changes: 65 additions & 12 deletions docs/whitepaper.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -318,6 +318,23 @@ Furthermore, insights from **VisualWebArena** demonstrate that pure visual
reasoning frequently fails in complex interfaces without strong textual and
structural grounding. Layer 2 provides this essential grounding.

**A populated AOM is the floor, not the finish line.** A three-week deployment
study of a screen-reader-accessible computer-use agent — 8 blind users, 1,258
real commands across 12 desktop applications, every model supplied with both
screenshots and UI trees carrying control roles and hierarchy — found the
strongest model, GPT-5, succeeding on only 52.5% of commands (95% CI 49.8-55.3),
followed by Claude Sonnet at 48.5%, Gemini 2.5 CU at 43.9%, UI-TARS at 39.8% and
Qwen3-VL at 37.9%. The failure classes the authors identify are grounding,
planning, constraint-tracking and termination (arXiv:2609.00524). Two things
follow, and the second constrains the first. The study makes **no** comparison
between agents with and without accessibility metadata, so it is not a
measurement of what Layer 2 buys; nothing here should be read as quantifying
this layer's contribution. What it does bound is the claim above: a well-formed
AOM supplies reliable interaction targets, and roughly half of real user
commands still fail for reasons that live in the agent's planning and
termination behaviour rather than in the interface's semantics. Layer 2 removes
Comment on lines +333 to +335

@chatgpt-codex-connector chatgpt-codex-connector Bot Sep 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid attributing every failed command to planning

The 52.5% success rate only establishes that roughly half the commands failed; it does not establish that all of those failures arose from planning or termination. The preceding sentence explicitly lists grounding and constraint-tracking as additional failure classes, and the study has no accessibility-metadata ablation, so concluding that the failures occurred for reasons outside interface semantics overstates the evidence and undermines the stated caveat about not measuring Layer 2.

Useful? React with 👍 / 👎.

a class of failure. It does not remove the rest.

As modern interfaces increasingly rely on Web Components, the encapsulation
provided by the Shadow DOM creates a new challenge: it hides critical state and
semantic information from the AOM. To solve this, developers must use the
Expand Down Expand Up @@ -616,6 +633,21 @@ it's measurable:
this framework names — making the tool route cheap to find and cheap to carry
— and it converges with the progressive-discovery direction the MCP roadmap
set out in August 2026.
- **CUA-Universe (2026)**: The complement to the previous entry, and a check on
reading the adoption gap as purely an interface problem. Its App-Forge
pipeline turns 16 real desktop applications into reproducible hybrid GUI+CLI
environments, Task-Weave synthesizes hybrid tasks over them, and Path-Steer
harvests verified efficient trajectories for post-training. A 9B model trained
on that data reaches 40.2% success on OSWorld against its own GUI-only
operating point of 23.4% (+16.8 pts), and 28.69% on OSWorld-MCP against the
untuned base model's 20.90% (+7.79 pts), with 27% fewer steps and 30% fewer
tokens (arXiv:2609.05374). The paper makes **no claim about interface or
website design** — it is an agent-training result, and is cited here only for
what it implies about Layer 5's boundary. An agent can be taught to route
around the GUI when a cheaper surface exists, which means tool adoption is not
the interface's problem alone; and 28.69% in absolute terms says that once
adoption is trained in, the binding constraint has moved somewhere other than
tool availability.
Comment on lines +648 to +650

@chatgpt-codex-connector chatgpt-codex-connector Bot Sep 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Do not infer the binding constraint from absolute success

A 28.69% OSWorld-MCP success rate does not show that tool availability is no longer the binding constraint. The reported comparison is between a post-trained model and its untuned base, not between complete and incomplete tool coverage, and the cited paper is expressly described here as making no interface-design claim. Missing or unsuitable tools could therefore still explain failures, so this causal conclusion should be removed or supported by a tool-availability ablation.

Useful? React with 👍 / 👎.

- **BenchJack (2026)**: Turned the evaluation lens on the benchmarks themselves.
Applied to 10 popular agent benchmarks spanning software engineering, web
navigation, desktop computing, and terminal operations, its automated
Expand Down Expand Up @@ -2355,6 +2387,20 @@ resilient, semantic, structured, and protocol-aware.
full-catalog `tools/list` exposure; in production at PayPal over 2,000+
indexed tools, tool-token consumption falls from 140.2k (70.1% of context)
to 1.3k (0.8%).
24. **Computer-Use Agents for Blind Users**: "Are We There Yet? Assessing
Computer-Use Agents for Blind Users' Accessible Interaction with Desktop
Applications" — Kodandaram, Padma Reddy, Bi, Zhou, Ramakrishnan, Ashok;
arXiv:2609.00524 (v1, 1 September 2026). Three-week IRB-approved diary study
with 8 blind screen-reader users; 1,258 commands across 12 desktop
applications; all models supplied with screenshots and UI trees. GPT-5 52.5%
(95% CI 49.8-55.3), Claude Sonnet 48.5%, Gemini 2.5 CU 43.9%, UI-TARS 39.8%,
Qwen3-VL 37.9%. No with/without accessibility-metadata comparison is made.
25. **CUA-Universe**: "A Scalable and Dynamic Environment for Hybrid GUI+CLI
Agents" — Shi, Wang, Fang, Liang, Jin, Zhao, Liu, Chen, Wang;
arXiv:2609.05374 (v1, 4 September 2026). 16 desktop applications; a
post-trained 9B model reaches 40.2% on OSWorld against its own GUI-only
23.4%, and 28.69% on OSWorld-MCP against the untuned base model's 20.90%. An
agent-training result; the paper makes no interface-design claim.

### **Agent Protocols**

Expand All @@ -2371,19 +2417,26 @@ resilient, semantic, structured, and protocol-aware.
commit, so it carries no stable publication date and should be cited by
retrieval date rather than by the date shown. Exposes
`document.modelContext`. Not a W3C Standard; not on the Standards Track.
(Header date 26 August 2026; retrieved 31 August 2026.) The `ModelContext`
interface now specifies `executeTool()` alongside `registerTool()` and
`getTools()`, added 14 August 2026 and revised 17 August 2026 to accept a
structured object rather than a JSON string; `RegisteredTool.inputSchema` was
retyped from `DOMString` to `object` on 14 August 2026. On 19 August 2026 the
draft specified `AbortSignal` integration for tool execution and preserved
in-flight executions after unregistration (PRs #247, #248) — cancellation and
teardown semantics, which are the properties a page needs before it can
safely expose a long-running Layer 5 tool to an agent. Implementation status
per the repository's `implementation-status.md` as of 26 August 2026: Chrome
(Retrieved 7 September 2026; the header then read 4 September 2026.) The
`ModelContext` interface now specifies `executeTool()` alongside
`registerTool()` and `getTools()`, added 14 August 2026 and revised 17 August
2026 to accept a structured object rather than a JSON string;
`RegisteredTool.inputSchema` was retyped from `DOMString` to `object` on 14
August 2026. On 19 August 2026 the draft specified `AbortSignal` integration
for tool execution and preserved in-flight executions after unregistration
(PRs #247, #248) — cancellation and teardown semantics, which are the
properties a page needs before it can safely expose a long-running Layer 5
tool to an agent. On 3 September 2026 `ToolAnnotations` gained
`consequentialHint`, defined as "If true, indicates that executing the tool
will result in consequential actions that are significant, real-world, or
non-reversible, ex: booking a flight, transferring money", joining
`readOnlyHint` and `untrustedContentHint`. Implementation status per the
repository's `implementation-status.md`, re-checked 7 September 2026: Chrome
149 and Edge 150 origin trials, Brave Leo AI chat experimental, ChatGPT
Desktop listed as supported (added 26 August 2026), Firefox and Safari
standards-positions entries only.
Desktop listed as supported (added 26 August 2026, the first non-browser
client), Firefox and Safari standards-positions entries only. The
repository's README lists headless task completion as a goal and fully
autonomous browser-less workflows as a non-goal.
3. **Agent-to-Agent Protocol (A2A)**: https://a2a-protocol.org — launched by
Google April 2025; the Linux Foundation announced A2A as the project's new
home on 23 June 2025. Specification v1.0.0 was released on 12 March 2026
Expand Down
Loading