Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 30 additions & 5 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,10 @@ agent, planner, model adapter, or robotics stack.
An **Operational Surface** is any bounded interface that can be observed, acted on through
declared runtime capabilities, and verified by a fresh observation. It may be a software GUI,
browser page, mobile screen, native app window, device panel, appliance control surface, or a
simple embodied-control surface. This is the protocol's **L0**: it defines what kind of object can
enter the visual-agent loop. The P0 contract covers the surface operation layer, not full robotics,
navigation, manipulation, or world modeling.
simple embodied-control surface. This is the protocol layer's **L0**, downstream of the
product-intent L0 in `README.md`, `README.zh-CN.md`, and `PRD.md`: it defines what kind of object
can enter the visual-agent loop. The P0 contract covers the surface operation layer, not full
robotics, navigation, manipulation, or world modeling.

## Narrative Rule

Expand Down Expand Up @@ -80,13 +81,37 @@ contract.

## Source Of Truth

The current documentation authority chain has exactly three files:
The current documentation authority chain has exactly four files:

1. `AGENTS.md` - build charter for agents.
2. `README.md` - public English thesis and product narrative.
3. `README.zh-CN.md` - public Chinese thesis and product narrative.
4. `PRD.md` - equivalent formal L0 projection, P0 scope, non-goals, downstream chain and owner boundary.

No other file is currently authoritative for public positioning, architecture, or product scope.
Protocol layer documents, schemas, fixtures, runtimes, examples and adapters are downstream
contract/manual assets under this root authority; they are not independent product authorities.

## Formal Development Flow

Use this in-repo L0-L4 chain as the portable development contract. Do not
drive durable protocol or runtime changes from repo-external process notes alone.
The protocol layer has its own L0-L3 terms, but those terms are downstream of
root product-intent L0:

```text
README.md / README.zh-CN.md / PRD.md
-> AGENTS.md
-> protocol layer documents
-> schemas + fixtures + conformance checks
-> runtimes/desktop-gui + adapters + examples
-> scripts/verify-local.sh evidence
```

- Product L0: `README.md`, `README.zh-CN.md`, and `PRD.md` define why visual-agent exists, P0 scope, non-goals, and owner boundary.
- Protocol L1/L2: protocol documents, schemas, fixtures, and conformance checks define the verifiable surface-operation contract.
- L3: runtime code, adapters, examples, manifests, and repo commands realize the protocol under the product boundary.
- L4: protocol conformance, Swift typecheck, shell syntax, smoke checks, and `VERIFY_LOCAL_OK` provide validation evidence; `ActionResult.ok == true` and passing local checks do not prove task success or product acceptance.

## Open-Source Surface

Expand Down Expand Up @@ -155,5 +180,5 @@ Needs confirmation:
- <small number of decisions>
```

Do not expand P1/P2 into P0. Do not rewrite the three-file root contract without explicit user
Do not expand P1/P2 into P0. Do not rewrite the four-file root contract without explicit user
approval.
70 changes: 70 additions & 0 deletions PRD.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# visual-agent PRD

`PRD.md` is the fixed formal L0 projection of the product narrative in `README.md`
and `README.zh-CN.md` for `visual-agent`. Together they define product intent
before protocol layer documents, schemas, fixtures, runtime code, adapters, or
integration guides are changed.

It is intentionally thin. It does not replace the public product narrative, and
it is not a fragmented spec set, implementation plan, or runtime manual.

## L0 Problem

Visual models and hosts can understand visible interfaces, but safe operation
requires a project-owned boundary around which surfaces are eligible, what a
runtime may execute, how surface-specific knowledge is represented, how task
intents expand into actions, and how success is verified by fresh observation.

`visual-agent` exists to provide that protocol-first boundary without becoming
a computer-use agent, planner, model adapter, browser automation framework, MCP
server, or robotics stack.

## P0 Scope

P0 is the protocol-first surface operation system:

- Operational Surface eligibility and boundary.
- Runtime capability declaration and machine-readable action results.
- Surface manuals for concrete operational surfaces.
- Scenario requirements and manifests that expand into safe action plans.
- Fresh-observation verification after declared runtime actions.

The current desktop GUI runtime is a reference implementation of this product
boundary, not the product boundary itself.

## Non-Goals

- Do not use per-layer protocol documents as independent product PRDs.
- Do not let runtime code redefine product scope or layer semantics.
- Do not hide surface eligibility inside runtime, L1, L2, or adapter code.
- Do not add app-specific, site-specific, or device-specific workflows to
generic runtime code.
- Do not treat `ActionResult.ok == true` as task success.

## Downstream Chain

Formal development flows from the co-equal public narrative and PRD L0 assets
into protocol and runtime assets:

```text
README.md / README.zh-CN.md / PRD.md
-> AGENTS.md
-> protocol layer documents
-> schemas + fixtures + conformance checks
-> runtime code + adapters + examples
-> verification evidence
```

Protocol layer documents are downstream contract/manual assets. They can define
protocol-layer L0/L1/L2/L3 behavior, but they must remain aligned with this PRD
and the public product narrative.

## Owner Boundary

Human owner or architect owns product intent, layer semantics, and P0 scope.
Agents and engineers may maintain protocol documents, schemas, fixtures, and
runtime code only under that frozen boundary.

Return to public narrative / PRD review when a change would alter product
scope, layer ownership, safety semantics, adapter boundaries, or verification
meaning.
9 changes: 8 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,12 @@ what a concrete surface means, which actions are safe, and how success is verifi

That structure is **visual-agent**.

This `README.md` and [`README.zh-CN.md`](./README.zh-CN.md) are the public
product narrative. [`PRD.md`](./PRD.md) is the fixed equivalent formal L0 projection:
it compresses the same product intent and P0 scope before protocol layer
documents, schemas, fixtures, runtime code, adapters, or integration guides are
changed.

## The Problem

Most computer-use, GUI automation, device-control, and embodied-control projects start from an
Expand Down Expand Up @@ -37,7 +43,8 @@ panels, appliance controls, instrument displays, and simple embodied-control sur
protocol is about operating the surface. Full robotics, navigation, grasping, force control, and
world modeling are outside P0.

This is the protocol's **L0: Operational Surface Contract**. L0 defines what kind of visible
Within the protocol layer, this is the **L0: Operational Surface Contract**. It is downstream of
the product-intent L0 in the public narrative and `PRD.md`, and defines what kind of visible
interface can enter the visual-agent loop before any runtime action, surface manual, or scenario
manifest is considered.

Expand Down
6 changes: 5 additions & 1 deletion README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@

这套结构就是 **visual-agent**。

本 `README.zh-CN.md` 和 [`README.md`](./README.md) 是面向人的公开产品叙事。[`PRD.md`](./PRD.md)
是固定的等价形式化 L0 投影:它压缩同一组产品意图和 P0 范围,再由协议层文档、schema、fixture、
运行时代码、adapter 和集成说明承接下游实现。

## 问题

很多计算机使用、图形界面自动化、设备控制和简单具身控制项目,都会从智能体循环、沙箱、浏览器、控制器或机器人栈开始。这些都重要,但它们中间缺了一层:从视觉理解到可信操作之间,需要协议和交互界面说明体系。
Expand All @@ -29,7 +33,7 @@

它可以是软件 GUI、浏览器页面、移动界面、原生应用窗口、打印机或 ATM 面板、家电控制面板、仪表显示,也可以是简单具身控制中的按钮、旋钮、指示灯和设备屏幕组合。核心协议处理的是“可见交互界面的操作”这一层;完整机器人、导航、抓取、力控和世界建模不属于 P0。

这就是协议的 **L0:Operational Surface Contract**。L0 定义什么样的可见交互界面可以进入 visual-agent 循环,先于任何运行时动作、交互界面说明书或场景清单。
在协议层内部,这就是 **L0:Operational Surface Contract**。它位于产品叙事和 `PRD.md` 的产品意图 L0 下游,用来定义什么样的可见交互界面可以进入 visual-agent 循环,先于任何运行时动作、交互界面说明书或场景清单。

## 运行模型

Expand Down
2 changes: 1 addition & 1 deletion integration/host-embedding.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Runtime Integration - Host Embedding Guide

- Kind: Engineering integration spec (NOT a protocol layer)
- Kind: Engineering integration guide (NOT a protocol layer)
- Version: 1
- Status: Normative for integration
- Builds on: the visual-agent L0/L1/L2/L3 contract in `../protocol/` and
Expand Down
24 changes: 13 additions & 11 deletions protocol/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,10 @@
- Audience: AI agents (planners, code editors, reviewers). For humans, start at `../README.md`.
- Status: Normative map for the current tracked public surface.
- Reading mode: this file fuses prose with YAML so agents can orient quickly.
- Authority rule: the root files (`AGENTS.md`, `README.md`, `README.zh-CN.md`) define public
positioning and P0 scope. Per-layer protocol files define current desktop GUI reference-runtime
behavior.
- Authority rule: the root files (`AGENTS.md`, `README.md`, `README.zh-CN.md`, `PRD.md`) define
product intent. `README.md` and `README.zh-CN.md` carry the public product narrative, while
`PRD.md` carries the equivalent formal L0 projection. Per-layer protocol files define current
desktop GUI reference-runtime behavior.

## 0. What This Document Is For

Expand All @@ -21,10 +22,11 @@ not make the desktop GUI runtime the product boundary.
documents:
root_contract:
- AGENTS.md
- PRD.md
- README.md
- README.zh-CN.md
overview: protocol/README.md
protocol_specs:
protocol_layer_docs:
L0: protocol/L0-operational-surface-contract.md
L1: protocol/L1-runtime-capability-manual.md
L2: protocol/L2-surface-manual.md
Expand All @@ -46,7 +48,7 @@ adapters:
reference_rule:
direction: runtime --> protocols
on_scope_conflict: "root contract wins for product boundary and P0 scope"
on_runtime_behavior_conflict: "protocol spec wins over code; update spec first, then code"
on_runtime_behavior_conflict: "update the relevant protocol layer document before changing observable runtime behavior"
```

## 1. L0-L3 Layer Map
Expand Down Expand Up @@ -78,7 +80,7 @@ layers:
- task intent expansion
- robotics navigation/grasping/force/world modeling
authority: root_contract
spec: protocol/L0-operational-surface-contract.md
layer_doc: protocol/L0-operational-surface-contract.md
L1:
name: Runtime Capability Manual
manual: runtime-capability-manual
Expand All @@ -92,7 +94,7 @@ layers:
- operational surface semantics
- business/task judgement
- scenario state
current_reference_spec: L1
current_reference_doc: L1
L2:
name: Surface Manual
manual: surface-manual
Expand All @@ -107,7 +109,7 @@ layers:
- reimplementing L1 mechanics
- task aggregation/reporting
- raw event dispatch
current_reference_spec: L2
current_reference_doc: L2
L3:
name: Scenario Requirements
manual: scenario-requirements
Expand All @@ -122,7 +124,7 @@ layers:
- raw pointer/device/actuator events
- runtime coordinate arithmetic as task logic
- deciding task success without fresh observation
current_reference_spec: L3
current_reference_doc: L3
```

## 2. Desktop GUI Reference Runtime Bridge
Expand Down Expand Up @@ -251,7 +253,7 @@ for semantics, layer boundaries, and future extension rules.

## 5. Cross-Cutting Invariants

These invariants bind the root contract, current reference specs, and runtime files.
These invariants bind the root contract, current reference layer documents, and runtime files.

```yaml
invariants:
Expand Down Expand Up @@ -320,7 +322,7 @@ playbooks:
- "if collecting data: add normalization + anonymization"
- "validate every intent with task_flow --manifest <manifest> preview-json"
edit_runtime_code:
before: "read the file's AGENT BINDING BLOCK and locate cited spec sections"
before: "read the file's AGENT BINDING BLOCK and locate cited protocol sections"
rule: "if a change alters observable behavior, update the protocol section first, then code"
never: "make the protocol merely describe current code"
review_change:
Expand Down
2 changes: 1 addition & 1 deletion runtimes/desktop-gui/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ runtimes/desktop-gui/
## Agent-Facing Documents

- `../../protocol/README.md` is the agent-facing protocol map. It connects the root contract to
L0/L1/L2/L3 specs, runtime files, examples, integration docs, adapters, and invariants.
L0/L1/L2/L3 protocol documents, runtime files, examples, integration docs, adapters, and invariants.
- Each `src/*.swift` file has an `AGENT BINDING BLOCK` that points code symbols to protocol
sections. Runtime files cite protocol sections; the protocol is not derived from code.

Expand Down
18 changes: 9 additions & 9 deletions runtimes/desktop-gui/src/action_executor.swift
Original file line number Diff line number Diff line change
Expand Up @@ -18,15 +18,15 @@
// l0: ../../../protocol/L0-operational-surface-contract.md
// overview: ../../../protocol/README.md
// implements:
// - { symbol: TargetApp, spec_section: "L1 section 3 Desktop GUI Target Binding" }
// - { symbol: appObservation, spec_section: "L1 section 4 AppObservation (desktop GUI reference)" }
// - { symbol: JSONActionPlan/runPlan, spec_section: "L1 section 5 ActionPlan, section 11 ActionResult" }
// - { symbol: executeAction, spec_section: "L1 section 6 Action Set" }
// - { symbol: eventPoint, spec_section: "L1 section 7 Coordinate Spaces" }
// - { symbol: ensurePointerSafe/runPlan, spec_section: "L1 section 8 Safety Invariants" }
// - { symbol: postMouseSequence/postEsc/postScroll, spec_section: "L1 section 9 Backends" }
// - { symbol: visualStats/imageUsableForVision/captureTargetWindowImage, spec_section: "L1 section 10 Capture Usability" }
// - { symbol: "diagnose/observe/window/screens", spec_section: "L1 section 13 Diagnostics" }
// - { symbol: TargetApp, protocol_section: "L1 section 3 Desktop GUI Target Binding" }
// - { symbol: appObservation, protocol_section: "L1 section 4 AppObservation (desktop GUI reference)" }
// - { symbol: JSONActionPlan/runPlan, protocol_section: "L1 section 5 ActionPlan, section 11 ActionResult" }
// - { symbol: executeAction, protocol_section: "L1 section 6 Action Set" }
// - { symbol: eventPoint, protocol_section: "L1 section 7 Coordinate Spaces" }
// - { symbol: ensurePointerSafe/runPlan, protocol_section: "L1 section 8 Safety Invariants" }
// - { symbol: postMouseSequence/postEsc/postScroll, protocol_section: "L1 section 9 Backends" }
// - { symbol: visualStats/imageUsableForVision/captureTargetWindowImage, protocol_section: "L1 section 10 Capture Usability" }
// - { symbol: "diagnose/observe/window/screens", protocol_section: "L1 section 13 Diagnostics" }
// invariants: # see L1 section 14; violating these breaks the protocol
// - actions_executed_in_order
// - coordinates_default_windowTopLeft
Expand Down
4 changes: 2 additions & 2 deletions runtimes/desktop-gui/src/scene_runner.swift
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@
// l0: ../../../protocol/L0-operational-surface-contract.md
// overview: ../../../protocol/README.md
// implements:
// - { symbol: loadProfile + --profile handling, spec_section: "L2 section 3 Surface Profile Binding (E1)" }
// - { symbol: process-delegation, spec_section: "L2 section 2 Layer Boundaries (no L1 re-impl)" }
// - { symbol: loadProfile + --profile handling, protocol_section: "L2 section 3 Surface Profile Binding (E1)" }
// - { symbol: process-delegation, protocol_section: "L2 section 2 Layer Boundaries (no L1 re-impl)" }
// delegates_to: ./action_executor.swift # L1 executor; resolved as a sibling file
// externalized_to_surface_data: # these L2 extension points live in profiles/manifests/docs, not here
// - "L2 section 4 Observed States (E2)"
Expand Down
10 changes: 5 additions & 5 deletions runtimes/desktop-gui/src/task_flow.swift
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,11 @@
// l0: ../../../protocol/L0-operational-surface-contract.md
// overview: ../../../protocol/README.md
// implements:
// - { symbol: loadManifest, spec_section: "L3 section 7 Scenario Manifest (E4)" }
// - { symbol: resolvedIntentFields, spec_section: "L3 section 6 State Persistence and Merge Rules" }
// - { symbol: "substitute/buildActions", spec_section: "L3 section 4 Action-Step Templates (E2)" }
// - { symbol: actionPlan, spec_section: "L3 section 3 Intent Set, section 10 Invariants" }
// - { symbol: executePlan, spec_section: "L3 section 2 Layer Boundary (delegate, no raw events)" }
// - { symbol: loadManifest, protocol_section: "L3 section 7 Scenario Manifest (E4)" }
// - { symbol: resolvedIntentFields, protocol_section: "L3 section 6 State Persistence and Merge Rules" }
// - { symbol: "substitute/buildActions", protocol_section: "L3 section 4 Action-Step Templates (E2)" }
// - { symbol: actionPlan, protocol_section: "L3 section 3 Intent Set, section 10 Invariants" }
// - { symbol: executePlan, protocol_section: "L3 section 2 Layer Boundary (delegate, no raw events)" }
// delegates_to: ./action_executor.swift # L1 executor; resolved as a sibling file
// not_implemented_here: # conditional L3 extension points realized by the scenario, not the engine
// - "L3 section 5 Field/Normalization Conventions (E3)" # data-collecting scenarios add their own tooling
Expand Down
53 changes: 53 additions & 0 deletions scripts/verify-local.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,59 @@ cd "$ROOT"
echo "== diff hygiene =="
git diff --check

require_contains() {
local file="$1"
local expected="$2"
grep -Fq -- "$expected" "$file" || {
echo "missing root authority marker in $file: $expected" >&2
exit 1
}
}

reject_contains() {
local file="$1"
local forbidden="$2"
if grep -Fq -- "$forbidden" "$file"; then
echo "forbidden source-method leakage marker in $file: $forbidden" >&2
exit 1
fi
}

echo "== root authority conformance =="
require_contains AGENTS.md "current documentation authority chain has exactly four files"
require_contains AGENTS.md "README.md / README.zh-CN.md / PRD.md"
require_contains AGENTS.md "Product L0"
require_contains AGENTS.md "Protocol L1/L2"
require_contains AGENTS.md "VERIFY_LOCAL_OK"
require_contains README.md "fixed equivalent formal L0 projection"
require_contains README.md '[`PRD.md`](./PRD.md)'
require_contains README.zh-CN.md "PRD.md"
require_contains PRD.md "fixed formal L0 projection"
require_contains PRD.md "## P0 Scope"
require_contains PRD.md "## Downstream Chain"
require_contains PRD.md "## Owner Boundary"
require_contains PRD.md "README.md / README.zh-CN.md / PRD.md"
require_contains protocol/README.md "AGENTS.md"
require_contains protocol/README.md "PRD.md"

echo "== source-method leakage scan =="
for file in AGENTS.md README.md README.zh-CN.md PRD.md protocol/README.md integration/host-embedding.md; do
reject_contains "$file" "source-side method"
reject_contains "$file" "execution-context detail"
reject_contains "$file" "agent-host"
reject_contains "$file" "source-function"
reject_contains "$file" "selected workflow"
reject_contains "$file" "selected skill"
reject_contains "$file" "target runtime external method"
reject_contains "$file" "agent runtime external method"
reject_contains "$file" "runtime dependency on source"
reject_contains "$file" "devops dependency on source"
reject_contains "$file" "l0_refs"
reject_contains "$file" "l1_l2_refs"
reject_contains "$file" "l3_refs"
reject_contains "$file" "l4_validation"
done

echo "== protocol conformance =="
node --check scripts/verify-protocol-conformance.mjs
node scripts/verify-protocol-conformance.mjs
Expand Down
Loading