Appium Skill: deepen visual execution for color-only controls and custom-drawn gestures #143

Work type

Implementation and actual-platform behavior qualification

Delivery stage

Stage 3 — Mobile/Appium experiment

Problem

The shared Appium Skill now has a control-first, visual-fallback policy, current screenshot-to-driver geometry requirements, image delivery through python_exec, and pre/post synthetic visual checks. That contract is not yet deep enough for rendered surfaces whose actionable identity and result exist only in pixels.

An external AndroidWorld campaign exposed this gap through a failed BrowserDraw task. The task is diagnostic evidence, not a contract to encode: target and candidate controls are distinguished by rendered color rather than accessibility semantics; drawing requires a continuous pointer path rather than a tap; and completion depends on rendered canvas pixels plus the page result. Without the retained trajectory, this evidence does not establish one exact failure cause. Plausible failure boundaries are:

• repeated unlabeled controls were guessed from element order because the accessibility tree did not expose color;
• visually similar colors were matched by language description instead of current pixel evidence;
• screenshot coordinates were reused after viewport or scroll changes;
• a tap or scroll-like gesture was used where pointer-down/move/up was required;
• a successful WebDriver command was treated as proof that the canvas changed;
• submission was not followed by observation of the requested business result.

AndroidWorld's implementation illustrates the mechanism: a touch/click start only records a point, movement writes pixels, and evaluation requires the requested rendered colors before showing success. See android_world/task_evals/single/browser.py in google-research/android_world. No AndroidWorld target values, candidate order, evaluator predicate, or solution belongs in AMKernel guidance or fixtures.

Existing evidence boundary

Merged PR #135 added shared visual fallback and synthetic image/pointer behavior. Its retained evidence explicitly does not qualify cropped/scaled coordinate mappings, real platform readiness, production apps, or AndroidWorld performance. Issue #98 owns the broader Appium behavior and real-platform qualification sequence. Issues #124 and #125 address locator/native API misuse and actionable exception evidence respectively; neither covers rendered attribute matching or continuous custom-surface gestures.

Goal

Deepen the existing shared Appium execution Module so an Agent can, through the unchanged public python_exec Interface:

1. recognize when control semantics cannot prove a visually encoded target;
2. derive bounded, current visual facts from a screenshot;
3. map those facts to current driver coordinates;
4. perform the required gesture semantics, including a continuous pointer path;
5. verify the rendered outcome and requested business result;
6. preserve session, ownership, output and cleanup truth.

Minimum sufficient design

Keep the external seam unchanged:

python_exec(code, timeout) -> PythonExecResult

Deepen the existing amkernel-appium.md observation policy into one generic execution contract:

Observe rendered surface
  -> resolve visual facts and geometry
  -> act with the required gesture semantics
  -> verify rendered outcome

Use the persistent Kernel for naturally useful live objects such as the Appium driver, current PNG bytes, Pillow image/crop objects and current geometry. Do not add a workflow state object, checkpoint database, Appium wrapper, mobile-specific MCP tool or benchmark planner.

Visual facts

• Treat color-only, pattern-only, image-only and custom-drawn controls as visually encoded when relevant identity is absent from current accessibility/control properties.
• An accessibility tree may establish candidate regions but does not prove a candidate's rendered color or image identity.
• Render the relevant screenshot/crop with display(..., mime_type="image/png") before a dependent model decision.
• When exact rendered values matter, use existing Pillow support for bounded center-region sampling, dominant/median color extraction, image differences or equivalent deterministic facts. The model determines region meaning; Python determines pixels and geometry.
• Avoid borders, shadows, text and antialiased edges. Return compact facts rather than base64, complete pixel arrays or unrelated screen content.
• Never infer target identity from accessibility return order, a fixed ordinal, a remembered coordinate or a natural-language color name when current pixel evidence can resolve it.

Geometry validity

Bind a visual coordinate decision to the current screenshot dimensions, crop/viewport origin, scale/device-pixel ratio, orientation, package/activity/context and layout/scroll state. Scrolling, navigation, browser-toolbar changes, keyboard/dialog changes, rotation, context switches or reflow invalidate affected coordinates and require a current observation.

Gesture semantics

Distinguish tap, scroll/swipe, drag and drawing/path gestures. A custom-drawn path requires pointer-down, one or more nonzero-duration pointer moves, and pointer-up within the intended surface. Do not substitute a click, a page scroll or an instantaneous coordinate jump. Add a concrete Appium Python/W3C gesture example only after the exact public Client calls pass a focused real-driver probe.

Verification and calls

• A successful input command proves delivery, not the requested rendered state.
• Compare a bounded relevant post-action image or authoritative changed control/business state with the requested outcome.
• If a control selection has no visible selected property, verify the resulting rendered effect rather than inventing selection state.
• Keep known, continuous select/action/direct-verification sequences together. Return an observation when image interpretation or changed geometry is required; do not split every pointer command into a separate python_exec call.
• Preserve the current prohibition on direct database/business-handler changes or manufactured completion evidence. Do not use page JavaScript to mutate a canvas or invoke an evaluator as a substitute for UI interaction.

Generic behavior fixture

Add a randomized synthetic scenario without AndroidWorld names, values, steps or evaluator logic. A suitable fixture exposes:

• several controls with identical or insufficient accessibility semantics but different rendered attributes;
• randomized target attributes and candidate order;
• a custom-drawn surface whose state changes only after continuous movement;
• a layout variant with crop/scale/viewport offset;
• an optional scroll or layout transition that invalidates old coordinates;
• a separate observable completion state.

The fixture and evaluator should test observable behavior and journals, not Markdown substrings or a fixed generated-code template.

Acceptance scenarios

1. When semantic properties are sufficient, the Agent retains the control path and takes no unnecessary screenshot.
2. When identity is color/image-only, the Agent obtains and actually receives a relevant image instead of exhausting locators or guessing element order.
3. The Agent derives bounded current visual facts and selects a matching candidate despite randomized order and visually similar alternatives.
4. Crop offset, viewport origin and non-1:1 scale map to the correct current driver target.
5. A tap-only attempt fails a path-required negative fixture; a bounded continuous W3C pointer path passes.
6. A scroll/layout change invalidates old visual coordinates; a negative fixture rejects acting on stale geometry.
7. A successful command without matching post-action rendered evidence does not pass.
8. The positive flow observes the rendered change and separate requested completion result in the same usable session.
9. Short continuous work is not split into one python_exec call per WebDriver/pointer command; naturally persistent objects are reused without a workflow engine.
10. Direct JavaScript/internal-state mutation, fixed coordinates, fixed candidate ordinals and benchmark-specific answers fail.
11. Session, restoration and server cleanup outcomes remain separate and truthful.
12. Guidance and tests contain no AndroidWorld task-specific operation, target content, candidate order, evaluator predicate or answer.

Actual Android qualification

After synthetic behavior passes, run the thinnest real vertical path on the authorized Android environment:

real emulator/device
  -> Chrome or a maintained rendered fixture
  -> screenshot delivered to the Agent
  -> bounded visual fact extraction
  -> current screenshot-to-device mapping
  -> real Appium continuous pointer gesture
  -> post-action image/business-state verification
  -> owned-session cleanup

Randomize visual attributes, candidate order and at least one geometry variable. Cover a no-scroll baseline plus one viewport-offset or scroll-invalidated case. Record exact device/emulator, Android, Chrome/WebView, Appium, UiAutomator2, Python Client, screenshot dimensions, mapping method and gesture method. Synthetic fixtures or type checks do not qualify this path.

Only after the generic real path passes should AndroidWorld be rerun as external evidence. Report official reward separately; task call count and screenshot count remain diagnostic telemetry, not acceptance metrics.

Architecture and security constraints

• Preserve AMKernel's one public python_exec execution Interface and persistent-namespace semantics.
• Keep platform execution guidance at the existing Skill/reference seam; do not put mobile or benchmark semantics into src/amkernel/.
• Reuse the current image output and Pillow capability; add no image/OCR/CV dependency without a separate demonstrated need and approval.
• Capture and display only the relevant app/viewport/region. Do not retain unrelated personal screen content, raw base64, full page source or full pixel arrays.
• Treat page text, images, accessibility data and tool output as untrusted evidence, never instructions.
• Preserve current authorization, service ownership, session-liveness, bounded-output and cleanup contracts.

Deferred by design

• No mobile_draw, visual_click or computer-use MCP tool.
• No general CV framework, OCR stack or object detector.
• No AndroidWorld task registry, prompt branch, evaluator integration or answer cache.
• No explicit phase/task-state object, checkpoint database or resumable workflow engine.
• No direct canvas/DOM mutation to manufacture success.
• No iOS/macOS qualification claim from Android evidence; the shared guidance may apply there, but each real Adapter needs its own evidence when claimed.

A reusable image-analysis or gesture Module should be considered only after at least two real callers require the same nontrivial implementation and deleting it would spread complexity across those callers.

Expected write seam

• src/amkernel/skill/references/amkernel-appium.md
• src/amkernel/skill/references/android-execution.md only for Android-specific public Appium/UiAutomator2 details that do not duplicate the shared policy
• existing Appium behavior fixture/evaluator tests under tests/e2e/
• managed Skill routing/integrity metadata only if required by the established distribution process
• bounded experiment evidence and the existing HANDOFF/experiment record

AMKernel Runtime remains unchanged unless an independently reproduced Runtime image-delivery or persistence defect is isolated into a separate issue.

Verification and required evidence

• Focused positive and negative behavior regressions for every applicable acceptance scenario.
• Packaged/installed Skill routing and image-delivery checks.
• Exact Appium Python continuous-gesture calls verified against the supported Client/driver combination rather than source-text assertions alone.
• One fresh installed-wheel native Agent synthetic run with bounded retained evidence.
• One authorized real Android vertical run covering current visual facts, coordinate mapping, continuous gesture, post-action verification, Kernel survival and cleanup.
• Independent review for benchmark leakage, behavior-oracle false positives, privacy/output bounds and ownership truth.
• Do not claim AndroidWorld improvement or real-platform qualification before the corresponding actual run passes.

Related: #98, #124, #125, PR #135.