Skip to content
Merged
12 changes: 9 additions & 3 deletions .docker/autobdd-base.dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -92,9 +92,15 @@ ARG OCULIX_VER
RUN chmod +x scripts/fetch-oculix.sh && \
OCULIX_VER="${OCULIX_VER}" ./scripts/fetch-oculix.sh

# Ensure the seam script is executable and on PATH
RUN chmod +x src/findTargetImage.js && \
ln -s /opt/autobdd/seam/src/findTargetImage.js /usr/local/bin/findTargetImage
# Ensure the seam is executable and wire the front door. Only the dispatcher and the
# deprecated alias are on PATH; the engine and the verb-named entry points live out of
# PATH so a derived image cannot shadow or collide with them.
RUN chmod +x src/findTargetImage.js bin/autobdd bin/findTargetImage && \
mkdir -p /usr/local/libexec/autobdd && \
ln -sf /opt/autobdd/seam/bin/autobdd /usr/local/bin/autobdd && \
ln -sf /opt/autobdd/seam/bin/findTargetImage /usr/local/bin/findTargetImage && \
ln -sf /opt/autobdd/seam/bin/autobdd /usr/local/libexec/autobdd/find-target && \
ln -sf /opt/autobdd/seam/bin/autobdd /usr/local/libexec/autobdd/read-text

# Bake the Oculix native libraries (NFR-P3: exact, self-contained image).
# The JAR bundles the Linux x86-64 natives (OpenCV + Leptonica + Tesseract) under
Expand Down
77 changes: 77 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,83 @@ the convention.
- One uniform screenshot watermark on every step/final capture — a dark bottom band with
the remark (green passed / red failed).

**Conformance suite — the front door is the default surface.** The suite drives
`/usr/local/libexec/autobdd/find-target` by default and prints the surface it used in its
banner; `TARGET_BIN=findTargetImage` runs the same matrix through the deprecated alias and
so keeps proving the alias transparent. The run lines therefore name the interface this
repo is moving to. Two guidance fixes came out of the same review: the banner now prints
invocations that actually work (the previous `make base-test` runs the script on the *host*,
which has none of the image's tooling), and the suite now detects that case and exits 2 with
a one-line pointer instead of failing once per assertion with missing-binary noise.

**Base seam — argument vocabulary.** The v1 names described *how we look* (`image` vs `ocr`)
and disagreed with themselves: `--imageSimilarity`/`--ocrSimilarity` were one concept, and
`--imageWaitTime` was **seconds** while `--ocrWaitTime` was **milliseconds**. The v2 names
describe *what you want*: `--match-image` / `--match-text` are the two ways to name the
target, and everything else is stated once — `--min-score`, `--max-score`, `--wait`,
`--limit`, `--box`, `--psm`, `--oem`. Actions became composable flags, so `--hover --click`
replaces the `hoverClick` enum string; `--match-text` is **literal by default** with
`--match-regex` opting in, so a search phrase containing `:` or `(` cannot silently change
meaning (v1 `--textHint` was regex and keeps that behaviour through the legacy path).
Durations now carry their unit (`--wait 5s`, `--wait 800ms`), which removes the s/ms split.

Every v1 name is still accepted: it is translated in the engine and warns **on stderr only**
(stdout carries the payload), so existing consumers keep working unchanged. See
`docs/CONTRACT.md` §2b. Verified: the whole conformance matrix runs green on the new
vocabulary, and a `legacy-flags` feature asserts the v1 names still match and still click.

**Base seam — front door.** The interface is now verb-object: `autobdd find-target …`
locates a target (optionally acting on it) and `autobdd read-text` reads the screen, so
"read the screen" is no longer spelled as "find a target called Screen". The engine moved
out of `PATH` to `/opt/autobdd/seam/src/`; only the front door and the deprecated alias are
on it, so a derived image cannot shadow or collide with them. `findTargetImage` is kept as a
**deprecated alias** that prints its notice on **stderr only** — stdout is parsed as the
payload — and defers to the front door, so argument translation added later applies to it
without a second implementation.

The target is now **required**: `autobdd find-target` with no target exits 2 and points at
`read-text`. Previously the engine defaulted to `--imagePath=Screen`, so omitting the target
silently ran a whole-screen scan (~3 s) and looked like a successful call, and it made
`find-target` and `read-text` behaviourally identical. `docs/CONTRACT.md` §1 always said the
target was required; this enforces it.

Verified by running the whole conformance matrix through **both** surfaces — the alias and
`/usr/local/libexec/autobdd/find-target`: **124 passed, 0 failed (43 features)** each time.

**Base seam — discovery surface.** `--help`/`-h`, `--version` and `--list` are answered
*before* the JVM and native engine start: measuring the old behaviour, `--help` returned
exit 0 after a **3079 ms whole-screen OCR scan** and the binary carried **0** usage strings,
so the canonical "what does this do?" move silently did work and reported success. Usage is
now ~40 ms and prints the flows, the output shape, the exit codes and every flag with its
default. Unusable values (a non-numeric threshold, an unknown `--imageAction`/`--ocrDetail`)
now print the offending flag on stderr and exit **2**, rather than silently behaving like the
default and looking like a successful call. Unknown arguments still warn and are ignored, so
the additive-argument guarantee is unchanged.

**Base seam — behaviour fixes found by the feature suite.** The base suite was rebuilt as a
34‑feature catalogue (`test-projects/autobdd-base-test/base-test/features.sh`), each feature
independently reproducible via `base-test/one.sh <feature>`. Writing it surfaced five defects
in `findTargetImage`, all fixed here:

- **`--imageWaitTime` was ignored.** SikuliX's `autoWaitTimeout` applies to `wait()`/
`exists()`, not to the `findAll()` the seam used, so a target that appeared late was
missed despite the caller asking to wait. The search now retries until the deadline, as
`docs/CONTRACT.md` §2 documents.
- **Clicking actions did not move the pointer.** `Region.click()`/`doubleClick()`/
`rightClick()` do not reposition the pointer in this Oculix build, so an action was
dispatched somewhere other than the reported `center` (measured: reported `900,1100`,
pointer landed at `131,160`). Actions now move to the region centre first; the suite
verifies **every** action against `xdotool`'s view of the pointer.
- **`--flash` never paused.** `java.lang.Thread.sleep()` through java-bridge is a no‑op
(`sleep(1000)` measured 0 ms), so the on‑screen flash had no hold time. Waits now use a
real Node‑side block (`Atomics.wait`), which also stops the OCR poll from spinning.
- **`--imageMaxCount` returned aliases.** One accumulator object was pushed per match, so N
results were N references to the last match (identical centres). Results are now built
fresh per match.
- **`clicked` was set for non‑clicking actions.** The image path recorded `clicked` even for
`--imageAction=hover`; it is now set only when a click was actually dispatched, matching
the OCR path and the field's documented meaning.

**Phase B — NFR hardening (in progress).**

- **Node 20 → Node 24 LTS.** Node 20 reached **EOL 2026-04-30**; the image now installs
Expand Down
181 changes: 171 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,15 +49,55 @@ object (or array) on **stdout**.
### Basic usage (image matching)

```bash
findTargetImage \
--imagePath=<path-to‑template.png> \
--imageSimilarity=0.8 \
--maxSim=1.0 \
--imageWaitTime=1000 \
--imageAction=click \
--imageMaxCount=1
autobdd find-target --match-image=<path-to-template.png> \
--min-score=0.8 --max-score=1.0 \
--wait=5s --click --limit=1
```

### The front door

```bash
autobdd find-target --match-image=logo.png --click # locate, then click
autobdd find-target --match-text="Submit" --box # find text, report its box
autobdd read-text # read the whole screen as text
autobdd --help # verbs, flags, exit codes
```

`findTargetImage` still works and is behaviourally identical — it is a **deprecated alias**
that prints one line to **stderr** and defers to the front door, so a v1 consumer keeps
working unchanged. **The conformance suite defaults to the front door**; set
`TARGET_BIN=findTargetImage` to run the same matrix through the alias. The engine lives at `/opt/autobdd/seam/src/`, out of `PATH`; only the
front door and the alias are on it, so a derived image cannot shadow either.

The target is **required**: `autobdd find-target` with no target exits `2` and points at
`read-text`. (It used to default to a whole-screen scan, so a mistyped flag silently cost
~3 s and looked like a successful call.)

The suite defaults to the **front door**. Run it through the deprecated alias instead to
prove the alias is still transparent:

```bash
AutoBDD_Ver=<v> make docker-run jobs="base-test" # default: the front door
AutoBDD_Ver=<v> docker compose run --rm \
-e TARGET_BIN=findTargetImage autobdd-base-test make base-test # the deprecated alias
```

### Discovery and exit status

```bash
findTargetImage --help # flows, output shape, exit codes, every flag + default
findTargetImage --list # the supported flows, one runnable example per line
findTargetImage --version # seam, image build stamp, Oculix, Node
```

These are answered **before the JVM and native engine start** — measured at ~40 ms, versus
~1.1 s for a real match, and they never touch the screen. Exit status is `0` when a result
was produced — **including `notFound`, which is an answer, not a fault** (branch on
`.[0].status`) — and `2` for a usage error. Unknown arguments warn on stderr and are
otherwise ignored, because the argument surface is additive; an *unusable value* (a
non‑numeric threshold, an unknown `--imageAction`) exits `2` instead of silently behaving
like the default.

### Opt‑in OCR mode (new)

All OCR‑related arguments are **optional**; if none of them are supplied the seam
Expand Down Expand Up @@ -161,17 +201,138 @@ an empty array – existing parsers that ignore this field see no change.
> **`xyteam/autobdd`** as a **deprecated alias**. See **Two images, two layers** above and
> [`docs/CONTRACT.md`](docs/CONTRACT.md) for the base's public seam.

## Feature conformance — what the base image can do, and how to check each bit

The base image's feature set is enumerated as a **catalogue** in
`test-projects/autobdd-base-test/base-test/features.sh` (currently **44 features**).
The suite is organised per feature, and every feature prints the single command that
reproduces it — so the output doubles as documentation.

**One-liner: run the whole feature matrix and see the full result**

```bash
cd test-projects/autobdd-base-test
AutoBDD_Ver=<v> make docker-run jobs="base-test" # ~3.5 min; ends "ALL FEATURES OK"
```

```text
════════════════════════════════════════════════════════════════════════════
AutoBDD base image — feature conformance
image : xyteam/autobdd-base (built 2026-09-22T19:02:59Z)
display : :1 1920x1200x24
surface : /usr/local/libexec/autobdd/find-target
catalogue: 44 features — see base-test/features.sh

reproduce ALL features (inside the image):
AutoBDD_Ver=<v> make docker-run jobs="base-test"
reproduce ONE feature:
AutoBDD_Ver=<v> make one FEATURE=image-match
list the catalogue:
AutoBDD_Ver=<v> make docker-run jobs="base-test/one.sh --list"
════════════════════════════════════════════════════════════════════════════

D — seam: image matching

▸ image-similarity — accept a weak match at a low floor and reject it at a high one
repro: base-test/one.sh image-similarity
run: findTargetImage --imagePath=/tmp/base-test.VGxClH/hello_blur.png \
--imageSimilarity=0.5 --flash=0 [rc=0]
✓ a low floor accepts the blurred template = hello_blur.png
run: findTargetImage --imagePath=/tmp/base-test.VGxClH/hello_blur.png \
--imageSimilarity=0.99 --flash=0 [rc=0]
✓ a high floor rejects it = notFound

▸ flash — hold the on-screen match flash for the requested time
repro: base-test/one.sh flash
timed without the fixture re-show (NOSHOW=1)
run: findTargetImage --imagePath=/tmp/.../hello.png [rc=0]
run: findTargetImage --imagePath=/tmp/.../hello.png --flash=0 [rc=0]
measured: default 2241 ms vs --flash=0 1338 ms -> delta 903 ms
✓ the default flash pauses for at least 0.5 s (0 ms would mean the flash is not applied) = 903 (>= 500)
✓ the flash pause is bounded = 903 ms (<= 2500)

════════════════════════════════════════════════════════════════════════════
feature conformance: 130 passed, 0 failed
ALL FEATURES OK
════════════════════════════════════════════════════════════════════════════
```

**Reading a run.** Each feature prints `▸ <id> — <one imperative sentence>` (the intent),
then one line per **invocation actually executed**:

| Line | Meaning |
|---|---|
| `run:` | the **exact argv** that ran, with its exit status. A check can never narrate a command it did not run, and a multi-invocation feature prints *every* invocation instead of one aspiration. |
| `probe:` | a host-side command, printed then run from the same string — so every printed line is literally runnable. |
| `✓ <what> = <observed>` | the assertion **and the value observed**, so a green run is evidence, not a checklist. |
| `known-gap:` | a documented limitation that is deliberately *not* asserted (see below). |
| `✗ <what> (got X, want Y)` | a failure, with both values; failures are collected per feature at the end. |


**One-liner: reproduce a single feature, or a whole group**

```bash
cd test-projects/autobdd-base-test
AutoBDD_Ver=<v> make one FEATURE=image-match # one feature
AutoBDD_Ver=<v> make one FEATURE=D # every D-group feature
AutoBDD_Ver=<v> make docker-run jobs="base-test/one.sh --list" # list the catalogue
```

Each feature is one line to reproduce inside the image: `base-test/one.sh <feature>`.

| Group | Features | What it establishes |
|---|---|---|
| **A — runtime substrate** | `tools` `java17` `natives` `screen-only` `provenance` | the L0 essentials are present, Java is the pinned 17 series, the Oculix natives are baked and loader‑visible, the image really is browser‑free, and `/etc/autobdd-versions` records the build inputs |
| **B — display + desktop** | `display` `wm` `vnc` `pointer` | Xvfb serves `DISPLAY` at the requested geometry, openbox runs, x11vnc exposes `:5900`, and the OS pointer can be driven |
| **C — whole‑screen OCR** | `screen-mode` | `--imagePath=Screen` returns the whole screen's text with `score: null` |
| **D — image matching** | `image-match` `image-similarity` `maxsim-ceiling` `text-hint` `image-wait` `image-maxcount` `image-missing` `flash` | every contract field; the similarity **floor** and `maxSim` **ceiling**; `--textHint` gating; `--imageWaitTime` waiting for a late target; several matches via `--imageMaxCount`; `notFound` as a status object; the `--flash` pause |
| **E — actions** | `action-click` `action-doubleclick` `action-rightclick` `action-hoverclick` `action-hover` `action-none` | each `--imageAction` is **verified against the OS pointer** (`xdotool`), so a reported click point must equal where the pointer actually went |
| **F — opt‑in OCR** | `ocr-detect` `ocr-detail-none` `ocr-detail-line` `ocr-similarity` `ocr-wait` `ocr-action` `ocr-psm-oem` | text search by `--ocrPath`, the optional `ocrDetails` box, its floor, wait, action dispatch and PSM/OEM pass‑through |
| **G — contract robustness** | `json-on-error` `additive-args` | an unusable display still yields JSON on stdout with exit 0, and unknown arguments are ignored (additive contract) |
| **H — non‑functional** | `latency` | NFR‑T2: a warm image match stays inside its budget |
| **I — discovery** | `help` `version` `list` `usage-error` | the tool explains itself *without* starting the engine (~40 ms, no screen scan), reports what is running, lists its flows, and fails loudly on an unusable value |
| **J — front door** | `front-door-help` `front-door-read-text` `front-door-find-target` `front-door-unknown-verb` `alias-transparent` | the `autobdd <verb>` layer; `read-text` is its own verb; `find-target` without a target is a usage error; and the deprecated alias stays transparent (notice on stderr, stdout carries exactly one result line) |

**Reading a run.** Each feature prints `▸ <feature> — <what it checks>`, the exact `cmd:`
line, then one `✓`/`✗` per assertion. Failures are collected at the end grouped by
feature, and groups can be run alone by letter, which makes a red run easy to bisect.

**Notes on coverage honesty**

* `SCREENSHOT` appears in `docs/CONTRACT.md` §4 but is **not implemented by the base
seam** (it is an L2/framework concern — the framework's hooks capture the flash frame).
It is deliberately absent from the catalogue rather than asserted falsely.
* Timing‑based checks (`flash`, `latency`) use the **minimum of three** runs, because the
minimum is the least noisy estimator of a fixed cost; their thresholds are stated in the
output so a regression is visible, not merely caught.
* `--flash=0` is used except where the flash is the subject: the flash is a pure visual
pause, and paying ~1 s for it on every feature would triple the run time for no extra
coverage. The `flash` feature measures the default path explicitly.
* **`--ocrSimilarity` is accepted but not applied.** Measured: with the text on screen,
`--ocrSimilarity=0.99` still matches, because this build's OCR path exposes no per‑match
confidence to filter on. The *image* floor **is** applied (`image-similarity` asserts both
directions). The report prints a `known-gap:` line rather than asserting a rejection that
would pass only when the screen happens to be blank — a false green.
* **`--ocrDetail=word` reports the matched region**, not one entry per token (a tight box
when the engine exposes one, otherwise the searched region's rectangle).
* Feature descriptions live in the catalogue table alone, so `--list`, the suite headers and
the run lines cannot drift apart.



## Try it and see the report

This repo ships two suites. Both run the **locally built** image only
(`pull_policy: never` — build it, or pre‑pull the published tag):

* **`test-projects/autobdd-base-test`** – a **no‑browser** conformance suite for the
**base** image (L0/L1 + the frozen CLI seam). Fast; no Chrome required:
* **`test-projects/autobdd-base-test`** – the **no‑browser** feature‑conformance suite for
the **base** image (L0/L1 + the frozen CLI seam). 44 features, no Chrome required. See
**Feature conformance** above for the one‑liners and the catalogue:

```bash
cd test-projects/autobdd-base-test
AutoBDD_Ver=<v> make docker-run jobs="base-test"
AutoBDD_Ver=<v> make docker-run jobs="base-test" # whole feature matrix
AutoBDD_Ver=<v> make one FEATURE=action-hover # one feature
```

* **`test-projects/autobdd-framework-test`** – the full framework suite. Run it to see
Expand Down
Loading
Loading