Skip to content

Base image: self-contained Oculix natives + working opt-in OCR text actions #169

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
xywang68 merged 2 commits into from
Sep 21, 2026
Merged

Base image: self-contained Oculix natives + working opt-in OCR text actions #169

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4e52869
Update README with OCR enhancements and native library handling; remo…
xywang68 Sep 20, 2026
9b8ec63
Bake Oculix natives at build time; fix seam OCR mode and OCR test exp…
xywang68 Sep 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 26 additions & 15 deletions .docker/autobdd-base.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,23 +82,34 @@ RUN curl -fsSL -o /tmp/node.tar.xz "https://nodejs.org/dist/v${NODE_VERSION}/nod

# the vendored bridge, installed world-readable under /opt/autobdd so the CLI seam works
# for ANY user — /root (where the framework tree lives) is not traversable by others.
COPY third_party/xysikulixapi /opt/autobdd/third_party/xysikulixapi
RUN cd /opt/autobdd/third_party/xysikulixapi && \
npm config set script-shell /bin/bash && \
npm ci --omit=dev --loglevel=error && \
rm -rf /tmp/*
# Copy seam source (no node_modules needed at runtime)
COPY seam/ /opt/autobdd/seam/
WORKDIR /opt/autobdd/seam
RUN npm install --omit=dev --loglevel=error

# Expose the seam on PATH and warm the Oculix natives at build (with a display), then bake
# them to a world-readable dir wired into ld.so (no per-container extraction, no env needed).
RUN mkdir -p /opt/oculix-natives && \
( Xvfb :99 -screen 0 400x300x24 >/dev/null 2>&1 & XPID=$!; sleep 2; \
DISPLAY=:99 node -e "const j=require('/opt/autobdd/third_party/xysikulixapi/node_modules/java-bridge'); j.ensureJvm({opts:['-Xms128m','-Xmx512m']}); try{ const X=require('/opt/autobdd/third_party/xysikulixapi/lib/xysikulixapi.js'); new X.Screen(); }catch(e){}" >/dev/null 2>&1; \
kill $XPID 2>/dev/null ) || true; \
cp -f /root/.cache/legerix/*/linux-x86-64*/*.so* /opt/oculix-natives/ 2>/dev/null || true; \
chmod 755 /opt/oculix-natives && \
echo /opt/oculix-natives > /etc/ld.so.conf.d/oculix-natives.conf && ldconfig && \
ln -sf /opt/autobdd/third_party/xysikulixapi/bin/findTargetImage.js /usr/local/bin/findTargetImage && \
# Fetch the latest stable Oculix JAR (allow override via build-arg)
ARG OCULIX_VER
RUN chmod +x scripts/fetch-oculix.sh && \
OCULIX_VER="${OCULIX_VER}" ./scripts/fetch-oculix.sh

# Ensure the seam script is executable and on PATH
RUN chmod +x src/findTargetImage.js && \
ln -s /opt/autobdd/seam/src/findTargetImage.js /usr/local/bin/findTargetImage

# Bake the Oculix native libraries (NFR-P3: exact, self-contained image).
# The JAR bundles the Linux x86-64 natives (OpenCV + Leptonica + Tesseract) under
# linux-x86-64/. We extract them into a stable, world-readable directory and
# register it with the dynamic loader, so the seam never depends on a writable
# /tmp or on an on-demand extraction at run time.
RUN mkdir -p /opt/oculix-natives /tmp/oculix-unpack && \
unzip -o -q lib/oculixapi-${OCULIX_VER:-4.0.0}-linux.jar 'linux-x86-64/*' -d /tmp/oculix-unpack && \
cp -f /tmp/oculix-unpack/linux-x86-64/*.so* /opt/oculix-natives/ && \
rm -rf /tmp/oculix-unpack && \
echo "/opt/oculix-natives" > /etc/ld.so.conf.d/oculix.conf && \
chmod 0755 /opt/oculix-natives && chmod 0644 /opt/oculix-natives/* && \
ldconfig && \
echo "baked oculix natives: $(ls /opt/oculix-natives 2>/dev/null | tr '\n' ' ')"

ENV LD_LIBRARY_PATH=/opt/oculix-natives

WORKDIR /root
Expand Down
188 changes: 135 additions & 53 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,12 @@ images**. You do **not** need to clone it to use AutoBDD.

* The images are built here and **published to Docker Hub**:
**`xyteam/autobdd-base`** and **`xyteam/autobdd-framework`**. **`xyteam/autobdd`**
is kept as a **deprecated alias** of `autobdd-framework` (same image, two tags), so
existing consumers keep working.
is kept as a **deprecated alias** of `autobdd-framework` (same image, two tags),
so existing consumers keep working.
* The image is **Ubuntu 24.04** based.
* **Test repositories pull and run the image directly** — e.g.
[AutoBDD-example](https://github.com/xyteam/AutoBDD-example) runs its suite against
`xyteam/autobdd:<version>` with no framework clone required.
[AutoBDD-example](https://github.com/xyteam/AutoBDD-example) runs its suite
against `xyteam/autobdd:<version>` with no framework clone required.
* **Clone only to inspect/build the AutoBDD docker image.**

```bash
Expand All @@ -33,90 +33,172 @@ AutoBDD_Ver=3.0.0 ABDD_PROJECT=AutoBDD-example \
The product is published as two tags that are also two **layers** of one stack:

| Tag | Layers | Contains | Use when |
|---|---|---|---|
| **`xyteam/autobdd-base`** | **L0** OS + essentials, X display + desktop (Xvfb/openbox/x11vnc) · **L1** screen engine (Java 17 + Node + the Oculix bridge) | **no browser, no WebdriverIO** | you want **screen-only** automation, or you are building **your own framework** on top |
| **`xyteam/autobdd-framework`** | L0 + L1 **+ L2** | Chrome + matching chromedriver · WebdriverIO · Cucumber · the AutoBDD step library | you want the ready-made BDD framework and HTML reports |
|-----|--------|----------|----------|
| **`xyteam/autobdd-base`** | **L0** OS + essentials, X display + desktop (Xvfb/openbox/x11vnc) · **L1** screen engine (Java 17 + Node + the Oculix bridge) | **no browser, no WebdriverIO** | you want **screen-only** automation, or you are building **your own framework** on top |
| **`xyteam/autobdd-framework`** | L0 + L1 **+ L2** | Chrome + matching chromedriver · WebdriverIO · Cucumber · the AutoBDD step library | you want the ready‑made BDD framework and HTML reports |

`xyteam/autobdd:<v>` is a **deprecated alias** of `xyteam/autobdd-framework:<v>`.

The base's **public interface** is a frozen CLI + JSON seam — `findTargetImage` —
documented in [`docs/CONTRACT.md`](docs/CONTRACT.md). A foreign framework shells out to it
instead of importing our code, so the base is usable **on its own** (screen-only mode).
## The public interface of the base image (the frozen seam)

**Philosophy:** image/action steps first; web/DOM steps are an assist. The engine sees
pixels and text, so the same steps cover web pages, canvas, native apps, and anything else
on screen.
The base image exposes a single command‑line program, **`findTargetImage`**, which is the
frozen public contract described in [`docs/CONTRACT.md`](docs/CONTRACT.md).
All communication with the base image happens through this program; it returns a JSON
object (or array) on **stdout**.

## The key difference: AutoBDD sees the screen
### Basic usage (image matching)

Most browser-automation tools drive the DOM through selectors. **AutoBDD can see
anything and test anything** because it operates on the **screen** using:
```bash
findTargetImage \
--imagePath=<path-to‑template.png> \
--imageSimilarity=0.8 \
--maxSim=1.0 \
--imageWaitTime=1000 \
--imageAction=click \
--imageMaxCount=1
```

### Opt‑in OCR mode (new)

All OCR‑related arguments are **optional**; if none of them are supplied the seam
behaves exactly as described above (pure image matching).
When `--ocrPath` is supplied the seam runs OCR instead of image matching.

| Argument | Type | Default | Meaning |
|----------|------|---------|---------|
| `--ocrPath <text>` | string | *(required for OCR mode)* | Text to search for via OCR. |
| `--ocrSimilarity <float>` | float | `0.8` | Minimum OCR confidence (0‑1) to treat as a match. |
| `--ocrMaxSim <float>` | float | `1.0` | Maximum OCR confidence (upper bound). |
| `--ocrWaitTime <ms>` | integer | `1000` | How long (ms) to wait for the OCR text to appear (polls until found or timeout). |
| `--ocrMaxCount <int>` | integer | `1` | Maximum number of OCR matches to return. |
| `--ocrAction <action>` | string | `none` | Action to perform on each OCR match: `none`, `click`, `doubleClick`, `rightClick`, `hover`, `hoverClick`. |
| `--ocrDetail <none|line|word>` | string | `none` | Include the OCR bounding box in the result. `line` and `word` both emit the matched text region: a tight box when the engine exposes one for the query, otherwise the searched region's rectangle. `none` omits `ocrDetails` entirely (backward‑compatible). |
| `--ocrPSM <int>` | integer | `7` | Tesseract Page Segmentation Mode (passed through to Oculix). |
| `--ocrOEM <int>` | integer | `3` | Tesseract OCR Engine Mode (passed through to Oculix). |

### Output format (backward‑compatible)

The seam always returns a JSON array (or a single `{status:'notFound'}` object).
Each result object contains the original fields:

```json
{
"name": "<string>", // matched text (or template name)
"score": <number>, // OCR confidence (0‑1) or image similarity
"text": [<string>, ...], // OCR lines (full text split by '\n')
"location": { "x":<num>, "y":<num>, "width":<num>, "height":<num> },
"dimension": { "width":<num>, "height":<num> },
"center": { "x":<num>, "y":<num> },
"clicked": { "x":<num>, "y":<num> } | null, // set only for clicking actions (click/doubleClick/rightClick/hoverClick)
"ocrDetails": [ // <-- ONLY present when --ocrDetail ≠ none
{
"text": "<string>", // the OCR word or line
"x":<num>, "y":<num>, // top‑left corner
"width":<num>, "height":<num>, // size in pixels
"confidence":<number> // OCR confidence for this token (0‑1)
},
...
]
}
```

If `--ocrDetail=none` (the default) the `ocrDetails` field may be omitted or set to
an empty array – existing parsers that ignore this field see no change.

### Example usages

* **Detect a word and get its bounding box (no action)**

```bash
findTargetImage --ocrPath="SUBMIT" --ocrDetail=word
```

* **Click the first occurrence of a word**

* **sample images you provide** — find, click, and assert on any on-screen image
(screen image matching), and
* **text** — read and act on on-screen text (OCR).
```bash
findTargetImage --ocrPath="SUBMIT" --ocrAction=click --ocrDetail=word
```

* **Double‑click the second occurrence of a phrase**

```bash
findTargetImage --ocrPath="END TEST" --ocrAction=doubleClick \
--ocrDetail=word --ocrMaxCount=2
```

* **Hover over a line of text (no click)**

```bash
findTargetImage --ocrPath="Status:" --ocrAction=hover --ocrDetail=line
```

That means it is not limited to DOM elements: it works for web pages, canvas /
non-DOM content, native apps, and any pixel you can see — image-in, action-out.
* **Combine image matching with OCR gating (textHint)** – unchanged from the original
seam; see `docs/CONTRACT.md` for details.

## Under the hood

* **Base** (`xyteam/autobdd-base`): Ubuntu 24.04 · Java 17 · Node 20 · Xvfb/openbox/x11vnc ·
the Oculix 4.0.0 screen engine exposed as the `findTargetImage` CLI seam.
The Oculix natives bundled in the engine JAR are extracted **at image‑build time**
into `/opt/oculix-natives` and registered with `ldconfig`, so the image is
self‑contained: no on‑demand extraction at run time and no need for a writable
`/tmp` or `/root`.
* **Framework** (`xyteam/autobdd-framework`): base + Chrome and a **matching
chromedriver** on PATH, WebdriverIO 9 (Cucumber), the AutoBDD step library.
* Runs with `docker compose` (Compose v2).
* Reports: HTML with step screenshots (pass/fail watermarks) and test movies.

## Version differences

| Release | Chrome | WebdriverIO | Node | Runtime |
|---|---|---|---|---|
| **v2.3.0** | 96 | 7 | 12 | original pinned runtime (Node 12.22.7 + Chrome 96) |
| **v2.4.0** | modern (latest stable, e.g. 153) | 7 | 14 | re-activated v2.3.0 line; builds against current Chrome; matching browser driver baked into the image |
| **v3.0.0** | modern (latest stable, e.g. 153) | 9 | 20 | runtime re-baseline: Node 20 + Java 17 + WebdriverIO 9 + Oculix 4.0.0 (image matching/OCR) |
|---------|--------|-------------|------|---------|
| **v2.3.0** | 96 | 7 | 12 | original pinned runtime (Node 12.22.7 + Chrome 96) |
| **v2.4.0** | modern (latest stable, e.g. 153) | 7 | 14 | re‑activated v2.3.0 line; builds against current Chrome; matching browser driver baked into the image |
| **v3.0.0** | modern (latest stable, e.g. 153) | 9 | 20 | runtime re‑baseline: Node 20 + Java 17 + WebdriverIO 9 + Oculix 4.0.0 (image matching/OCR) |

> From the next release the product is published as two tags — **`xyteam/autobdd-base`**
> (screen-only) and **`xyteam/autobdd-framework`** (the full product), with
> (screen‑only) and **`xyteam/autobdd-framework`** (the full product), with
> **`xyteam/autobdd`** as a **deprecated alias**. See **Two images, two layers** above and
> [`docs/CONTRACT.md`](docs/CONTRACT.md) for the base's public seam.

The Node version is chosen to support both the WebdriverIO version and the
bundled internal demo-app.

## Try it and see the report

This repo ships two suites. Both run the **locally built** image only
(`pull_policy: never` — build it, or pre-pull the published tag):
(`pull_policy: never` — build it, or pre‑pull the published tag):

* **`test-projects/autobdd-base-test`** — a **no-browser** conformance suite for the
* **`test-projects/autobdd-base-test`** – a **no‑browser** conformance suite for the
**base** image (L0/L1 + the frozen CLI seam). Fast; no Chrome required:

```bash
cd test-projects/autobdd-base-test
AutoBDD_Ver=<v> make docker-run jobs="base-test"
```
Dev (interactive shell, or ssh + VNC desktop) is extra `docker compose run`
parameters — e.g. `docker compose run --rm --entrypoint /root/autobdd-dev.startup.sh
-p 2225:22 -p 5925:5900 autobdd-base-test`; see the suite's README.
* **`test-projects/autobdd-framework-test`** — the full framework suite. Run it to see
AutoBDD's reports for yourself — step screenshots (with green/red pass-fail watermarks
and image-match markers), per-scenario movies, and the HTML report:

```bash
cd test-projects/autobdd-framework-test
AutoBDD_Ver=<v> ABDD_PROJECT=autobdd-framework-test \
USER=$(whoami) PASSWORD=ubuntu HOSTOS=Linux USERID=$(id -u) GROUPID=$(id -g) \
make docker-run jobs="clean e2e-test"
# then open test-results/e2e-test/*/index.html
```

## Under the hood
* **`test-projects/autobdd-framework-test`** – the full framework suite. Run it to see
AutoBDD's reports for yourself — step screenshots (with green/red pass/fail watermarks
and image‑match markers), per‑scenario movies, and the HTML report:

* **Base** (`xyteam/autobdd-base`): Ubuntu 24.04 · Java 17 · Node 20 · Xvfb/openbox/x11vnc ·
the Oculix 4.0.0 screen engine exposed as the `findTargetImage` CLI seam.
* **Framework** (`xyteam/autobdd-framework`): base + Chrome and a **matching chromedriver**
on PATH, WebdriverIO 9 (Cucumber), the AutoBDD step library.
* Runs with `docker compose` (Compose v2).
* Reports: HTML with step screenshots (pass/fail watermarks) and test movies.
```bash
cd test-projects/autobdd-framework-test
AutoBDD_Ver=<v> ABDD_PROJECT=autobdd-framework-test \
USER=$(whoami) PASSWORD=ubuntu HOSTOS=Linux USERID=$(id -u) GROUPID=$(id -g) \
make docker-run jobs="clean e2e-test"
# then open test-results/e2e-test/*/index.html
```

## Credits

* Screen image matching / OCR: **[oculix-org/Oculix](https://github.com/oculix-org/Oculix)** —
the engine AutoBDD uses to see the screen. OculiX carries the SikuliX lineage
([RaiMan/SikuliX1](https://github.com/RaiMan/SikuliX1)) forward in its own project.
* Keyboard/mouse: **[octalmage/robotjs](https://github.com/octalmage/robotjs)**.
* Demo app and pre-canned Cucumber-JS steps adapted from
* Demo app and pre‑canned Cucumber‑JS steps adapted from
**[webdriverio/cucumber-boilerplate](https://github.com/webdriverio/cucumber-boilerplate)**.
* Framework control from
**[webdriverio/webdriverio](https://github.com/webdriverio/webdriverio)**.

---

*This README reflects the current state of this branch, including the opt‑in OCR
arguments, native‑library handling, and the updated public contract of the
`findTargetImage` seam.*
10 changes: 10 additions & 0 deletions seam/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"name": "autobdd-base-seam",
"version": "1.0.0",
"description": "Dependencies for the AutoBDD base seam (findTargetImage)",
"main": "src/findTargetImage.js",
"dependencies": {
"java-bridge": "^2.7.0",
"minimist": "^1.2.5"
}
}
31 changes: 31 additions & 0 deletions seam/scripts/fetch-oculix.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
#!/usr/bin/env bash
set -euo pipefail

# Fetch the latest stable Oculix JAR from GitHub releases.
# If OCULIX_VER is set, use that version; otherwise, get the latest release.
if [ -n "${OCULIX_VER:-}" ]; then
VERSION="${OCULIX_VER}"
else
# Get the latest release tag (e.g., v4.0.0)
VERSION_TAG=$(curl -s https://api.github.com/repos/oculix-org/Oculix/releases/latest | jq -r .tag_name)
# Remove leading 'v' if present
VERSION="${VERSION_TAG#v}"
fi

# Construct the expected asset name
ASSET_NAME="oculixapi-${VERSION}-linux.jar"
# Get the release assets and find the download URL for our asset
DOWNLOAD_URL=$(curl -s https://api.github.com/repos/oculix-org/Oculix/releases/tags/${VERSION_TAG} | \
jq -r --arg ASSET "$ASSET_NAME" '.assets[] | select(.name == $ASSET) | .browser_download_url' || echo "")

if [ -z "$DOWNLOAD_URL" ]; then
# Fallback to the conventional URL pattern
DOWNLOAD_URL="https://github.com/oculix-org/Oculix/releases/download/${VERSION_TAG}/${ASSET_NAME}"
fi

DEST="lib/oculixapi-${VERSION}-linux.jar"

echo "Fetching Oculix ${VERSION} from $DOWNLOAD_URL..."
mkdir -p "$(dirname "$DEST")"
curl -L -f -o "$DEST" "$DOWNLOAD_URL"
echo "Saved to $DEST"
Loading
Loading