bmcap

Captures real Windows screens paired with UI Automation labels, as training data for a vision model that navigates a PC.

Every frame is three files written together:

file	what it holds
frame-NNNNNN.png	the monitor's framebuffer, with protected fields repainted
frame-NNNNNN.uia.json	the full raw UIA tree — everything UIA offered, minus password values
frame-NNNNNN.gt.json	ground truth: objects, zones, text runs, cursor, visibility

The .gt.json is written against schema/groundtruth.schema.json, which is also what the ButtonMasher renderer writes its synthetic frames against, so real and simulated frames pool into one dataset. That schema is a committed, generated copy — see Regenerating the schema.

Design notes: docs/2026-08-05-real-capture-tool-design.md.

Install

Windows only, Python 3.11+. A fresh virtualenv in the checkout:

py -m venv .venv
.venv\Scripts\python.exe -m pip install -e ".[dev]"

Quote the ".[dev]" — PowerShell treats bare [dev] as a wildcard. Drop [dev] if you do not want pytest and jsonschema.

That puts two launchers in .venv\Scripts\: bmcap.exe (console) and bmcap-tray.exe (no console — see below). Run the suite with:

.venv\Scripts\python.exe -m pytest

Configure

bmcap config                         # show everything
bmcap config --data-root D:\bmdata   # where frames are written

Settings live in %APPDATA%\bmcap\config.json (or $BMCAP_CONFIG_DIR if you set it):

key	default	meaning
data_root	<checkout>\dataset	sessions are created under <data_root>\real\
fallback_interval_s	20.0	how often record samples a screen nobody is touching
hotkey	ctrl+alt+f9	pauses record; stops crawl
crawl_allowlist	notepad, calc, mspaint, explorer	the only apps crawl will launch or click
encrypt	true	see Encryption

A session is <data_root>\real\<UTC timestamp>\, holding session.json, the frames, and salt.bin if encrypted.

The three modes

bmcap snap — one frame

bmcap snap
bmcap snap --monitor 1 --scope foreground
bmcap snap --session D:\bmdata\real\20260806-101500

Captures once and exits. Creates a new session directory unless --session points at an existing one to append to.

--scope defaults to desktop: the walk covers every top-level window intersecting the monitor, not just the foreground one, because the screenshot covers the whole monitor and a background window's controls would otherwise be pixels with no label. --scope foreground is much faster and much smaller.

A screenshot and a UIA walk cannot happen at the same instant — the walk takes seconds. So the screen is grabbed before and after the walk and the frame is thrown away if more than a small pixel budget changed (~1036px at 1080p, enough to absorb a clock rollover and a blinking caret, far below a moved window). Three attempts, then it gives up and says so. A screen with almost no image information in it — a sleeping, off, or locked display — is rejected before the walk is even paid for, because such a frame passes the stability check most easily and would be written confidently mislabelled.

bmcap record — passive, tray-controlled

bmcap record
bmcap record --monitor 1 --no-encrypt

Sits in the tray and captures while you work. It does not drive anything; it only watches. Captures are triggered by Win32 foreground and window lifecycle events — the screen becoming a different screen is what makes a frame worth having — with fallback_interval_s as an idle heartbeat so an untouched screen is still sampled. There is a hard floor of 8 s between captures so an active desktop cannot saturate UIA. Consecutive frames that are the same screen (structural hash plus a brightness bucket, so a caret blink or a clock digit is not a new example) are dropped.

The tray menu offers pause/resume, set and open the data folder, and a frame count. ctrl+alt+f9 pauses and resumes from anywhere. Exit — or Ctrl+C, which takes the same path — waits up to 20 s for an in-flight capture so a frame is never left half written.

This mode holds a console window open. To run it without one, use bmcap-tray.

bmcap crawl — drives the mouse (gated)

bmcap crawl --i-understand --steps 40

This one clicks the real UI on this machine, by itself, and steals focus for the duration. It exists to harvest click outcomes: each frame is written with the action taken from it, so a frame and its consequence are linked by the next frame.

Four independent barriers, in order:

1. The gate. --i-understand and an interactive prompt you answer by typing this machine's name. The flag is checked first, so a bare bmcap crawl prints an explanation and exits — it never falls through into a prompt you might answer by reflex.
2. Containment. The crawler launches the apps in crawl_allowlist itself and refuses to click any element not owned by one of the process ids it started. This is the primary defence: an exact, small, known set of pids. Immediately before each press it re-checks which process owns the window under that pixel now, because the tree it chose from is seconds old.
3. A name filter. Destructive-sounding targets are vetoed — deleting, ending the session, and acting on the outside world (send, post, buy). Deliberately broad and deliberately second: it reads one control at a time and it is English-only, so it cannot be the thing you rely on.
4. The stop hotkey. ctrl+alt+f9 ends the crawl from anywhere — which matters precisely because the crawler holds focus and a console Ctrl+C may therefore never reach it.

A target must also be at least 90% unobstructed before it may be clicked: a press lands on whatever window is topmost at that point, so a mostly-covered control is one whose name was checked and whose click goes somewhere else. The launched apps are terminated on the way out, however the crawl ended.

bmcap decrypt — read back what you collected

bmcap decrypt <session-dir> <out-dir>

Writes the plaintext of every .enc file into <out-dir>. On a session that was recorded unencrypted it says so rather than failing obscurely.

Running as a GUI (bmcap-tray)

bmcap record needs a console window to stay open. bmcap-tray is the same recorder with no console at all: pip installs it as a gui-scripts launcher, so it is built against pythonw.exe. Put a shortcut to .venv\Scripts\bmcap-tray.exe wherever you like — Startup folder, taskbar:

bmcap-tray
bmcap-tray --no-encrypt --monitor 1

An encrypted session asks for its passphrase in a dialog box, since there is no console to type into (getpass under pythonw does not fail — it blocks forever). An unencrypted one asks for nothing and starts straight into the tray.

If it fails to start, nothing can be printed, so failures go to a message box and to %APPDATA%\bmcap\bmcap-tray.log — next to config.json, or under $BMCAP_CONFIG_DIR if you set that. That log is the place to look when the tray icon never appears.

Encryption

Frames are encrypted at rest by default — they are pictures of a real desktop. A passphrase (scrypt-derived key, Fernet, per-session salt in salt.bin) is asked for when a session starts, or read from $BMCAP_PASSPHRASE for scripted use.

Plaintext is opt-in, for looking at what you just collected:

bmcap config --no-encrypt            # persist it
bmcap config --encrypt               # back to the default
bmcap snap --no-encrypt              # this run only (also record/crawl/bmcap-tray)

With encryption off the artifacts are written as frame-000000.png, .uia.json and .gt.json — no .enc, no salt file, and no passphrase is asked for at all. Each session records which mode it used in session.json ("encrypted": true|false), so nothing has to guess it from filenames. The mode is announced when a session starts and, while recording, in the tray tooltip (bmcap - NOT ENCRYPTED). Appending to an existing session with --session keeps that session's mode; a directory cannot hold half its frames encrypted and half not.

Only a literal false in config.json turns encryption off. A hand-edited 0, "", "no" or null reads as encrypted, because guessing wrong in the other direction writes someone's screen to disk unprotected.

Protected fields are always masked

This is not an option and there is no flag for it, in any mode, encrypted or not.

Every element UIA reports as a password field has its rectangle repainted in the PNG before the frame is written — before either the encrypting or the plaintext writer ever sees the bytes, so the encryption choice cannot affect it. The wipe is unconditional: whatever the OS actually rendered there, at minimum the real glyph count, never survives.

What replaces it is not a black box, because the ground truth says the field holds a run of bullets and black pixels would teach the model that a password box looks black. A field judged filled is repainted with bullet glyphs drawn in its own sampled background and contrasting foreground, so it reads correctly in dark themes. The number of bullets comes from the field's geometry alone — how many glyphs fit across a box that size — never from what was typed, so the picture cannot leak the length. The same predicate decides the pixels and the label, so they can never disagree.

Alongside that: the .uia.json sidecar keeps the element but blanks its value, and the .gt.json carries the object as type: "password" with a masked text run of exactly the bullet count that was painted.

The ground truth contract

schema/groundtruth.schema.json is the contract. tests/test_map_gt.py validates this mapper's output against it, which is what stops the Python and TypeScript writers from drifting apart.

A reader must reject the NaN, Infinity and -Infinity literals at parse time. They are not JSON, but Python's json accepts them by default and no schema keyword can catch a NaN afterwards — bounds are implemented as instance > maximum, and every comparison against NaN is false, so a NaN passes every bound in the document. load_frame in tests/test_map_gt.py is the one-line pattern; schema/README.md explains it in full.

Regenerating the schema

Only needed when the zod source changes. This repo needs no Node toolchain of its own — the schema is generated in ButtonMasher and the result is committed here:

cd C:\GitHub\ButtonMasher
npm install                      # first time only
npx tsx tools/gen-groundtruth-schema.mts C:\GitHub\ScreenMapper\schema\groundtruth.schema.json

schema/README.md documents where it comes from, the two things zod-to-json-schema gets wrong for this target and why both patches are load-bearing, and the one deliberate difference from zod (additionalProperties: false).

Layout

bmcap/            the package
  cli.py          argument parsing and the subcommands
  config.py       persisted settings
  uia.py          UIA tree extraction, DPI, monitors
  map_gt.py       raw UIA -> ground truth
  snap.py         one frame: grab, walk, mask, write
  record.py       the passive capture loop (no UI)
  tray.py         the tray icon around that loop
  crawl.py        the gated auto-clicker
  crypt.py        at-rest encryption, and the plaintext opt-out
  hooks.py        Win32 window events and the global hotkey
  gui.py          the console-less `bmcap-tray` launcher
tests/            pytest suite, no live desktop needed
schema/           the generated ground truth contract
docs/             design spec, implementation plan, build log

Where this came from

bmcap was built inside the ButtonMasher monorepo as capture/ and moved here to stand alone. The two repos share nothing but the schema file, which is copied, not linked — the tests pass with ButtonMasher absent. The design spec and plan under docs/ were written before and during that original build, so paths in them describe the old capture/ layout; each carries a note saying so.