Show the phone, live, and cut the settings down to two

.github/workflows/release.yml

@@ -40,24 +40,30 @@ env:
 
 jobs:
   windows:
-    name: Windows installer
+    name: Windows installer and WinUI front end
     runs-on: windows-latest
-    timeout-minutes: 30
+    # Two PyInstaller builds, a WiX build, a self-contained WinUI publish and a
+    # 400 MB zip. Thirty minutes was enough before the WinUI front end existed.
+    timeout-minutes: 50
     steps:
       - uses: actions/checkout@v4
 
       - uses: actions/setup-python@v5
         with:
           python-version: "3.12"
 
+      - uses: actions/setup-dotnet@v4
+        with:
+          dotnet-version: "8.0.x"
+
       - name: Install build dependencies
         run: |
           python -m pip install --upgrade pip
           python -m pip install -e ".[gui]" pyinstaller
           dotnet tool install --global wix --version ${{ env.WIX_VERSION }}
 
-      - name: Build app, installer and archive
-        run: python tools/build_app.py --msi --archive
+      - name: Build app, installer, WinUI front end and archive
+        run: python tools/build_app.py --msi --winui --archive
 
       - uses: actions/upload-artifact@v4
         with:
@@ -143,7 +149,8 @@ jobs:
 
             | | |
             |---|---|
-            | **Windows** | `MThreadDraw-*-x64.msi` — installs like any other program, Start Menu entry included. The `.zip` is the same app with no installer. |
+            | **Windows** | `MThreadDraw-*-x64.msi` — installs like any other program, Start Menu entry included. `MThreadDraw-*-windows-x64.zip` is the same app with no installer. |
+            | **Windows, native interface** | `MThreadDraw-WinUI-*-x64.zip` — the WinUI 3 front end. Unzip anywhere and run `MThreadDraw.exe`; it needs no runtime installed and carries its own copy of the engine. Larger, and the newer of the two. |
             | **macOS** | `MThreadDraw-*-arm64.dmg` for Apple Silicon, `MThreadDraw-*-x64.dmg` for Intel. Drag it to Applications. |
             | **Linux** | Run from source — see the [README](https://github.com/MAXAWER/MThread-Draw#install-variants). |
 

CHANGELOG.md

@@ -2,6 +2,18 @@
 
 ## Unreleased
 
+- **A live view of the device screen**, in its own shape, with the strokes laid
+  over exactly where they will land. `mthread.mirror.ScreenMirror` runs
+  `com.mthread.Mirror` from the same jar as the injector: the phone scales and
+  JPEG-compresses each frame, so the cable carries 27 kB instead of sixteen
+  megabytes. About 3.4 frames a second on a Pixel 8 Pro, against one frame every
+  two seconds for `adb exec-out screencap -p`.
+- **Two sliders instead of five.** Edge sensitivity and detail were always
+  turned together; speed and hand simulation were one axis pretending to be two.
+  The margin is fixed at six per cent.
+- The Windows front end connects by itself when there is exactly one device, and
+  opens an image named on its command line.
+
 - **The project is now MThread Draw.** Everything is renamed: `import adbtouch`
   becomes `import mthread`, `autodraw` becomes `mthread_draw`, the console
   script is `mthread` and the desktop entry point `mthread-draw`, the

README.md

@@ -1,8 +1,10 @@
-<h1 align="center">MThread Draw&nbsp;+&nbsp;mthread</h1>
+<p align="center">
+  <img src="docs/banner.gif" width="900" alt="MThread Draw — a cat's portrait assembling itself out of white touch points">
+</p>
 
 <p align="center">
-  <b>Draw any picture on an Android screen, and record and replay touch gestures.</b><br>
-  Over USB or wireless ADB. Nothing is installed on the phone.
+  <sub>Every white circle is one touch point, in the order it is sent to the phone. Nothing about that picture is illustrated:
+  it is a real <a href="tools/make_banner.py"><code>tools/make_banner.py</code></a> run over <code>examples/cat.jpg</code>.</sub>
 </p>
 
 <p align="center">
@@ -14,12 +16,12 @@
 </p>
 
 <p align="center">
-  <img src="docs/demo.gif" width="900" alt="A colour photograph of a cat being traced into touch strokes and drawn on a phone screen">
+  <img src="docs/demo.gif" width="900" alt="A colour photograph of a guitar being traced into touch strokes and drawn on a phone screen">
 </p>
 
 <p align="center">
-  <sub>An ordinary photograph in, 283 strokes and 1,528 touch points out: the exact path list <code>mthread</code> sends to the device, in the order it draws them.<br>
-  On a Pixel 8 Pro that draws in about five seconds. Rendered from a real run by <a href="tools/make_demo.py"><code>tools/make_demo.py</code></a>; playback speed here is arbitrary.</sub>
+  <sub>An ordinary photograph in, 57 strokes and 478 touch points out: the exact path list <code>mthread</code> sends to the device, in the order it draws them.<br>
+  On a Pixel 8 Pro that draws in under two seconds. Rendered from a real run by <a href="tools/make_demo.py"><code>tools/make_demo.py</code></a>; playback speed here is arbitrary.</sub>
 </p>
 
 ---

injector/src/com/mthread/Mirror.java

@@ -0,0 +1,134 @@
+package com.mthread;
+
+import android.graphics.Bitmap;
+import android.util.Base64;
+
+import java.io.BufferedReader;
+import java.io.ByteArrayOutputStream;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+
+/**
+ * Screen capture that scales and compresses on the phone, so the cable carries
+ * a thumbnail instead of a framebuffer.
+ *
+ * <p>Why this exists, in numbers measured on a Pixel 8 Pro at 1344x2992:
+ *
+ * <ul>
+ *   <li>{@code adb exec-out screencap -p} - 2.1 s per frame. The device spends
+ *       almost all of it PNG-encoding sixteen megabytes.</li>
+ *   <li>{@code adb exec-out screencap} - 2.9 s per frame. No encoding, but now
+ *       all sixteen megabytes cross the USB link, at about 5 MB/s.</li>
+ *   <li>This, at 480 pixels wide and quality 60 - the raw capture costs about
+ *       300 ms on the device, the scale and JPEG maybe 30, and the cable
+ *       carries some 40 kB.</li>
+ * </ul>
+ *
+ * <p>So the ceiling is the capture itself, which is roughly three frames a
+ * second. That is not video and this is not scrcpy: scrcpy is fast because it
+ * feeds the display straight into the hardware H.264 encoder, which needs a
+ * decoder on the other end. Three frames a second is enough to see what is on
+ * the phone and where a drawing is going to land, which is what this is for.
+ *
+ * <p>Pull-based: one line in, one frame out, so the host sets the rate and
+ * frames never queue up behind a slow reader.
+ *
+ * <pre>
+ *   &lt;maxWidth&gt; &lt;quality&gt;   capture, reply with "F " and the JPEG in base64
+ *   Q                          quit
+ * </pre>
+ *
+ * <p>Base64, for a picture, over a link that ought to be binary-safe. In
+ * practice it is not: adb on Windows still turns a line feed into a carriage
+ * return and a line feed on the way out, and a JPEG is full of line feeds - so
+ * the image arrives longer than it left, the declared length no longer matches,
+ * and every frame after the first is out of step by however many newlines the
+ * one before it happened to contain. Base64 has no byte the translation can
+ * touch. It costs a third more on a 25 kB frame and removes the whole class of
+ * problem.
+ */
+public final class Mirror {
+
+    public static void main(String[] args) throws Exception {
+        BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
+
+        System.out.println("READY");
+        System.out.flush();
+
+        String line;
+        while ((line = in.readLine()) != null) {
+            line = line.trim();
+            if (line.isEmpty()) {
+                continue;
+            }
+            if (line.startsWith("Q")) {
+                break;
+            }
+
+            String[] parts = line.split("\\s+");
+            int maxWidth = parts.length > 0 ? Integer.parseInt(parts[0]) : 480;
+            int quality = parts.length > 1 ? Integer.parseInt(parts[1]) : 60;
+
+            try {
+                byte[] jpeg = capture(maxWidth, quality);
+                System.out.println("F " + Base64.encodeToString(jpeg, Base64.NO_WRAP));
+            } catch (Throwable error) {
+                System.out.println("E " + error);
+            }
+            System.out.flush();
+        }
+    }
+
+    /** Capture the screen, scaled to *maxWidth* on its longer edge, as JPEG. */
+    private static byte[] capture(int maxWidth, int quality) throws Exception {
+        byte[] raw = readAll(Runtime.getRuntime().exec("screencap").getInputStream());
+        if (raw.length < 16) {
+            throw new IllegalStateException("screencap returned " + raw.length + " bytes");
+        }
+
+        ByteBuffer header = ByteBuffer.wrap(raw).order(ByteOrder.LITTLE_ENDIAN);
+        int width = header.getInt();
+        int height = header.getInt();
+        header.getInt();  // pixel format; always RGBA_8888 in practice
+
+        // Android 10 added a colour space to the header, so its size is not a
+        // constant. The pixels are the last width*height*4 bytes either way.
+        int offset = raw.length - width * height * 4;
+        if (offset < 12) {
+            throw new IllegalStateException(
+                    "screencap gave " + raw.length + " bytes for " + width + "x" + height);
+        }
+
+        Bitmap bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);
+        // ARGB_8888 is stored RGBA in memory on a little-endian machine, which
+        // is exactly what screencap produces, so the buffer copies straight in.
+        ByteBuffer pixels = ByteBuffer.wrap(raw, offset, width * height * 4);
+        bitmap.copyPixelsFromBuffer(pixels);
+
+        int longer = Math.max(width, height);
+        if (longer > maxWidth) {
+            float scale = (float) maxWidth / longer;
+            bitmap = Bitmap.createScaledBitmap(
+                    bitmap, Math.round(width * scale), Math.round(height * scale), true);
+        }
+
+        ByteArrayOutputStream buffer = new ByteArrayOutputStream();
+        bitmap.compress(Bitmap.CompressFormat.JPEG, quality, buffer);
+        return buffer.toByteArray();
+    }
+
+    private static byte[] readAll(InputStream stream) throws Exception {
+        ByteArrayOutputStream buffer = new ByteArrayOutputStream(1 << 22);
+        byte[] chunk = new byte[1 << 16];
+        int read;
+        while ((read = stream.read(chunk)) > 0) {
+            buffer.write(chunk, 0, read);
+        }
+        return buffer.toByteArray();
+    }
+
+    private Mirror() {
+    }
+}

mthread/__init__.py

@@ -21,7 +21,7 @@
 """
 
 from .adb import find_adb
-from .device import Device, DeviceInfo, list_devices
+from .device import Device, DeviceInfo, find_devices, list_devices, restart_server
 from .hand import HandSettings, reorder_strokes, simulate
 from .injector import InjectorUnavailableError, Pacing, TouchInjector
 from .errors import (
@@ -62,7 +62,7 @@ def __dir__():
 
 __all__ = [
     "__version__",
-    "Device", "DeviceInfo", "list_devices", "find_adb",
+    "Device", "DeviceInfo", "list_devices", "find_devices", "restart_server", "find_adb",
     "HandSettings", "simulate", "reorder_strokes",
     "TouchInjector", "Pacing", "InjectorUnavailableError",
     "Recorder", "Session", "InputEvent", "replay", "build_replay_script", "parse_getevent_line",

mthread/device.py

@@ -57,6 +57,32 @@ def list_devices(adb_path: str | None = None) -> list[DeviceInfo]:
     return devices
 
 
+def restart_server(adb_path: str | None = None) -> None:
+    """Stop and start the adb daemon.
+
+    The daemon can end up running but enumerating nothing - it was started
+    before the USB driver settled, or another copy of adb on the machine claimed
+    port 5037 first. ``adb devices`` then reports an empty list for a phone that
+    is plugged in and working, and no amount of asking again helps.
+
+    Only worth doing when the list came back empty: killing the daemon
+    disconnects anything else using it, Android Studio included.
+    """
+    adb = adb_path or find_adb()
+    run_adb(adb, ["kill-server"], check=False)
+    run_adb(adb, ["start-server"], check=False)
+
+
+def find_devices(adb_path: str | None = None) -> list[DeviceInfo]:
+    """List devices, reviving a wedged daemon once before giving up."""
+    adb = adb_path or find_adb()
+    devices = list_devices(adb)
+    if devices:
+        return devices
+    restart_server(adb)
+    return list_devices(adb)
+
+
 class Device:
     """A connected Android device.