Converge the two containerised fleet-box stacks (M3's broker.sh --docker vs Ms's compose)

CLAUDE.md

@@ -163,7 +163,7 @@ waypoint cross) from `/v1/maps/<site>/<floor>/zones.json`.
 
 ## Fleet: the server pipelines (Ms)
 
-Milestone Ms of `docs/design/fleet.md`: how the two **non-robot** machines are built and updated, runbook in `docs/fleet/server-pipelines.md`, measurements in `docs/fleet/ms-verification.md`. Both are container deploys **driven by their operator, not by the fleet server** — a robot is fleet-managed, a server is infrastructure — and the pipelines differ in exactly one thing, state. **The fleet server** (`mote_fleet/deploy/`: `Dockerfile` + `docker-compose.yml` + `fleet-deploy.sh`, image `ghcr.io/clachdev/mote-fleet` built by `.github/workflows/fleet-image.yml`) is two containers — `eclipse-mosquitto:2` mounting **`server/mosquitto.conf`, the same file `fleet-broker` uses**, so deployed and workstation brokers cannot drift, and a python image carrying the API, the registry and the M3 dashboard — plus two named volumes holding the only state that matters: `/var/lib/mote-fleet` (registry.db + the `sites/` bundles the dashboard's basemaps come from) and the broker's retained messages. `.env` is the declared state; `BROKER_HOST` is the one value that must be right (handed to robots verbatim at enrollment, so the compose file refuses to start without it) and `BROKER_WS_PORT` is published *and* passed as `--broker-ws-port`, because it is the port the browser is told to reach the broker on. It holds state, so its update is a **gated recreate**, not blue/green: `fleet-deploy.sh update` tags the running image `:previous` before pulling, recreates, health-gates on `/healthz` over the *published* port, and puts the old image back automatically if that fails; `backup`/`restore` snapshot both volumes (registry via sqlite3's online backup API, not `cp`). **The inference server** (`mote_perception/deploy/inference-deploy.sh`, one file curled onto the host) is stateless, so it *is* blue/green: the candidate runs on shadow ports 5611/5612 while the current one keeps serving, and must pass `mote_perception/tools/probe.py` — health **and a real synthetic frame**, because a health sentinel is answered before the model has ever loaded and cannot see a broken weight download or a CUDA mismatch. The **flip is a stop-then-start**, deliberately: pushing a new port out to robots (as the design sketch had it) means editing `perception.yaml` on every robot, a worse outage than the seconds this costs, which the robot's warn-and-skip fallback makes a non-event. Every check runs *inside the image being deployed*, so the GPU box still installs nothing. `mote_perception/deploy/test/drill.sh` (`pixi run deploy-test`) exercises that whole pipeline with stub images on any machine with docker — no GPU — and `mote_fleet/test/test_fleet_outage.py` is the other half of the milestone's acceptance: kill the broker under a live agent and the robot still finishes its task, then the agent reconnects by itself.
+Milestone Ms of `docs/design/fleet.md`: how the two **non-robot** machines are built and updated, runbook in `docs/fleet/server-pipelines.md`, measurements in `docs/fleet/ms-verification.md`. Both are container deploys **driven by their operator, not by the fleet server** — a robot is fleet-managed, a server is infrastructure — and the pipelines differ in exactly one thing, state. **The fleet server** (`mote_fleet/deploy/`: `Dockerfile` + `docker-compose.yml` + `fleet-deploy.sh`, image `ghcr.io/clachdev/mote-fleet` built by `.github/workflows/fleet-image.yml`) is two containers — mosquitto mounting **`server/mosquitto.conf`, the same file `fleet-broker` uses**, so deployed and workstation brokers cannot drift, with the **image tag pinned once** in the compose file and read back by `broker.sh` so they cannot drift on the binary either (a *minor* series, `2.1-alpine`, not the floating `:2`: 2.0 links libwebsockets and 2.1 implements websockets natively, so `:2` moving again could take the dashboard's read path with it, and the broker healthcheck probes 9001 as well as 1883 because mosquitto stays up and healthy-looking when only the websockets listener fails to open), and a python image carrying the API, the registry and the M3 dashboard — plus two named volumes holding the only state that matters: `/var/lib/mote-fleet` (registry.db + the `sites/` bundles the dashboard's basemaps come from) and the broker's retained messages. `.env` is the declared state; `BROKER_HOST` is the one value that must be right (handed to robots verbatim at enrollment, so the compose file refuses to start without it) and `BROKER_WS_PORT` is published *and* passed as `--broker-ws-port`, because it is the port the browser is told to reach the broker on. It holds state, so its update is a **gated recreate**, not blue/green: `fleet-deploy.sh update` tags the running image `:previous` before pulling, recreates, health-gates on `/healthz` over the *published* port, and puts the old image back automatically if that fails; `backup`/`restore` snapshot both volumes (registry via sqlite3's online backup API, not `cp`). **The inference server** (`mote_perception/deploy/inference-deploy.sh`, one file curled onto the host) is stateless, so it *is* blue/green: the candidate runs on shadow ports 5611/5612 while the current one keeps serving, and must pass `mote_perception/tools/probe.py` — health **and a real synthetic frame**, because a health sentinel is answered before the model has ever loaded and cannot see a broken weight download or a CUDA mismatch. The **flip is a stop-then-start**, deliberately: pushing a new port out to robots (as the design sketch had it) means editing `perception.yaml` on every robot, a worse outage than the seconds this costs, which the robot's warn-and-skip fallback makes a non-event. Every check runs *inside the image being deployed*, so the GPU box still installs nothing. `mote_perception/deploy/test/drill.sh` (`pixi run deploy-test`) exercises that whole pipeline with stub images on any machine with docker — no GPU — and `mote_fleet/test/test_fleet_outage.py` is the other half of the milestone's acceptance: kill the broker under a live agent and the robot still finishes its task, then the agent reconnects by itself.
 
 ## Sites (maps & zones)
 

docs/fleet/README.md

@@ -400,44 +400,85 @@ enrolling a robot that already has an id adopts that id.
 
 ## 6. The fleet server
 
-Two processes on one always-on box — a VPS, a home server, or a spare Pi. Both
-are ROS-free, so the box needs no robot software; the `fleet` pixi environment
-carries nothing but a broker and Python.
+Two processes on one always-on box — a VPS, a home server, or a spare Pi: an
+MQTT broker and the API that serves the registry and the dashboard. Both are
+ROS-free, so the box needs no robot software.
+
+**The fleet box runs them as containers**, from
+[`mote_fleet/deploy/`](../../mote_fleet/deploy) — a compose file, an image and
+one script — with `docker` as the only thing installed on it. That directory is
+also where the update, rollback, backup and restore story lives, and it is what
+"rebuild the fleet server from scratch" means in practice:
+
+```bash
+cp env.example .env && $EDITOR .env       # BROKER_HOST = the address robots dial
+./fleet-deploy.sh up                      # gated on /healthz answering
+./fleet-deploy.sh fleetctl operator new --name you   # dispatch needs one
+./fleet-deploy.sh fleetctl token new                 # then enroll a robot
+```
+
+The runbook is [`server-pipelines.md`](server-pipelines.md); what was measured
+is [`ms-verification.md`](ms-verification.md). Two differences from running the
+processes by hand: the containers restart themselves, so there are no systemd
+units to write (the fleet box is deliberately not covered by `pixi run setup`,
+which provisions robots), and `fleetctl` runs *inside* the server container,
+because that is where the registry file lives.
+
+### The same thing on a workstation
+
+For development, or to try the stack before committing a box to it, the two
+processes run in the foreground:
 
 ```bash
 pixi run fleet-broker                                 # MQTT: 1883, WebSockets: 9001
 pixi run -e fleet fleet-server -- --broker-host fleet-box   # API + UI, port 8080
 ```
 
-**Why the broker runs in a container.** The dashboard (§9) subscribes to the
+`fleet-broker` is **the same broker as the deployment** — the same image and the
+same `mosquitto.conf`, just in the foreground and without the API beside it. It
+is not a second way to run the fleet box: the image tag is pinned once, in
+`docker-compose.yml`, and `broker.sh` reads it from there, so a workstation
+cannot end up on a different mosquitto from the one that is deployed.
+
+**Why the broker is a container at all.** The dashboard (§9) subscribes to the
 control plane straight from the browser, and a browser cannot speak raw MQTT —
 it needs the broker's WebSocket listener. conda-forge's mosquitto is built
-without one, so `fleet-broker` runs `eclipse-mosquitto` under Docker with the
-same `mosquitto.conf` this repo ships — which is also exactly what the deployed
-fleet box runs. For a box with no Docker there is `fleet-broker-local`, which
-uses conda's binary: robots and `fleetctl` work exactly as before, and it tells
-you on startup that the dashboard will not.
+without one. For a box with no Docker there is `fleet-broker-local`, which uses
+conda's binary: robots and `fleetctl` work exactly as before, and it tells you on
+startup that the dashboard will not.
 
 ```bash
 pixi run -e fleet fleet-broker-local                  # a box with no docker
 ```
 
 It lives in the `fleet` environment rather than beside `fleet-broker` because
 that is where the binary it runs comes from — the container needs docker, not an
-environment.
+environment. It is also the only case where the shipped config is not used
+verbatim: `broker.sh` strips the websockets stanza out of a *copy*, rather than
+this repo carrying two configs that can drift.
 
 The reasoning and the measurement are in
 [`m3-verification.md`](m3-verification.md) §1.
 
+**If the dashboard is blank, suspect this listener first.** Mosquitto opens the
+MQTT listener and keeps running whether or not the WebSocket one came up, so
+robots and `fleetctl` stay perfectly healthy while the browser has nothing to
+subscribe to. The startup log names every listener it opened, which is the
+direct answer; the compose healthcheck probes both ports for the same reason.
+
+### The rest of it
+
 State lives in **`$MOTE_FLEET_HOME`** (default `~/.mote-fleet`) — the registry
 database, the broker's retained messages, and the site bundles the dashboard
 draws robots on. That is the server-side analogue of the robot's `MOTE_HOME`:
 redeploying the server software replaces code around it and never the fleet's
-memory of who is in it.
+memory of who is in it. Under compose those are two named volumes, which is what
+`fleet-deploy.sh backup` snapshots.
 
 `--broker-host` is the address **robots** should dial, so on a tailnet it is the
 fleet box's MagicDNS name (`fleet-box`), not `localhost` — it is handed out
-verbatim in every enrollment answer. It defaults to the box's hostname.
+verbatim in every enrollment answer. It defaults to the box's hostname. Under
+compose it is `BROKER_HOST` in `.env`, and the stack refuses to start without it.
 
 **Security, plainly:** the broker is anonymous and the API's *read* routes are
 unauthenticated. Dispatch is not — it needs an operator token (§8) — but that is
@@ -448,29 +489,6 @@ already trusted on. Per-robot broker credentials and operator auth everywhere
 are M7; the shape of what changes is in
 [`control-plane.md`](control-plane.md#security-posture-and-what-m7-changes).
 
-**That is the workstation shape.** A real fleet box runs both processes as
-containers, from [`mote_fleet/deploy/`](../../mote_fleet/deploy) — a compose
-file, an image and one script — with `docker` as the only thing installed on it.
-The broker is the same `eclipse-mosquitto` container `fleet-broker` runs,
-reading the same `mosquitto.conf`, but as a restarting service beside the API
-rather than a foreground command. That directory is also where the update,
-rollback, backup and restore story lives, and it is what "rebuild the fleet
-server from scratch" means in practice:
-
-```bash
-cp env.example .env && $EDITOR .env       # BROKER_HOST = the address robots dial
-./fleet-deploy.sh up                      # gated on /healthz answering
-./fleet-deploy.sh fleetctl operator new --name you   # dispatch needs one
-./fleet-deploy.sh fleetctl token new                 # then enroll a robot
-```
-
-The runbook is [`server-pipelines.md`](server-pipelines.md); what was measured
-is [`ms-verification.md`](ms-verification.md). Two differences from running it
-by hand: the containers restart themselves, so there are no systemd units to
-write (the fleet box is deliberately not covered by `pixi run setup`, which
-provisions robots), and `fleetctl` runs *inside* the server container, because
-that is where the registry file lives.
-
 ---
 
 ## 7. Enrolling a robot

docs/fleet/server-pipelines.md

@@ -144,20 +144,40 @@ registry at the end.
 
 ### The broker container, and the one config
 
-The deployed broker is `eclipse-mosquitto:2` and it mounts
+The deployed broker mounts
 [`../server/mosquitto.conf`](../../mote_fleet/server/mosquitto.conf) — the file
 the workstation broker uses too. That is deliberate: M3 already had to run
 mosquitto in a container, because conda-forge's build has no websockets
 (m1-verification.md §4) and the dashboard subscribes from the browser. `pixi run
 fleet-broker` is that container as a foreground command; this compose service
 is the same thing as a restarting service with a volume for its retained state
-and a healthcheck that asks the broker for its own uptime.
+and a healthcheck.
 
 Keeping one config means the listener set cannot drift between the box you
 develop on and the box you deploy. `working_dir: /mosquitto/data` is what puts
 `persistence_file mosquitto.db` in the volume — the same reason `broker.sh` cds
 to `$MOTE_FLEET_HOME`.
 
+**The image tag is pinned in one place**, the compose file's broker service;
+`broker.sh` reads that line rather than carrying a default of its own, so the
+foreground broker and the deployed one cannot end up on different mosquittos.
+It names a **minor** series (`2.1-alpine`) rather than the floating `:2`,
+because the thing that must not move under us is how websockets is built in:
+2.0 links libwebsockets, 2.1 implements it natively, and `:2` follows whatever
+comes next. `-alpine` is not a variant choice — upstream publishes 2.1 only as
+`-alpine` tags, and `:2`, `:2.1-alpine` and `:2.1.2-alpine` are one manifest
+today. `MOTE_BROKER_IMAGE` overrides it for both.
+
+**The healthcheck probes both listeners**, not just MQTT. Mosquitto opens the
+MQTT listener and keeps running when the websockets one is refused, so an
+MQTT-only probe reports a healthy stack whose dashboard is dead — and
+`fleet-deploy.sh up` gates on that probe, so it would report a good deploy.
+`mosquitto_sub` asks the broker for its own uptime on 1883, which is more than
+a TCP connect proves; 9001 is a TCP probe, because the failure being caught is
+"the listener never opened" and `mosquitto_sub` cannot speak websockets to do
+better. `mote_fleet/test/test_deploy_config.py` holds all of the above without
+needing docker.
+
 ### Security posture
 
 Unchanged from M3, and stated here because a container makes it easy to expose

mote_fleet/deploy/docker-compose.yml

@@ -12,15 +12,26 @@ name: mote-fleet
 
 services:
   # The MQTT control plane (docs/fleet/control-plane.md), including the
-  # WebSocket listener the browser dashboard subscribes on — `pixi run
-  # fleet-broker-ws` already runs mosquitto in a container for exactly that
-  # reason (conda-forge's build has no websockets, m1-verification.md §4); this
-  # is that same container as a permanent, restarting service.
+  # WebSocket listener the browser dashboard subscribes on — conda-forge's
+  # mosquitto is built without websockets (m1-verification.md §4), so the broker
+  # runs in a container everywhere. `pixi run fleet-broker` is this same image
+  # and config as a foreground command for a workstation; this is it as a
+  # permanent, restarting service, which is the deployment.
   #
   # It mounts ../server/mosquitto.conf rather than a copy: one config for the
   # workstation broker and the deployed one, so they cannot drift.
   broker:
-    image: ${MOTE_BROKER_IMAGE:-eclipse-mosquitto:2}
+    # THE broker image pin for this repo. `broker.sh` reads this line rather
+    # than carrying a default of its own, so a workstation cannot run a
+    # different mosquitto from the fleet box.
+    #
+    # Pinned to the 2.1 minor series, not the floating `:2`, because the thing
+    # that must not change under us is the websockets implementation: 2.0 links
+    # libwebsockets, 2.1 implements it natively, and `:2` follows whatever comes
+    # next. `-alpine` is not a variant choice — 2.1 is published only as
+    # `-alpine` tags, and `:2`, `:2.1-alpine` and `:2.1.2-alpine` are the same
+    # manifest today.
+    image: ${MOTE_BROKER_IMAGE:-eclipse-mosquitto:2.1-alpine}
     restart: unless-stopped
     ports:
       - "${BIND_PREFIX-}${BROKER_PORT:-1883}:1883"
@@ -33,9 +44,16 @@ services:
     # container's filesystem. Same reason broker.sh cds to $MOTE_FLEET_HOME.
     working_dir: /mosquitto/data
     healthcheck:
-      # Ask the broker for its own uptime: a listener that accepts a TCP
+      # Two listeners, two checks, because they fail independently.
+      #
+      # 1883: ask the broker for its own uptime — a listener that accepts a TCP
       # connection but cannot serve a subscription is not healthy.
-      test: ["CMD", "mosquitto_sub", "-p", "1883", "-t", "$$SYS/broker/uptime", "-C", "1", "-W", "3"]
+      # 9001: the dashboard's read path. mosquitto opens the MQTT listener and
+      # keeps running even when the websockets one is refused, so without this
+      # the stack reports healthy and the UI is silently dead. A TCP probe is
+      # the whole check: the failure being caught is "the listener never
+      # opened", and mosquitto_sub cannot speak websockets to do better.
+      test: ["CMD-SHELL", "mosquitto_sub -p 1883 -t '$$SYS/broker/uptime' -C 1 -W 3 >/dev/null && nc -z 127.0.0.1 9001"]
       interval: 30s
       timeout: 10s
       start_period: 5s