From 3a25ef69b67c66cf00b95604c321fff7c264608e Mon Sep 17 00:00:00 2001 From: Michael Johnson Date: Thu, 30 Jul 2026 18:37:42 +0100 Subject: [PATCH] Converge the fleet box on one containerised broker MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit M3 and Ms each grew a way to run mosquitto in a container and did not know about each other. Ms had already folded the two pixi tasks into one and pointed compose at `server/mosquitto.conf`, so the config was shared; what was left was an image tag pinned in two places and docs that led with the workstation command rather than the deployment. The compose stack is the fleet box's deployment. `broker.sh` is the workstation convenience and now defers to it in both directions: it already used the shared mosquitto.conf, and it reads the image tag out of docker-compose.yml rather than carrying a default of its own. A literal here would be a second pin nobody would remember to bump. --local resolves nothing from the deploy directory, so a box with no docker is unaffected. The tag names a minor series, `2.1-alpine`, not the floating `:2`. What must not move under us is how websockets is built in: 2.0 links libwebsockets, 2.1 implements it natively, and `:2` follows whatever comes next. `-alpine` is not a variant choice — upstream publishes 2.1 only as `-alpine` tags, and `:2`, `:2.1-alpine` and `:2.1.2-alpine` are one manifest today, so this changes no bytes. The broker healthcheck now probes 9001 as well as 1883, because that failure is silent: mosquitto opens the MQTT listener and keeps running when the websockets one is refused, so robots and fleetctl stay healthy while the dashboard has nothing to subscribe to — and `fleet-deploy.sh up` gates on this probe, so it would have reported a good deploy. Measured on a broker with the websockets stanza stripped: the MQTT-only check reports healthy, the new one reports unhealthy. 9001 is a TCP probe because the failure being caught is "the listener never opened" and mosquitto_sub cannot speak websockets. README §6 now leads with compose and presents the pixi tasks as the same thing in the foreground. Verified: compose broker up --wait passes the new health gate on the pinned image; both listeners open under 2.1.2; the dashboard's own MQTT codec (server/ui/mqtt.mjs) completes CONNECT/SUBSCRIBE over the published WebSocket port. test_deploy_config.py holds the invariants without needing docker — one pin, no second tag, broker.sh reaching for the pinned image against a stub docker, one mounted config, the websockets stanza and its strip markers, no log_type override, and both ports in the healthcheck. 194 passed, 6 skipped across mote_fleet; pre-commit clean. Co-authored-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01RdGDhvwcDBmcgK9Gc294th --- CLAUDE.md | 2 +- docs/fleet/README.md | 88 +++++++++------ docs/fleet/server-pipelines.md | 24 +++- mote_fleet/deploy/docker-compose.yml | 32 ++++-- mote_fleet/server/broker.sh | 34 +++++- mote_fleet/test/test_deploy_config.py | 151 ++++++++++++++++++++++++++ 6 files changed, 283 insertions(+), 48 deletions(-) create mode 100644 mote_fleet/test/test_deploy_config.py diff --git a/CLAUDE.md b/CLAUDE.md index 2e88716..31f23a2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -163,7 +163,7 @@ waypoint cross) from `/v1/maps///zones.json`. ## Fleet: the server pipelines (Ms) -Milestone Ms of `docs/design/fleet.md`: how the two **non-robot** machines are built and updated, runbook in `docs/fleet/server-pipelines.md`, measurements in `docs/fleet/ms-verification.md`. Both are container deploys **driven by their operator, not by the fleet server** — a robot is fleet-managed, a server is infrastructure — and the pipelines differ in exactly one thing, state. **The fleet server** (`mote_fleet/deploy/`: `Dockerfile` + `docker-compose.yml` + `fleet-deploy.sh`, image `ghcr.io/clachdev/mote-fleet` built by `.github/workflows/fleet-image.yml`) is two containers — `eclipse-mosquitto:2` mounting **`server/mosquitto.conf`, the same file `fleet-broker` uses**, so deployed and workstation brokers cannot drift, and a python image carrying the API, the registry and the M3 dashboard — plus two named volumes holding the only state that matters: `/var/lib/mote-fleet` (registry.db + the `sites/` bundles the dashboard's basemaps come from) and the broker's retained messages. `.env` is the declared state; `BROKER_HOST` is the one value that must be right (handed to robots verbatim at enrollment, so the compose file refuses to start without it) and `BROKER_WS_PORT` is published *and* passed as `--broker-ws-port`, because it is the port the browser is told to reach the broker on. It holds state, so its update is a **gated recreate**, not blue/green: `fleet-deploy.sh update` tags the running image `:previous` before pulling, recreates, health-gates on `/healthz` over the *published* port, and puts the old image back automatically if that fails; `backup`/`restore` snapshot both volumes (registry via sqlite3's online backup API, not `cp`). **The inference server** (`mote_perception/deploy/inference-deploy.sh`, one file curled onto the host) is stateless, so it *is* blue/green: the candidate runs on shadow ports 5611/5612 while the current one keeps serving, and must pass `mote_perception/tools/probe.py` — health **and a real synthetic frame**, because a health sentinel is answered before the model has ever loaded and cannot see a broken weight download or a CUDA mismatch. The **flip is a stop-then-start**, deliberately: pushing a new port out to robots (as the design sketch had it) means editing `perception.yaml` on every robot, a worse outage than the seconds this costs, which the robot's warn-and-skip fallback makes a non-event. Every check runs *inside the image being deployed*, so the GPU box still installs nothing. `mote_perception/deploy/test/drill.sh` (`pixi run deploy-test`) exercises that whole pipeline with stub images on any machine with docker — no GPU — and `mote_fleet/test/test_fleet_outage.py` is the other half of the milestone's acceptance: kill the broker under a live agent and the robot still finishes its task, then the agent reconnects by itself. +Milestone Ms of `docs/design/fleet.md`: how the two **non-robot** machines are built and updated, runbook in `docs/fleet/server-pipelines.md`, measurements in `docs/fleet/ms-verification.md`. Both are container deploys **driven by their operator, not by the fleet server** — a robot is fleet-managed, a server is infrastructure — and the pipelines differ in exactly one thing, state. **The fleet server** (`mote_fleet/deploy/`: `Dockerfile` + `docker-compose.yml` + `fleet-deploy.sh`, image `ghcr.io/clachdev/mote-fleet` built by `.github/workflows/fleet-image.yml`) is two containers — mosquitto mounting **`server/mosquitto.conf`, the same file `fleet-broker` uses**, so deployed and workstation brokers cannot drift, with the **image tag pinned once** in the compose file and read back by `broker.sh` so they cannot drift on the binary either (a *minor* series, `2.1-alpine`, not the floating `:2`: 2.0 links libwebsockets and 2.1 implements websockets natively, so `:2` moving again could take the dashboard's read path with it, and the broker healthcheck probes 9001 as well as 1883 because mosquitto stays up and healthy-looking when only the websockets listener fails to open), and a python image carrying the API, the registry and the M3 dashboard — plus two named volumes holding the only state that matters: `/var/lib/mote-fleet` (registry.db + the `sites/` bundles the dashboard's basemaps come from) and the broker's retained messages. `.env` is the declared state; `BROKER_HOST` is the one value that must be right (handed to robots verbatim at enrollment, so the compose file refuses to start without it) and `BROKER_WS_PORT` is published *and* passed as `--broker-ws-port`, because it is the port the browser is told to reach the broker on. It holds state, so its update is a **gated recreate**, not blue/green: `fleet-deploy.sh update` tags the running image `:previous` before pulling, recreates, health-gates on `/healthz` over the *published* port, and puts the old image back automatically if that fails; `backup`/`restore` snapshot both volumes (registry via sqlite3's online backup API, not `cp`). **The inference server** (`mote_perception/deploy/inference-deploy.sh`, one file curled onto the host) is stateless, so it *is* blue/green: the candidate runs on shadow ports 5611/5612 while the current one keeps serving, and must pass `mote_perception/tools/probe.py` — health **and a real synthetic frame**, because a health sentinel is answered before the model has ever loaded and cannot see a broken weight download or a CUDA mismatch. The **flip is a stop-then-start**, deliberately: pushing a new port out to robots (as the design sketch had it) means editing `perception.yaml` on every robot, a worse outage than the seconds this costs, which the robot's warn-and-skip fallback makes a non-event. Every check runs *inside the image being deployed*, so the GPU box still installs nothing. `mote_perception/deploy/test/drill.sh` (`pixi run deploy-test`) exercises that whole pipeline with stub images on any machine with docker — no GPU — and `mote_fleet/test/test_fleet_outage.py` is the other half of the milestone's acceptance: kill the broker under a live agent and the robot still finishes its task, then the agent reconnects by itself. ## Sites (maps & zones) diff --git a/docs/fleet/README.md b/docs/fleet/README.md index b6bbc98..67e77a2 100644 --- a/docs/fleet/README.md +++ b/docs/fleet/README.md @@ -400,23 +400,52 @@ enrolling a robot that already has an id adopts that id. ## 6. The fleet server -Two processes on one always-on box — a VPS, a home server, or a spare Pi. Both -are ROS-free, so the box needs no robot software; the `fleet` pixi environment -carries nothing but a broker and Python. +Two processes on one always-on box — a VPS, a home server, or a spare Pi: an +MQTT broker and the API that serves the registry and the dashboard. Both are +ROS-free, so the box needs no robot software. + +**The fleet box runs them as containers**, from +[`mote_fleet/deploy/`](../../mote_fleet/deploy) — a compose file, an image and +one script — with `docker` as the only thing installed on it. That directory is +also where the update, rollback, backup and restore story lives, and it is what +"rebuild the fleet server from scratch" means in practice: + +```bash +cp env.example .env && $EDITOR .env # BROKER_HOST = the address robots dial +./fleet-deploy.sh up # gated on /healthz answering +./fleet-deploy.sh fleetctl operator new --name you # dispatch needs one +./fleet-deploy.sh fleetctl token new # then enroll a robot +``` + +The runbook is [`server-pipelines.md`](server-pipelines.md); what was measured +is [`ms-verification.md`](ms-verification.md). Two differences from running the +processes by hand: the containers restart themselves, so there are no systemd +units to write (the fleet box is deliberately not covered by `pixi run setup`, +which provisions robots), and `fleetctl` runs *inside* the server container, +because that is where the registry file lives. + +### The same thing on a workstation + +For development, or to try the stack before committing a box to it, the two +processes run in the foreground: ```bash pixi run fleet-broker # MQTT: 1883, WebSockets: 9001 pixi run -e fleet fleet-server -- --broker-host fleet-box # API + UI, port 8080 ``` -**Why the broker runs in a container.** The dashboard (§9) subscribes to the +`fleet-broker` is **the same broker as the deployment** — the same image and the +same `mosquitto.conf`, just in the foreground and without the API beside it. It +is not a second way to run the fleet box: the image tag is pinned once, in +`docker-compose.yml`, and `broker.sh` reads it from there, so a workstation +cannot end up on a different mosquitto from the one that is deployed. + +**Why the broker is a container at all.** The dashboard (§9) subscribes to the control plane straight from the browser, and a browser cannot speak raw MQTT — it needs the broker's WebSocket listener. conda-forge's mosquitto is built -without one, so `fleet-broker` runs `eclipse-mosquitto` under Docker with the -same `mosquitto.conf` this repo ships — which is also exactly what the deployed -fleet box runs. For a box with no Docker there is `fleet-broker-local`, which -uses conda's binary: robots and `fleetctl` work exactly as before, and it tells -you on startup that the dashboard will not. +without one. For a box with no Docker there is `fleet-broker-local`, which uses +conda's binary: robots and `fleetctl` work exactly as before, and it tells you on +startup that the dashboard will not. ```bash pixi run -e fleet fleet-broker-local # a box with no docker @@ -424,20 +453,32 @@ pixi run -e fleet fleet-broker-local # a box with no docker It lives in the `fleet` environment rather than beside `fleet-broker` because that is where the binary it runs comes from — the container needs docker, not an -environment. +environment. It is also the only case where the shipped config is not used +verbatim: `broker.sh` strips the websockets stanza out of a *copy*, rather than +this repo carrying two configs that can drift. The reasoning and the measurement are in [`m3-verification.md`](m3-verification.md) §1. +**If the dashboard is blank, suspect this listener first.** Mosquitto opens the +MQTT listener and keeps running whether or not the WebSocket one came up, so +robots and `fleetctl` stay perfectly healthy while the browser has nothing to +subscribe to. The startup log names every listener it opened, which is the +direct answer; the compose healthcheck probes both ports for the same reason. + +### The rest of it + State lives in **`$MOTE_FLEET_HOME`** (default `~/.mote-fleet`) — the registry database, the broker's retained messages, and the site bundles the dashboard draws robots on. That is the server-side analogue of the robot's `MOTE_HOME`: redeploying the server software replaces code around it and never the fleet's -memory of who is in it. +memory of who is in it. Under compose those are two named volumes, which is what +`fleet-deploy.sh backup` snapshots. `--broker-host` is the address **robots** should dial, so on a tailnet it is the fleet box's MagicDNS name (`fleet-box`), not `localhost` — it is handed out -verbatim in every enrollment answer. It defaults to the box's hostname. +verbatim in every enrollment answer. It defaults to the box's hostname. Under +compose it is `BROKER_HOST` in `.env`, and the stack refuses to start without it. **Security, plainly:** the broker is anonymous and the API's *read* routes are unauthenticated. Dispatch is not — it needs an operator token (§8) — but that is @@ -448,29 +489,6 @@ already trusted on. Per-robot broker credentials and operator auth everywhere are M7; the shape of what changes is in [`control-plane.md`](control-plane.md#security-posture-and-what-m7-changes). -**That is the workstation shape.** A real fleet box runs both processes as -containers, from [`mote_fleet/deploy/`](../../mote_fleet/deploy) — a compose -file, an image and one script — with `docker` as the only thing installed on it. -The broker is the same `eclipse-mosquitto` container `fleet-broker` runs, -reading the same `mosquitto.conf`, but as a restarting service beside the API -rather than a foreground command. That directory is also where the update, -rollback, backup and restore story lives, and it is what "rebuild the fleet -server from scratch" means in practice: - -```bash -cp env.example .env && $EDITOR .env # BROKER_HOST = the address robots dial -./fleet-deploy.sh up # gated on /healthz answering -./fleet-deploy.sh fleetctl operator new --name you # dispatch needs one -./fleet-deploy.sh fleetctl token new # then enroll a robot -``` - -The runbook is [`server-pipelines.md`](server-pipelines.md); what was measured -is [`ms-verification.md`](ms-verification.md). Two differences from running it -by hand: the containers restart themselves, so there are no systemd units to -write (the fleet box is deliberately not covered by `pixi run setup`, which -provisions robots), and `fleetctl` runs *inside* the server container, because -that is where the registry file lives. - --- ## 7. Enrolling a robot diff --git a/docs/fleet/server-pipelines.md b/docs/fleet/server-pipelines.md index 5749b8a..eb87e19 100644 --- a/docs/fleet/server-pipelines.md +++ b/docs/fleet/server-pipelines.md @@ -144,20 +144,40 @@ registry at the end. ### The broker container, and the one config -The deployed broker is `eclipse-mosquitto:2` and it mounts +The deployed broker mounts [`../server/mosquitto.conf`](../../mote_fleet/server/mosquitto.conf) — the file the workstation broker uses too. That is deliberate: M3 already had to run mosquitto in a container, because conda-forge's build has no websockets (m1-verification.md §4) and the dashboard subscribes from the browser. `pixi run fleet-broker` is that container as a foreground command; this compose service is the same thing as a restarting service with a volume for its retained state -and a healthcheck that asks the broker for its own uptime. +and a healthcheck. Keeping one config means the listener set cannot drift between the box you develop on and the box you deploy. `working_dir: /mosquitto/data` is what puts `persistence_file mosquitto.db` in the volume — the same reason `broker.sh` cds to `$MOTE_FLEET_HOME`. +**The image tag is pinned in one place**, the compose file's broker service; +`broker.sh` reads that line rather than carrying a default of its own, so the +foreground broker and the deployed one cannot end up on different mosquittos. +It names a **minor** series (`2.1-alpine`) rather than the floating `:2`, +because the thing that must not move under us is how websockets is built in: +2.0 links libwebsockets, 2.1 implements it natively, and `:2` follows whatever +comes next. `-alpine` is not a variant choice — upstream publishes 2.1 only as +`-alpine` tags, and `:2`, `:2.1-alpine` and `:2.1.2-alpine` are one manifest +today. `MOTE_BROKER_IMAGE` overrides it for both. + +**The healthcheck probes both listeners**, not just MQTT. Mosquitto opens the +MQTT listener and keeps running when the websockets one is refused, so an +MQTT-only probe reports a healthy stack whose dashboard is dead — and +`fleet-deploy.sh up` gates on that probe, so it would report a good deploy. +`mosquitto_sub` asks the broker for its own uptime on 1883, which is more than +a TCP connect proves; 9001 is a TCP probe, because the failure being caught is +"the listener never opened" and `mosquitto_sub` cannot speak websockets to do +better. `mote_fleet/test/test_deploy_config.py` holds all of the above without +needing docker. + ### Security posture Unchanged from M3, and stated here because a container makes it easy to expose diff --git a/mote_fleet/deploy/docker-compose.yml b/mote_fleet/deploy/docker-compose.yml index 17016ef..865d341 100644 --- a/mote_fleet/deploy/docker-compose.yml +++ b/mote_fleet/deploy/docker-compose.yml @@ -12,15 +12,26 @@ name: mote-fleet services: # The MQTT control plane (docs/fleet/control-plane.md), including the - # WebSocket listener the browser dashboard subscribes on — `pixi run - # fleet-broker-ws` already runs mosquitto in a container for exactly that - # reason (conda-forge's build has no websockets, m1-verification.md §4); this - # is that same container as a permanent, restarting service. + # WebSocket listener the browser dashboard subscribes on — conda-forge's + # mosquitto is built without websockets (m1-verification.md §4), so the broker + # runs in a container everywhere. `pixi run fleet-broker` is this same image + # and config as a foreground command for a workstation; this is it as a + # permanent, restarting service, which is the deployment. # # It mounts ../server/mosquitto.conf rather than a copy: one config for the # workstation broker and the deployed one, so they cannot drift. broker: - image: ${MOTE_BROKER_IMAGE:-eclipse-mosquitto:2} + # THE broker image pin for this repo. `broker.sh` reads this line rather + # than carrying a default of its own, so a workstation cannot run a + # different mosquitto from the fleet box. + # + # Pinned to the 2.1 minor series, not the floating `:2`, because the thing + # that must not change under us is the websockets implementation: 2.0 links + # libwebsockets, 2.1 implements it natively, and `:2` follows whatever comes + # next. `-alpine` is not a variant choice — 2.1 is published only as + # `-alpine` tags, and `:2`, `:2.1-alpine` and `:2.1.2-alpine` are the same + # manifest today. + image: ${MOTE_BROKER_IMAGE:-eclipse-mosquitto:2.1-alpine} restart: unless-stopped ports: - "${BIND_PREFIX-}${BROKER_PORT:-1883}:1883" @@ -33,9 +44,16 @@ services: # container's filesystem. Same reason broker.sh cds to $MOTE_FLEET_HOME. working_dir: /mosquitto/data healthcheck: - # Ask the broker for its own uptime: a listener that accepts a TCP + # Two listeners, two checks, because they fail independently. + # + # 1883: ask the broker for its own uptime — a listener that accepts a TCP # connection but cannot serve a subscription is not healthy. - test: ["CMD", "mosquitto_sub", "-p", "1883", "-t", "$$SYS/broker/uptime", "-C", "1", "-W", "3"] + # 9001: the dashboard's read path. mosquitto opens the MQTT listener and + # keeps running even when the websockets one is refused, so without this + # the stack reports healthy and the UI is silently dead. A TCP probe is + # the whole check: the failure being caught is "the listener never + # opened", and mosquitto_sub cannot speak websockets to do better. + test: ["CMD-SHELL", "mosquitto_sub -p 1883 -t '$$SYS/broker/uptime' -C 1 -W 3 >/dev/null && nc -z 127.0.0.1 9001"] interval: 30s timeout: 10s start_period: 5s diff --git a/mote_fleet/server/broker.sh b/mote_fleet/server/broker.sh index 05bad30..b4458a4 100755 --- a/mote_fleet/server/broker.sh +++ b/mote_fleet/server/broker.sh @@ -6,13 +6,19 @@ # broker's retained state belongs with the registry database, not in a git tree # that a redeploy replaces. Same split as the robot's MOTE_HOME. # +# THE FLEET BOX'S BROKER IS THE COMPOSE STACK (mote_fleet/deploy/), not this +# script. This is the workstation convenience: the same image, the same +# mosquitto.conf and the same listeners, in the foreground and without the API +# beside it. It owns neither of those two facts -- the config is +# ../server/mosquitto.conf, which compose mounts too, and the image is the pin +# in ../deploy/docker-compose.yml, which this script reads. +# # WEBSOCKETS, and why the default is a container. M3's dashboard subscribes to # the control plane straight from the browser, which means the broker needs a # `protocol websockets` listener -- and conda-forge's mosquitto is built without # libwebsockets (measured: it refuses to start, # docs/fleet/m1-verification.md S4). So the broker anyone should run is -# mosquitto in a container, which is also exactly what the deployed fleet box -# runs (mote_fleet/deploy/docker-compose.yml): same image, same config file. +# mosquitto in a container. # # pixi run fleet-broker the container: MQTT 1883 + WS 9001 # pixi run -e fleet fleet-broker-local conda's mosquitto (--local), for a @@ -31,9 +37,28 @@ set -euo pipefail FLEET_HOME="${MOTE_FLEET_HOME:-$HOME/.mote-fleet}" HERE="$(cd "$(dirname "$0")" && pwd)" CONF="$HERE/mosquitto.conf" -IMAGE="${MOTE_BROKER_IMAGE:-eclipse-mosquitto:2}" +COMPOSE="$HERE/../deploy/docker-compose.yml" MODE=docker +# The image is pinned once, in the compose file, because that is the deployment +# -- this script is the workstation convenience and defers to it. Reading the +# line back is what stops the two from drifting; a literal default here would be +# a second pin that nobody would remember to bump. +default_image() { + local pinned + # The single quotes are the point: ${MOTE_BROKER_IMAGE:-...} is literal + # text being matched in the compose file, not an expansion to perform here. + # shellcheck disable=SC2016 + pinned="$(sed -n 's/^ *image: *${MOTE_BROKER_IMAGE:-\([^}]*\)}.*/\1/p' "$COMPOSE" 2>/dev/null | head -1)" + [ -n "$pinned" ] || { + echo "cannot read the broker image pin from $COMPOSE" >&2 + echo " (expected the broker service's: image: \${MOTE_BROKER_IMAGE:-:})" >&2 + echo " set MOTE_BROKER_IMAGE to override, or repair that file" >&2 + exit 1 + } + printf '%s\n' "$pinned" +} + case "${1:-}" in --local) MODE=local @@ -57,6 +82,9 @@ if [ "$MODE" = docker ]; then # published ports would put a NAT between the two for no benefit. # --user: the image's mosquitto user cannot write a state directory owned by # whoever runs this, and the fleet's retained state should stay theirs. + # Resolved here rather than at the top so that --local, which is the mode + # for a box with no docker, never depends on the deploy directory at all. + IMAGE="${MOTE_BROKER_IMAGE:-$(default_image)}" echo "broker: $IMAGE (docker) state: $FLEET_HOME config: $CONF" exec docker run --rm --name mote-broker --network host \ --user "$(id -u):$(id -g)" \ diff --git a/mote_fleet/test/test_deploy_config.py b/mote_fleet/test/test_deploy_config.py new file mode 100644 index 0000000..8bb8ce6 --- /dev/null +++ b/mote_fleet/test/test_deploy_config.py @@ -0,0 +1,151 @@ +"""Hold the fleet box to one broker: one image pin, one config, one WS listener. + +There were two containerised brokers here once — a workstation `docker run` and +the compose service — with their own image tags and their own idea of the +config. Everything below is a way for that to come back, so each check is a +seam where the two could drift apart again rather than a property of either one +alone. + +The websockets checks are the ones that matter most, because that failure is +silent: mosquitto opens the MQTT listener and keeps running whether or not the +WebSocket one came up, so robots and `fleetctl` look perfectly healthy while the +dashboard is dead. Nothing here needs docker — these are the text-level +invariants, and `docs/fleet/ms-verification.md` records the live run. +""" + +import re +import subprocess +from pathlib import Path + +import pytest +import yaml + +REPO = Path(__file__).resolve().parents[2] +COMPOSE = REPO / "mote_fleet" / "deploy" / "docker-compose.yml" +BROKER_SH = REPO / "mote_fleet" / "server" / "broker.sh" +MOSQUITTO_CONF = REPO / "mote_fleet" / "server" / "mosquitto.conf" + +PIN = re.compile(r"^ *image: *\$\{MOTE_BROKER_IMAGE:-([^}]*)\}", re.M) + + +@pytest.fixture +def compose(): + return yaml.safe_load(COMPOSE.read_text()) + + +def test_the_broker_image_is_pinned_exactly_once(): + """The compose file is the pin; nothing else may carry a default.""" + pins = PIN.findall(COMPOSE.read_text()) + assert len(pins) == 1, f"expected one broker image pin, found {pins}" + + others = [ + path + for path in (BROKER_SH, MOSQUITTO_CONF) + if "eclipse-mosquitto:" in path.read_text() + ] + assert others == [], ( + "a second broker image tag has appeared in " + f"{[p.name for p in others]} — the pin belongs only in {COMPOSE.name}" + ) + + +def test_the_broker_image_is_not_a_floating_major_tag(): + """`:2` moved from mosquitto 2.0 to 2.1, which changed how WS is built in. + + 2.0 links libwebsockets, 2.1 implements it natively, and whatever follows + may do neither — so the tag names a minor series. + """ + (pin,) = PIN.findall(COMPOSE.read_text()) + assert re.match(r"^eclipse-mosquitto:\d+\.\d+", pin), ( + f"{pin!r} does not pin a minor series" + ) + + +def test_broker_sh_runs_the_image_the_compose_file_pins(tmp_path): + """Run the script against a stub docker and see which image it reaches for. + + This is the deferral itself: `broker.sh` keeps no tag of its own, so the + only way it can name one is by reading the compose file. + """ + (pin,) = PIN.findall(COMPOSE.read_text()) + + bin_dir = tmp_path / "bin" + bin_dir.mkdir() + stub = bin_dir / "docker" + stub.write_text('#!/usr/bin/env bash\necho "DOCKER_ARGS: $*"\n') + stub.chmod(0o755) + + result = subprocess.run( + ["bash", str(BROKER_SH)], + capture_output=True, + text=True, + timeout=60, + env={ + "PATH": f"{bin_dir}:/usr/bin:/bin", + "HOME": str(tmp_path), + "MOTE_FLEET_HOME": str(tmp_path / "state"), + }, + ) + + assert result.returncode == 0, result.stderr + assert "DOCKER_ARGS: run " in result.stdout + assert pin in result.stdout, ( + f"broker.sh ran something other than the pinned {pin}:\n{result.stdout}" + ) + + +def test_the_deployed_broker_mounts_the_repos_own_mosquitto_conf(compose): + """One config file, mounted — not a copy that can fall behind.""" + mounts = compose["services"]["broker"]["volumes"] + conf = [m for m in mounts if m.endswith("/mosquitto/config/mosquitto.conf:ro")] + assert len(conf) == 1, f"expected one config mount, got {mounts}" + + source = COMPOSE.parent / conf[0].split(":")[0] + assert source.resolve() == MOSQUITTO_CONF.resolve() + + +def test_the_websockets_listener_survives_into_the_shipped_config(): + """The dashboard's whole read path is this listener.""" + text = MOSQUITTO_CONF.read_text() + block = re.search(r"^# >>> websockets$(.*?)^# <<< websockets$", text, re.M | re.S) + assert block, "the websockets stanza and its strip markers are gone" + assert "listener 9001" in block.group(1) + assert "protocol websockets" in block.group(1) + + +def test_mosquitto_does_not_narrow_its_own_log_types(): + """`log_type` replaces mosquitto's defaults rather than adding to them. + + Naming any subset drops `error`, and the errors it drops are the two that + make the broker exit: a port already in use, and a websockets listener the + build cannot honour (m3-verification.md §1). + """ + lines = [ + ln + for ln in MOSQUITTO_CONF.read_text().splitlines() + if ln.strip().startswith("log_type") + ] + assert lines == [], f"log_type is set, which silences errors: {lines}" + + +def test_the_healthcheck_covers_the_websocket_listener(compose): + """An MQTT-only probe reports healthy on a broker with no dashboard.""" + test = " ".join(compose["services"]["broker"]["healthcheck"]["test"]) + assert "1883" in test, "the healthcheck no longer probes the MQTT listener" + assert "9001" in test, ( + "the healthcheck does not probe the WebSocket listener, so a broker " + "that never opened it would report healthy" + ) + + +def test_the_browser_is_told_the_published_websocket_port(compose): + """The server hands the browser a port on the host, not the container's.""" + command = compose["services"]["server"]["command"] + assert "--broker-ws-port" in command + value = command[command.index("--broker-ws-port") + 1] + assert "BROKER_WS_PORT" in value, f"--broker-ws-port is hardcoded to {value!r}" + + published = compose["services"]["broker"]["ports"] + assert any("BROKER_WS_PORT" in p and p.endswith(":9001") for p in published), ( + f"the WebSocket listener is not published: {published}" + )