Dev instance keeps every image it ever pulled: the deploy must drop the old ones

[Devski]

On 09.09.2026 the dev instance filled its 24 GB root filesystem to 100% and stayed there. PR previews stopped starting at all (no space left on device while unpacking the image, #118), and dev's own health check could not run — Docker could not write the temporary file needed to exec it, so the container read unhealthy for thirty checks in a row while the application itself was serving normally.

What it was

129 images, 21.82 GB, three of them in use.

• 76 under ghcr.io/devski/platform-lite — one per commit, 578 MB each, pulled on every deploy since Dockerfile (standalone) + deploy on push to the dev instance, G9 #21 started deploying on 05.09.2026.
• 50 under ghcr.io/3dbdg/platform-lite — the registry namespace from before the repository moved, dead ever since.
• postgres:17, caddy:2-alpine, curlimages/curl — the three that matter.

Cleared by hand on 09.09.2026 down to six images: 24 GB used → 4.5 GB, 19 GB free, and the app went back to healthy by itself.

Why nothing caught it

Nothing in deploy/ or .github/workflows/ removes an image — no docker image prune, no rmi, no image rm. Images only ever arrive.

And the obvious reflex would not have helped: every image carries a tag (its commit sha), so there are no dangling images at all. docker image prune — which only removes untagged ones — frees exactly zero bytes here. It has to be removal by name, or prune -a with the running set protected.

What to build

A step at the end of a successful deploy that keeps the N most recent images of ghcr.io/devski/platform-lite (the running one plus a couple to roll back to) and removes the rest. Same for a preview coming down. Points to settle:

• Where it belongs: deploy/remote-deploy.sh after the container is up and healthy, and deploy/preview-down.sh. It must never run before the new container is proven, or a failed deploy would have nothing to go back to.
• N: three is what the by-hand cleanup kept.
• Whether to guard the disk as well as the count — refuse to deploy under, say, 3 GB free and say so, the way preview-up.sh refuses past its preview cap rather than letting the kernel choose.
• The dead 3dbdg namespace is gone from the instance; nothing needs to keep it out, but a prune by name should not assume a single namespace either.

Related: #111 (preview cleanup losing the race), #113 (a database per preview).

[Devski]

Relieved by hand on 12.09.2026 at ~20:36 UTC, minutes before #187 landed. The new hourly check (#167) reported disk at 97%, 886 MB free — 81 application images on the instance, with CI from several open pull requests adding more every hour.

Removed: docker image prune --all --force --filter "until=24h" — only images not used by any container and built more than a day ago (41 of them). Result: 97% → 56%, 11 GB free, all containers healthy throughout (dev, the proxy, PostgreSQL, a preview).

Correction to what I first wrote here: an image removed from the instance is not "a docker pull away" — the instance holds no registry credential (G9). Going back to one of those means re-running that commit's deploy-dev job from Actions, as docs/deployment.md says. In practice nothing was lost to a rollback: the images removed were all older than the last day, and #187's own cleanup, deployed at 20:45, then settled the instance on its rule (the last three healthy deployments in /opt/platform-lite/deployed-images, plus whatever a container uses) — which would have removed the same images.