The deploy clears the images it hoarded, and keeps the ones a rollback needs (#119)

[Devski]

Summary

Closes #119. On 09.09.2026 the dev
instance's 24 GB disk stood at 100% with 129 images — one per commit, nothing ever removing
one. Previews stopped starting, and dev read unhealthy because Docker could not write the
file its own health check needs. It was cleared by hand; nothing stopped it happening again.

Now a deployment clears what nothing needs:

• After the new container is healthy — never before, so a deployment that fails still has
  everything it could go back to.
• Before anything else, when under 5 GB is free — otherwise an instance that is already
  full cannot fetch the deployment that would fix it.

docker image prune would have freed nothing here: every image is tagged with its commit sha,
so none is ever dangling. It is removal by name.

What stays

Every image a container uses, running or stopped — so a preview on an older pull request keeps
its image — and the last three images that came up healthy on dev, listed in
/opt/platform-lite/deployed-images on the instance.

That rule is the third one. Review took the first two apart, and each time the casualty was
the same thing: the image docs/deployment.md tells an operator to roll back to, which it says
"needs no registry access" — true only while the image is on the instance, because the
instance holds no registry credential of its own (G9).

1. "The three newest images." Previews pull into the same repository as dev. After a pull
request with two pushes is merged, the three newest are the new dev image and the two preview
images, and the version dev was running a minute earlier comes fourth and goes. The 7-second
rollback drilled on 05.09.2026 would have quietly stopped working.

2. "The last three deployments", seeded from .env, recorded beside the .env write. Three
ways to lose it, each reproduced before it was fixed:

• After a hand rollback. The doc's rollback changes the container and leaves .env naming the
  version rolled back from. The next deployment trusted the file, so the image dev was
  actually running was the one it removed: deploy A, B, C; roll back to A; deploy D → on disk
  B C D. Now the running version is read from the container.
• After deployments that never came up. Recording before the health check gave broken images
  places on the list: A, B, C; D unhealthy, roll back to C; E unhealthy, roll back to C; F → on
  disk D E F, the last good version gone and both fallbacks known to be broken. Now a version is
  recorded once it is healthy, and the running one only if its container says healthy or
  it is on the list already — a version put back by hand stays even while a health check catches up.
• On a full disk. Recording the list and docker login both write a file, and came before the
  free-space check: at 100% the first write ended the run before any room was made. The check and
  its clearing now run first. And the list is written to a temporary file and moved only when
  the write succeeded — otherwise an empty file would have replaced every rollback target.

Also from review

• A failing df ended the deployment. Under set -euo pipefail the assignment carried
  df's status and the fallback was never reached. Unreadable now reads as tight, and clearing
  is safe. The old line exits 1 where the path does not exist; the new one carries on.
• And it measured the wrong place. With containerd's image store — which this instance uses —
  the images are under /var/lib/containerd, not /var/lib/docker. On one root filesystem the
  two read the same; on separate disks the rescue would have watched the wrong one. Both paths are
  read, whichever exist, and the tighter counts.
• An image under both registry names could never be removed. The owner was renamed once
  (3dbdg → devski); docker image rm <id> refuses an id tagged twice without --force.
  Removal is by repo:tag, so the dead name is untagged and the image stays with the kept one.
• Housekeeping never fails a deployment. Every call is fn || echo WARNING, so bash runs it
  with errexit off, and every read inside returns on failure rather than carrying on with an
  empty list — which would otherwise have been reported as "nothing to remove".

Deliberately not here

• Clearing when a preview comes down (preview-down.sh) — left out by Dawid's decision:
  preview images share dev's repository, so every deployment of dev sweeps them. A preview closed
  without a merge keeps its image until the next one.
• The registry's own growth — ~8 GB and over a gigabyte a day on GitHub's side. That stays
  with #163.
• A narrow race with a preview being started. preview-up.sh removes its old container,
  pulls its image, then copies the database before its first docker run; in that gap no
  container refers to the image. A dev deployment turning healthy inside it removes the image,
  and if that deployment's docker logout (older than this change, and shared by both scripts)
  has run by then, the preview's docker run cannot pull it back: the job goes red with the old
  preview already gone, and a re-run fixes it. Closing it would take a lock around both spans in
  preview-up.sh, which is the half of Dev instance keeps every image it ever pulled: the deploy must drop the old ones #119 left out.

Test plan

The Docker daemon was not available locally, so the logic was tested against stand-in docker
commands on PATH, with the functions cut verbatim from the script and run under
set -euo pipefail. A stateful simulator replays deployments in the script's order
(record the running version → up → if healthy, record and clear):

• Deploy A, B, C; hand rollback to A; deploy D → A still on the instance, rollback to A works
  (the same sequence on the previous version: A removed)
• A, B, C; D unhealthy → back to C; E unhealthy → back to C; F → C still there
  (previous version: C removed, only broken D and E left)
• Dev on dprev, two preview images pulled, m deployed → dprev kept, previews cleared
• Five healthy deployments → the two oldest cleared, rollback to either of the two before works
• D left unhealthy, nobody rolls back, E fixes it → C kept, D cleared
• A redeploy of the same sha counts once
• Rolled back by hand to the OLDEST on the list, dev still reporting unhealthy at the next
  deploy (a health check catching up after the disk was freed) → that version kept
  (before the last fix: removed); a broken deploy nobody rolled back is still not recorded
• Writing the list fails (full disk) → WARNING, the list untouched, the script carries on
• Free space: neither path, one, the other, both — the lines run verbatim under pipefail; the
  smaller of two disks counts, in either order; nothing readable clears first
• No dev container / an unhealthy one / a healthy one / docker compose itself failing
• One id under both registry names → only the dead name untagged
• A container vanishing between ps and inspect → WARNING, nothing removed
• bash -n deploy/remote-deploy.sh, pnpm check
• On dev after merge: the deploy log shows images: N removed, and
  cat /opt/platform-lite/deployed-images lists the running tag

🤖 Generated with Claude Code