Skip to content

Operations: the disk advice follows the rollback promise (#167, #119) #191

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Devski merged 1 commit into from
Sep 12, 2026
+17 −17
Merged

Operations: the disk advice follows the rollback promise (#167, #119) #191

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 17 additions & 17 deletions docs/operations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,23 +49,23 @@ reads — which is the failure this document exists to prevent, one level up.

## What each signal means, and what to do

| Signal | Tier | What to do |
| ------------------------------------------------------ | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Disk at 90% or more | ACT NOW | Full, PostgreSQL stops writing — dev and every preview go down together. The usual cause is application images: one per commit, nothing prunes (#119). `docker image ls ghcr.io/devski/platform-lite` and remove the oldest ones no container uses; keep the last few for rollback. |
| Disk at 80–89% | this week | The same, with time to plan it. |
| A container is missing, stopped or unhealthy | ACT NOW | `docker ps -a`; for the app, `journalctl CONTAINER_NAME=platform-lite-app-1 -n 100`. Rolling back is redeploying an older tag (SPEC §8). |
| The site does not answer from the instance itself | ACT NOW | The proxy or the app — the container lines in the same report say which. |
| The site does not answer from outside (`outage` issue) | ACT NOW | If the report from the instance still arrives, it is the network, DNS or the certificate; if it has also stopped, the instance is down — the OVH console. |
| The newest copy of the databases is over 36 h old | ACT NOW | The nightly copy has stopped (#168). `systemctl status platform-backup.service`, `journalctl -u platform-backup.service -n 50`. A refusal to shrink is explained in `deploy/backup-db.sh`. |
| No copy has ever succeeded | ACT NOW | A new instance where the timer was never installed, or it never once worked: `docs/backup-and-restore.md`. |
| Preview databases with no container | this week | Copies of dev's accounts, left by a preview removed some other way than closing its pull request. The next preview sweeps them; by hand: `docs/dev-environment.md`. |
| The R360 collector found frame sets no record names | this week | By design it deletes nothing it cannot account for, and waits for a person (#156). The list is in the log: `journalctl CONTAINER_NAME=platform-lite-app-1 \| grep collector`. Decide per set; nothing is lost by waiting a week. |
| The R360 collector failed | this week | The same log. A sweep that stops reporting is worse than one that fails loudly — this is the loud version. |
| Database deadlines hit | this week | #172. Which bound and which path are in the log (`grep '\[db\]'`); one a day is noise, a pattern is a query or a lock to look at. |
| Unhandled server errors | this week | The log for the last day — read it on the instance, and see "Logs" below before copying a line anywhere. |
| A quarter or more of mail failed (4+ sent) | ACT NOW | The sending domain's reputation or the provider — people are not receiving verification messages. Scaleway console → Transactional Email; SPF/DKIM/DMARC per G8. |
| Some mail failed | this week | A bounce or a rejection — usually one bad address. |
| The provider blocks more addresses than yesterday | this week | A hard bounce or a complaint. The addresses are in the Scaleway console, deliberately not in the report. |
| Signal | Tier | What to do |
| ------------------------------------------------------ | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Disk at 90% or more | ACT NOW | Full, PostgreSQL stops writing — dev and every preview go down together. Since #119 every deployment clears application images beyond the last three that came up healthy, so first look at what else grew: `sudo docker system df`, `sudo du -sh /var/lib/containerd /var/lib/docker /var/log`. **Never remove an image listed in `/opt/platform-lite/deployed-images`**: those are the rollback targets, and the instance holds no registry credential to pull them back (`docs/deployment.md`). |
| Disk at 80–89% | this week | The same, with time to plan it. |
| A container is missing, stopped or unhealthy | ACT NOW | `docker ps -a`; for the app, `journalctl CONTAINER_NAME=platform-lite-app-1 -n 100`. Rolling back is redeploying an older tag (SPEC §8). |
| The site does not answer from the instance itself | ACT NOW | The proxy or the app — the container lines in the same report say which. |
| The site does not answer from outside (`outage` issue) | ACT NOW | If the report from the instance still arrives, it is the network, DNS or the certificate; if it has also stopped, the instance is down — the OVH console. |
| The newest copy of the databases is over 36 h old | ACT NOW | The nightly copy has stopped (#168). `systemctl status platform-backup.service`, `journalctl -u platform-backup.service -n 50`. A refusal to shrink is explained in `deploy/backup-db.sh`. |
| No copy has ever succeeded | ACT NOW | A new instance where the timer was never installed, or it never once worked: `docs/backup-and-restore.md`. |
| Preview databases with no container | this week | Copies of dev's accounts, left by a preview removed some other way than closing its pull request. The next preview sweeps them; by hand: `docs/dev-environment.md`. |
| The R360 collector found frame sets no record names | this week | By design it deletes nothing it cannot account for, and waits for a person (#156). The list is in the log: `journalctl CONTAINER_NAME=platform-lite-app-1 \| grep collector`. Decide per set; nothing is lost by waiting a week. |
| The R360 collector failed | this week | The same log. A sweep that stops reporting is worse than one that fails loudly — this is the loud version. |
| Database deadlines hit | this week | #172. Which bound and which path are in the log (`grep '\[db\]'`); one a day is noise, a pattern is a query or a lock to look at. |
| Unhandled server errors | this week | The log for the last day — read it on the instance, and see "Logs" below before copying a line anywhere. |
| A quarter or more of mail failed (4+ sent) | ACT NOW | The sending domain's reputation or the provider — people are not receiving verification messages. Scaleway console → Transactional Email; SPF/DKIM/DMARC per G8. |
| Some mail failed | this week | A bounce or a rejection — usually one bad address. |
| The provider blocks more addresses than yesterday | this week | A hard bounce or a complaint. The addresses are in the Scaleway console, deliberately not in the report. |

## What the report never carries

Expand Down
Loading