diff --git a/docs/operations.md b/docs/operations.md index 0d82691..007bb83 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -49,23 +49,23 @@ reads — which is the failure this document exists to prevent, one level up. ## What each signal means, and what to do -| Signal | Tier | What to do | -| ------------------------------------------------------ | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Disk at 90% or more | ACT NOW | Full, PostgreSQL stops writing — dev and every preview go down together. The usual cause is application images: one per commit, nothing prunes (#119). `docker image ls ghcr.io/devski/platform-lite` and remove the oldest ones no container uses; keep the last few for rollback. | -| Disk at 80–89% | this week | The same, with time to plan it. | -| A container is missing, stopped or unhealthy | ACT NOW | `docker ps -a`; for the app, `journalctl CONTAINER_NAME=platform-lite-app-1 -n 100`. Rolling back is redeploying an older tag (SPEC §8). | -| The site does not answer from the instance itself | ACT NOW | The proxy or the app — the container lines in the same report say which. | -| The site does not answer from outside (`outage` issue) | ACT NOW | If the report from the instance still arrives, it is the network, DNS or the certificate; if it has also stopped, the instance is down — the OVH console. | -| The newest copy of the databases is over 36 h old | ACT NOW | The nightly copy has stopped (#168). `systemctl status platform-backup.service`, `journalctl -u platform-backup.service -n 50`. A refusal to shrink is explained in `deploy/backup-db.sh`. | -| No copy has ever succeeded | ACT NOW | A new instance where the timer was never installed, or it never once worked: `docs/backup-and-restore.md`. | -| Preview databases with no container | this week | Copies of dev's accounts, left by a preview removed some other way than closing its pull request. The next preview sweeps them; by hand: `docs/dev-environment.md`. | -| The R360 collector found frame sets no record names | this week | By design it deletes nothing it cannot account for, and waits for a person (#156). The list is in the log: `journalctl CONTAINER_NAME=platform-lite-app-1 \| grep collector`. Decide per set; nothing is lost by waiting a week. | -| The R360 collector failed | this week | The same log. A sweep that stops reporting is worse than one that fails loudly — this is the loud version. | -| Database deadlines hit | this week | #172. Which bound and which path are in the log (`grep '\[db\]'`); one a day is noise, a pattern is a query or a lock to look at. | -| Unhandled server errors | this week | The log for the last day — read it on the instance, and see "Logs" below before copying a line anywhere. | -| A quarter or more of mail failed (4+ sent) | ACT NOW | The sending domain's reputation or the provider — people are not receiving verification messages. Scaleway console → Transactional Email; SPF/DKIM/DMARC per G8. | -| Some mail failed | this week | A bounce or a rejection — usually one bad address. | -| The provider blocks more addresses than yesterday | this week | A hard bounce or a complaint. The addresses are in the Scaleway console, deliberately not in the report. | +| Signal | Tier | What to do | +| ------------------------------------------------------ | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Disk at 90% or more | ACT NOW | Full, PostgreSQL stops writing — dev and every preview go down together. Since #119 every deployment clears application images beyond the last three that came up healthy, so first look at what else grew: `sudo docker system df`, `sudo du -sh /var/lib/containerd /var/lib/docker /var/log`. **Never remove an image listed in `/opt/platform-lite/deployed-images`**: those are the rollback targets, and the instance holds no registry credential to pull them back (`docs/deployment.md`). | +| Disk at 80–89% | this week | The same, with time to plan it. | +| A container is missing, stopped or unhealthy | ACT NOW | `docker ps -a`; for the app, `journalctl CONTAINER_NAME=platform-lite-app-1 -n 100`. Rolling back is redeploying an older tag (SPEC §8). | +| The site does not answer from the instance itself | ACT NOW | The proxy or the app — the container lines in the same report say which. | +| The site does not answer from outside (`outage` issue) | ACT NOW | If the report from the instance still arrives, it is the network, DNS or the certificate; if it has also stopped, the instance is down — the OVH console. | +| The newest copy of the databases is over 36 h old | ACT NOW | The nightly copy has stopped (#168). `systemctl status platform-backup.service`, `journalctl -u platform-backup.service -n 50`. A refusal to shrink is explained in `deploy/backup-db.sh`. | +| No copy has ever succeeded | ACT NOW | A new instance where the timer was never installed, or it never once worked: `docs/backup-and-restore.md`. | +| Preview databases with no container | this week | Copies of dev's accounts, left by a preview removed some other way than closing its pull request. The next preview sweeps them; by hand: `docs/dev-environment.md`. | +| The R360 collector found frame sets no record names | this week | By design it deletes nothing it cannot account for, and waits for a person (#156). The list is in the log: `journalctl CONTAINER_NAME=platform-lite-app-1 \| grep collector`. Decide per set; nothing is lost by waiting a week. | +| The R360 collector failed | this week | The same log. A sweep that stops reporting is worse than one that fails loudly — this is the loud version. | +| Database deadlines hit | this week | #172. Which bound and which path are in the log (`grep '\[db\]'`); one a day is noise, a pattern is a query or a lock to look at. | +| Unhandled server errors | this week | The log for the last day — read it on the instance, and see "Logs" below before copying a line anywhere. | +| A quarter or more of mail failed (4+ sent) | ACT NOW | The sending domain's reputation or the provider — people are not receiving verification messages. Scaleway console → Transactional Email; SPF/DKIM/DMARC per G8. | +| Some mail failed | this week | A bounce or a rejection — usually one bad address. | +| The provider blocks more addresses than yesterday | this week | A hard bounce or a complaint. The addresses are in the Scaleway console, deliberately not in the report. | ## What the report never carries