What needs a decision reaches a person by itself (#167)

[Devski]

Refs #167 — the issue stays open until OPS_EMAIL is set on the instance and
the first daily report has arrived (the recipient is the owner's to choose).

The R360 collector had been reporting two frame sets that no record names for
two days, into a container's stdout, and the report reached a person exactly
once — because someone went in over SSH to look. The same silence covered the
disk, whether the site answers, the nightly copy, server errors and the mail
provider. The design kept leaving decisions to a human, and no code path ever
carried one to the human.

Two watchers, because a box cannot report its own death

	Where	How often	What it sees
deploy/ops-check.sh	the instance, platform-ops.timer	hourly	disk, containers, the site from inside, the nightly copy's age, preview copies nobody owns, the collector's findings, database deadlines, unhandled errors, mail failures and blocks
.github/workflows/watch.yml	GitHub	every 30 min	whether the site answers from outside

Three tiers and no more. ACT NOW is mailed when a condition starts, again a
day later if it is still true, and once when it stops — keyed on the condition,
not the numbers inside it, so a disk going from 91% to 92% is one alert, not
one an hour. THIS WEEK and RECORD go in the report that arrives every
morning either way: an empty inbox means "checked", and a missing report is
itself the signal.

The outside check speaks through an outage issue — opened when the site
stops answering (three tries a minute apart; a 4xx is an answer), closed with
the time it came back. No new provider, no cost while the repository is
public, the owner's phone through GitHub's own notifications, and the label's
state is the answer.

Logs move to the host journal, capped at 300 MB and thirty days. A log in
the container's own directory died with the container, which is every deploy —
the hour before a bad deploy was always gone by the time anyone looked.
Preview containers keep a file log, capped at three 10 MB files.

What it never sends

No address, handle, name or line of any log — counts and the instance's own
figures only. The mailbox may be outside the EU (§7). The mail statistics and
the provider's blocklist are read as total_count / failed_count for exactly
that reason (shapes read from the real API on 12.09.2026, not assumed).

Decisions taken

• The instance speaks for itself; logs are not shipped anywhere. Shipping
  means a new provider, a recurring cost and log lines carrying personal data
  out of the EU — all §7 "ask first", and the need is covered without them.
• Mail through the provider the application already uses, with its key.
  Nothing new to pay for or to sign up to.
• The outside check is a GitHub workflow. This is a CI configuration change
  (SPEC §7 "ask first"), and it is the one Operations: the state of an environment, and the reports that reach nobody #167's acceptance criteria require:
  "from somewhere that is not the instance". If the repository goes private
  (Take the repository private again — what it costs and what must be in place first #163) it costs minutes: 48 runs a day, well under a minute each.

Verified

The check was run on dev by hand before this pull request, with no recipient
set so nothing could be sent. It found two real things at once:

ACT NOW
  - disk at 92%, 2.0G free — when it fills, PostgreSQL stops writing and previews stop starting (#119)

THIS WEEK
  - 1 preview database(s) with no container — copies of dev's accounts; ...

RECORD
  - 76 application images kept on the instance (#119)
  - platform-lite-app-1: running/healthy
  - https://dev.architektow3d.pl/ answers 200
  - databases copied 16 h ago: 2026-09-12T03:18:21Z backups/devski/pg/sat.sql.gz 1813858 bytes
  - 0 messages in 24 h, none failed
  - the provider blocks 0 address(es)

To verify after merge: the deploy installs the timer and the journal cap, dev's
app and proxy log to the journal (docker inspect shows journald), the
collector's startup run lands in the journal and the next check reports it,
and the Watch workflow run by hand against https://down.invalid/ opens an
outage issue that the next run against dev closes.

Not verified, on purpose: stopping dev to watch the outside check notice (the
hand-run against an address that cannot answer tests the same path without an
outage), and a hard bounce (sending to a bouncing address costs the sending
domain reputation).

🤖 Generated with Claude Code

---

What the two review lanes changed

A refused mail was recorded as sent. send was only ever called as an if
condition, and bash switches set -e off inside a function called that way —
the trap remote-deploy.sh already documents, written again here. A provider
error ran on to "mailed", returned success, and the state said delivered: no
retry for a day, the daily report marked done. Every step in send now says
itself what happens on failure (0 sent, 1 failed, 2 nowhere to send), and a
missing OPS_EMAIL is no longer recorded as delivered — so the first real
recipient still receives the alert that was standing before they existed.

Each of the rest ended in silence exactly when the check was needed:

• Docker down failed the first pipeline and stopped the script before any
  report; Requires=docker.service would have stopped it starting. Now "Docker
  is not answering" is an ACT NOW line, every call that can hang has a time
  limit, the unit Wants Docker, and its timeout is the sum of the call limits.
• A full disk stopped the mail at the header file. The key reaches curl
  through a file descriptor: no disk, no argv, exported to nothing else.
• An unreadable journal read as "no unhandled server errors". A failed read
  says so, and zero lines from a container up longer than the collector's
  twelve-hour interval says the log is not being read.
• A provider error reset the blocklist baseline to 0, so the next good read
  reported a rise that never happened. Only a count actually read becomes the
  baseline, and the domain is matched to EMAIL_FROM by name.
• The subject was always the first check's line: a backup stopping while
  the disk alert stood would have looked like the disk alert re-sent. Subjects
  name what is new; each condition resolves on its own.
• One slow answer on one core would have mailed ACT NOW and "resolved" an
  hour apart. The site, the containers and Docker need two looks in a row.
• The daily report is retried the next hour if its hour's run failed or was
  late; the body is one function, so a deploy rewriting the file mid-run cannot
  make bash execute half of each version; numbers from the API and from state
  files are validated before they reach shell arithmetic.

From the security lane:

• Retention made real. journald deletes whole files, and at dev's volume a
  file covers a month, so "30 days" meant up to sixty. Now one file a day, two
  weeks, and SystemKeepFree=500M — the default 15% is already broken by a disk
  at 92%, and journald would otherwise keep almost nothing.
• No second copy in /var/log/syslog. Ubuntu forwards the journal to
  rsyslog (confirmed on the instance: active, ForwardToSyslog=yes, weekly
  rotation for a month, no size cap). The containers now log under a tag and an
  rsyslog rule drops those lines, and only those — auth.log is untouched.
• The unit is sandboxed: NoNewPrivileges, ProtectSystem=strict,
  ProtectHome=read-only, PrivateDevices, RestrictSUIDSGID, MemoryMax=128M.
• The watcher: https addresses only, passed with --proto '=https' --url;
  the token lives in the one step that needs it; permissions is issues: write alone; the issue it opens is locked, since the thread notifies the
  owner; and a hand-run drill uses its own outage-drill label, so it can never
  open or close dev's real alarm.
• Failed queries are logged with their parameters — session tokens and
  addresses among them. Two weeks bounds that, it does not remove it: filed as
  Failed queries are logged with their parameters — session tokens and addresses in the journal #186, with a "never paste a log line" rule in docs/operations.md until then.

Verified on the instance under the unit's exact sandbox (systemd-run with
the same properties, then removed): Docker answers under NoNewPrivileges and
a read-only system, the journal is readable through the supplementary group,
the mail domain is found by name, the state directory is written, and no alert
state is recorded while there is no recipient.