Operations: the state of an environment, and the reports that reach nobody

[Devski]

Goal

A written, repeatable way to know what a deployed environment is doing — disk, database,
storage, mail, errors, and the collector's findings — that does not depend on a person
deciding to log in and look. What needs a decision comes to us; when nothing does, the
silence means "checked, nothing to do" rather than "nobody looked".

What to call it

Three different things get called one name, and only one of them is missing here:

• Monitoring — the checks we decided to run, on a schedule. Some exist (the collector).
• Observability — being able to answer a question we did not plan for, after the fact.
  Partly there: the logs hold a lot, for as long as the container lives.
• Alerting, and the process around it — the signal reaching a person, and that person
  knowing what it means and what to do. This is what does not exist at all.

So the work is mostly the third, and this issue is named for the practice, not the tooling.

Why now

Found on 12.09.2026. The R360 collector (#127, #156) had been reporting two frame sets on
dev that no record names — 240 objects, ~29.6 MB — and the report had reached a person
exactly once: when I went in over SSH with docker logs because I was asked to. It is a
console.warn in the app container (src/lib/r360/collector-timer.ts:78). The whole #156
design rests on the sentence written above that line — "said out loud, left where it is —
a human decides" — and there is no path from the saying to the human.

The collector is only the instance of it that we happened to notice:

• Disk. One image per commit, nothing prunes (Dev instance keeps every image it ever pulled: the deploy must drop the old ones #119). Nothing in deploy/compose.yaml
  caps container logs either, so Docker's default keeps every line for as long as the
  container lives. Both end on the same €7 instance's disk, and the first symptom will be
  a deploy that fails or a Postgres that cannot write.
• Whether it is up. Nothing checks. Dev, the previews and — after Production environment: instance, managed Postgres, prod bucket, CDN, G10 drill #24 — production are
  watched by whoever happens to open the site.
• Errors. Unhandled exceptions go to stdout and are read never. A real one from
  12.09.2026: Uncaught TypeError: ... cannot have a negative time stamp, seen only
  because I happened to be reading that log for another reason.
• Backups. Nothing copies dev's database anywhere at all — there is no pg_dump in the
  repository, and G10's written restore procedure has nothing to restore from. Split out
  as No copy of dev's database exists anywhere: losing one Docker volume orphans the whole bucket #168 on 12.09.2026, because that is a standing risk rather than a missing process.
  What stays here is only the other half: being told when a copy stops being made.
• Mail. Scaleway TEM records bounces and complaints (Scaleway TEM + SPF/DKIM/DMARC + switch email.ts to TEM #22, Deliverability test: Gmail / Onet / WP / Interia #23). A domain's reputation
  degrades quietly, and the first sign we would get is people saying they never received
  the verification message.
• Storage. What the bucket actually holds, per account and in total, is unknown until
  Storage meter per account: count the largest representation of every asset, show the usage #69. The collector's report is the only number anyone has ever read out of it.

It is the same shape as #34: the design anticipates the situation, and the code path that
would carry it to a human was never built.

Acceptance criteria

• One document in docs/ says what is measured, how often, where it appears, who it
  reaches, and what each signal means someone should do
• What needs a decision arrives by itself: the collector's unrecorded sets, disk over a
  threshold, a database copy that did not happen, a deploy that did not come up
• Silence is informative — either a report arrives on its schedule even when everything
  is fine, or its absence is itself visible; "no message" must never be ambiguous
• An outside check confirms dev and production answer, from somewhere that is not the
  instance (a box cannot report its own death)
• Application logs survive a restart and a deploy, and are capped so that keeping them
  cannot fill the disk we are watching
• Bounces and complaints from the mail provider reach the same place as everything else
• No control surface is exposed on the instance and no port opens (G9); nothing new
  carries personal data outside the EU/EEA (§7)
• A new provider or a recurring cost is a decision taken first, not a side effect (§7)

Open decisions

• Which channel. Transactional mail already works (Scaleway TEM + SPF/DKIM/DMARC + switch email.ts to TEM #22) and is the cheapest channel we
  have; the risk is that a routine digest to the same address trains us to ignore it. A
  plausible split: a phone-level channel for the few things that mean act now, mail for
  the periodic record.
• Does the instance speak, or do we ship its logs out? Shipping means a new provider, a
  recurring cost, and personal data leaving our control if the logs carry any. The instance
  speaking for itself is smaller and sits inside G9, but it cannot report its own death —
  which is why the outside check above is separate from this decision.
• What is allowed to wake someone. At most three tiers — act now, look this week, the
  record. More than that and the channel dies of noise, which is the failure mode this
  issue is about, one level up.
• Before or after Production environment: instance, managed Postgres, prod bucket, CDN, G10 drill #24. Building it for dev only means building it twice; folding it
  into Production environment: instance, managed Postgres, prod bucket, CDN, G10 drill #24 delays production.

Verification

• Fill dev's disk past the threshold on purpose (a large temporary file): the warning
  arrives, names the number, and stops once the file is gone
• Stop the dev container: the outside check notices within the window we agreed
• Leave an unfinished R360 set on dev: it shows up in the next report instead of only
  in docker logs
• Send to an address that hard-bounces: it lands in the same place as the rest
• Follow the document as someone who was not here: answering "is everything fine right
  now" takes one place to look, not an SSH session

Dependencies

None blocking. Wants #24 to exist before it can be called done for production; #119 is one
of the numbers this would have caught before it mattered. #168 was split out of this
one and is the more urgent half. Related: #156 (the report that found no reader), #111 and
#113 (previews leaving things behind), #69 (storage figures), #34 (same shape — a design
without the code path to a person).

Spec: SPEC.md §7 (G9, G10), §8 · Size: M · Labels: deployment, enhancement

[Devski]

Built and deployed on 12.09.2026 in #184, with a fix from the first drill in #189. What was verified, and the one step left.

Verified on dev

• The deploy installed platform-ops.timer (hourly), capped the journal at 300 MB / two weeks, and moved the app's and the proxy's logs to the journal (docker inspect → journald). docker compose logs still works.
• rsyslog keeps no second copy: after it restarted, a minute of traffic added no line to /var/log/syslog.
• The collector's startup run landed in the journal, and the installed service's next report carried it — "the R360 collector found 2 frame set(s) that no record names" under THIS WEEK. That is verification item 3: the finding reaches the report instead of only docker logs.
• Disk past the threshold did not need faking: the first real run reported ACT NOW: disk at 97%, 886M free. (41 unused application images older than a day were removed after that run — 97% → 56% — see Dev instance keeps every image it ever pulled: the deploy must drop the old ones #119, which still needs its permanent fix.)
• The outside check, drilled without breaking dev: Watch by hand against https://down.invalid/ opened a locked outage-drill issue (https://down.invalid/ is not answering #190); run again against dev, it closed it with "Answering again at 2026-09-12T20:52Z (200)". The first drill (https://down.invalid/ is not answering #188) found that a locked issue refused that comment — fixed in The outage issue says when the site came back (#167) #189.
• Every run so far ended with OPS_EMAIL is not set: … reached this journal and nobody else, and recorded no alert state — so the first real recipient will receive the standing disk alert rather than a day late.

Not verified, on purpose

• Stopping dev's container to watch the outside check notice — the drill exercises the same path without an outage.
• A hard bounce — sending to a bouncing address costs the sending domain reputation.

Left before this closes

• OPS_EMAIL=<address> in /opt/platform-lite/.env on the instance — the recipient is the owner's to choose
• the first daily report has arrived (05:00 UTC)

Follow-ups filed from the reviews: #186 (failed queries are logged with their parameters).

[Devski]

Both remaining steps done on 12.09.2026: OPS_EMAIL=ops@architektow3d.pl is in the instance's .env, and the first daily report — [dev] daily: 1 to look at this week — was sent at 21:40 UTC; the provider reports it sent / success, i.e. accepted by the receiving mail server. Whether it landed in the inbox or in spam is visible only in the mailbox itself.