Server: a system unit for --service, so an OOM kill can't strand the phone

[pallaoro]

The bug

A box's iOS access went down for ~21 hours while the desktop kept working. Not an app bug — the daemon was dead, and only the phone had no way to say so.

From the box's journal:

Aug 14 09:15:53  ateam.service: killed by OOM, restart counter at 2
Aug 14 09:16:18  killed by OOM again (25s later)
Aug 14 09:16:18  user@1001.service: Main process exited, code=killed, status=9/KILL
   ... 21 hours of nothing ...
Aug 15 06:30:44  Started ateam.service   ← a desktop SSH login revived the user manager

The unit install.sh --service writes has three defects that only bite under memory pressure:

1. The daemon is scored as the preferred OOM victim. user@.service ships OOMScoreAdjust=100, inherited by every user service. The kernel killed a 36 MB daemon (oom_score_adj:200) and spared the 442 MB agents it supervises (score 0).
2. systemd gives up permanently. Default StartLimitBurst=5 per 10s; a burst of OOM kills exhausts it and nothing restarts the daemon again.
3. It dies with the systemd user manager, which was itself OOM-killed here. Linger=yes does not resurrect a killed manager — only a new login does.

The desktop survives all three by starting a daemon over SSH on connect, which is exactly what masks the problem. The phone's WebSocket only exists while a daemon is already running, so the one client --service exists to serve is the one that can never recover.

The fix

--service now writes a system unit wherever passwordless sudo is available — every box Ateam provisions, since provision.ts bakes NOPASSWD:ALL into cloud-init:

OOMScoreAdjust=-500        # what distros already do for udevd/dbus/journald
StartLimitIntervalSec=0    # never permanently give up restarting
Restart=on-failure         # unchanged: exit 0 means another daemon owns the socket
KillMode=process           # unchanged: a restart must not kill live agent PTYs

Without sudo it still writes the user unit, plus StartLimitIntervalSec=0 and a warning naming what stays unprotected. It deliberately does not write OOMScoreAdjust there — measured: systemd silently ignores a negative value in a user unit (the unit starts, /proc still reads 100), so that line would read as protection while doing nothing.

Also fixes a stale-socket false positive in the handover check: it tested for the socket file, which a crashed daemon leaves behind, so a stale socket made the installer decline to start the service. Now tests for a live process.

Verified

• System path, on a real box: ran the actual installer block — detected sudo, inherited ATEAM_WS_ADDR from the existing unit, wrote the unit, service active with /proc/<pid>/oom_score_adj = -500.
• Self-heal: kill -9 the daemon twice → back in ~3s, listening, WebSocket handshake answering {protocolVersion:2,agents:["claude"]}.
• OOM ranking inverted the right way: daemon oom_score=337 vs biggest agent 692.
• Rootless fallback: sandboxed run with failing sudo → correct user unit, StartLimitIntervalSec=0 present, zero OOMScoreAdjust lines, warning printed.
• Migration: user→system upgrade carries ATEAM_WS_ADDR across and leaves running agents untouched (they live outside the unit's cgroup; the PTY daemon is never signalled).
• 95 server tests pass.

Not in scope

This keeps the daemon alive; it does not stop a box from running out of memory. Agents are unbounded (200–450 MB each, weeks-long) and, separately, the PTY daemon and every agent live in whichever SSH login-session scope first started them rather than an Ateam-owned cgroup — so agent memory can't be bounded as a group, and reaping that scope would kill every agent at once. The fix is spawning the PTY daemon into an Ateam-owned slice, which is a design change with real UX stakes (throttling a live agent mid-task) and belongs in its own PR.