fix(slots): bound the restart teardown, converge config on load, kill drift false positives (#1224, #1226)

[thinmintdev]

Closes #1224. Closes #1226.

#1224a — restart wedges on errored slots

SlotManager.terminate() documented timeout_s as "preserved in the signature for caller compatibility" and awaited run_in_executor(provider.unload_sync) with no bound at all. Underneath, ContainerProvider.unload_sync ran systemctl stop <unit> with no child timeout.

Against a unit systemd had already parked in failed, that stop never returns — so restart() blocked forever, the REST caller ReadTimeout"d, and load() was never reached (hence zero journald activity for the unit). The pre-existing contextlib.suppress(Exception) only catches exceptions; it does nothing against a hang.

#1224b — slot load no-ops, config never converges

load() short-circuits on READY/SERVING/IDLE and returns status() without touching the unit. The unit bakes its launch config at render time, so PUT /config {"port": 8091} → slot load left the container on 8089.

#1226 — false drift, two compounding bugs

1. compute_config_drift resolved the model from the raw TOML id, while load() first runs it through _resolve_servable_model — so for a catalog id that landed locally under a different id, the registry lookup missed and the bare id was compared against the running containers real --model path
2. The id→path substitution compared id spellings with ==, which never fires when the TOML holds the catalog spelling (Qwopus3.5-4B-Coder-MTP-Q6_K) and the registry the slug (qwopus3-5-4b-coder-mtp-q6-k)

Fixes

• terminate() enforces timeout_s via asyncio.wait_for (caller-side bound); unload_sync bounds its systemd verbs (_UNIT_STOP_TIMEOUT_S = 20.0, worker-side) and on an abandoned stop still removes the Quadlet source + daemon-reloads, dropping the failed generated .service
• SystemCtlSeam.systemctl gained a timeout passthrough so the bound also applies on the sudo-seam route — the deployment where the bug was seen. Default stays None everywhere else
• restart() takes the bounded force path for ERROR or any slot whose unit is not active; teardown is best-effort-with-logging so restart always converges
• load()s short-circuit is gated on _unit_needs_convergence(), which re-renders through rerender_unit_sync — the same renderer load_sync uses, whose byte-identical guarantee is already pinned by tests/providers/test_container.py. Compares the whole effective launch config rather than a hand-maintained key list. Conservative: NPU trio shadows, adopted containers with no unit on disk, and render failures all keep the fast no-op
• Drift resolves the servable model before _resolve_model_info and matches ids on _normalise_id

Did not touch name-keying, the state graph, or decompose manager.py/slots.py.

Verification

• tests/slots tests/api tests/providers tests/system -m "not podman and not systemd and not network" → 2470 passed, 2 deselected
• Adjacent sweep (updater/capabilities/stacks/slot_view/dispatcher/install/systemd/golden_paths/cli) → 1370 passed, 1 skipped
• Every new test verified failing pre-fix against stashed source, e.g. test_terminate_enforces_its_timeout → assert (36648.76 - 36633.74) < 5.0 — 15s elapsed against a 0.25s bound

One for a reviewers eye: restart() now calls _is_active() before choosing its path and coerces probe errors to False, routing to the bounded force path. Safe and convergent, but a probe outage makes every restart take the force path rather than the graceful drain.

🤖 Generated with Claude Code

[thinmintdev]

⚠️ Converted to draft — collision detected.

610150d7 ("fix(tests): hermes uninstall tests hit real system bus; RC 1.0 carryover", #1357) landed on main and already closes #1302, #1301, #1300, #1224 via an independent implementation.

This PR is being reconciled against it. It will either be closed as superseded or reduced to the genuinely unique delta (e.g. the security/exposure.py ADMIN route pinning and #1024, which 610150d7 does not appear to cover). Not merging duplicate implementations.

[thinmintdev]

Reduced after reconciliation with 610150d7

Rebased onto main, cut 679 → 290 lines.

Dropped — mains slot-manager work is the better of the two:

• terminate. Main does asyncio.wait_for(asyncio.shield(stop), timeout=_terminate_timeout_s) raising a typed SlotTerminateTimeout (504). The shield() is correct — plain wait_for cancels the future, which does nothing to the thread but surfaces a spurious CancelledError. This branch did neither. Main wins.
• Convergence check. This branchs _unit_needs_convergence compared the whole rendered unit (image, mounts, env, argv) vs mains 6-key argv comparator — genuinely broader, but it wrote the unit file as a side effect of a "check", and mains fixes the filed symptom (--port). Dropped; recorded as a follow-up rather than re-litigated.

Where main is wrong

610150d7 says #1226 "needed no code … verified by the existing tests." That is not correct — both new tests fail against mains drift.py:

1. _resolve_drift_flags compares ids with ==. The slot TOML keeps the catalog spelling (Qwopus3.5-4B-Coder-MTP-Q6_K) while the registry key — and the running containers --alias — is the slug (qwopus3-5-4b-coder-mtp-q6-k), so the path substitution never fires and status prints a permanent bogus --model drift.
2. compute_config_drift looks up the raw TOML id while load() launches from _resolve_servable_model(...). For exactly the slots this matters for, the registry lookup misses, the renderer emits the bare id, and the warning is permanent.

Mains existing tests only cover the case where TOML id == registry key exactly.

Mains bound also stops at the caller. terminate cannot kill the executor thread, so on the seamed route (SystemCtlSeam → sudo -n hal0-systemctl, i.e. a real hal0-service-user install — the deployment the wedge was reported from) the thread stays on the child indefinitely. The thread leak is the lesser problem: unload_sync never reaches the Quadlet-source removal + daemon-reload, so the generated .service stays on disk in failed and mains own claim that "the subsequent load still converges" does not hold. Verified: mains SystemCtlSeam.systemctl takes no timeout and container.py::_run passes none.

Surviving delta:

• drift.py — _resolve_servable_model on the DriftHost protocol, _normalise_id-based --model compare, servable-model resolution before render (a raising heuristic falls back to the raw id, so status can never fail because of it). This also makes mains new _should_converge more accurate, since it reads the same comparator.
• seam.py — timeout forwarded on both routes (default None = unbounded, unchanged).
• container.py — _UNIT_STOP_TIMEOUT_S = 20.0, deliberately under mains 30s caller budget so the worker unwinds first; catch TimeoutExpired, log container.unit_stop_timeout, continue the teardown.

Non-vacuity proven for all three: #1226 tests fail on mains drift.py; seam tests fail with TypeError: systemctl() got an unexpected keyword argument "timeout"; container test fails with ImportError: _UNIT_STOP_TIMEOUT_S.

Verification: ruff check ✅ · ruff format (984 files) ✅ · import smoke ✅ · sunset ✅ · tests/slots tests/system tests/providers tests/api tests/dispatcher → 2648 passed, 2 deselected.