[1.7.0-beta2][Bug] OTA update fails on ZimaCube Pro — dbus-broker drops rauc (and other core services) from the system bus during the boot mount-storm

[jmakumbi]

What happened?

OTA updates fail on my ZimaCube Pro, and the failure is a symptom of a larger problem: ~28 seconds into every boot, dbus-broker disconnects several core system services from the D-Bus at once. The processes keep running but lose their bus names and cannot re-acquire them until the next reboot.

rauc is always one of the casualties, which is why OTA install fails:

RAUC.Info() → "The name de.pengutronix.rauc was not provided by any .service files"

The same disconnect event also explains two other things I'd been seeing, which are all one root cause:

D-Bus name lost	Process	User-visible symptom
de.pengutronix.rauc	alive	OTA update fails (RAUC.Info() error above)
org.freedesktop.NetworkManager	alive	Can't change network settings — web UI / nmcli says "NetworkManager is not running" (it is)
org.freedesktop.login1 + org.freedesktop.systemd1	alive	Software reboot/shutdown fails (systemctl reboot → "Call to Reboot failed: Launch helper exited with unknown return code 1"); a physical power-cycle is required

systemctl still works (it uses systemd's private socket, not a bus name), which is why the box looks "up" while its D-Bus control plane is dead. rauc is lost on every boot; the exact set of other casualties varies with how hard the storm hits.

Environment

• Device: ZimaCube Pro (12 logical cores) — the core count matters (see below)
• Affected builds: 1.6.1, 1.6.2, 1.7.0-beta1, 1.7.0-beta2 (still unfixed; re-verified 2026-07-22)
• Not affected: my ZimaBoard / ZimaBoard2 / ZimaBlade (4-core) on the same builds
• Workload: 26 auto-starting Docker apps
• Component: dbus-broker (system bus) under the systemd mount-unit signal storm from Docker + ZimaOS's 3× media bind-mount layout

The exact knockout point (pinned with an active-polling probe)

I ran a probe that polls NameHasOwner on a fresh connection every 0.3 s (ground truth — the same view a new CLI client or rauc status gets), started before docker/rauc. It pins the failure to a single instant:

uptime 27.6s  overlay2 mounts = 0    rauc, NetworkManager, logind, systemd1 : all TRUE
uptime 27.9s  overlay2 mounts = 18   ← the 78 Docker overlay mounts begin landing in a rush
uptime 28.2s  overlay2 mounts = 78   all four STILL true
uptime 28.65s overlay2 mounts = 78   ALL FOUR = false  ← simultaneous mass-disconnect, never recovers

• All core names drop together, in a ≤0.3 s window, at system load ≈ 3.7 (low).
• It coincides exactly with the UnitNew/UnitRemoved burst peaking at ~830 signals/100 ms (~8,000/s) as the mounts are created.
• It is not CPU/load-driven — peak load (~19) happens ~75 s later and is irrelevant. I confirmed this directly (see "what doesn't work" below).

Root cause

1. dbus-broker is single-threaded and drains message queues serially.
2. ZimaOS bind-mounts the same storage at three paths, so every Docker overlay mount is tracked as three systemd mount units:
   • /media/SSD-Storage/docker/overlay2/<hash> (real)
   • /DATA/.media/SSD-Storage/docker/overlay2/<hash> (bind)
   • /var/lib/casaos_data/.media/SSD-Storage/docker/overlay2/<hash> (bind)
3. At boot, ~26 containers start in parallel and PID 1 broadcasts a burst of UnitNew/UnitRemoved signals (measured peak ~830/100 ms). The single-threaded broker can't drain the burst before its per-peer queues overflow, so it sheds all root-owned peers at once (it does not crash — NRestarts=0, same instance the whole boot).
4. The names are then unrecoverable while dbus-broker lives — only a reboot clears it. rauc is worse: it is registered both Type=dbus and D-Bus-activatable (SystemdService=rauc.service), so a later call to the unowned name triggers an activation that never completes, permanently wedging the name.

Why only the ZimaCube Pro: app count is similar to my 4-core boards. The difference is core count — 12 cores create the mounts far more in parallel, so the signal burst arrives faster than the single-threaded broker can drain. The 4-core boards create mounts more serially, stay under the drain rate, and never lose their services.

What generates the storm: across the 26 apps the mount churn splits evenly across the three bind-trees (measured 86/86/86) — a clean confirmation of the 3× multiplication. No single app is at fault; it's the aggregate of all apps' overlay mounts, each replicated ×3 by the stock layout.

Steps to reproduce

1. On a ZimaCube Pro (≥ ~8 logical cores), install ~15–24 apps so they auto-start at boot.
2. Reboot.
3. ~25–90 s in, watch the core bus names:

   for n in de.pengutronix.rauc org.freedesktop.NetworkManager org.freedesktop.login1; do
     printf "%s " "$n"; dbus-send --system --dest=org.freedesktop.DBus --print-reply \
       /org/freedesktop/DBus org.freedesktop.DBus.NameHasOwner string:$n | awk '/boolean/{print $2}'
   done
   

   They transition to false and never return.
4. Confirm the symptoms: OTA install fails (RAUC.Info()); nmcli general status → "NetworkManager is not running"; systemctl reboot → "Launch helper exited with unknown return code 1".

See the trigger: dbus-monitor --system "type='signal',interface='org.freedesktop.systemd1.Manager',member='UnitRemoved'" — thousands of docker-*.mount events in the boot window.

Things that do NOT fix it (tested on the affected unit)

• CPU-priority protection of rauc.service (Nice=-15, CPUWeight=10000, OOMScoreAdjust=-900) — still dropped.
• CPU-priority protection of dbus-broker.service itself (CPUWeight=10000, Nice=-15) — no effect, mass-disconnect still fired at t≈27.8 s / load 2.2. This proves it's burst-volume/queue-overflow, not CPU scheduling.
• Delaying rauc start until the storm settles — worse: while rauc is nameless, zimaos-mark-good calls it over D-Bus and triggers the activation wedge, so rauc's later RequestName fails outright. Please don't delay a Type=dbus+activatable service.

Suggested fixes (first two attack the cause)

1. Cut the 3× mount-unit multiplication — mount the /DATA/.media and /var/lib/casaos_data/.media bind trees with slave/private propagation so Docker overlay mounts don't replicate into them (~3× fewer mount signals).
2. Serialize/throttle container auto-start so mount creation doesn't arrive as one wide parallel burst.
3. Raise dbus-broker's per-user/per-connection message-queue limits so a transient boot burst can't force it to shed peers.
4. Drop the SystemdService= activation coupling for rauc (it is already Type=dbus); the dual mechanism is what makes its name unrecoverable after a drop.
5. Make core daemons re-RequestName after a bus disconnect where possible.

Notes for triage

• dbus-broker does not crash (NRestarts=0), so crash-log hunting finds nothing. The tell is NameHasOwner=false for core services while their processes are alive.
• The journal is volatile (tmpfs) — capture across the boot with a persistent logger or the evidence is lost on reboot.
• This is the intersection of three separately-documented upstream issues: dbus-broker kicking core services off the bus (bus1/dbus-broker#361, #192), Docker+systemd mount-unit flooding (docker/for-linux#679), and heavy-container ZimaOS setups (#474).

I have per-boot captures (0.3 s ownership timelines, signal-rate CSVs, /proc curves) for both the baseline and the CPU-shield control boot, and can attach them on request.

Screenshots

No response