What happened?
On our production master (3006.27, official onedir deb, Debian 11, ~87 minions, default worker_threads: 5), individual MWorker processes deadlock permanently while storing a minion job return, roughly 2 workers/day, until after ~2.5 days the last worker dies and the master stops serving requests entirely. Restarting salt-master clears it; the cycle then repeats. We first saw chronic symptoms in May 2026, proved the mechanism with py-spy in July, upgraded 3006.26 → 3006.27 hoping the event-system fixes there (#66282 / #65702 backports) would cover it — it recurred on 3006.27 within 2.5 days, with the same signature.
Symptom (minion side)
Requests round-robined to a wedged worker vanish: the master TCP-ACKs the request bytes and then never replies (verified with tcpdump). Depending on the code path minions see:
SaltReqTimeoutError + 60 s retry per lost request (salt-call wall time jumps 9 s → 70 s),File client timed out after 180 seconds,- or an infinite hang in
master_tops during state.apply — the fileclient's _channel_send (fileclient.py, master_tops → top_matches → call_highstate path) has no timeout, so a scheduled highstate never completes and never returns.
The master looks completely healthy the whole time: load ~0.25, workers epoll-idle, zero relevant log lines at any level, no OOM.
Unambiguous detection signature (master side)
# healthy: Send-Q 0 on all worker connections; wedged: sustained nonzero
$ ss -x -m | grep workers.ipc
u_str ESTAB 0 214016 /var/run/salt/master/workers.ipc 414840336 * 414840335 skmem:(...,tb212992,...)
A wedged MWorker never reads workers.ipc again, so the MWorkerQueue→worker kernel pipe fills to its buffer limit (~212992). Once the pipe hits the high-water mark, ZeroMQ excludes that worker from the round-robin and service appears to recover on the surviving workers — which is why each wedge shows up as a short mysterious outage window and the underlying worker deaths go unnoticed until the last one dies.
py-spy: where the wedged worker sits
MainThread of the wedged MWorker (py-spy dump, 3006.x):
_wait_for_tstate_lock (threading.py)
join (threading.py)
_wrap (salt/utils/asynchronous.py) # SyncWrapper: thread.join() — no timeout
fire_event (salt/utils/event.py:883) # pusher.send(msg)
_store_job (salt/utils/job.py:91) # store_job fires salt/job/<jid>/ret event
...
run (salt/master.py) # MWorker handling a minion return
Because _store_job fires the event before writing the return cache, the wedged worker also eats the job return itself (no return.p is written); the minion's 60 s retry then lands on a healthy worker, which is why the return appears in the job cache ~60–90 s into each wedge.
Root cause
fire_event → SyncWrapper (salt/utils/asynchronous.py) runs the IPC send on a helper thread / run_sync and blocks on thread.join() with no timeout; the underlying IPC client write to master_event_pull.ipc (salt/transport/ipc.py in 3006) also has no timeout (and connect retries forever). Any event-bus write that never resolves therefore turns into a permanent, silent MWorker death — there is no log line, no exception, no watchdog.
What makes the write never resolve is the remaining open question — our evidence points at large event payloads: the wedges correlate with our fleet's largest job returns (3-hourly scheduled highstates of two minions with ~700 states, ~284 KB return payload; 3–4 of 5 wedges in our atop forensics coincide with exactly those returns, ~12% wedge probability per large return). That matches the already-fixed minion-side member of this bug family, #66562 ("Publishing large event data ... via ipc socket hangs", partial write + EAGAIN, fixed by #67096) — but the master-side _store_job path is still exposed.
This is not fixed by the 3007/3008 transport rework
salt/transport/ipc.py is gone in 3007+, but the hazard pair survives — verified against the current branches:
salt/utils/event.py (3007.x): fire_event still calls self.pusher.publish(msg) through SyncWrapper(salt.transport.ipc_publish_server, ...); the timeout argument is only used for connect_pull, never for the publish itself.salt/utils/asynchronous.py (3007.x, line ~259; same on master and v3008.2): SyncWrapper._wrap still does thread.start(); thread.join() with no timeout (and the no-running-loop fast path, io_loop.run_sync, equally has no timeout).salt/transport/tcp.py (3007.x/master): _TCPPubServerPublisher.send() literally carries # FIXME timeout unimplemented and ends in await self.stream.write(pack) with no timeout.
So the same permanent-silent-death mode exists in 3006.x, 3007.x, and 3008.x.
Related issues (same family, none covering this path)
Suggested fix
A bounded timeout on the master event-bus publish path, i.e.:
SyncWrapper._wrap: thread.join(timeout) (and run_sync with a timeout) with a raise/log on expiry instead of blocking forever, and/or- an actual write timeout in the event pusher (
IPCClient.send in 3006.x; _TCPPubServerPublisher.send in 3007+ — resolving the existing FIXME timeout unimplemented),
so that a stuck event write becomes a logged, retried (or at worst dropped-event) error instead of a permanently dead worker that silently discards minion requests. Given 3006.x is the LTS line still receiving stability fixes (3006.26/3006.27 wave), we'd love to see this land there.
We're happy to provide more evidence: we run a watchdog on the master that, on the next confirmed wedge, captures py-spy dumps of all MWorker/MWorkerQueue/EventPublisher processes plus the master_event_pull.ipc socket state before restarting, and we can test candidate patches.
Workaround for anyone else hitting this
Watch ss -x | grep workers.ipc for sustained nonzero Send-Q (check twice a few minutes apart) and restart salt-master when it triggers. Raising worker_threads extends the runway between first wedge and full outage but doesn't prevent it.
Type of salt install
Official deb (onedir, packages.broadcom.com apt repo)
Major version
3006.x (reproduced on 3006.26 and 3006.27; code inspection shows 3007.x/3008.x affected)
OS
debian-11
salt --versions-report output
Salt Version:
Salt: 3006.27
Python Version:
Python: 3.11.15 (main, Jun 29 2026, 22:21:49) [GCC 11.2.0]
Dependency Versions:
cffi: 2.0.0
cherrypy: 18.10.0
cryptography: 47.0.0
dateutil: 2.9.0.post0
docker-py: Not Installed
gitdb: 4.0.12
gitpython: 3.1.50
Jinja2: 3.1.6
libgit2: Not Installed
looseversion: 1.3.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.1.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 24.0
pycparser: 3.00
pycrypto: Not Installed
pycryptodome: 3.23.0
pygit2: Not Installed
python-gnupg: 0.5.6
PyYAML: 6.0.3
PyZMQ: 27.1.0
relenv: 0.22.16
smmap: 5.0.2
timelib: 0.3.0
Tornado: 6.5.5
ZMQ: 4.3.5
System Versions:
dist: debian 11.11 bullseye
locale: utf-8
machine: x86_64
release: 5.10.0-45-amd64
system: Linux
version: Debian GNU/Linux 11.11 bullseye
What happened?
On our production master (3006.27, official onedir deb, Debian 11, ~87 minions, default
worker_threads: 5), individual MWorker processes deadlock permanently while storing a minion job return, roughly 2 workers/day, until after ~2.5 days the last worker dies and the master stops serving requests entirely. Restarting salt-master clears it; the cycle then repeats. We first saw chronic symptoms in May 2026, proved the mechanism with py-spy in July, upgraded 3006.26 → 3006.27 hoping the event-system fixes there (#66282 / #65702 backports) would cover it — it recurred on 3006.27 within 2.5 days, with the same signature.Symptom (minion side)
Requests round-robined to a wedged worker vanish: the master TCP-ACKs the request bytes and then never replies (verified with tcpdump). Depending on the code path minions see:
SaltReqTimeoutError+ 60 s retry per lost request (salt-callwall time jumps 9 s → 70 s),File client timed out after 180 seconds,master_topsduringstate.apply— the fileclient's_channel_send(fileclient.py,master_tops→top_matches→call_highstatepath) has no timeout, so a scheduled highstate never completes and never returns.The master looks completely healthy the whole time: load ~0.25, workers epoll-idle, zero relevant log lines at any level, no OOM.
Unambiguous detection signature (master side)
A wedged MWorker never reads
workers.ipcagain, so the MWorkerQueue→worker kernel pipe fills to its buffer limit (~212992). Once the pipe hits the high-water mark, ZeroMQ excludes that worker from the round-robin and service appears to recover on the surviving workers — which is why each wedge shows up as a short mysterious outage window and the underlying worker deaths go unnoticed until the last one dies.py-spy: where the wedged worker sits
MainThread of the wedged MWorker (py-spy dump, 3006.x):
Because
_store_jobfires the event before writing the return cache, the wedged worker also eats the job return itself (noreturn.pis written); the minion's 60 s retry then lands on a healthy worker, which is why the return appears in the job cache ~60–90 s into each wedge.Root cause
fire_event→SyncWrapper(salt/utils/asynchronous.py) runs the IPC send on a helper thread /run_syncand blocks onthread.join()with no timeout; the underlying IPC client write tomaster_event_pull.ipc(salt/transport/ipc.pyin 3006) also has no timeout (andconnectretries forever). Any event-bus write that never resolves therefore turns into a permanent, silent MWorker death — there is no log line, no exception, no watchdog.What makes the write never resolve is the remaining open question — our evidence points at large event payloads: the wedges correlate with our fleet's largest job returns (3-hourly scheduled highstates of two minions with ~700 states, ~284 KB return payload; 3–4 of 5 wedges in our atop forensics coincide with exactly those returns, ~12% wedge probability per large return). That matches the already-fixed minion-side member of this bug family, #66562 ("Publishing large event data ... via ipc socket hangs", partial write + EAGAIN, fixed by #67096) — but the master-side
_store_jobpath is still exposed.This is not fixed by the 3007/3008 transport rework
salt/transport/ipc.pyis gone in 3007+, but the hazard pair survives — verified against the current branches:salt/utils/event.py(3007.x):fire_eventstill callsself.pusher.publish(msg)throughSyncWrapper(salt.transport.ipc_publish_server, ...); thetimeoutargument is only used forconnect_pull, never for the publish itself.salt/utils/asynchronous.py(3007.x, line ~259; same on master and v3008.2):SyncWrapper._wrapstill doesthread.start(); thread.join()with no timeout (and the no-running-loop fast path,io_loop.run_sync, equally has no timeout).salt/transport/tcp.py(3007.x/master):_TCPPubServerPublisher.send()literally carries# FIXME timeout unimplementedand ends inawait self.stream.write(pack)with no timeout.So the same permanent-silent-death mode exists in 3006.x, 3007.x, and 3008.x.
Related issues (same family, none covering this path)
_store_jobequivalent.Suggested fix
A bounded timeout on the master event-bus publish path, i.e.:
SyncWrapper._wrap:thread.join(timeout)(andrun_syncwith a timeout) with a raise/log on expiry instead of blocking forever, and/orIPCClient.sendin 3006.x;_TCPPubServerPublisher.sendin 3007+ — resolving the existingFIXME timeout unimplemented),so that a stuck event write becomes a logged, retried (or at worst dropped-event) error instead of a permanently dead worker that silently discards minion requests. Given 3006.x is the LTS line still receiving stability fixes (3006.26/3006.27 wave), we'd love to see this land there.
We're happy to provide more evidence: we run a watchdog on the master that, on the next confirmed wedge, captures py-spy dumps of all MWorker/MWorkerQueue/EventPublisher processes plus the
master_event_pull.ipcsocket state before restarting, and we can test candidate patches.Workaround for anyone else hitting this
Watch
ss -x | grep workers.ipcfor sustained nonzero Send-Q (check twice a few minutes apart) and restart salt-master when it triggers. Raisingworker_threadsextends the runway between first wedge and full outage but doesn't prevent it.Type of salt install
Official deb (onedir, packages.broadcom.com apt repo)
Major version
3006.x (reproduced on 3006.26 and 3006.27; code inspection shows 3007.x/3008.x affected)
OS
debian-11
salt --versions-report output