Summary
During a September 9, 2026 incident in a Coder dogfood workspace, disk-full errors preceded detection of an incomplete Mux config and fallback to an empty registry. The UI appeared to have lost all projects/workspaces; the user restored the config.
Disk exhaustion is the leading trigger, not a proven explanation of the exact write failure. It is not yet established whether exhaustion occurred at the workspace volume, quota, or underlying shared host layer.
Evidence reported from the workspace investigation
- At 16:39 and 16:43 UTC, the live Mux server log reported
ENOSPC: no space left on device during metadata saves, status persistence, workspace-recency updates, and workflow lock-directory creation.
- At 16:43:41 UTC, Mux rejected the config and preserved it as
~/.mux/config.json.corrupt-1788972221254.
- That backup was exactly 684,032 bytes; JSON parsing failed at end of file. Separate extension metadata also became invalid around the same time.
- The running Mux Node process had been alive since 08:04:37 UTC. The observed disappearance therefore did not require a server-process restart; a UI reload/reconnect may have looked like one.
- After recovery, the config parsed successfully and contained 31 projects. This does not establish that every historical session was recovered.
- Relevant server log:
~/.coder-modules/coder/mux/logs/mux.log. In this instance, ~/.xum resolves to ~/.mux.
These findings were reported in the incident investigation; this issue does not include raw logs or config contents because they may contain sensitive data.
Secondary risk: orphan cleanup after empty fallback
The code audit identified that cleanupOrphanSessionDirs removes session directories absent from the config when idle for more than 24 hours. If config loading fails and Mux substitutes an empty registry, otherwise valid sessions could consequently be treated as orphans on startup.
No actual orphan-session deletion was found in the retained server logs for this incident. This is a potential amplification path, not a confirmed loss mechanism here.
Expected behavior / investigation
- Reproduce config persistence under
ENOSPC and other write/flush/rename failures; identify how incomplete content reached the live config despite the apparent atomic-replacement write path.
- Preserve the last known-good config on failed saves, and surface a clear actionable storage/persistence error rather than silently presenting an empty installation.
- Fail closed for destructive orphan cleanup when config loading failed or the registry is an untrusted fallback. A failed load must not establish that sessions are orphaned.
- Add regression tests covering failed persistence, invalid/incomplete config recovery, and cleanup suppression after config-load failure.
Unknowns
The exact writer/failing operation, affected build/commit, and exhausted storage layer still need confirmation. Another user reported a similar empty-config incident, but a shared cause has not been established. No PR is identified as the cause.
Summary
During a September 9, 2026 incident in a Coder dogfood workspace, disk-full errors preceded detection of an incomplete Mux config and fallback to an empty registry. The UI appeared to have lost all projects/workspaces; the user restored the config.
Disk exhaustion is the leading trigger, not a proven explanation of the exact write failure. It is not yet established whether exhaustion occurred at the workspace volume, quota, or underlying shared host layer.
Evidence reported from the workspace investigation
ENOSPC: no space left on deviceduring metadata saves, status persistence, workspace-recency updates, and workflow lock-directory creation.~/.mux/config.json.corrupt-1788972221254.~/.coder-modules/coder/mux/logs/mux.log. In this instance,~/.xumresolves to~/.mux.These findings were reported in the incident investigation; this issue does not include raw logs or config contents because they may contain sensitive data.
Secondary risk: orphan cleanup after empty fallback
The code audit identified that
cleanupOrphanSessionDirsremoves session directories absent from the config when idle for more than 24 hours. If config loading fails and Mux substitutes an empty registry, otherwise valid sessions could consequently be treated as orphans on startup.No actual orphan-session deletion was found in the retained server logs for this incident. This is a potential amplification path, not a confirmed loss mechanism here.
Expected behavior / investigation
ENOSPCand other write/flush/rename failures; identify how incomplete content reached the live config despite the apparent atomic-replacement write path.Unknowns
The exact writer/failing operation, affected build/commit, and exhausted storage layer still need confirmation. Another user reported a similar empty-config incident, but a shared cause has not been established. No PR is identified as the cause.