Skip to content

restore: Don't hang on a filesystem that never preallocates - #2

Open
msaggiorato wants to merge 1 commit into
masterfrom
upstream/preallocation-guard
Open

msaggiorato wants to merge 1 commit into
masterfrom
upstream/preallocation-guard

Conversation

@msaggiorato

@msaggiorato msaggiorato commented Sep 16, 2026

Copy link
Copy Markdown
Member

Upstream submission candidate, off master, one commit.

The defect

The restorer reserves space for each file before filling it (upstream restic#2195, so that out-of-order blob writes don't fragment large files). On some sandboxed filesystems that syscall is accepted and never answered, and the damage goes well past a slow restore. Reproduced outside the original host — branch repro/preallocation-hang has a FUSE filesystem that never answers fallocate, plus the probes below:

  • the restore stops at the first file needing data, leaves it at 0 bytes, prints nothing, and never finishes — the same stack as the production goroutine dump (createFileensureSizePreallocateFileunix.Fallocate);
  • the kernel holds the inode lock, so pwrite, ftruncate and even unlink on that file block forever, and a blocked unlink wedges its directory;
  • FALLOC_FL_KEEP_SIZE behaves identically, so no flag avoids it;
  • the waiting thread sits in state=D wchan=request_wait_answer and ignores SIGKILL, so the process can never exit.

Why a deadline is not the fix

An earlier version of this branch put a 5 s deadline around the call. The reproduction shows that cannot work: by the time it expires, that file is unwritable and unremovable and the process is unkillable. Issuing the call is an irreversible commitment.

The change

Ask the filesystem first, then use it: preallocate a throwaway file whose name is unlinked before the call, from a process that exists only to ask, so a filesystem that never answers strands that process rather than the restore. The answer is kept per filesystem. Whether the call succeeded is not the question, only whether it came back — one that fails does so promptly, and both callers already tolerate failure.

Evidence it works

upstream with this change
restore on the hanging filesystem never finishes Restored 8 files/dirs (73.009 MiB) in 0:05, exit 0
preallocations the filesystem sees one per restored file one, for the throwaway probe file
checksums of restored data files left empty all match
process exit unkillable (state=D) clean
healthy ext4 (512 MiB file) 3 extents, fully allocated 3 extents, fully allocated, no warning

Open before this goes upstream

CI for this branch runs in #4, which adds the minio fix from #3, because master's own test workflow cannot get past Get programs.

🤖 Generated with Claude Code

https://claude.ai/code/session_017d5y4KNmm2N5ySPPH1H49Z

@msaggiorato
msaggiorato force-pushed the upstream/preallocation-guard branch from 9c66b0b to 2d9eef8 Compare September 17, 2026 01:26
@msaggiorato msaggiorato changed the title fileio: Don't hang the restore when preallocation never returns restore: Don't hang on a filesystem that never preallocates Sep 17, 2026
@msaggiorato
msaggiorato force-pushed the upstream/preallocation-guard branch from 2d9eef8 to 0b9ea9c Compare September 17, 2026 01:30
The restorer reserves space for each file before filling it, so that writing its
blobs out of order does not fragment it. Some sandboxed filesystems, seen on
managed WordPress hosting, accept that syscall and never answer it, and the
consequences run deeper than a slow restore: the kernel holds the inode lock
while it waits, so the file can no longer be written, shortened or unlinked —
unlinking it blocks in turn, and takes its directory with it — and the waiting
thread cannot be interrupted, not even by SIGKILL, so the process it belongs to
can never exit. A restore stopped at the first file that needed data, left it
empty, and printed nothing.

Making the call is therefore an irreversible commitment, and no deadline around
it can take one back: by the time the deadline passes, that file and that
process are already lost. Ask the filesystem beforehand instead, with a
throwaway file whose name is unlinked before the call, from a process that
exists only to ask — so a filesystem that never answers strands that process
instead of the restore. The answer is kept per filesystem. Whether the call
succeeded is not the question, only whether it came back at all: one that fails
does so promptly, and the caller tolerates a failure anyway.

Both callers already treat a failed preallocation as harmless and let the writes
extend the file, so where the question goes unanswered, restoring simply carries
on without reserving space.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017d5y4KNmm2N5ySPPH1H49Z
@msaggiorato
msaggiorato force-pushed the upstream/preallocation-guard branch from 0b9ea9c to 58b941c Compare September 17, 2026 01:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant