[earthlingdavey]

Part of #177.

A storage that accepts requests and never answers held a backup for 100 minutes at the opening and 106 mid-upload [M92]. restic's stuck-request watchdog ends each try at five minutes, minio-go makes ten tries, and restic's own retry layer always goes round once more. The upload, the fetch and the drill's check now run with -o s3.retries=1 (oneTry in backups/engine/engine.go). That leaves restic's own quarter of an hour of retries, each one printed to the journal, as all there is.

The listings keep minio-go's ten tries. With one try, a single 500 on a listing is retried by restic rather than minio-go, which puts a line on stderr, and hotserve does not believe a listing with anything beside it (besides) [M92]. Every other backend (local, rest:, sftp:) ignores the option.

Measured

Through the real units, with the product binaries (before: 58c4aab; after: this branch) in an e2e box. The fault proxy runs in the box's network namespace [M93]:

Fault	Before	After
No answer from the upload's first request	6003 s	904 s, blog failed and named, the others not attempted
No answer from 20–60 s into a 300 MB upload	6343 s	1245, 2109, 2146 s
Host stops reading (paused) mid-upload	—	1245 s
No answer from the fetch's first request	—	904 s; the drill then exits 1 at 2736 s
No answer from 31 / 61 s into a fetch	—	2108 s exit 1, 3040 s exit 130 (restic's lock monitor); drill exit 1 at 3999 / 4901 s
No answer from the check's first request	—	904 s; drill exit 1 at 939 s, unreachable
10-min blackhole mid-upload, then normal	—	exit 0, both apps ok
60 s of 500 on every request mid-upload	—	exit 0, both apps ok

M94, bare restic: the check blackholed mid-read gives up five packs every quarter of an hour, and has no lock monitor to end it: 2711 s for 14 packs, exit 1.

Triage

Scored by .claude/skills/triage/SKILL.md. The bar, for reviewers who cannot read the skill: every failure is scored severity × occurrence × detection, each on 1–5.

• Severity 5 is always fixed.
• A failure under one fault must end correct, or red and named.
• Anything else is fixed only if its product is 20 or more, or its fix scores 1 for both size and risk.
• Otherwise it is accepted, and recorded in backups/README.md's "Accepted residuals".

Please flag only findings that clear that bar, and score them.

Finding	S	O	D	Product	Fix	Outcome
A storage that answers a byte at a time is never given up on	4	3	1	12	size 3, risk 3	accept: existing row narrowed, stays with #177
The drill's check, blackholed mid-read: a quarter of an hour for each five packs left in its group (about 60 h for a 1 TB repository)	4	3	1	12	size 3, risk 3	accept: new row, stays with #177
A drill's fetch, blackholed, is followed by the next app's 30-minute listing clock: 46–82 min in all	2	3	1	6	size 2, risk 2	accept: new row
With no region set, a region error is no longer retried inside minio-go (restic's retry recovers, except AccessDenied)	2	3	1	6	size 2, risk 2	accept: new row (read from the source, not measured)
KillSignal=SIGQUIT	—	—	—	—	—	not a failure: a unit's stop already ends restic after systemd's 90 s, by SIGKILL

Note for the merger: this change is wide. It changes how every upload, fetch and check talks to off-site storage. It is not one-way: reverting this commit restores minio-go's ten tries.

Tests

• TestOnlyTheUnclockedUnitsMakeOneTry: the upload, fetch and check each carry -o s3.retries=1 exactly once, and no other restic unit carries it. It fails when oneTry is emptied, and when a listing is given it.
• make test, make vet and make lint pass on the rebased branch.
• make e2e-backup passes, all suites.

🤖 Generated with Claude Code

[Copilot]

🔵 Needs a closer look

The change affects every off-site upload, fetch, and integrity check, requiring maintainer confirmation of provider-specific fault measurements and acceptance of documented long-running-check and region-error residuals.

No open issues 🎉

What changed in this PR

This PR advances #177 by reducing S3 request retries for backup uploads, restores, and repository checks, shortening stalls when storage stops responding. Listings retain their existing retries to avoid stderr output that would invalidate their results.

Changes:

• Adds shared -o s3.retries=1 arguments to upload, fetch, and check commands.
• Adds regression coverage for which restic units receive the option.
• Documents measured failure timings and accepted remaining limitations.

File	Description
docs/​design/​backups/​plan.md	Records fault measurements and the revised retry decision.
docs/​backups.md	Updates operator guidance on storage stalls and remaining limits.
backups/​README.md	Documents retry behavior and accepted residual failures.
backups/​engine/​restore.go	Applies one-attempt S3 requests to fetches and updates explanatory comments.
backups/​engine/​engine.go	Defines shared retry arguments and applies them to uploads.
backups/​engine/​engine_test.go	Verifies only upload, fetch, and check units receive the option.
backups/​engine/​check.go	Applies the retry option to repository checks.
backups/​engine/​check_test.go	Updates the expected repository-check command.

_{Review effort: Balanced}