From 6c8d9caae5c3b1e7225c456a19c9dd56d98e83df Mon Sep 17 00:00:00 2001 From: refmerge Date: Thu, 29 Oct 2026 18:29:11 +0000 Subject: [PATCH] =?UTF-8?q?docs(backups):=20#177=20closed=20=E2=80=94=20wh?= =?UTF-8?q?at=20is=20left=20of=20it=20is=20accepted,=20not=20owed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #368 bounded a storage that never answers. What is left, a storage that answers a byte at a time and the drill's check blackholed mid-read, scored 12 under the triage rule, so it is accepted and #177 is closed with its scores. The rows and the guide that called it #177's question still now say so; the watchdog and the dead man's switch are features. Co-Authored-By: Claude Opus 5.5 --- backups/README.md | 4 ++-- docs/backups.md | 6 ++++-- docs/design/backups/README.md | 3 ++- docs/design/backups/plan.md | 2 +- 4 files changed, 9 insertions(+), 6 deletions(-) diff --git a/backups/README.md b/backups/README.md index 8e1ce73c..6e6382c1 100644 --- a/backups/README.md +++ b/backups/README.md @@ -1444,8 +1444,8 @@ severity × occurrence × detection, then the fix's size and risk. | `setup`'s look (`restic cat config`) exiting 1 within its clock for a reason other than the storage refusing — a config file restic cannot read, a TLS trust failure — is said as every exit 1 is, "restic could not open the repository (exit 1): " then "the storage refused the key, or could not be reached: the key id and secret again", and setup ends after three askings with restic's words; before #175 the look was given up on and init said the same words after a password had been shown. | `look` (`backups/engine/setup.go`), `lookRefused` | 2 | 3 | 1 | 6 | size 1, risk 2 (reading the look's exit 1 by restic's wording, as init's is, would send a storage that refuses at once in another shape — S3's `AccessDenied` comes through the config's Stat, not the opening — back to the password-first path) | Under rule 3 the product is under 20 and the fix is not size 1, risk 1 (2026-10-23, #175). Rule 2 holds: red, with restic's own words on the line, and nothing written. | | A storage outage that begins during a backup leaves the command's lock file in the repository: the backup ends exit 1 after 20–35 minutes, its lock still there [M87], which restic never passes by itself [M75]. The box's own commands go on (a backup, a fetch and setup's write check take shared locks, the drill's check none; the next backup measured ok); an off-box `prune` waits out its `--retry-lock 1h` and exits 11, since restic refuses an exclusive lock over any lock and never asks whether it is stale (restic 0.18.0, `internal/restic/lock.go:188-189`; M75, measured for a backup). | restic, not this code; docs/backups.md, "Retention, off the box" | 2 | 3 | 1 | 6 | size 2, risk 2 (a `restic unlock` unit after a run the storage failed would delete in the repository on every such run, and would remove a lock another machine's restic still held) | Rule 2 holds: red and named where it bites — `prune` says the repository is locked (exit 11) — and the guide gives `restic list locks --no-lock` and the unlock. Under rule 3 the product is under 20 (2026-10-23, #354). | | An off-box `prune` that is stopped (killed, its machine asleep or cut off) or whose deletes all fail leaves an exclusive lock, which restic does not take for stale with its process gone [M90]: every run and drill of the box's then waits its `--retry-lock 2h` for it, holding the run lock, nothing said meanwhile, and ends "the repository stayed locked by something else for 2h (exit 11)"; backups stop until `restic unlock` from your own machine. | restic, not this code; `resticFailure` (`backups/engine/engine.go`); docs/backups.md, "Retention, off the box" | 4 | 4 | 1 | 16 | size 3, risk 3 (the box removing another machine's lock would end a prune that is still running, whose lock is all that keeps a backup from writing beside its deletions) | Rule 2 holds: red and named — every run fails naming the lock, `status` is stale from the third hour — and the guide gives the remedy and says to read `prune`'s exit status, not its words (one whose every delete failed printed "2 snapshots have been removed", M90). Under rule 3 the product is under 20 (2026-10-23, #354). Whether `restic unlock` without `--remove-all` removes such a lock is not measured; the guide gives `--remove-all` with the box's timers stopped. | -| A storage that answers a byte at a time is never given up on by restic — each byte read resets its stuck-request watchdog, at any timeout [M92] — so an upload, a fetch or the check runs on, holding the run lock, and no later run starts. One that takes connections and never answers, or is paused, is no longer this row: those three units make one try at a request (`oneTry`) and end after 15 minutes at the opening, 21–36 mid-upload and 35–51 mid-fetch, exit 1, or 130 where restic's lock monitor gives up first [M93] (the check mid-read, the next row), where ten tries took 100–106 minutes with nothing on stderr until the end [M92]. | the upload, fetch and check units (`backups/engine/engine.go`, `restore.go`, `check.go`); restic's stuck-request watchdog, kicked by every byte | 4 | 3 | 1 | 12 | size 3, risk 3 (a no-progress watchdog that ends a unit, the option #177 weighs: a new state per unit, and a cut that must never end a slow upload that is moving — restic's `bytes_done` stands still while the last packs are sent, and the bytes a unit writes grow in a stall, as restic sends a pack again every five minutes [M92]) | Rule 2 holds: `status` exits 1 from the third hour — "no run has finished since …", the app stale, and `running: upload of , since …` naming the stuck unit. Under rule 3 the product is under 20 (2026-10-23, #354; narrowed 2026-10-29, #177). The switch to the S3 endpoint removed the hang #177 found (M87: every fault that answers ends), and one try a request the one that never answers [M93]; this one is left with #177 (on hold), where the watchdog and a dead man's switch are weighed. | -| A storage that stops answering while the drill's check reads its data group holds the check a quarter of an hour for each five packs of the group not yet read — 45 minutes for 14 packs [M94] — since the check takes no lock, and so has no lock monitor to end it as a fetch's does [M93]: a group of 1,200 packs, a 1 TB repository's fifty-second in restic's 16 MiB packs, some 60 hours, the run lock held and every hourly run refused through it. It ends exit 1, and the probe after it says `unreachable` (`checkRepository`). | `checkArgv` (`backups/engine/check.go`: `--no-lock`, no clock, D4); restic 0.18's five connections and its quarter of an hour of retries a pack | 4 | 3 | 1 | 12 | size 3, risk 3 (a clock on the check from its group's size, which the drill does not know before the check, and which must never cut a slow link that is reading; or the lock, which the owner refused for the check, since one killed hard leaves an exclusive lock restic never passes [M75]; or #177's no-progress watchdog) | Rule 2 holds: `status` exits 1 from the third hour, "no run has finished since …", and its `running:` line names the check and since when; the check ends by itself, `unreachable` and named, and the next drill reads the group again. One try a request made it ten times shorter: ten took 100 minutes for each five packs [M92]. Under rule 3 the product is under 20 (2026-10-29, #177); left with #177, with the watchdog. | +| A storage that answers a byte at a time is never given up on by restic — each byte read resets its stuck-request watchdog, at any timeout [M92] — so an upload, a fetch or the check runs on, holding the run lock, and no later run starts. One that takes connections and never answers, or is paused, is no longer this row: those three units make one try at a request (`oneTry`) and end after 15 minutes at the opening, 21–36 mid-upload and 35–51 mid-fetch, exit 1, or 130 where restic's lock monitor gives up first [M93] (the check mid-read, the next row), where ten tries took 100–106 minutes with nothing on stderr until the end [M92]. | the upload, fetch and check units (`backups/engine/engine.go`, `restore.go`, `check.go`); restic's stuck-request watchdog, kicked by every byte | 4 | 3 | 1 | 12 | size 3, risk 3 (a no-progress watchdog that ends a unit, the option #177 weighed: a new state per unit, and a cut that must never end a slow upload that is moving — restic's `bytes_done` stands still while the last packs are sent, and the bytes a unit writes grow in a stall, as restic sends a pack again every five minutes [M92]) | Rule 2 holds: `status` exits 1 from the third hour — "no run has finished since …", the app stale, and `running: upload of , since …` naming the stuck unit. Under rule 3 the product is under 20 (2026-10-23, #354; narrowed 2026-10-29, #177). The switch to the S3 endpoint removed the hang #177 found (M87: every fault that answers ends), and one try a request the one that never answers [M93]; this one is accepted, and #177 closed with these scores (2026-10-29): the watchdog and a dead man's switch it lists are features, not fixes owed. | +| A storage that stops answering while the drill's check reads its data group holds the check a quarter of an hour for each five packs of the group not yet read — 45 minutes for 14 packs [M94] — since the check takes no lock, and so has no lock monitor to end it as a fetch's does [M93]: a group of 1,200 packs, a 1 TB repository's fifty-second in restic's 16 MiB packs, some 60 hours, the run lock held and every hourly run refused through it. It ends exit 1, and the probe after it says `unreachable` (`checkRepository`). | `checkArgv` (`backups/engine/check.go`: `--no-lock`, no clock, D4); restic 0.18's five connections and its quarter of an hour of retries a pack | 4 | 3 | 1 | 12 | size 3, risk 3 (a clock on the check from its group's size, which the drill does not know before the check, and which must never cut a slow link that is reading; or the lock, which the owner refused for the check, since one killed hard leaves an exclusive lock restic never passes [M75]; or #177's no-progress watchdog) | Rule 2 holds: `status` exits 1 from the third hour, "no run has finished since …", and its `running:` line names the check and since when; the check ends by itself, `unreachable` and named, and the next drill reads the group again. One try a request made it ten times shorter: ten took 100 minutes for each five packs [M92]. Under rule 3 the product is under 20 (2026-10-29; #177 closed with these scores). | | A drill whose fetch the storage stops answering gives that fetch up, `restore not proven` and that app's own — 15 minutes at the fetch's opening, exit 1; 35 and 51 minutes from mid-download, exit 1 and exit 130 — then asks the next app's snapshots, which waits out the listing's 30-minute clock before nothing more is asked: 46 minutes in all from the opening, 67 and 82 from mid-download, the run lock held and the hourly runs refused through it [M93: the fetch 904, 2108 and 3040 s after the fault; the drill 2736, 3999 and 4901 s]. A storage that stops answering before the drill ends it at the first listing's clock instead. | `Drill` (`backups/engine/restore.go`: a fetch's exit 1 is app-own, and `refused` is set by a wide error or `x.unanswered` alone); `listClock` (`check.go`) | 2 | 3 | 1 | 6 | size 2, risk 2 (take a fetch's exit 1 after a request timeout as the repository's: restic's words are in the journal, not the unit's stdout, as in the read-only key's row above; or give the listings one try too, which puts a single 500's retry lines beside a listing that is then not believed [M92]) | Rule 2 holds: the drill exits 1, each app named, `restore not proven` or that the repository did not answer, and the bound does not grow with the apps or the data: one fetch, ended by restic's lock monitor where its packs are not all asked first, one listing's half hour and the check's 30-second probe, however many there are. Under rule 3 the product is under 20 (2026-10-29, #177). | | Where no region is set (a hand-written file for an S3 host other than AWS's regional ones and B2's), a storage that answers a request with a region error (`AuthorizationHeaderMalformed`, `InvalidRegion`, `AccessDenied`) naming another region than its own bucket-location answer gave is no longer retried by minio-go inside the try, since the upload, fetch and check make one (`oneTry`): restic's retry layer tries again, a retry line in the journal, and the region minio-go cached by then answers; an `AccessDenied` restic takes for permanent, and that unit ends exit 1. | `oneTry` (`backups/engine/engine.go`); minio-go 7.0.89 `executeMethod` in `api.go` (the region retry counts against `MaxRetry`, after `bucketLocCache.Set`); restic 0.18 `IsPermanentError` in `internal/backend/s3/s3.go` | 2 | 3 | 1 | 6 | size 2, risk 2 (leave out `oneTry` where no region is set, a branch on the credential file that brings back the 100-minute hang there) | Read from the sources, not measured: rclone, the e2e storage, answers no region error. It takes a storage whose two answers disagree; the run warns of a B2 endpoint with no region ("What it refuses"), and on AWS minio-go reads the region from a regional host. Rule 2 holds: the unit's exit 1 is said for its app, restic's line in its journal. Under rule 3 the product is under 20 (2026-10-29, #177). | | On an S3 host other than AWS's and B2's, for which setup writes no region, every restic command's first request is minio-go's bucket-location lookup, which minio-go does not retry on a 5xx that carries no S3 error code: one such 500 puts restic's two retry lines on stderr beside a listing, exit 0, and the listing is not believed [M89: B2's documented JSON 5xx body, "Codes in the 5xx range … include JSON diagnostics" (Backblaze, Introduction to the S3-Compatible API), injected through a proxy]. An S3 XML 500 there is retried by minio-go in silence. | `besides` (`backups/engine/engine.go`); minio-go 7.0.89 `bucket-cache.go`, `executeMethod` in `api.go` | 2 | 3 | 1 | 6 | size 2, risk 3 (a region for every host would have to be asked for, and a wrong one is refused; reading restic's retry lines as harmless would believe a listing beside a line that means a snapshot was left out) | Rule 2 holds: the listing is a warning of the run, said as restic having something to say beside it, and the next run lists again. On AWS minio-go knows the region from a regional host alone (minio-go `pkg/s3utils/utils.go`, `GetRegionFromURL`, gives none for `s3.amazonaws.com`); there the guide's policy grants `s3:GetBucketLocation`, and AWS's errors are XML, which minio-go retries. On B2 setup writes it ("Setup", 4), and with it the same fault is silent (M89). Under rule 3 the product is under 20 (2026-10-23, #354). | diff --git a/docs/backups.md b/docs/backups.md index 0af2dc45..4e445044 100644 --- a/docs/backups.md +++ b/docs/backups.md @@ -874,8 +874,10 @@ are the endpoint's own questions after them. 21–36 mid-upload and 35–51 mid-fetch, exit 1 (or 130, restic giving up its lock) [M93]. The check, which takes no lock, holds a quarter of an hour mid-read for each five packs it has left [M94]. One that - answers a byte at a time is never given up on, which is #177's - question still. + answers a byte at a time is never given up on; `status` is red from + the third hour, naming the unit still running. Both are accepted + ([#177](https://github.com/smallhoursorg/hotserve/issues/177), + closed with their scores). - **Retention off the box**: **unverified**. **Through B2's S3-compatible endpoint**, besides every line above: diff --git a/docs/design/backups/README.md b/docs/design/backups/README.md index 56099e60..0a2ed80f 100644 --- a/docs/design/backups/README.md +++ b/docs/design/backups/README.md @@ -12,7 +12,8 @@ operator guide is [docs/backups.md](../../backups.md) and the reference is bucket passed or were answered; check 5 (retention) and check 8's egress figure were not done. - The run found a hang that stops backups: a run's upload sat for 27 hours - on B2 with no output — **#177**, with the evidence and the options. + on B2 with no output — **#177**, with the evidence and the options; + closed 2026-10-29, after #368, with what is left accepted. - Every RC check on B2 ran through B2's native API — restic's own `b2:` backend, and rclone's and `backblaze-b2`'s B2 code. Since D13 (2026-10-23, #354) B2 is reached through restic's S3 backend at diff --git a/docs/design/backups/plan.md b/docs/design/backups/plan.md index f84545c2..b720c13a 100644 --- a/docs/design/backups/plan.md +++ b/docs/design/backups/plan.md @@ -1018,7 +1018,7 @@ Settled (owner, 2026-09-19): **Revised 2026-10-23 (D13):** "restic bounds its own waiting" holds through its S3 backend for a storage that answers or refuses — exit 1 within 14.3–14.9 min per operation, 20.5–34.7 min for an outage from mid-backup [M87] — and not for one that takes connections and never answers, or answers a byte at a time (no exit in 40 min [M88]), or is paused (no exit in 25 min, #346). Those stay #177's. An off-box prune that fails or is stopped leaves an exclusive lock that holds every run for its two-hour `--retry-lock` [M90]; a residual in the README. - **Revised 2026-10-29 (#177):** a storage that takes connections and never answers is bounded by restic after all, but at ten times what D13 measured: each try at a request is ended by restic's five-minute stuck-request watchdog, minio-go makes ten, and restic's own retries go once more — 100 minutes at the opening, 106 mid-upload, a SIGTERM not ending it sooner [M92]. The upload, the fetch and the check make one try (`-o s3.retries=1`, `oneTry`): 15 minutes at the opening, 21–36 mid-upload, 35–51 mid-fetch (packs given up five at a time, every quarter of an hour, until restic's lock monitor ends the command; once exit 130), exit 1, the check mid-read, with no lock and so no lock monitor, a quarter of an hour for each five packs left in its group [M94], while every fault that answers is retried by restic as long as before, a ten-minute silence still ends exit 0, and a slow link that is moving is not cut [M92, M93]. Shortening the watchdog instead (`--stuck-request-timeout`) cut uploads still moving at 2 Mbit/s [M92], so it stays at restic's five minutes. The listings keep minio-go's ten: one 500 retried by restic is a line on stderr, and a listing beside one is not believed. A storage that answers a byte at a time is still never given up on; that stays #177's. + **Revised 2026-10-29 (#177):** a storage that takes connections and never answers is bounded by restic after all, but at ten times what D13 measured: each try at a request is ended by restic's five-minute stuck-request watchdog, minio-go makes ten, and restic's own retries go once more — 100 minutes at the opening, 106 mid-upload, a SIGTERM not ending it sooner [M92]. The upload, the fetch and the check make one try (`-o s3.retries=1`, `oneTry`): 15 minutes at the opening, 21–36 mid-upload, 35–51 mid-fetch (packs given up five at a time, every quarter of an hour, until restic's lock monitor ends the command; once exit 130), exit 1, the check mid-read, with no lock and so no lock monitor, a quarter of an hour for each five packs left in its group [M94], while every fault that answers is retried by restic as long as before, a ten-minute silence still ends exit 0, and a slow link that is moving is not cut [M92, M93]. Shortening the watchdog instead (`--stuck-request-timeout`) cut uploads still moving at 2 Mbit/s [M92], so it stays at restic's five minutes. The listings keep minio-go's ten: one 500 retried by restic is a line on stderr, and a listing beside one is not believed. A storage that answers a byte at a time is still never given up on; accepted with the check mid-read, and #177 closed with their scores (2026-10-29). - **D13 — Backblaze B2 through restic's S3 backend (the owner, 2026-10-23, #354).** B2 is reached at its S3-compatible endpoint, `s3:https://s3..backblazeb2.com/`; restic's own `b2:` backend is refused by `setup` before any prompt, naming the S3 form with the bucket and path as restic's b2 backend reads them and `` for the operator (`b2Refused`, `b2AsS3`), by `status` as root as a warning in the same words, and by every run, restore and drill before any unit, in words that open "the repository is b2:, which is not supported:" and name the `setup` to run before the reason, so that the record's cut (`record.Text`) drops the reason first (`b2RunRefused`) — so the run reads the credential file's `RESTIC_REPOSITORY` (`repositoryInUse`: `envfile.Parse`, the last assignment, as the manager reads it), which it otherwise only hands the manager. For a host that is `s3..backblazeb2.com` in ASCII, in any case, read past a port or a trailing dot (the region is the name's; whether B2 takes a host written so is not measured), and never a longer name ending so (`b2Host`, `b2RegionOf`), setup writes `AWS_DEFAULT_REGION=` in lower case, not asked, and says so; any other host gets nothing more. Why: Debian 13's restic 0.18.0 is built against Debian's blazer 0.6.1 (its `Build-Depends`, not restic's `go.mod`), whose upload retries a connection that fails before any answer on the same upload URL, every second, without limit — the 27-hour hang of #177, a run's lock write that never returned: `makeNetRequest` makes a transport error `b2err{retry: 1}` (`base/base.go:317-320` in 0.6.1), `Action` answers Retry for it (`base/base.go:70-71`), and `withBackoff` (`b2/backend.go:749-767`), under `beURL.uploadFile` (`:505`), tries it again on that URL with no limit. blazer fixed it in v0.9.0 (2026-10-22): Backblaze/blazer PR #81 (`fd652a4`: a transport fault on `b2_upload_file` or `b2_upload_part` asks for a new upload URL) and #32 (`95963ac`: a failed upload URL is not pooled again); no restic release carries it, nor Debian 13. restic's documentation points B2 users to the S3 endpoint "due to issues with error handling in the current B2 library that restic uses" (`doc/030_preparing_a_new_repo.rst`), and the e2e suites already run restic's S3 backend. The S3 endpoint is load-balanced (Backblaze, "Load balancing 2.0", 2023; four IPv4 and four IPv6 addresses for `s3.eu-central-003.backblazeb2.com`, looked up 2026-10-23), so no request is tied to one storage pod. Rejected: a bundled or patched restic (Debian's security updates become this project's, for the life of Debian 13); keeping both doors (two sets of key and hide semantics to measure and word, for a backend restic no longer develops).