docs(backups): #177 closed — what is left of it is accepted, not owed

backups/README.md

@@ -1444,8 +1444,8 @@ severity × occurrence × detection, then the fix's size and risk.
 | `setup`'s look (`restic cat config`) exiting 1 within its clock for a reason other than the storage refusing — a config file restic cannot read, a TLS trust failure — is said as every exit 1 is, "restic could not open the repository (exit 1): <restic's words>" then "the storage refused the key, or could not be reached: the key id and secret again", and setup ends after three askings with restic's words; before #175 the look was given up on and init said the same words after a password had been shown. | `look` (`backups/engine/setup.go`), `lookRefused` | 2 | 3 | 1 | 6 | size 1, risk 2 (reading the look's exit 1 by restic's wording, as init's is, would send a storage that refuses at once in another shape — S3's `AccessDenied` comes through the config's Stat, not the opening — back to the password-first path) | Under rule 3 the product is under 20 and the fix is not size 1, risk 1 (2026-10-23, #175). Rule 2 holds: red, with restic's own words on the line, and nothing written. |
 | A storage outage that begins during a backup leaves the command's lock file in the repository: the backup ends exit 1 after 20–35 minutes, its lock still there [M87], which restic never passes by itself [M75]. The box's own commands go on (a backup, a fetch and setup's write check take shared locks, the drill's check none; the next backup measured ok); an off-box `prune` waits out its `--retry-lock 1h` and exits 11, since restic refuses an exclusive lock over any lock and never asks whether it is stale (restic 0.18.0, `internal/restic/lock.go:188-189`; M75, measured for a backup). | restic, not this code; docs/backups.md, "Retention, off the box" | 2 | 3 | 1 | 6 | size 2, risk 2 (a `restic unlock` unit after a run the storage failed would delete in the repository on every such run, and would remove a lock another machine's restic still held) | Rule 2 holds: red and named where it bites — `prune` says the repository is locked (exit 11) — and the guide gives `restic list locks --no-lock` and the unlock. Under rule 3 the product is under 20 (2026-10-23, #354). |
 | An off-box `prune` that is stopped (killed, its machine asleep or cut off) or whose deletes all fail leaves an exclusive lock, which restic does not take for stale with its process gone [M90]: every run and drill of the box's then waits its `--retry-lock 2h` for it, holding the run lock, nothing said meanwhile, and ends "the repository stayed locked by something else for 2h (exit 11)"; backups stop until `restic unlock` from your own machine. | restic, not this code; `resticFailure` (`backups/engine/engine.go`); docs/backups.md, "Retention, off the box" | 4 | 4 | 1 | 16 | size 3, risk 3 (the box removing another machine's lock would end a prune that is still running, whose lock is all that keeps a backup from writing beside its deletions) | Rule 2 holds: red and named — every run fails naming the lock, `status` is stale from the third hour — and the guide gives the remedy and says to read `prune`'s exit status, not its words (one whose every delete failed printed "2 snapshots have been removed", M90). Under rule 3 the product is under 20 (2026-10-23, #354). Whether `restic unlock` without `--remove-all` removes such a lock is not measured; the guide gives `--remove-all` with the box's timers stopped. |
-| A storage that answers a byte at a time is never given up on by restic — each byte read resets its stuck-request watchdog, at any timeout [M92] — so an upload, a fetch or the check runs on, holding the run lock, and no later run starts. One that takes connections and never answers, or is paused, is no longer this row: those three units make one try at a request (`oneTry`) and end after 15 minutes at the opening, 21–36 mid-upload and 35–51 mid-fetch, exit 1, or 130 where restic's lock monitor gives up first [M93] (the check mid-read, the next row), where ten tries took 100–106 minutes with nothing on stderr until the end [M92]. | the upload, fetch and check units (`backups/engine/engine.go`, `restore.go`, `check.go`); restic's stuck-request watchdog, kicked by every byte | 4 | 3 | 1 | 12 | size 3, risk 3 (a no-progress watchdog that ends a unit, the option #177 weighs: a new state per unit, and a cut that must never end a slow upload that is moving — restic's `bytes_done` stands still while the last packs are sent, and the bytes a unit writes grow in a stall, as restic sends a pack again every five minutes [M92]) | Rule 2 holds: `status` exits 1 from the third hour — "no run has finished since …", the app stale, and `running: upload of <app>, since …` naming the stuck unit. Under rule 3 the product is under 20 (2026-10-23, #354; narrowed 2026-10-29, #177). The switch to the S3 endpoint removed the hang #177 found (M87: every fault that answers ends), and one try a request the one that never answers [M93]; this one is left with #177 (on hold), where the watchdog and a dead man's switch are weighed. |
-| A storage that stops answering while the drill's check reads its data group holds the check a quarter of an hour for each five packs of the group not yet read — 45 minutes for 14 packs [M94] — since the check takes no lock, and so has no lock monitor to end it as a fetch's does [M93]: a group of 1,200 packs, a 1 TB repository's fifty-second in restic's 16 MiB packs, some 60 hours, the run lock held and every hourly run refused through it. It ends exit 1, and the probe after it says `unreachable` (`checkRepository`). | `checkArgv` (`backups/engine/check.go`: `--no-lock`, no clock, D4); restic 0.18's five connections and its quarter of an hour of retries a pack | 4 | 3 | 1 | 12 | size 3, risk 3 (a clock on the check from its group's size, which the drill does not know before the check, and which must never cut a slow link that is reading; or the lock, which the owner refused for the check, since one killed hard leaves an exclusive lock restic never passes [M75]; or #177's no-progress watchdog) | Rule 2 holds: `status` exits 1 from the third hour, "no run has finished since …", and its `running:` line names the check and since when; the check ends by itself, `unreachable` and named, and the next drill reads the group again. One try a request made it ten times shorter: ten took 100 minutes for each five packs [M92]. Under rule 3 the product is under 20 (2026-10-29, #177); left with #177, with the watchdog. |
+| A storage that answers a byte at a time is never given up on by restic — each byte read resets its stuck-request watchdog, at any timeout [M92] — so an upload, a fetch or the check runs on, holding the run lock, and no later run starts. One that takes connections and never answers, or is paused, is no longer this row: those three units make one try at a request (`oneTry`) and end after 15 minutes at the opening, 21–36 mid-upload and 35–51 mid-fetch, exit 1, or 130 where restic's lock monitor gives up first [M93] (the check mid-read, the next row), where ten tries took 100–106 minutes with nothing on stderr until the end [M92]. | the upload, fetch and check units (`backups/engine/engine.go`, `restore.go`, `check.go`); restic's stuck-request watchdog, kicked by every byte | 4 | 3 | 1 | 12 | size 3, risk 3 (a no-progress watchdog that ends a unit, the option #177 weighed: a new state per unit, and a cut that must never end a slow upload that is moving — restic's `bytes_done` stands still while the last packs are sent, and the bytes a unit writes grow in a stall, as restic sends a pack again every five minutes [M92]) | Rule 2 holds: `status` exits 1 from the third hour — "no run has finished since …", the app stale, and `running: upload of <app>, since …` naming the stuck unit. Under rule 3 the product is under 20 (2026-10-23, #354; narrowed 2026-10-29, #177). The switch to the S3 endpoint removed the hang #177 found (M87: every fault that answers ends), and one try a request the one that never answers [M93]; this one is accepted, and #177 closed with these scores (2026-10-29): the watchdog and a dead man's switch it lists are features, not fixes owed. |
+| A storage that stops answering while the drill's check reads its data group holds the check a quarter of an hour for each five packs of the group not yet read — 45 minutes for 14 packs [M94] — since the check takes no lock, and so has no lock monitor to end it as a fetch's does [M93]: a group of 1,200 packs, a 1 TB repository's fifty-second in restic's 16 MiB packs, some 60 hours, the run lock held and every hourly run refused through it. It ends exit 1, and the probe after it says `unreachable` (`checkRepository`). | `checkArgv` (`backups/engine/check.go`: `--no-lock`, no clock, D4); restic 0.18's five connections and its quarter of an hour of retries a pack | 4 | 3 | 1 | 12 | size 3, risk 3 (a clock on the check from its group's size, which the drill does not know before the check, and which must never cut a slow link that is reading; or the lock, which the owner refused for the check, since one killed hard leaves an exclusive lock restic never passes [M75]; or #177's no-progress watchdog) | Rule 2 holds: `status` exits 1 from the third hour, "no run has finished since …", and its `running:` line names the check and since when; the check ends by itself, `unreachable` and named, and the next drill reads the group again. One try a request made it ten times shorter: ten took 100 minutes for each five packs [M92]. Under rule 3 the product is under 20 (2026-10-29; #177 closed with these scores). |
 | A drill whose fetch the storage stops answering gives that fetch up, `restore not proven` and that app's own — 15 minutes at the fetch's opening, exit 1; 35 and 51 minutes from mid-download, exit 1 and exit 130 — then asks the next app's snapshots, which waits out the listing's 30-minute clock before nothing more is asked: 46 minutes in all from the opening, 67 and 82 from mid-download, the run lock held and the hourly runs refused through it [M93: the fetch 904, 2108 and 3040 s after the fault; the drill 2736, 3999 and 4901 s]. A storage that stops answering before the drill ends it at the first listing's clock instead. | `Drill` (`backups/engine/restore.go`: a fetch's exit 1 is app-own, and `refused` is set by a wide error or `x.unanswered` alone); `listClock` (`check.go`) | 2 | 3 | 1 | 6 | size 2, risk 2 (take a fetch's exit 1 after a request timeout as the repository's: restic's words are in the journal, not the unit's stdout, as in the read-only key's row above; or give the listings one try too, which puts a single 500's retry lines beside a listing that is then not believed [M92]) | Rule 2 holds: the drill exits 1, each app named, `restore not proven` or that the repository did not answer, and the bound does not grow with the apps or the data: one fetch, ended by restic's lock monitor where its packs are not all asked first, one listing's half hour and the check's 30-second probe, however many there are. Under rule 3 the product is under 20 (2026-10-29, #177). |
 | Where no region is set (a hand-written file for an S3 host other than AWS's regional ones and B2's), a storage that answers a request with a region error (`AuthorizationHeaderMalformed`, `InvalidRegion`, `AccessDenied`) naming another region than its own bucket-location answer gave is no longer retried by minio-go inside the try, since the upload, fetch and check make one (`oneTry`): restic's retry layer tries again, a retry line in the journal, and the region minio-go cached by then answers; an `AccessDenied` restic takes for permanent, and that unit ends exit 1. | `oneTry` (`backups/engine/engine.go`); minio-go 7.0.89 `executeMethod` in `api.go` (the region retry counts against `MaxRetry`, after `bucketLocCache.Set`); restic 0.18 `IsPermanentError` in `internal/backend/s3/s3.go` | 2 | 3 | 1 | 6 | size 2, risk 2 (leave out `oneTry` where no region is set, a branch on the credential file that brings back the 100-minute hang there) | Read from the sources, not measured: rclone, the e2e storage, answers no region error. It takes a storage whose two answers disagree; the run warns of a B2 endpoint with no region ("What it refuses"), and on AWS minio-go reads the region from a regional host. Rule 2 holds: the unit's exit 1 is said for its app, restic's line in its journal. Under rule 3 the product is under 20 (2026-10-29, #177). |
 | On an S3 host other than AWS's and B2's, for which setup writes no region, every restic command's first request is minio-go's bucket-location lookup, which minio-go does not retry on a 5xx that carries no S3 error code: one such 500 puts restic's two retry lines on stderr beside a listing, exit 0, and the listing is not believed [M89: B2's documented JSON 5xx body, "Codes in the 5xx range … include JSON diagnostics" (Backblaze, Introduction to the S3-Compatible API), injected through a proxy]. An S3 XML 500 there is retried by minio-go in silence. | `besides` (`backups/engine/engine.go`); minio-go 7.0.89 `bucket-cache.go`, `executeMethod` in `api.go` | 2 | 3 | 1 | 6 | size 2, risk 3 (a region for every host would have to be asked for, and a wrong one is refused; reading restic's retry lines as harmless would believe a listing beside a line that means a snapshot was left out) | Rule 2 holds: the listing is a warning of the run, said as restic having something to say beside it, and the next run lists again. On AWS minio-go knows the region from a regional host alone (minio-go `pkg/s3utils/utils.go`, `GetRegionFromURL`, gives none for `s3.amazonaws.com`); there the guide's policy grants `s3:GetBucketLocation`, and AWS's errors are XML, which minio-go retries. On B2 setup writes it ("Setup", 4), and with it the same fault is silent (M89). Under rule 3 the product is under 20 (2026-10-23, #354). |

docs/backups.md

@@ -874,8 +874,10 @@ are the endpoint's own questions after them.
   21–36 mid-upload and 35–51 mid-fetch, exit 1 (or 130, restic giving
   up its lock) [M93]. The check, which takes no lock, holds a quarter
   of an hour mid-read for each five packs it has left [M94]. One that
-  answers a byte at a time is never given up on, which is #177's
-  question still.
+  answers a byte at a time is never given up on; `status` is red from
+  the third hour, naming the unit still running. Both are accepted
+  ([#177](https://github.com/smallhoursorg/hotserve/issues/177),
+  closed with their scores).
 - **Retention off the box**: **unverified**.
 
 **Through B2's S3-compatible endpoint**, besides every line above:

docs/design/backups/README.md

@@ -12,7 +12,8 @@ operator guide is [docs/backups.md](../../backups.md) and the reference is
   bucket passed or were answered; check 5 (retention) and check 8's
   egress figure were not done.
 - The run found a hang that stops backups: a run's upload sat for 27 hours
-  on B2 with no output — **#177**, with the evidence and the options.
+  on B2 with no output — **#177**, with the evidence and the options;
+  closed 2026-10-29, after #368, with what is left accepted.
 - Every RC check on B2 ran through B2's native API — restic's own
   `b2:` backend, and rclone's and `backblaze-b2`'s B2 code. Since
   D13 (2026-10-23, #354) B2 is reached through restic's S3 backend at