check: survive I/O errors while reading repository objects

[ThomasWaldmann]

Fixes #3509.

An OSError from the store (failing disk, flaky network filesystem, ...) escaped up to the top-level handler, so borg check died with a "Local Exception" traceback and the results collected so far were lost. The issue is from the segment-file days, but master has the same hole in the new code paths:

$ chmod 000 repo/packs/39/39414...   # simulates an unreadable pack
$ borg check
Local Exception
Traceback (most recent call last):
  ...
  File "src/borg/repository.py", line 1677, in verify
    ok = self.store.hash(key, algorithm=STORE_HASH_NAME) == name
  ...
PermissionError: [Errno 13] Permission denied: '.../repo/packs/39/39414...'

Reads of repository objects now raise Repository.StoreReadError (rc 35), keeping the original OSError as __cause__. It is deliberately distinct from ObjectNotFound and IntegrityError: borg never saw the content, so it can tell neither that the object is missing nor that it is corrupt. This covers all read paths: whole-pack loads, range reads, batched reads (gather_many, used for archive metadata and item metadata streams), namespace listings and the repository config.

Following the three points from the issue:

• The original exception's information is kept: the errno, the message and the file name are in the reported message, and the OSError itself stays as __cause__.
• The check finishes: the repository check reports the object, goes on and fails at the end, so one run lists everything that is affected. No result is recorded for such a pack or archive, so a later check verifies it again rather than remembering it as corrupt (or as fine). The archives check reports the affected archive and continues with the next one.
• Nothing acts on data we could not read: because it may well be readable again once the underlying problem is fixed, --repair refuses to repair (new Repository.RepairUnsafe, rc 36), neither the chunk index is rebuilt nor are corrupt packs salvaged while some objects could not be read, and --verify-data leaves those chunks in place instead of deleting them. Previously --verify-data --repair could only reach the delete path via an IntegrityError, so this is about keeping the two apart as the code grows.

Commands other than borg check stop with the read error instead of working with a partially readable repository: e.g. prune must not see a placeholder for an archive whose metadata could not be read. Only borg check lists such an archive with a placeholder entry, so it can go on with the other archives.

Same repository as above, with the fix:

$ borg check -v
Starting full repository check
1 pack check results on record, verifying every pack.
Store object packs/39414... could not be read: [Errno 13] Permission denied: '.../repo/packs/39/39414...'
Finished checking packs.
Checked 2 index files (0 errors) and 2 packs (0 errors).
1 store object(s) could not be read, so they could not be checked. This usually means a
hardware, filesystem or network problem - see the "Data integrity" section of the docs.
The repository was not modified; fix the underlying problem, then run the check again.
Finished full repository check, errors found.
Starting archive consistency check...
Analyzing archive archive-metadata-could-not-be-read Thu, 1970-01-01 01:00:00 +0100 cb9ecb... (1/1)
Archive metadata block cb9ecb... could not be read: Error reading pack 39414... from the
repository: [Errno 13] Permission denied: '...'. Check the storage hardware / filesystem.
Analyzed 1 archive(s).
Archive consistency check complete, problems found.
rc=1

$ borg check --repair
...
Not repairing: 1 repository object(s) could not be read. Fix the underlying problem, then check again.
rc=36

(The archive's name is stored in its unreadable metadata, so the check can only show the placeholder and the archive id.)

A listing that fails (store.list() on an unreadable directory) is still fatal — we do not know what to check then — but it ends the check with that one-line message instead of a traceback. Likewise, an unreadable repository config ends a command with that message, instead of looking like a missing repository.

Fifteen regression tests cover it: the basics (reported and the check continues, no tracker record written, --repair refuses), each read path (packs, index objects, namespace listings, archive metadata, the archives listing, the repository config), and the cases that would otherwise lose data - a defect chunk whose re-read fails is kept, --repair does not salvage corrupt packs while another pack is unreadable, and --repair stops at unreadable archive metadata and leaves the archive intact. They patch the posixfs backend to raise EIO rather than using file permissions, so they also work when the tests run as root.

[codecov]

Codecov Report

❌ Patch coverage is 90.44118% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.30%. Comparing base (4c563dc) to head (18f415b).
⚠️ Report is 37 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/borg/archive.py	82.00%	7 Missing and 2 partials ⚠️
src/borg/cache.py	77.77%	1 Missing and 1 partial ⚠️
src/borg/manifest.py	91.66%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10194      +/-   ##
==========================================
- Coverage   89.30%   89.30%   -0.01%     
==========================================
  Files         104      104              
  Lines       20027    20125      +98     
  Branches     3144     3157      +13     
==========================================
+ Hits        17886    17973      +87     
- Misses       1482     1488       +6     
- Partials      659      664       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[ThomasWaldmann]

Two follow-ups pushed:

• 5d2724986: the FreeBSD job found a real gap, not a flake. Archives.list() reads every archive's metadata chunk, so an unreadable pack holding one killed the archives check before the per-archive handling. Archives.list() now takes an opt-in tolerate_read_errors; only the archives check passes it. Everywhere else the read error propagates, because prune calls list() with no exists filter and sorts by timestamp, so a placeholder's 1970 date would put an only-transiently-unreadable archive at the top of the deletion list.
• 34bc9fea0: replaced the wide try around the per-archive body with two precise handlers (reading the archive metadata block, reading the item metadata stream). Better scoped, and it keeps the body where it was, so the earlier git diff -w review note no longer applies. Plus tests for the branches that were untested: patch coverage of the new lines goes from 76% to ~98%.

One scope note I should have put in the description from the start: this catches OSError, which covers the posixfs backend (and the sftp failures that surface as IOError). The other borgstore backends report a read failure as a BackendError, so they are not covered — and for ssh:// the information is gone before it reaches the client: a server-side EIO falls into the generic else in borgstore's server/rest.py and goes out as HTTP 500 with the body "Internal Server Error", losing errno, message and file name. Making this work for remote repositories needs a borgstore-side ObjectReadError in the existing BackendError family, mapped per backend and carried across the REST wire; the policy in this PR would then hang off that instead. Happy to do that separately if you want it.