known-repos: new command to list locally known repositories, fixes #10453

[KazukiKandaKK]

Summary

Closes #10453. Adds borg known-repos, which lists local security info,
cache, and/or key files for repositories this installation knows about.
It does not open any repository.

$ borg known-repos
Repository ID                                                    Layout Cache Security           Key files Last written        Location
401606173d54eab22cd98de67278d093e3bae7b187da3af8c6430f18fa1112bb borg2  180 B aes256-ocb, sha256 -         2026-10-12T17:15:19 /path/to/repo

--json is supported, with the full repository id either way.

Notes

layout (borg2/legacy) is determined from the security or cache
root it was found in, or the cache's own content. If it can't be
determined, it shows as unknown in the table, null in --json.
It's not the on-disk format version - see borg repo-info for that.

Inspection failures are logged and make the command exit nonzero.

A legacy repo's security location and cache previous_location are
compared through the same unicode normalization assert_location_matches()
uses, so NFC vs. NFD alone isn't a mismatch on macOS (#2913).

Cache size sums file sizes found while walking the cache directory - not
disk usage. "Last written" is the cache config's mtime.

Key files in the current and legacy keys directories are counted from
their plaintext header.

18 tests cover discovery, legacy paths, inspection-failure handling,
layout detection, and the NFC/NFD location comparison.

[codecov]

Codecov Report

❌ Patch coverage is 90.17094% with 23 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.30%. Comparing base (9c08f32) to head (c0c4934).
⚠️ Report is 41 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/borg/archiver/known_repos_cmd.py	90.00%	20 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10513      +/-   ##
==========================================
+ Coverage   89.12%   89.30%   +0.18%     
==========================================
  Files         103      105       +2     
  Lines       19799    20173     +374     
  Branches     3103     3161      +58     
==========================================
+ Hits        17645    18015     +370     
- Misses       1492     1500       +8     
+ Partials      662      658       -4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[ThomasWaldmann]

Thanks for working on this! The feature is useful and the basic behaviour works: I ran it against a real local state (~1100 known repos, borg2 + legacy security dirs, separate legacy cache root on macOS), it finished in about 0.5s and did not create anything. But there are a few defects, and the PR is much larger than the feature needs.

Blockers

1. lint fails: black wants to reformat the process_epilog("""...""") call in known_repos_cmd.py.
2. test_known_repos_deep_cache_tree_does_not_hit_recursion_limit errors on Python 3.11: the 1500-level tree breaks pytest's tmpdir teardown, because shutil.rmtree is still recursive on 3.11 (RecursionError). It also needs to raise RLIMIT_NOFILE above 1500. A 1500-deep cache directory is not a realistic case, so please just drop this test.
3. "Layout" is wrong on Linux (and whenever BORG_BASE_DIR is set): there the borg2 and legacy cache roots are the same path (~/.cache/borg), so only the borg2 scan runs and every borg 1.x repo that only has a cache is reported as layout: borg2. It only works out on platforms where the two roots differ (e.g. macOS). Either drop the field, or determine it from the cache contents (a borg 1.x cache dir has chunks / chunks.archive.d).

Design / maintainability

This is the main concern. The command basically lists 2-4 directories and reads two small files per entry, but the implementation is ~750 lines:

• fd-relative openat walks, O_NOFOLLOW everywhere, FIFO / trailing-newline / symlink defences, BaseException fd-cleanup paths, an explicit-stack tree walker... Anybody who could exploit what these defend against already has write access to the user's own config/cache dirs, so I don't think they are worth the complexity.
• The yes/no/unknown tri-state (incomplete counters, listing_ok, security_fully_searched, "unreadable" kinds) drives most of the code and test volume. Logging a warning and exiting with rc 1 on any OSError would be enough.
• The epilog is ~90 lines and mostly documents internal failure semantics. A short paragraph is enough for help text.
• 35 tests / ~1065 lines, mostly for these edge cases, is a lot to maintain.

Please consider a rewrite using plain Path / os.scandir. I'd expect something around 150 lines plus a handful of tests.

Functional suggestions

• Output format: 6-7 lines per repo gets unwieldy fast (with ~1100 repos it is >6000 lines). A one-line-per-repo table, as proposed in #10453, would be much more usable.
• Location: (unknown) for cache-only legacy repos: a borg 1.x cache config has previous_location (and timestamp), so you could show it from there. A borg2 cache config has no location, so this only helps the legacy case.
• Unknown key types print as a bare number, e.g. Security info: yes (144). Something like unknown key type 144 would be clearer.
• _legacy_security_dir() duplicates legacy.fs.get_security_dir() just because that calls get_config_dir() with create=True. Passing create through there would be the smaller change. Also, the legacy/fs.py module docstring says borg.security is its only caller and that the module goes away together with the borg 1.x transfer support. This PR makes that docstring stale, and makes a new user-facing command depend on code that is scheduled for removal, so please keep that in mind.
• The generated docs/man/*.1 and docs/usage/*.rst.inc files are usually regenerated at release time, so you don't need to include them.

Minor

• JSON has both key_type_raw (a string, e.g. "16") and key_type. Is the raw value needed?
• Using the cache config mtime for "Cache last written" is fine as a best-effort proxy, since there is no manifest timestamp in the security dir anymore.

[KazukiKandaKK]

Thanks for the detailed review. I'll rewrite it with Path / os.scandir and simpler error handling, like you suggested, and sort out the rest too. Will push once it's ready.

[PhrozenByte]

I haven't checked the code, but one thing came to mind: what happens if ~/.config/borg/keys/ contains a stale key for a repo that no longer has any associated security information or local cache? Would borg known-repos detect it, and if so, how would it be presented? No answer needed, just something to consider.

Thanks for working on this! 🎉

[KazukiKandaKK]

@ThomasWaldmann
Rewrote this with Path and a much thinner _Scanner, as suggested. Landed
at 358 lines — here's how each point was addressed.

Design / maintainability

• fd-relative walks, O_NOFOLLOW, the explicit-stack walker: all removed,
  per the threat-model argument.
• The yes/no/unknown tri-state is gone. has_security_info/has_cache are
  plain bools now. An inspection failure is logged and exits nonzero; if a
  repo-id directory is the only evidence for an id, a failure there drops
  the id entirely instead of keeping it as "unknown" like the old version.
• Epilog: a short paragraph now, not ~90 lines.
• Tests: 457 lines / 17 functions, covering layout detection across
  borg1/borg2/coinciding-roots, the Linux fix, and previous_location
  fallback.

Blockers

• lint: fixed.
• 1500-deep recursion test: dropped.
• Linux layout bug: this hit cache-only repos (no security info at all,
  so layout started as unresolved) whenever the cache roots coincide -
  which they do by default on Linux and whenever BORG_BASE_DIR is set.
  They got reported as borg2 even when the cache content was actually
  legacy. Fixed in _cache_layout(), which checks the cache entry's own
  content (chunks / chunks.archive.d) in that case instead of
  defaulting to borg2.

Functional suggestions

• Output format: one line per repo in a table, full id only in --json.
• previous_location for cache-only legacy repos: added, read from the
  cache config.
• unknown key type N: added.
• _legacy_security_dir(): removed, now threads create=False straight
  through to legacy.fs.get_security_dir(). Updated that module's
  docstring since known-repos is now a second caller of code scheduled for
  removal.
• Dropped the generated man page / usage.rst.inc.

Minor

• Dropped key_type_raw. For a known type its only remaining use was the
  numeric value, which isn't in the key_type label (e.g.
  "aes256-ocb, sha256"); for an unknown type that same number is already
  visible in "unknown key type N".

358 lines is still above the ~150 estimate. Some of that is two things I
added after the version you reviewed: scanning $BORG_KEYS_DIR (closing
the issue's "including keys" part) and the location_mismatch check.

[ThomasWaldmann]

Thanks, this is a big improvement! It's much easier to read and review now, and all points from my first review were addressed.

I checked c0c4934 locally: the new tests and legacy_fs_test.py pass. I also ran it against a real local state (~1100 known repos). The Linux layout detection via the cache content works, cache-only legacy repos now show their previous_location, and key-file-only repos show up with layout unknown, which also answers @PhrozenByte's question. Old borg2 beta key type bytes show as "unknown key type N", which is fine. It does not create anything beyond what borg's startup creates anyway.

Still blocking

1. black still fails on the process_epilog(""" call in build_parser_known_repos(). Black wants the opening """ on its own line after process_epilog( and the closing ) on a separate line. Please run black (or pre-commit run --all-files) before pushing.
2. Docs: please restore the generated usage docs. Sorry, my earlier remark about the generated docs was wrong. docs/usage/known-repos.rst still includes known-repos.rst.inc, which this commit deleted, so the rendered page silently loses its usage section (the docs build only warns). Other recently added commands (e.g. copy, find) committed docs/usage/<cmd>.rst.inc and docs/man/borg-<cmd>.1 together with the command, so please regenerate both with python scripts/make.py build_usage and python scripts/make.py build_man, and keep only the known-repos files.

Minor

• The PR description is outdated: it still describes the tri-state, "BORG_KEYS_DIR is not scanned" and "35 tests".
• location_mismatch compares the security-dir location with the cache's previous_location without the unicode normalization that SecurityManager.assert_location_matches() does (normalize_local_path(), see #2913), so on macOS the same path in NFC vs NFD form would be flagged as a mismatch.
• dir_size() uses os.path.getsize(), which follows symlinked files. os.lstat() (counting only regular files) would match the "regular files" semantics. Borg never creates symlinks there, so this is cosmetic.
• The table truncates the repo id to 15 chars. To act on an entry (e.g. remove a stale cache dir), users need the full id, so showing it in full may be worth the extra width. Your call.
• legacy/fs.py docstring: "(and now also the known-repos command)" - "now also" describes history. Just listing both callers (as the second paragraph already does) is enough.
• Epilog: with BORG_SECURITY_DIR set, the borg2 and legacy security roots coincide, so security-only repos (no cache) get layout unknown. That's fine, but it could be mentioned briefly in the epilog.

With black and the generated docs fixed, this looks good to me.

[KazukiKandaKK]

@ThomasWaldmann
Fixed the two blockers:

• lint: black reformatted the process_epilog("""...""") call.
• docs: restored docs/usage/known-repos.rst.inc and docs/man/borg-known-repos.1 (scripts/make.py build_usage/build_man). Docs build passes now.

Minors:

• location_mismatch now normalizes both sides with normalize_local_path() (same as assert_location_matches()), so an NFC vs. NFD difference alone isn't flagged on macOS (macOS NFD normalization confuses repo move check #2913). Added a regression test.
• The table no longer truncates the repository id to 15 chars.
• legacy/fs.py's docstring drops the "(and now also ...)" phrasing.
• The epilog now notes that a security-only repo gets unknown when $BORG_SECURITY_DIR makes the borg2 and legacy security roots coincide.
• Left dir_size()'s symlink-following as-is: borg never creates symlinks there, so getsize() vs. lstat() wouldn't change the result.

Also updated the PR description, so it no longer describes the tri-state, "BORG_KEYS_DIR is not scanned", or "35 tests".