Repository navigation
borg2: avoid doing an unnecessary rebuild of the index #8476
Description
Note: updated for situation WITH packs.
For high-latency stores (sftp: and a lot of what's available via rclone:) a full index rebuild needs to list all objects in all packs in the store and there are potentially a lot of objects. This is slow as it needs to scan over each pack file - only borg check --repair-index shall do that.
Most commands will use the existing chunks index. Most commands only add objects, so they will just write an additional partial index to index/sha256(partial_index) with just the new index entries.
For borg create:
- it's not a big problem if the index does not have all objects that exist in the repo. If that happens, borg create will just store something to the repo that's already there.
- it would be a severe problem though if the index would falsely say "we have that object" and borg would not store it to the repo. the archive would then reference a non-existing object.
- similarly it is fatal if the index points to a wrong pack_id
Activity
[-]borg2: which commands shall rebuild a completely new ChunkIndex?[/-][+]borg2: which commands shall do a complete rebuild of the ChunkIndex?[/+]ThomasWaldmann commentedon Nov 16, 2024
borg compact emits some stats and for that it needs the stored sizes of the repository objects.
With packs, we have pack_id, obj_offset and obj_size in the chunks index - that is enough to do stats for compact.
As borg compact moves object to other pack ids, it needs to write the complete updated index and get rid of old index files.
ThomasWaldmann commentedon Nov 19, 2024
borg check could just try to work with the existing chunks index and not rebuild a new one.
repository check: hash all index files, hash all pack files - if it does not find issues we assume all is fine on the repository level.
If borg check finds index issues it would already terminate at that step.
Otherwise it would continue with hashing the packs and report all corrupted pack ids.
borg check --repair-index would rebuild the index by scanning packs (slow). it could either read the full pack (and then also hash it) or iterate only over the object headers.
borg check --repair-pack <pack_ids> could repair just the given pack ids without doing a full scan over all packs. repairing a pack means determining defective objects in the pack (by reading and decrypting them, if AEAD decryption fails, the object is corrupted). all non-corrupt objects will be written to a new pack, index fixed with the new pack_id.
59 remaining items
check --repair: clamp the header walk to the pack, document blobs fro…
check --repair: report why a header failed and what a repair leaves b…
check --repair: move object_validator to repoobj, test the keyless re…
check --repair: report the objects a resync drops as a check problem, b…
check --repair: let only unusable metadata make the object validator …
check --repair: drop a damaged pack's tail when there is no validator, …
check --repair: split the tail-drop policy out of on_drop, borgbackup…
check --repair: validate the lazy repository chunks index rebuild, bo…
check: validate the lazy repository chunks index rebuild, borgbackup#…
ThomasWaldmann commentedon Sep 11, 2026
@mr-raj12 There is a ton of commits referencing this issue, but it is still open. Please check.
mr-raj12 commentedon Sep 11, 2026
I checked all the commits referencing this are already in master, most of the timeline noise is from the #10094 rebases. The main parts are done: partial index writes, the read-only check without a rebuild, the header-based index rebuild in check --repair, and finish() keeping the index.
Only two things are pending:
pack repair (#10026) and avoiding the extra rebuilds in check --repair (#10318)
so we can close this one and track the rest there, or would you prefer to keep it open till pack repair is done?
ThomasWaldmann commentedon Sep 11, 2026
Guess we rather keep this one open until all these places are fixed.
mr-raj12 commentedon Sep 27, 2026
Current state on master (f1ca9f8):
check --repairstill rebuilds the index from all packs even when the stored index is intact and the repository phase just verified every pack in the same run (archive.py:2334,slow_rebuild=repair). check: a full --repair rebuilds a corrupt index once, in the archives phase, fixes #10434 #10445 does not touch this, it only removes the second rebuild when the index is corrupt. This is the same point you made in borg2: check [--repair] improvements #8466.salvage_pack()is merged (repository: add salvage_pack, keeping only the authenticated objects of a corrupt pack, refs #10026 #10419) but nothing calls it yet, sorepository.py:1793still reports pack repair as not implemented. That is borg2 check: implement repository repair (index and pack rebuild) #10026 item 4, and items 5 and 7 are open too.#8572is closed but still referenced incheck_cmd.py,repository.pyand the tests. I will repoint those to borg2 check: implement repository repair (index and pack rebuild) #10026.So after #10445 what is left here is #10026 plus that
--repairrebuild. Should I handle the rebuild under #8466, or do you want--repairto always rebuild, so this issue only waits on #10026?ThomasWaldmann commentedon Oct 5, 2026
This is done on master:
borg checkwithout--repairhashes the index and pack objects (repository: read-only check by hashing pack/index objects #9794) and uses the stored index. It does not rebuild it from the packs.check --repairrebuilds the index from the pack object headers and validates every object (repository: add PackReader to walk pack object headers #9795, check --repair: rebuild a corrupt repository index from the packs, #10026 #10048, validate object headers when walking a pack (PackReader.iter_headers) #10083, check --repair: resync past corrupt object headers when rebuilding the chunks index #10094).check --repairbuilds the index only once per run (check: use one chunk index for the checker and the repository, refs #10364 #10368, check: a full --repair rebuilds a corrupt index once, in the archives phase, fixes #10434 #10445).finish()re-reads only the packs the repair wrote (check --repair: re-read only the packs the repair wrote, refs #8466 #10379) and writes the index back, so the next command does not rebuild it (borg2: check [--repair] improvements #8466).check --repairsalvages corrupt packs withsalvage_pack()and updates the index.check --repairstill rebuilds the index from all packs once per run, even if the stored index is intact. We accept that for now: the full walk is what finds archives that reference chunks in missing packs.@mr-raj12 this answers your question:
--repairkeeps doing that one rebuild.