seal the files cache with the repository key, fixes #5041

docs/faq.rst

@@ -641,13 +641,20 @@ Make sure that only you have access to the Borg data directory.
 Do I need to take security precautions regarding the cache?
 -----------------------------------------------------------
 
-The cache contains a lot of metadata information about the files in
-your repositories and it is not encrypted.
-
-However, the assumption is that the cache is being stored on the very
-same system which also contains the original files which are being
-backed up. So someone with access to the cache files would also have
-access the original files anyway.
+The files cache contains a lot of metadata about the files in your backups
+(file sizes, timestamps, the chunk lists of the files and, if you use
+``borg create --digests``, digests of the file contents). It is protected with the
+repository key, like the data in the repository: it is encrypted and authenticated
+(authenticated only in the ``authenticated-*`` modes), so someone who can read the
+cache directory, but does not have the key, can not read it, and a modified files
+cache is detected (borg ignores it with a warning and rebuilds it). The pack cache
+(``BORG_STORE_CACHE``) only holds copies of the encrypted packs of the repository.
+
+What is not protected: the names, sizes and timestamps of the files in the cache
+directory, the repository id (the cache directory is named after it) and the cache
+``config`` file (it only contains the cache version and the repository id). Someone
+who can write to the cache directory can delete files there, which only makes the
+next backup slower.
 
 The Internals section contains more details about :ref:`cache`. If you ever need to move the cache
 to a different location, this can be achieved by using the appropriate :ref:`env_vars`.

docs/internals/data-structures.rst

@@ -138,6 +138,35 @@ missing or fails the authentication, ``borg check --repair`` replaces it by empt
 (see :ref:`repo_defaults`). The ``chunkindex-invalid`` marker has no content and is
 stored as is.
 
+.. _sealed_stream:
+
+Sealed streams
+~~~~~~~~~~~~~~
+
+Data that may be too big to be kept in memory twice is protected as a **sealed
+stream** (``borg.crypto.sealed_stream``): the byte stream is cut into frames of 1 MiB
+(the last frame holds the rest, it may be empty), and every frame is put into the
+key's envelope, so it is encrypted and authenticated in the encrypting modes and
+authenticated only in the ``authenticated-*`` modes::
+
+    stream := frame+
+    frame  := flag (1 byte: 0 = more frames follow, 1 = last frame)
+              length (4 bytes, unsigned, little endian: the length of the envelope)
+              envelope (the key's encrypt_oneshot() of the payload, empty id)
+
+The AAD of a frame's envelope is a *context* given by the user of the sealed stream,
+followed by the frame number (8 bytes, big endian, counting from 0) and the flag (1
+byte). The context starts with a domain prefix of its own and contains what the stream
+belongs to (e.g. the repository id and a name). So a frame can not be changed, moved
+to another position or into another stream, and a stream that ends before its last
+frame, claims an earlier frame to be the last one or has data after its last frame is
+rejected. What a sealed stream can not detect is the replacement of the whole stream
+by an older stream with the same context.
+
+Every frame is sealed in a one-off session (a new random session id, IV 0), so
+sealed streams can be used from any thread. Each frame costs a session key derivation
+(a few microseconds), which is negligible for 1 MiB of payload.
+
 
 Keys
 ~~~~
@@ -729,11 +758,27 @@ Borg can also work without using the files cache (saves memory if you have a
 lot of files or not much RAM free), then all files are assumed to have changed.
 This is usually much slower than with files cache.
 
-The on-disk format of the files cache is a stream of msgpacked tuples (key, value).
-There, the chunks list is stored in its uncompressed form (chunk id and size), as
-the chunks index indexes are only valid for one specific in-memory chunks index.
-Loading the files cache involves reading the file, one msgpack object at a time,
-unpacking it, and compressing the entry as described above.
+On disk, the files cache is a stream of msgpacked tuples (key, value). There, the
+chunks list is stored in its uncompressed form (chunk id and size), as the chunks
+index indexes are only valid for one specific in-memory chunks index. This stream
+is stored as a :ref:`sealed stream <sealed_stream>`, so it is encrypted and
+authenticated with the repository key (authenticated only in the ``authenticated-*``
+modes), like the objects in the repository. The context of the sealed stream is
+``b"borg-files-cache\0"``, followed by the repository id and the file name of the files
+cache (``files.<SUFFIX>``): a files cache copied from another repository (using the
+same key material) or from another archive series is rejected. Loading the files
+cache involves reading and verifying the file frame by frame, unpacking one msgpack
+object at a time and compressing the entry as described above.
+
+Without the key, one can only see the names, sizes and timestamps of the files
+cache files. A files cache that fails the verification (it is corrupted, was
+modified, or was not written with the key of this repository) is ignored with a
+warning: borg then rebuilds the files cache from the most recent archive of the
+series in the repository, or, failing that, starts with an empty files cache. The
+same happens, without a warning, if there is no files cache file (yet). Replacing a
+files cache file by an older version of it is not detected, but harmless: its
+entries were correct when it was written, and entries that refer to chunks that are
+no longer in the repository are dropped when loading it.
 
 .. _index:
 
@@ -1258,19 +1303,14 @@ just retry later.
 Checksumming data structures
 ----------------------------
 
-As detailed in the previous sections, Borg generates and stores files
-containing important meta data, currently the files cache.
-
-Data corruption in the files cache could create incorrect archives, e.g. due
-to wrong object IDs or sizes in the files cache.
-
-Therefore, Borg calculates checksums when writing these files and tests checksums
-when reading them. Checksums are generally 256-bit sha256 hashes.
+Borg 2 protects its own local files with the repository key (see :ref:`the files
+cache <cache>`). Checksums are only used for reading borg 1.x repositories (see
+``borg transfer``): borg 1.x stores the repository index and hints files of such a
+repository (``index.N``, ``hints.N``) with checksums, so that data corruption in them
+is detected. Checksums are generally 256-bit sha256 hashes.
 Checksums are stored as hexadecimal ASCII strings.
 
 For compatibility, checksums are not required and absent checksums do not trigger errors.
-The mechanisms have been designed to avoid false-positives when various Borg
-versions are used alternately on the same repositories.
 
 Checksums are a data safety mechanism. They are not a security mechanism.
 
@@ -1299,9 +1339,8 @@ mixed into the checksum state first (encoded as an ASCII string via `%10d`
 printf format), then the name of the part is mixed in as an UTF-8 string.
 Lastly, the current position (length) in the file is mixed in as well.
 
-Borg 2 uses parts only when reading a borg 1.x repository (see ``borg
-transfer``): its index and hints files have a ``HashHeader`` part. The files
-cache is written and read as a single part.
+The index file of a borg 1.x repository has a ``HashHeader`` part, its hints file
+is a single part.
 
 The checksum state is not reset at part boundaries.
 
@@ -1338,35 +1377,13 @@ DetachedIntegrityCheckedFile, which automatically writes and reads it from a
 Upper layer
 ~~~~~~~~~~~
 
-.. rubric:: The files cache
-
-The files cache is the only file borg 2 protects this way. Its integrity data
-is stored in the ``[integrity]`` section of the cache ``config`` file, keyed by
-the file's name (see :ref:`the files cache <cache>` about that name):
-
-.. code-block:: none
-
-    [cache]
-    version = 1
-    repository = 3c4...e59
-
-    [integrity]
-    files.9f8...a08 = {"algorithm": "SHA256", "digests": {"final": "e2a...b24"}}
-
-The chunks index is not in this list: it is not a local file, but lives in the
-repository below ``index/`` and has its own integrity mechanism, see
-:ref:`pack-index-namespace`.
-
-The cache config file is read in its entirety (using the Python ConfigParser),
-modified and written back, so sections and values a Borg version does not
-understand are preserved. There is no guard against an older Borg version
-updating the files cache without updating its integrity data: every Borg
-version that can open a version 5 repository knows the ``[integrity]`` section.
+The integrity data of ``index.N`` and ``hints.N`` is stored in ``integrity.N`` next
+to them, a msgpacked dict with the integrity data version (2) and the integrity data
+of both files (keys ``index`` and ``hints``). Without an ``integrity.N`` file, or if
+its version is unknown, the files are read without checking them.
 
-A files cache that fails its integrity check (or can not be read at all) is
-discarded, not used: borg then rebuilds the files cache from the most recent
-archive of the series in the repository, or, failing that, starts with an empty
-files cache.
+If the index or hints file fails its integrity check, borg rebuilds the index
+and hints from the segments, as for any other error reading them.
 
 
 HardLinkManager and the hlid concept

docs/internals/frontends.rst

@@ -214,7 +214,6 @@ Saving the local cache at the end of :ref:`borg_create`::
 
     {"message": "Saving files cache", "operation": 1, "msgid": "cache.close", "type": "progress_message", "finished": false, "time": 1787900398.719723}
     {"message": "Saving index", "operation": 1, "msgid": "cache.close", "type": "progress_message", "finished": false, "time": 1787900398.728792}
-    {"message": "Saving cache config", "operation": 1, "msgid": "cache.close", "type": "progress_message", "finished": false, "time": 1787900398.7294679}
     {"message": "", "operation": 1, "msgid": "cache.close", "type": "progress_message", "finished": true, "time": 1787900398.739775}
 
 A debug log message::
@@ -1028,7 +1027,7 @@ Warnings
 Operations
     - cache.close
 
-      Saving the local cache (files cache, chunks index, cache config) at the end of a command.
+      Saving the local cache (files cache, chunks index) at the end of a command.
     - cache.merge_chunkindex_fragments
 
       Loading the chunk index by merging the index fragments stored in the repository.

docs/internals/security.rst

@@ -42,6 +42,14 @@ includes the store objects borg keeps next to the packs: the chunk index fragmen
 and the per-archive reference caches are encrypted with the key, too, see
 :ref:`store object envelope <store_object_envelope>`.
 
+The client-local files cache (see :ref:`the files cache <cache>`) is protected with
+the key as well, as a :ref:`sealed stream <sealed_stream>`: someone who can read the
+borg cache directory, but does not have the key, can not read the file sizes,
+timestamps, chunk lists and content digests stored in it (in the encrypting modes),
+and a files cache that was modified is detected in all modes (it is ignored with a
+warning and rebuilt). The names, sizes and timestamps of the files cache files are
+not protected.
+
 The attacker can always impose a denial of service by definition (they could
 block connections to the repository, or delete it partly or entirely).