cockpit: use the JSON API, with per-command screens and docs; JSON API additions and fixes, #9454

docs/changes.rst

@@ -197,6 +197,12 @@ New features:
 - mount: support Windows using WinFsp (via mfusepy), #2316
 - import-tar --strip-components: strip leading path components, #6461
 - add the BORG_NEW_PASSCOMMAND and BORG_NEW_PASSPHRASE_FD env vars
+- extract/export-tar --list --log-json: output a file_status JSON object per listed item,
+  like create does. The text listing of export-tar has the same "+" prefix as extract's now.
+  prune/delete/undelete --list --log-json: output an archive_status JSON object per listed
+  archive, #9454.
+- create/recreate --dry-run --progress: show the progress (also as archive_progress JSON
+  objects with --log-json), it showed nothing
 
 Fixes:
 
@@ -247,6 +253,14 @@ Fixes:
 - locking: try at least once before a lock acquire times out, also with --lock-wait 0
 - repoobj: catch get() errors in --find-lost-archives, #10318
 
+- recreate/transfer --log-json --progress: output archive_progress JSON objects (as
+  documented), not text progress lines
+- import-tar/recreate: count the items by their status, like create does. The counts
+  were always 0: "Added files" of import-tar --stats, files_stats in the JSON output.
+- --log-json: the final archive_progress object (finished: true) has the final statistics.
+  The progress output is rate limited, so a frontend could not know what was processed
+  after the previous object (or at all, for a short operation), see also #6570.
+
 Other changes:
 
 - update pyinstaller to 6.22.3
@@ -274,6 +288,12 @@ Other changes:
   but was never really used)
 - security: drop the manifest timestamp replay check (not needed anymore)
 - debug get-obj, put-obj, delete-obj: need the key now (to access the chunk index)
+- cockpit: process borg's --log-json output (progress, file list, log messages, prompts)
+  instead of parsing text lines, #9454. The display depends on the command: archive
+  statistics for create/import-tar/recreate/transfer (with the final statistics from
+  --json), a progress bar for extract/export-tar, the progress phases for the other
+  commands. Yes/no prompts are shown as a dialog. The cockpit exits with the exit code
+  of the borg command.
 - docs:
 
   - extract: document the metadata that can only be restored as root, #8088
@@ -283,6 +303,7 @@ Other changes:
   - derive the borg passphrase from a YubiKey (challenge-response), #4549
   - protect the borg passphrase with age (which also supports crypto tokens,
     TPM, Apple Secure Enclave, ... via age plugins), #4549
+  - add a usage page for the cockpit TUI, #9454
 - tests:
 
   - add an archiver-level test for BORG_WORKAROUNDS=authenticated_no_key

docs/installation.rst

@@ -193,7 +193,7 @@ development header files (sometimes in a separate `-dev` or `-devel` package).
   - borgstore[rest,blake3,sftp] ~= 0.7.0 (use `pip install borgbackup[sftp]`)
 * Optionally, if you wish to use rclone Backend:
   - borgstore[rest,blake3,rclone] ~= 0.7.0 (use `pip install borgbackup[rclone]`)
-* Optionally, if you wish to use the TUI (``borg --cockpit``):
+* Optionally, if you wish to use the cockpit TUI (``borg --cockpit``, see :ref:`cockpit`):
   - textual >= 6.8.0 (use `pip install borgbackup[cockpit]`)
 
 If you have troubles finding the right package names, have a look at the

docs/internals/frontends.rst

@@ -90,13 +90,15 @@ it is not produced unless ``--progress`` is specified.
 archive_progress
     Output during operations creating archives (:ref:`borg_create`, :ref:`borg_import-tar`,
     :ref:`borg_recreate` and :ref:`borg_transfer`).
-    The following keys exist, each represents the current progress.
+    The following keys exist, each represents the current progress. The output is rate limited, so
+    only the last object (*finished* is *true*) tells about everything that was processed.
 
     original_size
         Original size of the data processed so far (before compression and deduplication)
     deduplicated_size
         Deduplicated size of the data processed so far (before compression): the size of the
-        chunks that were new to the repository
+        chunks that were new to the repository. Absent for a ``--dry-run`` of :ref:`borg_create`
+        or :ref:`borg_recreate`: it only knows the number of files and the original size.
     nfiles
         Number of (regular) files processed so far
     hashing_time
@@ -105,7 +107,8 @@ archive_progress
         Seconds spent chunking file contents so far (float)
     files_stats
         Object mapping the single-character file status (as used by ``--list``) to the number of
-        files that got that status so far, e.g. ``{"A": 3, "d": 3}``
+        items that got that status so far, e.g. ``{"A": 3, "d": 3}``. It is empty for
+        :ref:`borg_transfer`, which has no file status.
     store_stats
         Object with the storage backend statistics. It is empty here, it is only filled in for the
         final :ref:`borg_create` ``--json`` output on *stdout*, see `Archive formats`_.
@@ -115,8 +118,8 @@ archive_progress
         Unix timestamp (float)
     finished
         boolean indicating whether the operation has finished, only the last object for an *operation*
-        can have this property set to *true*. That last object has no keys besides *time*, *type*
-        and *finished*.
+        can have this property set to *true*. That last object has the final statistics of the
+        archive and no *path*.
 
 progress_message
     A message-based progress information with no concrete progress information, just a message
@@ -158,15 +161,49 @@ progress_percent
         Unix timestamp (float)
 
 file_status
-    This is only output by :ref:`borg_create`, :ref:`borg_import-tar` and :ref:`borg_recreate` if
-    ``--list`` is specified. The usual rules for the file listing applies, including the
-    ``--filter`` option.
+    One object per listed item, output by :ref:`borg_create`, :ref:`borg_import-tar`,
+    :ref:`borg_recreate`, :ref:`borg_extract` and :ref:`borg_export-tar` if ``--list`` is
+    specified. The usual rules for the file listing apply, including the ``--filter`` option.
 
     status
-        Single-character status as for regular list output
+        Single-character status as for regular list output: the item flags of :ref:`borg_create`,
+        or ``+`` (item extracted / exported) and ``-`` (item excluded) for :ref:`borg_extract`
+        and :ref:`borg_export-tar`.
     path
         Path of the file system object
 
+archive_status
+    One object per listed archive, output by :ref:`borg_prune` (``--list``, ``--list-kept`` and
+    ``--list-pruned``), :ref:`borg_delete` and :ref:`borg_undelete` (``--list``). With ``--json``,
+    :ref:`borg_prune` outputs the archives on *stdout* instead.
+
+    status
+        *kept* or *pruned* (:ref:`borg_prune`), *deleted* (:ref:`borg_delete`) or *undeleted*
+        (:ref:`borg_undelete`). With ``--dry-run``, this is what would be done.
+    dry_run
+        *true* for a ``--dry-run``: *status* tells what would be done, nothing was changed
+    name, archive
+        Name of the archive
+    id
+        Archive ID (hex)
+    time
+        Archive timestamp
+    message
+        The text line of the ``--list`` output, e.g. *Keeping archive (rule: daily #1): ...*
+
+    :ref:`borg_prune` additionally gives the keys of the archive objects of ``borg prune --json``:
+    the keys requested via ``--format`` and
+
+    group
+        Object mapping the ``--group-by`` keys to the values of this archive
+    kept
+        *true* if the archive is kept, *false* if it is pruned
+    keep_rule, kept_oldest, kept_archive_number
+        For a kept archive: the rule keeping it (e.g. *daily*), whether it is the oldest archive kept
+        by the rule, and its number within the rule (1 = the most recent one)
+    deleted_archive_number
+        For a pruned archive: its number among the pruned archives (1 = the first one pruned)
+
 log_message
     Any regular log output invokes this type. Regular log options and filtering applies to these as well.
 
@@ -208,7 +245,35 @@ See Prompts_ for the types used by prompts.
     {"type": "file_status", "status": "A", "path": "src/linux/file1"}
     {"type": "file_status", "status": "d", "path": "src/linux"}
     {"type": "file_status", "status": "d", "path": "src"}
-    {"time": 1787900398.686938, "type": "archive_progress", "finished": true}
+    {"original_size": 250012, "deduplicated_size": 250012, "nfiles": 3, "hashing_time": 0.002,
+     "chunking_time": 0.001, "files_stats": {"A": 3, "d": 3}, "store_stats": {}, "time": 1787900398.686938,
+     "type": "archive_progress", "finished": true}
+
+:ref:`borg_extract` file listing, with ``--exclude src/linux/baz/file3``::
+
+    {"type": "file_status", "status": "+", "path": "src"}
+    {"type": "file_status", "status": "+", "path": "src/linux"}
+    {"type": "file_status", "status": "+", "path": "src/linux/baz"}
+    {"type": "file_status", "status": "+", "path": "src/linux/baz/file2"}
+    {"type": "file_status", "status": "-", "path": "src/linux/baz/file3"}
+    {"type": "file_status", "status": "+", "path": "src/linux/file1"}
+
+:ref:`borg_prune` archive listing, with ``--list --dry-run --keep-daily=1``::
+
+    {"name": "daily", "archive": "daily", "id": "2c77c68a...", "time": "2026-09-09T02:00:00.000000+02:00",
+     "group": {"name": "daily", "host": "host"}, "kept": true, "keep_rule": "daily", "kept_oldest": false,
+     "kept_archive_number": 1, "status": "kept", "dry_run": true, "type": "archive_status",
+     "message": "Keeping archive (rule: daily #1):            daily   Wed, 2026-09-09 02:00:00 +0200 [2c77c68a...]"}
+    {"name": "daily", "archive": "daily", "id": "99a5671a...", "time": "2026-09-08T02:00:00.000000+02:00",
+     "group": {"name": "daily", "host": "host"}, "kept": false, "deleted_archive_number": 1, "status": "pruned",
+     "dry_run": true, "type": "archive_status",
+     "message": "Would prune:                                 daily   Tue, 2026-09-08 02:00:00 +0200 [99a5671a...]"}
+
+:ref:`borg_delete` archive listing, with ``--list``::
+
+    {"name": "daily", "archive": "daily", "id": "99a5671a...", "time": "2026-09-08T02:00:00.000000+02:00",
+     "status": "deleted", "dry_run": false, "type": "archive_status",
+     "message": "Deleted archive: daily   Tue, 2026-09-08 02:00:00 +0200 [99a5671a...] (1/1)"}
 
 Saving the local cache at the end of :ref:`borg_create`::
 

docs/usage.rst

@@ -35,6 +35,7 @@ Usage
 
 .. toctree::
    usage/general
+   usage/cockpit
 
    usage/repo-create
    usage/repo-space

docs/usage/cockpit.rst

@@ -0,0 +1,105 @@
+.. highlight:: none
+.. _cockpit:
+
+Cockpit
+-------
+
+The cockpit is a full-screen terminal user interface showing what a borg command does
+while it runs: its progress, its statistics, the file list and the log messages, all
+updated live. To use it, put ``--cockpit`` in front of the command::
+
+    $ borg --cockpit -r /path/to/repo create --list my-files ~/Documents
+    $ borg --cockpit -r /path/to/repo extract --list my-files
+    $ borg --cockpit -r /path/to/repo check --repair
+
+The cockpit needs the ``textual`` package: ``pip install borgbackup[cockpit]`` installs
+it, the binary releases include it (see :ref:`installation`). It needs a terminal of at
+least 80x24 characters, a taller terminal gives the log more room. It does not start if
+stdin, stdout or stderr is not a terminal (e.g. when run by cron or with redirected
+output): it is an interactive display and stays on the screen until you quit it.
+
+How it works
+~~~~~~~~~~~~
+
+The cockpit runs the borg command as a subprocess with ``--log-json`` and ``--progress``
+added, and builds its display from the JSON output borg produces for frontends, see
+:ref:`json_output`. Apart from that, the command runs exactly like it does without
+``--cockpit``, with the options you gave it.
+
+.. note::
+
+    ``--progress`` makes ``extract`` and ``export-tar`` read the archive metadata once
+    more before they start, to determine the total amount of data for the progress bar,
+    so they start a bit later than usual, see :ref:`borg_extract`.
+
+The lower part of the screen is the log: borg's messages, warnings and errors, the file
+list if you gave ``--list``, and everything else borg outputs. Control characters (a file
+name can contain them, e.g. an ESC starting a terminal escape sequence) are shown as the
+replacement character (U+FFFD) everywhere in the cockpit, so they can not affect the
+terminal. The panel in the upper right depends on the command:
+
+``create``, ``import-tar``, ``recreate``, ``transfer``
+    The statistics of the archive being created: the number of files, the original and
+    the deduplicated size, the counts of added, modified and unchanged files, the path
+    being processed and the throughput in files and bytes per second, with a history
+    graph. For ``create`` and ``import-tar``, the exact final statistics of the new
+    archive (what ``--stats`` prints) are shown in the panel and in the log when borg
+    has finished.
+
+    These numbers are the statistics borg reports. They do not depend on ``--list`` and
+    ``--filter``, which only determine what the log shows. A ``-`` means that borg does
+    not report that number: ``transfer`` has no counts by status, and a ``--dry-run``
+    only reports the number of files and the original size.
+
+``extract``, ``export-tar``
+    A progress bar with the percentage and the estimated remaining time, the amount of
+    data extracted so far, the throughput and the counts of the ``--list`` lines.
+
+All other commands
+    The phases of the operation borg reports progress for, e.g. "Checking index" and
+    "Checking archives" for ``check``, each with a progress bar. For ``prune``, ``delete``
+    and ``undelete`` with ``--list``, also the numbers of kept, pruned, deleted or undeleted
+    archives (marked as "dry-run" if nothing was changed).
+
+Every panel also shows the elapsed time, the number of warnings and errors and, when
+borg has finished, its exit code. The cockpit stays on the screen until you press ``q``,
+so you can have a look at the log and the numbers. It then exits with the exit code of
+the borg command, see :ref:`return_codes`.
+
+Prompts and passphrases
+~~~~~~~~~~~~~~~~~~~~~~~
+
+When borg asks a yes/no question (e.g. ``check --repair`` asks whether you know what you
+are doing), the cockpit shows a dialog: answer with the YES or NO button, or type another
+answer into the input field.
+
+The cockpit can not enter a passphrase. Give it to borg via the environment, e.g. by
+setting ``BORG_PASSPHRASE`` or ``BORG_PASSCOMMAND`` (see :ref:`env_vars`). Otherwise the
+cockpit shows a hint that borg is waiting for a passphrase, and you have to quit and try
+again.
+
+Commands the cockpit can not run
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The cockpit uses the terminal and borg's stdin is connected to the cockpit (that is how
+the answers to prompts get to borg). Thus, the cockpit refuses to run:
+
+- commands reading from stdin: ``create`` with ``-`` as a path or with
+  ``--paths-from-stdin``, ``import-tar`` reading from ``-``, ``key import`` reading from
+  ``-`` or with ``--paper``, and ``serve``.
+- commands writing their data to stdout: ``extract --stdout`` and ``export-tar`` writing
+  to ``-``.
+
+Keys
+~~~~
+
+``q`` (or Ctrl-C)
+    Quit. If borg is still running, quitting means terminating it, so the cockpit asks
+    for confirmation first: ``y`` terminates borg (SIGTERM), waits until it has exited and
+    quits; ``n``, Escape or Enter continue. When the cockpit gets a SIGTERM, SIGHUP or
+    SIGINT signal (e.g. because its terminal window gets closed), it terminates borg,
+    waits and exits without asking.
+
+``t``
+    Toggle the universal translator: the labels are shown in Borg speak. Resistance is
+    futile.

docs/usage_general.rst.inc

@@ -14,6 +14,8 @@
 
 .. include:: usage/general/logging.rst.inc
 
+.. _return_codes:
+
 .. include:: usage/general/return-codes.rst.inc
 
 .. include:: usage/general/config.rst.inc