Reuse cached root images during installation

.github/workflows/build.yml

@@ -32,8 +32,28 @@ jobs:
       - uses: actions/setup-python@v5
         with:
           python-version: '3.12'
+      # Development can require a hardware version before its release exists.
+      - name: Check out pinned development hardware dependency
+        if: github.ref_type != 'tag' && github.event_name != 'workflow_dispatch'
+        uses: actions/checkout@v4
+        with:
+          repository: OrchardLinux/hardware
+          ref: f3f14c91940e3d9ec83db6980abb2e8857fad439
+          path: out/hardware-source
+          persist-credentials: false
+      - name: Build pinned development hardware dependency
+        if: github.ref_type != 'tag' && github.event_name != 'workflow_dispatch'
+        run: |
+          python -m pip install build
+          python -m build --wheel --outdir out/hardware out/hardware-source
+          python - <<'PY' >> "$GITHUB_ENV"
+          from pathlib import Path
+          import tomllib
+          project = tomllib.loads(Path('out/hardware-source/pyproject.toml').read_text())
+          print('HARDWARE_TAG=v' + project['project']['version'])
+          PY
       - name: Download pinned released hardware dependency
-        if: inputs.hardware_run_id == ''
+        if: (github.ref_type == 'tag' || github.event_name == 'workflow_dispatch') && inputs.hardware_run_id == ''
         env:
           GH_TOKEN: ${{ github.token }}
         run: |

README.md

@@ -29,6 +29,18 @@ orchard install omarchy-a
 
 The registered name stores a URL. Each resolution fetches the current index; verified images are cached by SHA-256. Publishing a new distro index updates future installations, not an already installed operating system.
 
+### Reinstalling the same image
+
+With a recovery image that supports root-image caching, Orchard retains a verified
+public root archive on the selected Linux-data partition, outside the installed
+system directory. Reinstalling the same image uses that cache instead of sending
+the archive over USB again. Changed, missing or corrupt images are transferred
+in full. The progress output identifies cache verification and local extraction.
+
+No extra partition is needed. Cache reuse still performs local verification and
+extraction; it is not an in-place upgrade or a backup of the replaced directory.
+Older recovery images continue to use the normal streaming installation path.
+
 ## Launch an installed system
 
 Connect the device in DFU mode, then run:

docs/device-protocol.md

@@ -13,12 +13,12 @@ Images expose `orchard-device` and image-owned generic `ready`. Omarchy-specific
 
 Initial USB serial invokes the versioned `orchard-device bootstrap BASE64_JSON` operation with a public key. The runtime owns authorization, daemon/PID/key-file state and startup; it returns the host key and endpoint. The host only pins that response and waits for authenticated connectivity. No device filesystem or service-management shell snippets belong in the host.
 
-Authenticated USB-network SSH invokes `orchard-device OPERATION BASE64_JSON`. Requests include `protocol: 1`. `prepare` and `install` carry their byte stream on stdin. Other operations do not consume payload data.
+Authenticated USB-network SSH invokes `orchard-device OPERATION BASE64_JSON`. Requests include `protocol: 1`. `prepare` and streamed `install` calls carry their byte stream on stdin. A capability-negotiated cached `install` has no stdin payload. Other operations do not consume payload data.
 
 Stdout is newline-delimited JSON:
 
 - `phase`: named device activity, no fabricated percentage.
-- `progress`: a bounded descriptive message.
+- `progress`: a bounded descriptive message, or structured cached-extraction byte progress.
 - `result`: one completed operation result.
 - `error`: actionable error, with `retry: false`.
 
@@ -33,20 +33,57 @@ All records include `protocol: 1`. Lost connections during mutations mean unknow
 | `prepare` | release, bytes, sha256 + mixin stdin | Verify fresh-ADT binding/data allowlist; apply in RAM, prepare hardware once, and clean owned leftover staging before target selection. |
 | `targets` | none | Boot ID, device-issued selection token, capacity estimate and GPT Linux-data partition/signature descriptions; free space for verified ext4. |
 | `format-target` | token, target ID, boot ID, operation ID, explicit confirmation | Once per recovery boot after preparation: revalidate a Linux-data target without recognized signatures, create ext4 without force/discard, verify UUID and restore fencing. Return a matching receipt; invalidate the old selection. |
-| `destination` | token, target ID, relative directory | Read-only inspection; return existence, destination identity, free/reclaimable bytes, estimated needed bytes and inode sufficiency. No ownership is inferred from installation markers. |
-| `install` | selected target/directory, destination_identity, explicit replace boolean, bytes, sha256 + root stdin | Revalidate selection; stream into fresh staging, verify, apply mixin and promote. |
+| `destination` | token, target ID, relative directory, optional `cache_root: true` | Read-only inspection; return existence, destination identity, free/reclaimable bytes, estimated needed bytes and inode sufficiency. Cache-aware estimates also return separately reclaimable owned cache bytes. No ownership is inferred from installation markers. |
+| `root-cache` | selected target/directory, bytes, sha256 | With `root-image-cache-v1` capability: read-only verification of the exact public root archive. Returns `cached` (boolean), `bytes` and `sha256`. |
+| `install` | selected target/directory, destination_identity, explicit replace boolean, bytes, sha256; optional `root_cache: store` or `reuse` | Revalidate selection; extract into fresh staging, verify, apply mixin and promote. Legacy/`store` calls send root stdin; `reuse` calls send no payload. |
 | `handoff` | none | Recheck the completed installation, preserve diagnostics and ask recovery PID 1 to switch root. |
 | `diagnostics` | none | Phase/failure state and bounded kernel/system logs. |
 | `abort` | none | Device-owned incomplete-staging cleanup, unmount/fence storage, retain used-phase guards and require a fresh recovery boot. |
 | `revoke-ssh` | current profile, ECID, `boot_id`, `release` | Only after successful readiness and matching installed/handoff receipt: retire listener/other connections, clear RAM credentials, return the final receipt over this non-multiplexed connection. No hardware/storage actions. |
 
 Host code must not infer device paths or encode module loads, filesystem offsets, mounts, permissions or PID 1 commands. Target IDs are displayable identifiers, not host-authored storage paths.
 
+## Device root-image cache
+
+Only use cache operations when the installer advertises `root-image-cache-v1`
+in `status.capabilities`. Otherwise the host retains the existing streaming
+workflow. Old hosts may omit all cache fields when using a new runtime.
+
+The host requests cache-aware capacity estimates before confirmation, then
+probes the selected partition using the exact release root hash and size. It
+checks all probe response fields before choosing `reuse` or `store`. It never
+sends device cache paths. A verified hit skips the USB root payload, but not
+local integrity checking or extraction. A miss transfers the whole image;
+there is no delta-update protocol.
+
+The runtime owns the persistent cache outside the installed root and rechecks
+cached bytes before replacement. Store mode retains only a verified public
+archive; private firmware/mixins remain separate. Missing or corrupt completed
+entries are cache misses, while unsafe storage paths fail closed. Incomplete
+entries are never cache hits. Capacity checks include the compressed archive,
+and cleanup is limited to owned cache entries. Replacing the installation
+directory does not remove this cache.
+
+Progress identifies cache verification, uploading and local extraction. During
+cached extraction, optional `progress` records carry `phase: cached-extraction`,
+integer `bytes` and `total` for compressed bytes consumed, and boolean `complete`.
+An initial zero starts the meter; intermediate counts are monotonic and below
+the fixed total. Only a successful extractor completion emits the total with
+`complete: true`. This does not mean installation finalization or boot has
+completed. The host labels the rate as cached extraction, not USB transfer,
+limits terminal refreshes and leaves failed/unfinished operations below 100%.
+Old phase-only runtimes remain supported; malformed or unknown events stay in
+diagnostics rather than producing a misleading meter.
+
+A failed or disconnected install is never automatically retried with a stream,
+even after a successful cache probe. Cache support is not an installation backup
+or a change to replacement confirmation semantics.
+
 ## Streaming safety
 
 - Root size, digest, compression and expansion limit are baked into recovery from the public release receipt.
 - Package-defined privileged entries have a public metadata manifest. Producers validate contents before restoring special modes from pacman MTREE; runtime verifies content, UID/GID and modes before activation. Missing `sudo` setuid/root ownership is a finalization error, not a broken system to promote.
-- Decode Zstandard and extract concurrently with incoming bytes; no temporary compressed archive.
+- Decode Zstandard and extract concurrently with incoming bytes. Negotiated `store` mode also retains a temporary compressed archive and promotes it only after validation; legacy streaming retains no archive. `reuse` extracts a verified local archive instead.
 - All file writes use no-follow directory descriptors; links are deferred. Traversal, duplicate paths, special files, reserved metadata and escaping hardlinks are rejected.
 - Verify compressed-byte count/digest and decoder completion before creating links or promoting. Preserve file ownership, modes and timestamps.
 - Replacement requires explicit approval and matching destination identity. Delete the selected directory before extraction; never create an automatic backup. Reject symlinks, non-directories and nested mounts. Estimate required bytes as expanded payload × 1.2 + 1 GiB and check free inodes (archive member count + 1024; legacy fallback 65536 + 1024). Recheck actual capacity after deletion. Failures must warn that replacement deletion cannot be undone.

src/orchard_cli/progress.py

@@ -50,6 +50,8 @@ def __init__(self, output=None, *, verbose=False):
         self.rows = {}
         self.rendered = 0
         self.log_path = None
+        self.device_phase = None
+        self.device_progress = None
 
     def task(self, name):
         return TaskProgress(self, name)
@@ -66,7 +68,7 @@ def phase(self, message):
             self.end_line()
             print(message, file=self.output, flush=True)
 
-    def diagnostic(self, message):
+    def diagnostic(self, message, *, display=True):
         with self.lock:
             if self.log_path is None:
                 from .cache import cache_root
@@ -76,7 +78,7 @@ def diagnostic(self, message):
                 os.close(fd)
             with open(self.log_path, 'a') as stream:
                 stream.write(str(message) + '\n')
-            if self.verbose:
+            if self.verbose and display:
                 self.phase(str(message))
 
     def transfer(self, sent, total):
@@ -102,8 +104,58 @@ def reset_transfer(self):
             self.end_line()
             self.started, self.last, self.initial = time.monotonic(), None, None
 
+    def reset_device_progress(self):
+        """End a device meter without inventing completion, including on RPC failure."""
+        with self.lock:
+            if self.device_progress is not None:
+                self.end_line()
+            self.device_phase = self.device_progress = None
+
+    def cached_extraction(self, record):
+        """Render validated compressed-byte consumption, not USB transfer speed."""
+        if self.device_phase != 'cached-extraction':
+            return False
+        count, total, complete = record.get('bytes'), record.get('total'), record.get('complete')
+        if (type(count) is not int or type(total) is not int or type(complete) is not bool or
+                not 0 < total <= 2**63 - 1 or not 0 <= count <= total or
+                (complete and count != total) or (not complete and count == total)):
+            return False
+        now = time.monotonic()
+        with self.lock:
+            state = self.device_progress
+            if state is None:
+                if count != 0 or complete:
+                    return False
+                state = self.device_progress = {'bytes': 0, 'total': total, 'started': now,
+                                               'last': None, 'complete': False}
+            elif (state['total'] != total or count < state['bytes'] or state['complete']):
+                return False
+            state['bytes'], state['complete'] = count, complete
+            interval = 0.2 if self.interactive else 10
+            if not complete and state['last'] is not None and now - state['last'] < interval:
+                return True
+            state['last'] = now
+            message = 'Cached extraction  ' + transfer_text(count, total, state['started'], 0, now,
+                                                          complete=complete)
+            if self.interactive:
+                print('\r\033[2K' + message, end='', file=self.output, flush=True)
+                self.active_line = True
+                if complete:
+                    self.end_line()
+            else:
+                print(message, file=self.output, flush=True)
+        return True
+
     def event(self, record):
-        # Internal hardware events are diagnostics, not user-facing phases.
+        if record.get('event') == 'phase':
+            self.reset_device_progress()
+            self.device_phase = record.get('phase')
+        if (record.get('event') == 'progress' and record.get('phase') == 'cached-extraction' and
+                self.cached_extraction(record)):
+            # Keep structured diagnostics without interrupting the live meter in verbose mode.
+            self.diagnostic(str(record), display=False)
+            return
+        # Old and unknown hardware events retain their diagnostic-only behavior.
         self.diagnostic(str(record))
 
     def row(self, name, message, *, complete=False):
@@ -120,11 +172,12 @@ def row(self, name, message, *, complete=False):
                 print(f'  {name:<20.20} {message}', file=self.output, flush=True)
 
 
-def transfer_text(sent, total, started, initial, now):
+def transfer_text(sent, total, started, initial, now, *, complete=None):
     unit, divisor = ('KiB', 1024) if total < 1048576 else ('MiB', 1048576)
     rate = max(0, sent - initial) / max(now - started, 0.001)
     eta = f'{(total - sent) / rate:.0f}s' if rate and sent < total else '—'
-    return (f'{sent * 100 / max(total, 1):3.0f}%  '
+    percent = sent * 100 / max(total, 1) if complete is None else (100 if complete else sent * 100 // total)
+    return (f'{percent:3.0f}%  '
             f'{sent / divisor:5.0f} / {total / divisor:5.0f} {unit}  '
             f'{rate / 1048576:5.1f} MiB/s  {eta:>5}')
 
@@ -156,5 +209,5 @@ def transfer(self, sent, total):
         self.parent.row(self.name, transfer_text(sent, total, self.started, self.initial, now),
                         complete=sent == total or not self.parent.interactive)
 
-    def diagnostic(self, message):
-        self.parent.diagnostic(message)
+    def diagnostic(self, message, *, display=True):
+        self.parent.diagnostic(message, display=display)

src/orchard_cli/protocol.py

@@ -21,14 +21,15 @@ def __init__(self, connection, progress):
         self.ssh, self.progress = connection.argv, progress
 
     def call(self, operation, request=None, *, source=None, size=0, timeout=120):
-        if operation not in ('status', 'prepare', 'targets', 'destination', 'install', 'handoff', 'diagnostics', 'abort', 'bootstrap', 'revoke-ssh', 'format-target', 'installations', 'launch'):
+        if operation not in ('status', 'prepare', 'targets', 'destination', 'root-cache', 'install', 'handoff', 'diagnostics', 'abort', 'bootstrap', 'revoke-ssh', 'format-target', 'installations', 'launch'):
             raise ValueError('Unsupported phase operation')
         try:
             self.connection.check()
         except EndpointUnavailable as error:
             raise DeviceError(str(error)) from error
         payload = base64.urlsafe_b64encode(json.dumps({'protocol': PROTOCOL, **(request or {})}).encode()).decode()
         argv = self.ssh + ['/usr/bin/orchard-device ' + operation + ' ' + payload]
+        self.progress.reset_device_progress()
         process = subprocess.Popen(argv, stdin=subprocess.PIPE if source else subprocess.DEVNULL,
                                    stdout=subprocess.PIPE, stderr=subprocess.PIPE)
         records, errors, stderr = [], [], []
@@ -89,6 +90,7 @@ def collect_errors():
             if process.poll() is None:
                 process.kill()
                 process.wait()
+            self.progress.reset_device_progress()
             for stream in (process.stdin, process.stdout, process.stderr):
                 if stream:
                     stream.close()