Harden first-time Core installation, cleanup and progress

deploy/install/README.md

@@ -41,7 +41,7 @@ This directory holds the Core/Web installer, the `oac` command and the node inst
 
 ## Versions and the lock
 
-- Install only into an empty directory or over an [incomplete installation](#new-installations), or repair a complete installation of the same source revision. Require the current state format and matching source revision before changing installation files or services. Preserve data when those checks fail.
+- Install only into an empty directory or over an [incomplete installation](#new-installations), or repair a complete installation of the same source revision. Require the current state format before changing installation files or services, and a matching source revision when repairing a complete installation. An incomplete installation of that state format is removed before installing the selected release, even when its source revision differs. Preserve data when those checks fail.
 - The packaged `oac.pyz` embeds its build revision and refuses a `state.json` whose `source_commit` differs.
 - The installer and every mutating `oac` command share `.oac.lock`. The installer holds it across creation, payload, native service and launcher repair, and apply, calling the already-locked apply implementation without locking again. Never replace the lock file; its inode must stay stable. Only the cleanup of a new installation, with the directory the installer created, and `oac uninstall` unlink it, last and while holding it. `locked` refuses a lock whose path no longer names the file it locked.
 
@@ -50,7 +50,7 @@ This directory holds the Core/Web installer, the `oac` command and the node inst
 - An installation is complete after its first start: apply converged, the services are healthy and, for Web-only, its Core accepts the Core key. `create()` writes `state.json` before anything else, atomically, with the project name and `"complete": false`, and the installer sets it to `true` at that moment. A later failure, such as a refused sandbox selection, keeps the installation.
 - Until then, `oac apply`, `start`, `domain` and `rotate-core-key` and Web's domain setup refuse the installation and point to the installer and `oac uninstall`; `oac status` reports it. Only the installer starts an incomplete installation; the installer or `oac uninstall` removes it.
 - Any failure or interrupt (Ctrl-C, SIGTERM, SIGHUP) of an installation that is not complete runs `oac_cli.remove` before anything is printed, so a closed terminal can't stop it. The original error then goes on, and the installer prints it followed by `Nothing was kept; fix the problem and rerun the same command.` `remove` disables native Core's unit, runs `docker compose -p <project> down --volumes --remove-orphans` from `/` without `COMPOSE_*` variables and with its output discarded, and deletes every file in the installation directory. The directory goes too when the installer created it; otherwise it stays with its lock. Loaded images stay.
-- The release downloader execs the installer, so the installer itself receives Ctrl-C, SIGTERM and SIGHUP and nothing stops it while it removes a failed installation.
+- The release downloader waits for the bundled installer in a separate process group. It forwards the first Ctrl-C, SIGTERM or SIGHUP and waits for cleanup before removing the temporary bundle; repeated signals do not interrupt that wait. The child inherits the download lock, so killing the downloader alone cannot expose its active files to another download.
 - `remove` touches only the `oac-<10 hex digits>` project and its `<project>-core.service` named in `state.json`, and never follows a link. `state.json`, then the `oac` command, then `.oac.lock` are removed last, and the files stay when a service can't be removed, so the next run still recognizes the installation and no other command locks it afresh mid-removal. Those final unlinks ignore SIGINT, SIGTERM and SIGHUP; if the command cannot be unlinked, cleanup restores `state.json` for a retry. SIGKILL or power loss during those final unlinks can leave files that need manual removal. The error lists what is left and the commands that remove it; the printed Compose command uses the same directory and environment isolation as automatic cleanup.
 - A rerun over an incomplete installation, a `state.json` without `"complete": true`, removes it the same way under the lock and then checks the settings and ports and installs with the flags given now. Only that explicit marker proves completion. A directory without `state.json` is refused whatever it holds, and nothing in it is removed.
 
@@ -89,7 +89,7 @@ A default combined Docker installation adds two Compose services from one pinned
 
 ## Output
 
-- Progress describes the operation about to run. Do not imply fresh health checks on a no-change repair.
+- Progress describes the operation about to run. Release downloads and archive hashing show byte percentages; extraction shows bytes processed, and quiet bundle verification and image loading show elapsed time. Dynamic output is terminal-only; redirected logs stay plain. Do not imply fresh health checks on a no-change repair.
 - Terminal styling is optional: honor `NO_COLOR` and keep redirected logs plain.
 - Summaries show credential file locations, never their values.
 - `install_display.py` owns shared terminal formatting; `install_output.py` and `node_output.py` own the completion guidance. Ship and checksum the display modules in both the node bootstrap and its retained helper.
@@ -120,6 +120,8 @@ The distribution manifest is the one download contract for the Core and node ins
 - Only artifact requests may follow HTTPS redirects, and only without credentials or cookies. Metadata and enrollment requests stay on the configured console. The console publishes only fixed non-secret files and declared artifact names.
 - Download into private temporary files, verify size and SHA-256 before an atomic rename, resume interrupted transfers, and reuse only verified cache entries or exact image identities. Never select a release other than the pinned one.
 - Python zipapps bundle the shared resolver with each remote bootstrap. The node asset is the `oac-node` binary.
+- The public release bootstrap retries transient network failures up to three times and resumes an interrupted asset within that run, using its immutable ID, declared size and final checksum. It checks free space before downloading and before extracting each file. These checks cannot reserve storage or measure a remote Docker engine's disk.
+- The bootstrap serializes downloads with `~/.oac/releases/.download.lock`. It discards the private `.download/` staging directory left by an interrupted run before accessing the network, never another active downloader's files. Failed downloads and installations remove staging; only a successful installation retains its verified bundle for repair. Loaded Docker images follow the installation cleanup rule above.
 - Release downloads are anonymous. Never add repository credentials to installed node or Runtime configuration.
 - Manual builds use the `build-<full SHA>` release tag and tag builds the `v*` tag. The manifest's download base must match the release tag; artifact file names and source provenance keep the full source SHA.
 

deploy/install/install.py

@@ -8,6 +8,7 @@
 import argparse
 import base64
 import contextlib
+import errno
 import hashlib
 import ipaddress
 import json
@@ -75,6 +76,10 @@ def error_text(error):
         text = str(error)
     elif isinstance(error, KeyboardInterrupt):
         text = "interrupted"
+    elif isinstance(error, OSError) and error.errno in (errno.ENOSPC, errno.EDQUOT):
+        text = "Disk space or quota exhausted; free space on the installation filesystem and rerun"
+    elif isinstance(error, PermissionError):
+        text = "Permission denied; use a directory writable by your current account (--install-dir)"
     else:
         text = "inspect prerequisites and private deployment files"
     removal = getattr(error, "removal", None)
@@ -352,7 +357,11 @@ def read_core_key_file(source):
 
 
 def check_compose():
-    version = run(["docker", "compose", "version", "--short"], capture_output=True, text=True).stdout.strip()
+    try:
+        version = run(["docker", "compose", "version", "--short"], capture_output=True, text=True,
+                      timeout=30).stdout.strip()
+    except (OSError, subprocess.SubprocessError):
+        raise InstallError("Docker with the Compose plugin is required; check docker compose version") from None
     match = re.fullmatch(r"v?(\d+)\.(\d+)\.(\d+)(?:[-+].*)?", version)
     if not match or tuple(map(int, match.groups())) < (2, 26, 0):
         raise InstallError("Docker Compose 2.26.0 or newer is required for literal Core environment values")
@@ -362,7 +371,11 @@ def check_host():
     if platform.system() != "Linux" or platform.machine() not in ("x86_64", "amd64"):
         raise InstallError("Core installation requires Linux amd64 with Docker access")
     check_compose()
-    run(["docker", "info", "--format", "{{.ServerVersion}}"], stdout=subprocess.DEVNULL)
+    try:
+        run(["docker", "info", "--format", "{{.ServerVersion}}"], capture_output=True, timeout=30)
+    except (OSError, subprocess.SubprocessError):
+        raise InstallError("Cannot reach Docker as the current account. Check docker info, the daemon and this "
+                           "account's Docker access; the installer does not require root or invoke sudo") from None
 
 
 def image_names(mode, native, managed=False):
@@ -376,8 +389,10 @@ def image_loader(manifest, bundle):
     def load(names):
         images = {}
         for name in names:
-            step("Preparing " + {"database": "PostgreSQL", "core": "Core", "web": "Web"}.get(name, name) + " image")
-            images[name] = ensure_docker_image(manifest, name, lambda name=name: bundle / f"images/{name}.tar")
+            message = "Preparing " + {"database": "PostgreSQL", "core": "Core", "web": "Web"}.get(name, name) + " image"
+            step(message)
+            with install_display.busy(message):
+                images[name] = ensure_docker_image(manifest, name, lambda name=name: bundle / f"images/{name}.tar")
         return images
     return load
 
@@ -661,7 +676,8 @@ def main(argv=None):
         step("Checking installation settings")
         prepared = prepare_fresh(args)
     step("Verifying installation files")
-    manifest = verify_bundle(bundle)
+    with install_display.busy("Verifying installation files"):
+        manifest = verify_bundle(bundle)
     # Refuse foreign state before even creating a lock; repeat under the lock to
     # protect against another current installer finishing between these reads.
     check_release(root, manifest)
@@ -677,7 +693,7 @@ def main(argv=None):
 def check_release(root, manifest):
     if (root / "state.json").exists():
         state = oac_cli.load_state(root)
-        if state.get("source_commit") != manifest["source_commit"]:
+        if state.get("complete") is True and state.get("source_commit") != manifest["source_commit"]:
             raise InstallError(oac_cli.UNSUPPORTED_VERSION)
 
 
@@ -768,7 +784,7 @@ def interrupted(signum, frame):
 if __name__ == "__main__":
     # SIGTERM and SIGHUP, such as from a dropped SSH session, stop the installer as Ctrl-C does,
     # so a new installation still removes what it created.
-    for signum in (signal.SIGTERM, signal.SIGHUP):
+    for signum in (signal.SIGINT, signal.SIGTERM, signal.SIGHUP):
         signal.signal(signum, interrupted)
     try:
         main()

deploy/install/install_display.py

@@ -1,7 +1,10 @@
 """Shared terminal presentation for the Core and node installers."""
 import os
+import contextlib
 import sys
 import textwrap
+import threading
+import time
 
 
 def color(message, code, stream=None):
@@ -15,6 +18,36 @@ def step(message):
     print("\n" + color("==> " + message + "...", "36"), flush=True)
 
 
+@contextlib.contextmanager
+def busy(message):
+    """Show elapsed time for a quiet operation whose completion percentage is unknown."""
+    stream = sys.stderr
+    if not stream.isatty() or os.environ.get("TERM") == "dumb":
+        yield
+        return
+    stopped = threading.Event()
+    started = time.monotonic()
+
+    def update():
+        frames = "|/-\\"
+        while not stopped.wait(0.5):
+            try:
+                elapsed = int(time.monotonic() - started)
+                print(f"\r{frames[elapsed % 4]} {message} ({elapsed}s)", end="\033[K", file=stream, flush=True)
+            except OSError:
+                return
+
+    worker = threading.Thread(target=update, daemon=True)
+    worker.start()
+    try:
+        yield
+    finally:
+        stopped.set()
+        worker.join()
+        with contextlib.suppress(OSError):
+            print("\r\033[K", end="", file=stream, flush=True)
+
+
 def heading(message):
     print("\n" + color(message, "1"))
 

deploy/install/test_install.py

@@ -2,6 +2,7 @@
 """install.sh acceptance behavior: fresh install, repair and bundle checks."""
 
 import contextlib
+import errno
 import io
 import json
 import os
@@ -108,6 +109,44 @@ def test_host_check_accepts_current_account_including_root(self):
                     ["docker", "info", "--format", "{{.ServerVersion}}"],
                 ])
 
+    def test_host_failures_explain_the_required_action_without_secrets(self):
+        for command, failure, message in (
+                ("compose", FileNotFoundError(), "Compose plugin"),
+                ("info", PermissionError("private detail"), "current account"),
+                ("info", subprocess.TimeoutExpired(["docker", "info"], 30), "current account")):
+            with self.subTest(command=command, failure=failure):
+                original = self.host.run
+                def run(args, **kwargs):
+                    if command in args:
+                        raise failure
+                    return original(args, **kwargs)
+                with mock.patch.object(subprocess, "run", side_effect=run):
+                    with self.assertRaisesRegex(install.InstallError, message) as raised:
+                        self.install()
+                self.assertNotIn("private detail", str(raised.exception))
+                self.assertFalse(self.root.exists())
+
+    def test_space_and_permission_errors_are_actionable_and_leave_no_installation(self):
+        for error, message in ((OSError(errno.ENOSPC, "full"), "Disk space"),
+                               (OSError(errno.EDQUOT, "quota"), "quota"),
+                               (PermissionError("private detail"), "current account")):
+            with self.subTest(error=error), mock.patch.object(install, "create", side_effect=error):
+                with self.assertRaises(OSError) as raised:
+                    self.install()
+                self.assertIn(message, install.error_text(raised.exception))
+                self.assertIn(install.NOTHING_KEPT, install.error_text(raised.exception))
+                self.assertFalse(self.root.exists())
+
+    def test_new_release_cleans_an_incomplete_installation_before_starting_again(self):
+        self.install()
+        state = self.document("state.json")
+        previous_id = state["installation_id"]
+        install.oac_cli.save_state(self.root, dict(state, complete=False, source_commit="b" * 40))
+        self.install("--ingress", "external")
+        self.assertNotEqual(self.document("state.json")["installation_id"], previous_id)
+        self.assertTrue(self.document("state.json")["complete"])
+        self.assertTrue(any("down" in command for command in self.host.commands))
+
     def test_root_identity_is_preserved_in_service_configuration(self):
         with mock.patch.object(install.os, "getuid", return_value=0), \
                 mock.patch.object(install.os, "getgid", return_value=0):

deploy/install/test_install_output.py

@@ -2,6 +2,7 @@
 import contextlib
 import io
 import os
+import threading
 from pathlib import Path
 import unittest
 from unittest import mock
@@ -16,6 +17,28 @@ def isatty(self):
 
 
 class OutputTests(unittest.TestCase):
+    def test_busy_indicator_stops_after_failure_and_keeps_logs_plain(self):
+        for stream in (Terminal(), io.StringIO()):
+            wrote = threading.Event()
+            original = stream.write
+            def write(value):
+                result = original(value)
+                if "Loading" in value:
+                    wrote.set()
+                return result
+            with mock.patch.object(stream, "write", side_effect=write), \
+                    mock.patch.dict(os.environ, {"TERM": "xterm"}), contextlib.redirect_stderr(stream):
+                with self.assertRaises(RuntimeError):
+                    with display.busy("Loading"):
+                        if stream.isatty():
+                            self.assertTrue(wrote.wait(2))
+                        raise RuntimeError("stop")
+            if stream.isatty():
+                self.assertIn("Loading (", stream.getvalue())
+                self.assertTrue(stream.getvalue().endswith("\r\033[K"))
+            else:
+                self.assertEqual(stream.getvalue(), "")
+
     def test_color_only_in_capable_terminals_and_no_color_wins(self):
         for stream, environment, colored in ((Terminal(), {"TERM": "xterm"}, True),
                                               (Terminal(), {"TERM": "dumb"}, False),

docs/getting-started/install.md

@@ -43,7 +43,11 @@ The script picks the latest stable release, verifies its checksum and runs the b
 
 It creates no Project or key and makes no model request. It ends by printing the console address, the API base URL and the next steps.
 
-If installation fails or is interrupted before the installer reports that the services are running, including by Ctrl-C or a closed SSH session, the installer removes everything it created: its Compose project with its volumes, native Core's service and the files in the installation directory, and the directory itself when the installer created it. Only the loaded images stay. Fix the cause it prints and run the same command again. A rerun also removes an installation that stopped without this cleanup, for example after a power loss, and then installs with the options given now; until then, the `oac` command refuses to change that installation. Once the installer has reported that the services are running, the installation is kept, and a rerun only [repairs](operations.md#installation-version-policy) it.
+Downloads retry temporary network failures automatically. The terminal shows download progress and activity during long steps.
+
+If installation fails or is interrupted before the services first become healthy, fix the reported cause and rerun the same command. The installer removes its temporary download, new service project, volumes and installation files; loaded Docker images remain reusable. A rerun first clears an incomplete installation or download left by a forced exit or power loss. It never clears another active installation process or an unrelated directory. Once the services have started successfully, failures preserve the installation and its data; use [same-release repair](operations.md#installation-version-policy).
+
+For insufficient space or quota, free space on the filesystem named by the error. Image-loading failures can also require space in Docker's storage, which may be on a different filesystem.
 
 ## Sign in to Web