Skip to content

deployment.md

Latest commit

 

History

History
487 lines (416 loc) · 26.3 KB

deployment.md

File metadata and controls

487 lines (416 loc) · 26.3 KB
Raw

Production install and deployment

This document covers taking repowatch from source to a running system service: prerequisites, the make install/make activate flow, the two ways to run it (a single daemon vs. split status API + timer), the nginx cache in front of it, backups, and upgrades. For config.yaml itself, see configuration.md; for authentication and access control, see access.md.

Prerequisites

For distribution-specific package commands and first login, start with Quick start. The shipped configuration has repos: []; it starts an administrative service and an empty nginx site. Add sources through YAML or the dashboard after installation.

The shipped profile enables purge and cache inventory: install and load both ngx_cache_purge and the njs HTTP module from packages compatible with nginx. make check does not prove their availability; activation's nginx -t does.

  • Python 3.11+ with venv and pip available (some distros split pip out of the base Python package — make check will tell you if it's missing).
  • nginx, unless you're running the caching layer yourself and setting WITH_NGINX=0 (see below).
  • systemd, unless you're supervising the process yourself with WITH_SYSTEMD=0.
  • gpgv, if any repository has verify_signature: true (apt, pacman, RPM-MD/dnf, apt-rpm, Slackware).
  • openssl, or apk-tools >= 3.0, if any apk repository has verify_signature: true — see apk_signature_backend in configuration.md.
  • Enough local disk for state_db (SQLite — must be local disk, not NFS; see configuration.md) and for the nginx package cache itself (NFS is fine for the cache directory, just not for state_db).

Run make check at any point — it's a read-only diagnostic (OK/WARN/ FAIL per item: Python version, required stdlib modules, gpgv/openssl/ nginx/systemctl availability, filesystem permissions, an existing installation's consistency, nginx include conflicts) and never touches disk or the network beyond reading local files.

Install

sudo make check                               # read-only diagnostics for system paths
sudo make install PREFIX=/usr/local           # builds wheels, lays down files — no services touched
sudoedit /etc/repowatch/config.yaml            # review settings before activation
sudo make activate                            # creates the system user, enables and starts services
sudo -u repowatch /usr/local/bin/repowatch set-password

sudo is required for make install itself with the defaults shown above — PREFIX=/usr/local, SYSCONFDIR=/etc, LOCALSTATEDIR=/var are all real system paths a normal user can't write to. It's only optional if every one of those is overridden to somewhere you already own (e.g. for local testing, or packaging into a DESTDIR — see Packaging).

make install is intentionally inert with respect to any running system: it builds/installs into a venv under PREFIX, writes the CLI wrapper, copies systemd unit templates and scripts, and writes an install manifest ($PREFIX/share/repowatch/install.json) — but it does not create the repowatch system user, does not touch systemctl, and does not write or link any nginx config. make activate is the separate, explicit step that does all of that; it requires root and refuses to run under DESTDIR.

Paths are independently configurable (all via make variables, forwarded as environment variables to the installer):

Variable Default Meaning
PREFIX /usr/local Application venv, CLI wrapper, install manifest.
SYSCONFDIR /etc config.yaml lives at $SYSCONFDIR/repowatch/config.yaml.
LOCALSTATEDIR /var state_db/backups at $LOCALSTATEDIR/lib/repowatch.
CACHE_DIR $LOCALSTATEDIR/cache/nginx/repowatch nginx's proxy_cache_path, when WITH_NGINX=1.
SYSTEMD_UNIT_DIR /etc/systemd/system Where unit files are linked.
NGINX_CONF / NGINX_ENABLED_DIR auto-detected Explicit override if make check can't unambiguously find your nginx's main config / sites-enabled-style include directory.
WITH_NGINX 1 Set to 0 to skip nginx integration entirely — bring your own caching layer/config.
WITH_SYSTEMD 1 Set to 0 to skip systemd unit installation/activation — repowatch run doesn't depend on systemd at all, run it under any supervisor.
DESTDIR (empty) Staging root for packaging — see Packaging.

Re-running make install against an already-installed layout is safe (it detects and preserves an existing config.yaml rather than overwriting it), but changing PREFIX/SYSCONFDIR/LOCALSTATEDIR after the fact is refused by make check — that's a data-migration decision the installer won't make silently on your behalf. "Safe" here means it won't corrupt anything, not that it always proceeds: the same preflight also refuses outright if any repowatch service or timer is currently active — stop them first (or use make upgrade below, which handles the stop/start itself as part of one transaction) rather than reinstalling underneath a running process.

What make activate does

  1. Creates the repowatch system user/group if they don't already exist (--system, no login shell, no home directory).
  2. chowns config.yaml's directory, config.yaml itself, and the state directory (including backups/) to that user — deliberately not recursive over an existing package cache (which may be large, and may sit on NFS).
  3. If WITH_NGINX=1: renders the cache server block from your config.yaml (see nginx/render.py / configuration.md), validates it with nginx -t, and symlinks it into your nginx's enabled- sites directory — rolling back the symlink/file if validation fails. Initial bootstrap publishes active.conf, purge.conf, dedup.map, probe.conf and probe.js before validation. It reads existing dedup data without creating SQLite as root; a fresh installation uses an empty map. Failed writes/validation remove newly created bootstrap files and the link, preserving pre-existing files. Also writes a small root-owned policy.json (cache dir, access log path, nginx binary, site-link path) that later reconciliation runs against — see the next section.
  4. If WITH_SYSTEMD=1: links the unit files from $PREFIX/share/repowatch/systemd/ into $SYSTEMD_UNIT_DIR, reloads systemd, and enables+starts repowatch.service and repowatch-backup.timer (and, if nginx is also managed, repowatch-nginx.timer).

make activate re-runs check-config against the installed venv before touching anything, and refuses to proceed if make check reports any failure — it will not activate a broken installation.

Two ways to run it

repowatch.service (the default from make activate) runs repowatch run — a single process that does everything: the scheduled index checks, cache warming, the syslog listener (if syslog_listener. enabled), and the status.json/dashboard HTTP server, all together.

The daemon keeps an independent task for each repository and reloads configuration every ten seconds. A long warm does not hold up that reload or the separate retention scheduler. Index checks and cache work share check_concurrency, with capacity reserved for checks; see the configuration reference. A repository's next check waits for its own previous check and cache work to finish. Most setting edits take effect on the next operation. Disabling prefetch stops queued automatic downloads after the next scheduler reload; active transfers may finish. Removing a repository cancels and drains its active task. Ordinary interrupted warming is not a persistent retry queue.

repowatch-status.service + repowatch-check.timer is the split alternative: a long-running serve-status process for the HTTP side, paired with a timer that runs check-once periodically instead of the built-in scheduler. The real tradeoff here is that this combination has no syslog listener — that only runs inside repowatch run — so warmed_packages won't reflect real client downloads and /api/requests stays empty, not "no dashboard" as the naming might suggest (the dashboard is served by repowatch-status.service either way). Reach for this split if you want independent restart/scaling of "answer status queries" vs. "go check upstreams", or if you'd rather drive checks from your own scheduler instead of repowatch run's internal one.

Only one of the two shapes should be enabled against the same state_db at a time — make activate enables the first one; switching to the second is a manual systemctl disable --now repowatch.service followed by enabling the other two units.

repowatch.service ships with baseline systemd hardening (NoNewPrivileges, ProtectSystem=strict, ProtectHome, PrivateTmp), with ReadWritePaths scoped to exactly state_db's directory and config.yaml's directory — the latter is required because the dashboard's POST /api/repos writes config.yaml directly at runtime.

Neither shape needs systemd at all, actually — see Running without systemd for repowatch supervise, a third option for environments without it.

nginx cache and auto-reconciliation

The nginx server block itself is rendered once at make activate time (see above), from your config.yaml. But config.yaml can keep changing afterward — new repositories, changed url_templates, edited nginx settings — through the dashboard, without a redeploy. repowatch-nginx.timer (every 15s) checks whether regeneration is needed. Unchanged ticks compare a fingerprint of the configuration, privileged policy, installed Python code and probe template, and catalog revisions when deduplication is enabled. They also hash all five managed output files, so missing or altered files trigger repair. Client request writes and successful checks of unchanged catalogs do not cause the expensive duplicate query or rendering to run again. Discovering or losing a package hash does invalidate the catalog revision, even without a version change.

Changed inputs run the renderer; changed output is validated with nginx -t and reloaded transactionally. A failed apply cannot authorize the fast path, and a catalog change during rendering prevents caching that reconciliation result. The successful input fingerprint is kept in the root-owned status.json, so it survives separate timer invocations. sudo repowatch nginx-apply --force bypasses this optimization and repeats validation/reload. Use it after out-of-band database edits or changes to nginx configuration outside the five managed files. Normal no-op ticks do not reload nginx; the CLI and systemd may still log their outcome. Harmless dedup pairs mapping a URI to itself are debug messages; conflicting canonical targets and case-colliding paths remain warnings.

This reconciliation runs as a separate root-owned oneshot systemd unit (repowatch-nginx.service, invoked by the timer), not as a sudo call from the main repowatch.service process — the main process keeps running as the unprivileged repowatch user with no elevated privileges of its own. If a render or nginx -t fails, the previously-active config is left in place and the failure surfaces in that unit's systemd logs (journalctl -u repowatch-nginx.service); the daemon itself isn't affected.

If you're not using the generator (WITH_NGINX=0, or nginx.enabled: false in config.yaml), none of this applies — write and manage your own nginx config. repowatch -c config.yaml nginx-render prints a concrete example for your own repos[] (the caching approach: proxy_cache, short TTL on mutable index files, long TTL on immutable package files, IPv4-only upstream resolution — see configuration.md for why).

Backups

repowatch-backup.timer runs daily (with up to 30 minutes of randomized delay, and Persistent=true to catch up after downtime) and backs up state_db — the SQLite file holding repository snapshots, change history, and warmed-package state — using SQLite's own online backup API (sqlite3 <db> ".backup '<dest>'"), not a plain file copy. That matters because a plain cp/rsync over a file the daemon is actively writing to (WAL mode) risks capturing it mid-transaction; the backup API produces a consistent snapshot without blocking the running service. Backups are gzip-compressed and rotated by age (RETENTION_DAYS, default 14 — independent of event_retention_days, which retains individual history rows in the live database).

config.yaml itself is not backed up by this timer — back it up the way you back up any other config file under version control or your usual config- management tooling. (The upgrade procedure below separately snapshots it as part of every upgrade attempt, but that's incidental to the upgrade, not a substitute for your own config backups.)

Restoring a backup:

sudo systemctl stop repowatch.service          # or repowatch-status.service + repowatch-check.timer
sudo -u repowatch gunzip -k /var/lib/repowatch/backups/state-<timestamp>.sqlite3.gz
sqlite3 /var/lib/repowatch/backups/state-<timestamp>.sqlite3 'PRAGMA integrity_check;'
sudo -u repowatch cp /var/lib/repowatch/backups/state-<timestamp>.sqlite3 /var/lib/repowatch/state.sqlite3
sudo systemctl start repowatch.service

Restoring only rolls back state_db (history, warmed-package tracking, issued tokens, admin sessions) — it does not touch config.yaml, so make sure the config you're running matches what you expect separately.

Upgrading

sudo make upgrade-plan MANIFEST=/usr/local/share/repowatch/install.json WHEELHOUSE=build/wheels
sudo make upgrade  MANIFEST=/usr/local/share/repowatch/install.json WHEELHOUSE=build/wheels

This is an offline upgrade path — it expects a pre-built wheelhouse with target-compatible dependencies already present (make wheel on a matching environment, or copied over ahead of time); it never reaches out to the network itself.

upgrade-plan loads the install manifest, resolves PREFIX/SYSCONFDIR/LOCALSTATEDIR/etc. against it, and prints exactly what upgrade --apply would operate on — then runs a set of cheap, read-only checks: does config.yaml load in the currently installed venv, does that venv exist, does the wheelhouse contain exactly one repowatch-*.whl, and are repowatch.service/nginx.service already active (both required before --apply will proceed). It exits non-zero and prints each failing check as a PROBLEM line if any of these don't hold. This is still not a full rehearsal of the upgrade — it does NOT build the candidate venv from the wheelhouse, does NOT install the new release's dependencies, and can't catch an incompatibility that only shows up once the candidate is actually built (that only happens inside step 2 below, with --apply). Use it to catch a wrong MANIFEST/WHEELHOUSE path, a missing/misnamed wheel, or services that aren't up yet, before you commit to the real upgrade.

make upgrade (root required):

  1. Refuses to run unless the current installation has the required active services and passes installation preflight and unless state_db lives outside the installation/config directories being replaced.
  2. Builds a candidate venv in a temp directory and validates it against the current config.yaml before touching the running service — an incompatible candidate is caught with zero downtime.
  3. Only then: stops services, snapshots the installed files and does an online backup of state_db into a timestamped directory under state_db's backups/, installs the new version, re-activates (including nginx re-render/reload if applicable), and runs a smoke check.
  4. On most failures, it restores the snapshotted files and database from that same backup directory and resumes the services in their prior enabled/active state, rather than leaving you with a half-upgraded installation. Two caveats worth knowing: the smoke check tolerates a 503 from /healthz as a pass (it only confirms the service answers with valid JSON, not that every repository is healthy), so a post-upgrade instance that's technically up but degraded won't trigger a rollback on that basis alone; and the rollback step itself isn't wrapped in its own exception handling — if restoring files, restoring the database, or reloading nginx during rollback fails, that failure propagates instead of being silently swallowed, but it also means rollback isn't unconditionally guaranteed to finish cleanly on every kind of failure.

If a managed service is still activating or deactivating (for example a running repowatch-nginx.service reconciliation), the upgrade refuses to proceed. Wait for that operation to finish and retry; do not bypass the state check.

It refuses to run against a DESTDIR-staged layout or with WITH_SYSTEMD=0 — this specific automated path assumes a systemd-managed installation is what's being upgraded in place.

Checking GitHub for a new release (repowatch self-update)

repowatch self-update --repo erophey7/repowatch --check   # just report, don't install
repowatch self-update --repo erophey7/repowatch            # install if newer

A separate, lighter mechanism from make upgrade above: it checks the given GitHub repository's latest (non-draft, non-prerelease) release, downloads its wheel and the <wheel>.sha256 checksum file published alongside it in the same release, verifies the checksum, and pip installs the wheel into whichever environment repowatch itself is currently running from. It never touches config.yaml, state_db, or nginx, and it doesn't require the source repository or a wheelhouse to be present — just network access to GitHub.

A release without exactly one .whl asset, without a matching .sha256 asset, or with a checksum that doesn't match is refused outright — there is no "install anyway" fallback. This protects against a corrupted download or a network-level tamper, but not against a compromised GitHub account: the checksum file travels in the same release as the wheel it describes, so whoever could replace one could replace both. If that stronger guarantee matters for your deployment, use the offline make upgrade path above (with a wheelhouse you built and reviewed yourself) instead.

self-update reinstalls repowatch and resolves its Python dependencies via pip install --upgrade --force-reinstall. Dependency versions can change even when the release has not raised its minimum requirements. It leaves config.yaml, state_db, and nginx untouched. It does not restart the service: restart it afterwards using your usual supervisor.

Running without systemd

repowatch run itself has no systemd dependency at all — it's a single process with its own asyncio scheduler (runtime.scheduler.run_forever), same as make install WITH_SYSTEMD=0 already assumes. What's missing without systemd is a replacement for the other two units: repowatch-nginx.timer (periodic nginx reconciliation) and repowatch-backup.timer (daily backup). repowatch supervise is a single foreground command that replaces all three, for any environment with no systemd (and, in the general case covered here, not even cron — a minimal or container environment being the main target):

repowatch -c /etc/repowatch/config.yaml supervise \
  --pid-file /var/run/repowatch/repowatch.pid \
  --backup-dir /var/lib/repowatch/backups --backup-interval-hours 24 \
  --nginx --nginx-policy /etc/nginx/repowatch/policy.json

This runs exactly what repowatch run runs (status API, optional syslog listener, the check scheduler) plus, if the corresponding flags are given, a backup loop and an nginx-reconciliation loop in the same process — see repowatch supervise --help for the full flag list (intervals, retention). Nothing is auto-detected: leave out --backup-dir/--nginx to skip those parts entirely, same as not enabling the corresponding systemd timer.

Both run and supervise bind HTTP/TLS and the enabled UDP socket before starting the watcher. Syslog must finish its initial index load within 30 seconds. A bind/TLS/startup failure aborts startup and releases prepared sockets. During operation, an unexpected watcher or mandatory listener exit terminates the service with an error; configure the process manager to restart it. Listener exits are checked every 100 ms when the event loop can run. Signals and external cancellation cancel and drain child coroutines, stop listener loops and remove the PID file. Blocking work already dispatched to a thread remains subject to its own timeout; this is not a hard real-time shutdown guarantee.

Unexpected non-SQLite errors in an individual repository operation are logged and isolated from other repositories. Shared SQLite failures are fatal and cancel and drain running repository tasks, so the service cannot continue retention with a known failed database or listener. check-once returns nonzero for these isolated operation failures; existing fetch-error logging behavior remains unchanged.

The --nginx root requirement, and what you give up without systemd. Under systemd, nginx reconciliation deliberately runs as a separate root-owned oneshot unit (repowatch-nginx.service), so the main daemon never needs elevated privileges (see nginx cache and auto-reconciliation and Hardening notes above) — that split exists because systemd can supervise two independently-privileged units against one config.yaml. The combined supervise --nginx command performs both jobs in one process, so it requires the whole process to run as root (it refuses to start otherwise, same check nginx-apply already makes) — there is no way to keep the rest of supervise unprivileged while it also reconciles nginx in-process.

If that tradeoff isn't acceptable, don't pass --nginx: run supervise unprivileged for the main daemon (and, if you want, --backup-dir too — backups only need write access to state_db's directory, not root), and apply nginx changes through whatever periodic mechanism you do have available — repowatch nginx-apply --policy ... as root from cron if you have it, or your own small root-owned loop/timer if you don't. The pieces are independent; --nginx is only a convenience for the case where running as root isn't a concern (a common posture for a single-purpose container, for instance).

PID file and manual start/stop, for environments with no supervisor to hand the command to at all:

repowatch -c /etc/repowatch/config.yaml supervise --pid-file /var/run/repowatch/repowatch.pid &
# ...later:
kill "$(cat /var/run/repowatch/repowatch.pid)"

supervise writes its PID to --pid-file on start and removes it on a clean shutdown (SIGTERM/SIGINT); it doesn't restart itself if it crashes — if you have any process supervisor available (even a minimal one), point it at repowatch supervise as the command to run, and let it handle restart-on-crash. Two short examples:

OpenRC 
#!/sbin/openrc-run
command="/usr/local/bin/repowatch"
command_args="-c /etc/repowatch/config.yaml supervise --pid-file /var/run/repowatch/repowatch.pid"
command_background=true
pidfile="/var/run/repowatch/repowatch.pid"
runit 
#!/bin/sh
# /etc/sv/repowatch/run
exec /usr/local/bin/repowatch -c /etc/repowatch/config.yaml supervise

(runit itself tracks the process's own PID via the pipe it holds open, so --pid-file is optional here — pass it only if something else also wants to read it, e.g. a health check script.)

Standalone backups without supervise. repowatch backup <dir> [--retention-days N] is also a first-class, one-shot command on its own — useful if you have cron but not systemd: 0 3 * * * repowatch -c /etc/repowatch/config.yaml backup /var/lib/repowatch/backups. It performs the same pure-Python online SQLite backup (no dependency on the sqlite3 CLI binary or scripts/backup-state.sh, which requires the make install layout) that supervise --backup-dir uses internally.

Packaging (DESTDIR)

make install DESTDIR=/path/to/stage PREFIX=/usr/local ... lays out files under the staging root for downstream packaging (e.g. building a distro package), without writing anything outside it and without any of the activate steps (which are refused entirely under DESTDIR). Runtime paths baked into the installed files (the CLI wrapper, systemd units, nginx policy) are always the target paths (PREFIX, etc.) — DESTDIR never leaks into them, only into where install itself writes on the build host.

Hardening notes

  • repowatch.service runs as the dedicated, unprivileged repowatch system user, with ProtectSystem=strict and a minimal ReadWritePaths.
  • Privileged operations — writing the nginx site config, reloading nginx — are isolated in a separate root-owned oneshot unit (repowatch-nginx.service) triggered by its own timer, not by granting the main process sudo or filesystem access outside its state/config directories.
  • admin_password_hash is a PBKDF2 hash, never the plaintext password — see access.md for the full authentication model, session/CSRF handling, and host-token scoping.
  • The dashboard and status API themselves aren't rate-limited by design — see the "What's deliberately not there" section of access.md for the reasoning; put a reverse proxy or firewall in front if you need that.

Disabling a repository's prefetch stops new queued automatic package downloads once the daemon reloads the edit (normally on the next ten-second tick). Active transfers may finish, and index checking and cleanup continue. Manual warming is independent. See demand and update selection.