Cream Cloud Backup

Introduction

Cream Cloud Backup is a lightweight, open-source backup wrapper for Restic. It securely backs up files, folders, and MySQL databases to an OpenStack Object Store (Swift) backend, runs on a daily cron schedule, and provides commands to inspect, restore and verify backups.

It is a Symfony Console based PHP application providing a creamcloud-backup command with several subcommands (backup:run, backup:restore, backup:list, ...), one per action.

Table of Contents

1. Introduction
2. Requirements
3. Installation
4. Configuration
5. Commands
6. Backup
7. Restore
8. Notifications
9. Updating
10. Uninstall
11. License
12. Authors

Requirements

• PHP 8.2 or newer (cli, with the mbstring, xml and curl extensions)
• Composer
• restic
• swift (python3-swiftclient), used for status uploads and storage statistics
• mysql / mysqldump, if you want MySQL databases to be backed up

install.sh installs all of the above on Debian, Ubuntu, CentOS, Fedora, Rocky Linux and AlmaLinux.

Installation

You need to run the installer as the root user.

Clone the repository:

git clone https://github.com/creamcloud/backup.git /usr/local/creamcloud-backup
cd /usr/local/creamcloud-backup

Run the installer:

bash install.sh

install.sh installs the required system packages, Composer and the PHP dependencies (composer install), then hands off to the application's own install command (bin/creamcloud-backup install), which:

• asks for this server's hostname (defaults to the machine's own hostname) and your OpenStack Object Store username, password, project ID, region and domain names (your password is not echoed back), and stores them - together with SWIFT_CONTAINER/RESTIC_REPOSITORY set to this server's hostname, so each server gets its own isolated Swift container - in /etc/creamcloud-backup/backup.conf;
• asks for a restic repository password (leave empty to generate one), and stores it in /etc/creamcloud-backup/restic-password.conf; pass --restic-password to set it non-interactively;
• initializes the restic repository;
• symlinks bin/creamcloud-backup to /usr/local/bin/creamcloud-backup;
• installs a daily cron job at /etc/cron.d/creamcloud-backup that runs a backup at a randomized time, prunes old snapshots once a day, and updates Cream Cloud Backup once a month.

All of the above is safe to re-run: existing configuration, the restic password and the cron job are left alone if they already exist. Pass --reinstall to overwrite the local configuration (hostname, OpenStack credentials, ...) and the cron job - the restic password is never overwritten, even with --reinstall, since that would lock you out of your existing backups.

To install unattended, pass the OpenStack credentials as options:

bash install.sh --username='user@example.org' --password='P@ssw0rd' --project-id='project-id' \
    --region='NL' --user-domain-name='transip' --project-domain-name='transip'

(this is equivalent to running creamcloud-backup install with the same options directly, e.g. to reconfigure a server without re-running the system package installation.) Add --hostname=<hostname> to also set the hostname non-interactively, and see creamcloud-backup install --help for the full list of options (including --restic-password and --reinstall).

Configuration

Configuration is loaded from two files (later overrides earlier):

1. etc/backup.conf - in the project directory, committed to git. Defaults shared by every server; do not put secrets here.
2. /etc/creamcloud-backup/backup.conf (path configurable via LOCAL_CONFIG_FILE in the project's etc/backup.conf) - per-server configuration, including credentials. Any variable from the project's etc/backup.conf may be overridden here. Generated by creamcloud-backup install, lives outside the application folder so it survives a fresh git clone and is never committed.

The most important variables (see etc/backup.conf for the full, documented list):

Variable	Description
HOSTNAME	Identity of this server. Used as the restic/Swift container identity, must be unique among backed up servers.
KEEP_DAILY	Number of daily snapshots to keep. Default 7.
KEEP_WEEKLY	Number of weekly snapshots to keep. Default 2.
BACKUP_PATH	Path to back up. Default /.
RESTIC_REPOSITORY	Restic repository URL, e.g. swift:creamcloud-backup:/. Set by the installer to swift:<hostname>:/, one container per server.
RESTIC_PASSWORD_FILE	Path to the file containing the restic repository password.
LOCAL_CONFIG_FILE	Path to the per-server configuration file (see above).
EXCLUDE_FILE	Path to a restic exclude file. Defaults to etc/exclude.conf.
SWIFT_CONTAINER	Swift container used for status objects and storage statistics. Set by the installer to this server's hostname, matching RESTIC_REPOSITORY.
OS_USERNAME, OS_PASSWORD, OS_PROJECT_NAME, OS_USER_DOMAIN_NAME, OS_PROJECT_DOMAIN_NAME, OS_REGION_NAME, OS_AUTH_URL, OS_IDENTITY_API_VERSION	OpenStack authentication for the Swift backend.
MAILER_DSN	Symfony Mailer DSN used for failure notifications. Defaults to null://null (do nothing).
NOTIFICATION_EMAILS	Comma separated list of addresses to email when a backup fails.
MAIL_FROM	Sender address for notification emails.

Service wiring and non-environment configuration (parameter names, default exclude file, working directories) lives in config/packages/creamcloud_backup.yaml and config/services.yaml.

If you want to use a different Restic backend (not Swift), set RESTIC_REPOSITORY (and any backend-specific environment variables it needs) in /etc/creamcloud-backup/backup.conf.

Commands

Everything is run through the creamcloud-backup command, installed as /usr/local/bin/creamcloud-backup (or php bin/creamcloud-backup from the project directory). Run creamcloud-backup (or creamcloud-backup list) on its own to see the full list of available commands.

creamcloud-backup <command> [<args>]

Command	Description
install [--reinstall]	Configure this server: hostname, OpenStack credentials, restic password, creamcloud-backup command and cron job. Run by install.sh, or directly to reconfigure a server. --reinstall overwrites the local config and cron job (not the restic password).
backup:init	Initialize the restic repository, if it has not been already.
backup:run	Run a backup: dispatches pre/post-backup events (status upload, lock check, MySQL dump), runs restic backup, then restic forget to apply the retention policy.
backup:cleanup	Remove stale locks and prune old data (restic unlock --remove-all followed by restic prune).
backup:list [time]	List available snapshots as a table, optionally restricted to those newer than time (e.g. 2d for 2 days ago, or 2024-01-01).
backup:delete <snapshot-id> / backup:delete --all	Delete a snapshot, or every snapshot with --all. Asks for confirmation unless --force/-f is given. Cannot be undone.
backup:stats	Show configuration, Swift storage usage and the snapshot list.
backup:verify	Check the integrity of the repository (restic check).
backup:restore	Interactively restore a file, folder or MySQL database.
self-update	Update Cream Cloud Backup itself (git pull + composer install). Requires the application to be a git checkout.

Backup

To run a backup manually:

creamcloud-backup backup:run

By default, install.sh installs a cron job at /etc/cron.d/creamcloud-backup which runs a backup once a day at a randomized time between 00:00 and 06:59. Edit this file to change the schedule.

Before the backup runs, the following happens:

• a started status object (and a version-x.y.z marker) is uploaded to the Swift container;
• leftover restic locks are checked and cleared if stale (a still-locked repository aborts the run);
• every non-system MySQL database is dumped with mysqldump to gzip files under /var/backups/sql/, so they are picked up by the backup.

After a successful restic backup and restic forget, a completed status object is uploaded. If the backup or the retention cleanup fails, a failed status object is uploaded and, if MAILER_DSN and NOTIFICATION_EMAILS are configured, a failure notification email is sent.

Restore

To restore a file, folder or MySQL database, run:

creamcloud-backup backup:restore

This starts an interactive wizard:

1. Hostname - pre-filled with the configured HOSTNAME. To restore data from another server, enter that server's hostname instead.
2. Type - choose "File or folder" or "MySQL database".
3. Path / database name:
   • For a file or folder, enter the full path (e.g. /home/user/test/), then choose whether to restore it to its original location (overwriting any existing files) or to a timestamped directory under /var/backups/restore/.
   • For a database, enter its name. It is restored into a new database named <database>_backup, leaving the original database untouched. To restore a database, MySQL superuser credentials must be available in /root/.my.cnf (this is created automatically during run if a supported control panel is detected).
4. Snapshot ID - the restic snapshot to restore from. Use creamcloud-backup backup:list to see available snapshots.
5. Confirmation - an overview of what will happen before anything is changed.

Notifications

To receive an email when a backup fails, set in /etc/creamcloud-backup/backup.conf:

MAILER_DSN='smtp://user:pass@smtp.example.com:587'
NOTIFICATION_EMAILS='ops@example.com,oncall@example.com'

See the Symfony Mailer documentation for supported DSN formats (SMTP, Sendmail, SES, etc.).

Updating

To update Cream Cloud Backup manually, run:

creamcloud-backup self-update

This runs git pull --ff-only followed by composer install in the application directory, and requires the application to have been installed via git clone. A cron job runs this automatically on the first day of each month.

Uninstall

To remove Cream Cloud Backup's cron job and the creamcloud-backup command, run:

bash uninstall.sh

This does not remove your restic repository, /etc/creamcloud-backup (which holds the restic password) or the application directory itself.

License

Cream Cloud Backup - Restic wrapper to back up to OpenStack Object Store

Copyright (C):          Cream Commerce B.V., https://www.cream.nl/
Based on the work of:   Remy van Elst, https://raymii.org/

This program is free software; you can redistribute it and/or modify it
under the terms of the GNU General Public License as published by the
Free Software Foundation; either version 2 of the License, or (at your
option) any later version.

This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
General Public License for more details.

You should have received a copy of the GNU General Public License along
with this program; if not, write to the Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.

Also see LICENSE.md for full text of GPLv2.

Authors

• Copyright: Cream Commerce B.V., https://www.cream.nl/
• Author: Danny Verkade
• Based on the work of: Remy van Elst, https://raymii.org/