Add foremanctl restore command - Complete offline backup restore

[Chyenne8]

Summary

Implements the foremanctl restore command to restore Foreman instances from offline backups created by foremanctl backup.

This PR adds complete end-to-end restore functionality with validation, error recovery, and comprehensive verification of all restored components including databases, Pulp content, encryption keys, and OAuth credentials.

Features

Command Usage

# Validate a backup without making changes
foremanctl restore /path/to/backup --dry-run

# Perform full restore
foremanctl restore /path/to/backup

What Gets Restored

• ✅ Databases (foreman, candlepin, pulpcore)
• ✅ Pulp content (media files)
• ✅ Pulp encryption keys (database_fields.symmetric.key, django_secret_key)
• ✅ OAuth keys and secrets
• ✅ Database passwords
• ✅ Foreman configuration (parameters.yaml)

Implementation Phases

Phase 1: Validation

• Validates backup directory exists
• Checks metadata.yml present
• Verifies all required dump files exist
• Supports --dry-run mode for validation-only

Phase 2: Prepare System

• Stops Foreman services safely
• Starts PostgreSQL for restore operations
• Waits for PostgreSQL readiness
• Comprehensive error handling with rescue block

Phase 3: Database Restore

• Reads backup metadata to determine which databases to restore
• Drops existing databases
• Creates empty databases with correct ownership
• Restores data from pg_dump files using pg_restore
• Fixes database ownership after restore
• Supports Katello (3 databases) and Vanilla Foreman (1 database)

Phase 4: Restore Pulp Content

• Backs up existing media directory
• Extracts Pulp content archive
• Verifies encryption keys restored:
  • database_fields.symmetric.key (CRITICAL)
  • django_secret_key (CRITICAL)
• Counts and reports restored media files
• Gracefully skips if backup used --skip-pulp-content

Phase 4b: Restore Foremanctl State

• Restores foremanctl-state.tar.gz
• Verifies all critical files:
  • foreman-oauth-consumer-key (CRITICAL)
  • foreman-oauth-consumer-secret (CRITICAL)
  • postgresql-admin-password
  • foreman-db-password
  • candlepin-db-password
  • pulp-db-password
  • parameters.yaml
• Required before starting services

Phase 5: Deploy and Verify

• Stops PostgreSQL (no longer needed)
• Starts all Foreman services
• Waits for services to stabilize
• Verifies Foreman API is responding
• Confirms all critical services are active
• Displays comprehensive success message

Error Handling

• Rescue block catches failures and restores system to running state
• Automatically restarts services on failure
• Uses state tracking flags to know what to clean up
• Clear error messages show exactly what failed
• System always left in a safe, working state

Testing

Comprehensive testing performed:

• ✅ Phase 1 validation with --dry-run
• ✅ Phase 2 success path (services stop/start correctly)
• ✅ Phase 2 error path (rescue block works)
• ✅ Phase 3 database restore (all 3 databases)
• ✅ Phase 4 Pulp content + encryption key verification
• ✅ Phase 4b OAuth keys and passwords verification
• ✅ Phase 5 services start and API responds
• ✅ Full end-to-end restore: 63 tasks, 0 failures

Files Changed

src/playbooks/restore/
├── metadata.obsah.yaml           (NEW - command definition)
└── restore.yaml                  (NEW - playbook entry point)

src/roles/restore/
├── defaults/main.yaml            (NEW - configuration)
└── tasks/
    ├── main.yaml                 (NEW - orchestration + error handling)
    ├── validate.yaml             (NEW - Phase 1)
    ├── prepare_system.yaml       (NEW - Phase 2)
    ├── restore_databases.yaml    (NEW - Phase 3)
    ├── restore_pulp_content.yaml (NEW - Phase 4)
    ├── restore_foremanctl_state.yaml (NEW - Phase 4b)
    └── deploy_and_verify.yaml    (NEW - Phase 5)

Total: ~560 lines of code across 7 new files

Acceptance Criteria

All requirements have been met:

• ✅ foremanctl restore /path restores a working system from a foremanctl backup
• ✅ --dry-run validates without making changes
• ✅ Hostname mismatch is caught before any destructive action
• ✅ Validation adapts required files based on instance type
• ✅ Works with backups that omit pulp_data.tar (gracefully skips)
• ✅ System verified healthy after restore (API ping, services up)

Security Considerations

• All encryption keys are verified after restore
• OAuth secrets are properly restored before services start
• Database passwords are restored from backup
• No secrets are logged (using no_log: true where appropriate)

Testing Instructions

1. Create a test backup:

   foremanctl backup /var/tmp/test-backup --wait-for-tasks

2. Test validation only (safe):

   foremanctl restore /var/tmp/test-backup/foreman-backup-TIMESTAMP --dry-run

3. Perform actual restore (destructive):

   foremanctl restore /var/tmp/test-backup/foreman-backup-TIMESTAMP

4. Verify services are running:

   systemctl status foreman.target
   curl -k https://$(hostname -f)/api/status

Checklist

• ✅ Code follows project conventions
• ✅ All phases tested individually
• ✅ Full end-to-end test successful
• ✅ Error handling tested
• ✅ Encryption keys verified
• ✅ Services health checked
• ✅ Clear commit messages
• ✅ No secrets exposed in logs
• ✅ Rebased on latest upstream/master

[sjha4]

This will be different for deployments..Use the obsah_state_path..Something similar to backup does for taking the backup..

[Chyenne8]

updated to use obsah_state_path instead of a hardcoded path.

[sjha4]

We probably need a foremanctl deploy in these steps somewhere after the foremanctl state is restored for everything to take effect.

[Chyenne8]

Added foremanctl deploy and tested it in foremanctl install environment.