feat(lab-09): Ansible playbook to rebuild the server in one command

.gitignore

@@ -21,6 +21,10 @@ crash.log
 
 # Ansible
 *.retry
+# Real inventory and vault stay local (commit vault.yml only encrypted: git add -f)
+ansible/inventory/production/hosts.yml
+ansible/inventory/production/group_vars/servers/vault.yml
+ansible/collections/
 
 # Backups / dumps
 *.sql

README.md

@@ -42,7 +42,7 @@ Status: ⬜ todo · 🟨 in progress · ✅ done
 ### Phase 4 — Infrastructure as Code
 | # | Lab | Status |
 |---|---|---|
-| 09 | [Ansible: rebuild the server in one command](labs/09-ansible/) | ⬜ |
+| 09 | [Ansible: rebuild the server in one command](labs/09-ansible/) | 🟨 |
 | 10 | [Terraform: provision VPS, DNS, firewall](labs/10-terraform/) | ⬜ |
 
 ### Phase 5 — Observability & Kubernetes

ansible/README.md

@@ -1,3 +1,65 @@
 # ansible
 
-Server configuration as code: `inventory/`, `roles/<concern>/`, `site.yml`. Built in lab 09 — goal: a fresh VPS becomes the full server with one command.
+Turns a fresh Ubuntu 24.04 server into the OpsForge server with one command (lab 09).
+
+```bash
+ansible-playbook site.yml                                      # fake VPS (inventory/test)
+ansible-playbook -i inventory/production site.yml --ask-vault-pass
+```
+
+Safe to re-run: a second run reports `changed=0`. The first play detects whether the server is fresh (root on port 22) or already hardened (admin user on `ssh_port`), so the same command works for both.
+
+## Roles
+
+Run in this order by [site.yml](site.yml):
+
+| Role | Does | From lab |
+|---|---|---|
+| [common](roles/common/) | apt upgrade, base packages, timezone, hostname, automatic security updates | 01 |
+| [users](roles/users/) | Admin user with SSH key only, passwordless sudo | 01 |
+| [firewall](roles/firewall/) | ufw: deny incoming; allow SSH, 80, 443 | 01 |
+| [ssh](roles/ssh/) | Hardened sshd on `ssh_port`, reconnects, closes port 22 | 01 |
+| [fail2ban](roles/fail2ban/) | sshd jail, systemd backend | 01 |
+| [docker](roles/docker/) | Docker Engine + Compose from Docker's repo, log rotation | 04 |
+| [opsforge](roles/opsforge/) | `deploy` user, `proxy` network, Traefik (+ Let's Encrypt), hello-api files and `.env` for CI deploys | 05, 07, 08 |
+
+Run a subset with tags: `ansible-playbook site.yml --tags ssh,firewall`.
+
+## Layout
+
+```
+ansible/
+├── site.yml                 # the playbook
+├── ansible.cfg              # defaults to inventory/test
+├── group_vars/all.yml       # defaults for every server
+├── inventory/
+│   ├── test/                # fake VPS: hosts.yml + group_vars/servers.yml (test values)
+│   └── production/          # your VPS: hosts.yml + group_vars/servers/{main,vault}.yml
+├── roles/<role>/{tasks,handlers,templates,defaults}/
+└── test/                    # fake VPS: Dockerfile + fake-vps.sh
+```
+
+Overrides go in `inventory/<env>/group_vars/servers…`, not `group_vars/all`: a playbook's `group_vars/all` takes precedence over an inventory's `group_vars/all`.
+
+## Test locally (no VPS needed)
+
+```bash
+ansible/test/fake-vps.sh up          # fresh Ubuntu 24.04 with systemd + sshd, as root on 127.0.0.1:2201
+cd ansible && ansible-playbook site.yml
+ansible/test/fake-vps.sh recreate    # destroy and start fresh
+ansible/test/fake-vps.sh down
+```
+
+The fake VPS is a privileged container. Fine for testing on your Mac; never a real server.
+
+## Production
+
+```bash
+cp inventory/production/hosts.yml.example inventory/production/hosts.yml
+cp inventory/production/group_vars/servers/vault.yml.example inventory/production/group_vars/servers/vault.yml
+ansible-vault encrypt inventory/production/group_vars/servers/vault.yml
+# edit inventory/production/group_vars/servers/main.yml (domain, email, deploy key)
+ansible-playbook -i inventory/production site.yml --ask-vault-pass
+```
+
+`hosts.yml` and `vault.yml` are git-ignored. Commit `vault.yml` only if it's encrypted (`git add -f`).

ansible/ansible.cfg

@@ -0,0 +1,11 @@
+[defaults]
+inventory = inventory/test/hosts.yml
+roles_path = roles
+interpreter_python = auto_silent
+callback_result_format = yaml
+retry_files_enabled = False
+# Never accept unknown SSH host keys silently (see lab 01: known_hosts).
+host_key_checking = True
+
+[ssh_connection]
+pipelining = True

ansible/group_vars/all.yml

@@ -0,0 +1,30 @@
+# Defaults for every server. Override per inventory in inventory/<env>/group_vars/servers/
+# (a named group beats "all"; inventory group_vars/all would NOT override these).
+
+# --- Connection -------------------------------------------------------------
+# A fresh server: root on port 22. After the ssh role: admin_user on ssh_port.
+# site.yml detects which one works, so the same command runs on both.
+bootstrap_user: root
+bootstrap_port: 22          # how to reach the fresh server's sshd
+ssh_initial_port: 22        # the port sshd listens on before hardening
+
+# --- Users ------------------------------------------------------------------
+admin_user: ops
+admin_ssh_pubkey: "{{ lookup('ansible.builtin.file', '~/.ssh/id_ed25519.pub') }}"
+deploy_user: deploy
+deploy_ssh_pubkey: ""       # public half of the CI deploy key (lab 07); empty = none
+
+# --- SSH and firewall -------------------------------------------------------
+ssh_port: 2222
+firewall_allowed_tcp_ports: [80, 443]
+
+# --- System -----------------------------------------------------------------
+server_hostname: opsforge
+server_timezone: Asia/Ho_Chi_Minh
+
+# --- OpsForge stacks --------------------------------------------------------
+domain: opsforge.localhost
+traefik_acme_email: ""      # set to enable Let's Encrypt (needs a public domain)
+# Secrets: define these in the inventory's vault (see inventory/production).
+traefik_dashboard_users: "{{ vault_traefik_dashboard_users }}"
+hello_api_postgres_password: "{{ vault_hello_api_postgres_password }}"

ansible/inventory/production/group_vars/servers/main.yml

@@ -0,0 +1,4 @@
+# Production settings (non-secret). Secrets go in vault.yml next to this file.
+domain: <your-domain.com>
+traefik_acme_email: <you@example.com>
+# deploy_ssh_pubkey: "ssh-ed25519 AAAA... github-actions-deploy"

ansible/inventory/production/group_vars/servers/vault.yml.example

@@ -0,0 +1,5 @@
+# Copy to vault.yml, fill in, then ENCRYPT before doing anything else:
+#   ansible-vault encrypt inventory/production/group_vars/servers/vault.yml
+# vault.yml is git-ignored. Commit it only encrypted, with `git add -f`.
+vault_traefik_dashboard_users: "admin:<bcrypt hash from: htpasswd -nB admin>"
+vault_hello_api_postgres_password: "<long random string: openssl rand -hex 24>"

ansible/inventory/production/hosts.yml.example

@@ -0,0 +1,5 @@
+# Copy to hosts.yml (git-ignored) and fill in your VPS.
+servers:
+  hosts:
+    opsforge:
+      ansible_host: <VPS_IP>

ansible/inventory/test/group_vars/servers.yml

@@ -0,0 +1,11 @@
+# Test overrides for the fake VPS. Its published ports differ from a real server:
+#   127.0.0.1:2201 -> container :22 (fresh sshd), 127.0.0.1:2202 -> :2202 (hardened)
+bootstrap_port: 2201
+ssh_port: 2202
+
+# Containers can't change their hostname or kernel settings like a VM can.
+common_set_hostname: false
+
+# Test-only values, NOT secrets: a real server gets these from an encrypted vault.
+vault_traefik_dashboard_users: "admin:$2y$05$1v9HS6HsMDzluqBjEk77oeKjenPLeRbd2T0oZQrJO8O/E0e1Lb/we"   # password: test
+vault_hello_api_postgres_password: test-only-password

ansible/inventory/test/hosts.yml

@@ -0,0 +1,7 @@
+# The disposable fake VPS from test/fake-vps.sh (a local container).
+# Hosts live in the "servers" group: inventory group_vars/servers/ override the
+# defaults in ../../group_vars/all.yml (a plain group beats "all").
+servers:
+  hosts:
+    fake-vps:
+      ansible_host: 127.0.0.1

ansible/requirements.yml

@@ -0,0 +1,6 @@
+# Collections used by the roles. Included in `brew install ansible`;
+# with ansible-core only: ansible-galaxy collection install -r requirements.yml
+collections:
+  - name: ansible.posix
+  - name: community.general
+  - name: community.docker

ansible/roles/common/defaults/main.yml

@@ -0,0 +1,12 @@
+common_upgrade: true          # apt dist-upgrade on every run
+common_set_hostname: true
+common_packages:
+  - ca-certificates
+  - curl
+  - git
+  - gnupg
+  - htop
+  - python3-debian            # used by deb822_repository (docker role)
+  - python3-requests          # used by the community.docker modules
+  - rsync
+  - unattended-upgrades

ansible/roles/common/tasks/main.yml

@@ -0,0 +1,27 @@
+- name: Update the apt cache and upgrade all packages
+  ansible.builtin.apt:
+    update_cache: true
+    cache_valid_time: 3600
+    upgrade: "{{ 'dist' if common_upgrade else 'no' }}"
+
+- name: Install base packages
+  ansible.builtin.apt:
+    name: "{{ common_packages }}"
+    state: present
+
+- name: Set the timezone
+  community.general.timezone:
+    name: "{{ server_timezone }}"
+
+- name: Set the hostname
+  ansible.builtin.hostname:
+    name: "{{ server_hostname }}"
+  when: common_set_hostname
+
+- name: Enable automatic security updates
+  ansible.builtin.copy:
+    dest: /etc/apt/apt.conf.d/20auto-upgrades
+    content: |
+      APT::Periodic::Update-Package-Lists "1";
+      APT::Periodic::Unattended-Upgrade "1";
+    mode: "0644"

ansible/roles/docker/defaults/main.yml

@@ -0,0 +1,5 @@
+docker_daemon_config:
+  log-driver: json-file
+  log-opts:
+    max-size: 10m
+    max-file: "3"

ansible/roles/docker/handlers/main.yml

@@ -0,0 +1,4 @@
+- name: Restart docker
+  ansible.builtin.service:
+    name: docker
+    state: restarted

ansible/roles/docker/tasks/main.yml

@@ -0,0 +1,41 @@
+# Docker Engine from Docker's own apt repository (newer than Ubuntu's docker.io).
+- name: Add Docker's apt signing key
+  ansible.builtin.get_url:
+    url: https://download.docker.com/linux/ubuntu/gpg
+    dest: /etc/apt/keyrings/docker.asc
+    mode: "0644"
+
+- name: Add Docker's apt repository
+  ansible.builtin.deb822_repository:
+    name: docker
+    types: deb
+    uris: https://download.docker.com/linux/ubuntu
+    suites: "{{ ansible_facts['distribution_release'] }}"
+    components: stable
+    architectures: "{{ 'arm64' if ansible_facts['architecture'] == 'aarch64' else 'amd64' }}"
+    signed_by: /etc/apt/keyrings/docker.asc
+
+- name: Install Docker Engine and the Compose plugin
+  ansible.builtin.apt:
+    name:
+      - docker-ce
+      - docker-ce-cli
+      - containerd.io
+      - docker-buildx-plugin
+      - docker-compose-plugin
+    state: present
+    update_cache: true
+
+# Without limits, container logs grow until the disk is full.
+- name: Configure log rotation for containers
+  ansible.builtin.copy:
+    dest: /etc/docker/daemon.json
+    content: "{{ docker_daemon_config | to_nice_json }}\n"
+    mode: "0644"
+  notify: Restart docker
+
+- name: Start Docker at boot
+  ansible.builtin.service:
+    name: docker
+    state: started
+    enabled: true

ansible/roles/fail2ban/defaults/main.yml

@@ -0,0 +1,3 @@
+fail2ban_bantime: 1h
+fail2ban_findtime: 10m
+fail2ban_maxretry: 5

ansible/roles/fail2ban/handlers/main.yml

@@ -0,0 +1,4 @@
+- name: Restart fail2ban
+  ansible.builtin.service:
+    name: fail2ban
+    state: restarted

ansible/roles/fail2ban/tasks/main.yml

@@ -0,0 +1,17 @@
+- name: Install fail2ban
+  ansible.builtin.apt:
+    name: fail2ban
+    state: present
+
+- name: Configure the sshd jail
+  ansible.builtin.template:
+    src: jail.local.j2
+    dest: /etc/fail2ban/jail.local
+    mode: "0644"
+  notify: Restart fail2ban
+
+- name: Start fail2ban at boot
+  ansible.builtin.service:
+    name: fail2ban
+    state: started
+    enabled: true

ansible/roles/fail2ban/templates/jail.local.j2

@@ -0,0 +1,11 @@
+# {{ ansible_managed }}
+[DEFAULT]
+bantime  = {{ fail2ban_bantime }}
+findtime = {{ fail2ban_findtime }}
+maxretry = {{ fail2ban_maxretry }}
+# Ubuntu 24.04 has no /var/log/auth.log by default: read the journal.
+backend  = systemd
+
+[sshd]
+enabled = true
+port    = {{ ssh_port }}

ansible/roles/firewall/tasks/main.yml

@@ -0,0 +1,32 @@
+- name: Install ufw
+  ansible.builtin.apt:
+    name: ufw
+    state: present
+
+- name: Deny incoming and allow outgoing by default
+  community.general.ufw:
+    direction: "{{ item.direction }}"
+    default: "{{ item.policy }}"
+  loop:
+    - { direction: incoming, policy: deny }
+    - { direction: outgoing, policy: allow }
+
+# On a fresh server we are still connected through the initial SSH port:
+# keep it open until the ssh role has moved sshd and reconnected.
+- name: Keep the initial SSH port open while bootstrapping
+  community.general.ufw:
+    rule: allow
+    port: "{{ ssh_initial_port | string }}"
+    proto: tcp
+  when: ansible_user == bootstrap_user
+
+- name: Allow the hardened SSH port and web ports
+  community.general.ufw:
+    rule: allow
+    port: "{{ item | string }}"
+    proto: tcp
+  loop: "{{ [ssh_port] + firewall_allowed_tcp_ports }}"
+
+- name: Enable the firewall
+  community.general.ufw:
+    state: enabled

ansible/roles/opsforge/defaults/main.yml

@@ -0,0 +1,2 @@
+opsforge_dir: "/home/{{ deploy_user }}/opsforge"
+opsforge_stacks_src: "{{ playbook_dir }}/../stacks"

ansible/roles/opsforge/handlers/main.yml

@@ -0,0 +1,4 @@
+- name: Restart traefik
+  community.docker.docker_compose_v2:
+    project_src: "{{ opsforge_dir }}/stacks/traefik"
+    state: restarted