linux-ops-observability-lab

Technical lab demonstrating Linux administration, Ansible-driven deployment, an Nginx reverse proxy in front of a Python application, and an observability stack (Prometheus, Grafana, Loki, Alloy) capable of detecting and helping diagnose a real incident: a backend outage and its recovery.

Grafana dashboard provisioned automatically through Ansible.

What this project demonstrates

• A Python application (Flask + gunicorn) managed as a systemd service.
• Nginx as a reverse proxy, with verified behavior on backend outage (502) and recovery (200).
• A complete, idempotent deployment with Ansible (roles, handlers, templates, variables separated from tasks).
• Metrics via Prometheus + Node Exporter, with a basic alert rule.
• Version-controlled Grafana dashboards, provisioned automatically.
• Centralized Nginx logs via Grafana Alloy → Loki, queryable from Grafana.
• A real incident reproduced and documented with log evidence (see docs/incidents/).

Architecture

flowchart LR
    Client["curl / browser"] -->|":8080"| N["nginx :8080\nreverse proxy"]
    N -->|"proxy_pass"| G["gunicorn :8000\n(linux-ops-app.service)"]
    N -->|"access/error log"| A["Grafana Alloy"]
    A -->|push| L["Loki :3100"]
    P["Prometheus :9090"] -->|scrape| NE["Node Exporter :9100"]
    P -->|scrape self| P
    P -->|rule_files| Alert["InstanceDown rule"]
    GF["Grafana :3000"] -->|datasource| P
    GF -->|datasource| L

Loading

Technologies

Linux (systemd), Python (Flask, gunicorn), Nginx, Ansible, Prometheus, Node Exporter, Grafana, Loki, Grafana Alloy.

Repository structure

app/                        Flask application source code
ansible/                    Deployment playbook and roles
  roles/common/              Base system packages
  roles/application/         systemd service for the Python app
  roles/nginx/                Reverse proxy on :8080
  roles/node_exporter/        System metrics
  roles/prometheus/           Scraping + alert rules
  roles/grafana_repo/         Official Grafana APT repository
  roles/loki/                 Log storage
  roles/alloy/                 Nginx log collection
  roles/grafana/               Provisioned datasources and dashboard
docs/incidents/              Documented evidence of real incidents
scripts/                     Failure simulation and recovery
screenshots/                 Dashboard screenshots (pending)

Requirements

• Ubuntu 24.04 (or similar, with apt and systemd).
• Ansible core ≥ 2.13 (tested with 2.16.3) on the control host.
• sudo access on the target host (the playbook uses become: true).
• Connectivity to apt.grafana.com to install Grafana, Loki and Alloy.

Quick start — deployment with Ansible

cd ansible
ansible-playbook site.yml --ask-become-pass

ansible.cfg already points to the inventory/local.ini inventory (a single local host). To deploy to another machine, replace the inventory with one pointing to the real host and its SSH credentials.

Syntax and idempotency validation

cd ansible
ansible-playbook --syntax-check site.yml
ansible-playbook site.yml --check --diff --ask-become-pass

A second run should report no changes, except for a known warning from the ansible.builtin.pip module in --check mode (see "Known limitations").

Manual validation

curl -i http://127.0.0.1:8000/        # app directly -> 200
curl -i http://127.0.0.1:8080/        # through Nginx -> 200
curl -i http://127.0.0.1:8000/health  # -> 200 {"status":"ok"}

Incident simulation and recovery

./scripts/simulate-failure.sh   # stops linux-ops-app and confirms HTTP 502
./scripts/recover.sh            # starts linux-ops-app and confirms HTTP 200

simulate-failure.sh saves the service's initial state and, if the script fails or is interrupted (Ctrl+C), restores it automatically. If the simulation completes successfully, it deliberately leaves the service stopped so that recover.sh performs the recovery explicitly.

Evidence of a real run of this flow, with timestamps and Nginx logs: docs/incidents/2026-07-20-app-down.md.

Diagnosing an incident

1. Logs: journalctl -u linux-ops-app -n 50 and tail -f /var/log/nginx/linux-ops-error.log show the connect() failed (111: Connection refused) error when the backend is down.
2. Grafana → Loki: the "Nginx Access Logs" panel on the linux-ops-overview dashboard shows 502 lines in real time.
3. Prometheus: the InstanceDown rule (up == 0 for 30s) is evaluated against the scraped targets (Prometheus, Node Exporter).

Troubleshooting

Symptom	Likely cause	Verification
Nginx returns 502	linux-ops-app is not active	systemctl status linux-ops-app
nginx -t fails	Syntax error in the vhost	nginx -t shows the exact file and line
Grafana shows no Loki data	Alloy is not reading the logs	systemctl status alloy and journalctl -u alloy
Prometheus target down	The corresponding exporter is not active	curl 127.0.0.1:9100/metrics (Node Exporter)

Known limitations

• The application does not expose /metrics; Prometheus does not monitor the app directly, only the host (Node Exporter) and Prometheus itself.
• The InstanceDown alert rule does not cover a direct HTTP probe against :8080; see "Next improvements".
• Loki storage uses /tmp/loki (Loki's default configuration), so logs do not persist across a host restart.
• The observability services (Prometheus, Grafana, Loki, Node Exporter) listen on all interfaces (0.0.0.0), which is acceptable for a local lab but should be restricted to 127.0.0.1 or protected with a firewall in any shared environment.
• There is no Alertmanager: alerts are evaluable in Prometheus (/alerts) but are not routed to any notification channel.

Technical decisions

• Ansible with per-component roles instead of a single flat playbook, so each piece (app, nginx, prometheus, grafana, loki, alloy) is independent and has its own handlers.
• Handlers instead of unconditional restarts: each service is only restarted or reloaded when its configuration actually changes.
• gunicorn instead of Flask's development server: app.run() is not suitable for production; the systemd unit runs gunicorn with 2 workers.
• docs/incidents/ with real log evidence instead of a purely descriptive simulation.

Next improvements

• Add blackbox_exporter to alert directly on the HTTP availability of :8080 (would detect the 502 without relying on logs).
• Dashboard screenshots taken during the incident, in screenshots/.
• Loki persistence outside of /tmp.
• Restrict Prometheus/Grafana/Loki/Node Exporter binding to 127.0.0.1 when remote access is not needed.

License

MIT