From 1478a18e5e2ec6e7decac11cce8182970913bf2f Mon Sep 17 00:00:00 2001 From: Tarit Witworrasakul <60599564+wtarit@users.noreply.github.com> Date: Sun, 5 Apr 2026 05:48:44 -0700 Subject: [PATCH 1/3] remove docker stuff --- RUNBOOK.md | 156 +++++++++++++++++------------------------------------ 1 file changed, 49 insertions(+), 107 deletions(-) diff --git a/RUNBOOK.md b/RUNBOOK.md index 8924702..750b245 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -2,41 +2,35 @@ ## In Case of Emergency -**First, identify your environment:** -- **Docker Development**: If running `docker-compose ps` shows containers -- **Kubernetes Production**: If `kubectl get pods -n url-shortener` shows pods - When an alert fires, follow this checklist in order: 1. **Access the environment** - - **Docker Development**: Ensure you're on the local machine with Docker running - - **Kubernetes Production**: Access services directly via NodePorts: + - Access services via NodePorts: - Prometheus: `http://192.168.1.112:30090` (or .110, .111) - Grafana: `http://192.168.1.112:30030` (or .110, .111) - App: `http://192.168.1.112:30080` (or .110, .111) - - Alertmanager: Run `kubectl port-forward svc/alertmanager-svc -n monitoring 9093:9093` from your local machine (with kubectl configured for the cluster), then access `http://localhost:9093` - - For SSH access to run commands: `ssh k9@192.168.1.112` (ensure kubectl is configured) - - Verify you have access to monitoring tools and can run diagnostic commands + - Alertmanager: Run `kubectl port-forward svc/alertmanager-svc -n monitoring 9093:9093`, then access `http://localhost:9093` 2. **Confirm the alert** - - **Docker**: Open Alertmanager: `http://localhost:9093` - - **Kubernetes**: - - Use NodePort services: `http://192.168.1.112:30090` (Prometheus), `http://192.168.1.112:30030` (Grafana) - - For Alertmanager: Run `kubectl port-forward svc/alertmanager-svc -n monitoring 9093:9093` from your local machine, then access `http://localhost:9093` - - Check targets page to verify service status + - Check Prometheus targets: `http://192.168.1.112:30090/targets` + - Check Grafana dashboards: `http://192.168.1.112:30030` + - For Alertmanager: Run `kubectl port-forward svc/alertmanager-svc -n monitoring 9093:9093`, then access `http://localhost:9093` - Identify alert name, severity, affected target, and status -2. **Check current service state** - - **Docker**: Confirm app containers: `docker ps --format 'table {{.Names}}\t{{.Status}}' | grep pe-hackathon-app` - - **Kubernetes**: Check pods: `kubectl get pods -n url-shortener` - - Confirm Prometheus target status: `http://localhost:9090/targets` (or `http://:30090/targets` in K8s) -3. **Capture evidence** + +3. **Check current service state** + - Check pods: `kubectl get pods -n url-shortener -o wide` + - Confirm Prometheus target status: `http://192.168.1.112:30090/targets` + +4. **Capture evidence** - Save the alert details and timestamps - Collect relevant logs and metrics screenshots -4. **Triage by alert type** - - ServiceDown → check container/pod availability + +5. **Triage by alert type** + - ServiceDown → check pod availability - HighErrorRate → inspect app errors and traffic patterns - HighLatency → inspect resource usage and slow queries -5. **Resolve and document** + +6. **Resolve and document** - Apply fix, then verify the alert returns to normal - Note root cause, resolution, and next preventative action @@ -44,57 +38,43 @@ When an alert fires, follow this checklist in order: ## Service Down Alert -**Symptoms**: Prometheus alert `ServiceDown` fired +**Symptoms**: Prometheus alert `ServiceDown` fired (app instance not reachable) -**What it means**: Prometheus cannot scrape a Flask app target. This usually means the app container is stopped, unreachable, or the service is not listening. +**What it means**: Prometheus cannot scrape a Flask app target. This usually means the pod is crashed, unreachable, or the service is not listening. **Immediate Actions**: -1. Check app container state: - - `docker ps -a --format 'table {{.Names}}\t{{.Status}}' | grep pe-hackathon-app` -2. If the container is stopped, inspect its logs: - - `docker logs pe-hackathon-app1-1 --tail 50` -3. Confirm the target address resolves in the app network: - - `docker exec pe-hackathon-prometheus-1 wget -qO- http://app1:5000/metrics` -4. Check if the app is listening on port 5000: - - `docker exec pe-hackathon-app1-1 netstat -tlnp | grep 5000` -5. If the container is healthy but unreachable, check Docker networking and DNS. +1. Check pod status: `kubectl get pods -n url-shortener -o wide` +2. If pod is in CrashLoopBackOff, check logs: `kubectl logs -n url-shortener --previous` +3. Check deployment events: `kubectl describe deployment url-shortener -n url-shortener` +4. Verify database connectivity: `kubectl exec -n url-shortener -- pg_isready -h postgres-cluster-rw -U postgres` +5. Check node resources if pods can't schedule: `kubectl describe node ` **Diagnosis**: -- `Exited (0)` or similar: container was stopped cleanly -- `OOM killed` / `signal`: resource or process failure -- App startup error: configuration or dependency issue -- DNS failure: target name not resolvable inside Prometheus +- CrashLoopBackOff: app is crashing on startup — check logs for config or dependency errors +- Pending pods: insufficient resources or scheduling constraints +- OOMKilled: pod exceeded memory limits +- DNS failure: target name not resolvable inside the cluster **Resolution**: -- Restart the app container: `docker start pe-hackathon-app1-1` -- If using compose, restart the service: `docker-compose up -d app1` -- If the app repeatedly exits, fix the root cause from logs and restart - -**Kubernetes Production Resolution**: -- Check pod status: `kubectl get pods -n url-shortener -o wide` -- If pod is in CrashLoopBackOff, check logs: `kubectl logs -n url-shortener --previous` - Restart the pod: `kubectl delete pod -n url-shortener` (deployment will recreate it) -- If issue persists, check deployment events: `kubectl describe deployment url-shortener -n url-shortener` -- Verify database connectivity: `kubectl exec -n url-shortener -- pg_isready -h postgres-cluster-rw -U postgres` -- Check node resources if pods can't schedule: `kubectl describe node ` +- Restart the deployment: `kubectl rollout restart deployment/url-shortener -n url-shortener` +- If the app repeatedly crashes, fix the root cause from logs and redeploy --- ## High Error Rate Alert -**Symptoms**: `HighErrorRate` fired +**Symptoms**: `HighErrorRate` fired (> 5% 5xx error rate for 2 minutes) **What it means**: A high percentage of requests returned 5xx responses over the last 5 minutes. **Immediate Actions**: -1. Check app error logs: - - `docker logs pe-hackathon-app1-1 --tail 100 | grep -i error` - - `docker logs pe-hackathon-app2-1 --tail 100 | grep -i error` -2. Identify failing request patterns in Grafana or Prometheus. -3. Confirm if the issue is isolated to a single endpoint or system-wide. -4. Check the database and external dependencies: - - `docker exec pe-hackathon-db-1 pg_isready -U postgres` - - `docker stats pe-hackathon-db-1` +1. Check logs from all app pods: `kubectl logs -n url-shortener -l app=url-shortener --tail 100 | grep -i error` +2. Identify failing request patterns in Grafana (`http://192.168.1.112:30030`) +3. Confirm if the issue is isolated to a single endpoint or system-wide +4. Check database cluster health: `kubectl get cluster -n url-shortener` +5. Verify Valkey connectivity: `kubectl exec -n url-shortener -- redis-cli -h valkey ping` +6. Check pod resource usage: `kubectl top pods -n url-shortener` **Diagnosis**: - Database connection failures @@ -104,31 +84,23 @@ When an alert fires, follow this checklist in order: **Resolution**: - Fix the failing code or configuration -- Roll back recent deployments if necessary -- Restart app services after the root cause is fixed - -**Kubernetes Production Resolution**: -- Check logs from all app pods: `kubectl logs -n url-shortener -l app=url-shortener --tail 100 | grep -i error` -- If using Loki, query logs in Grafana for error patterns -- Check database cluster health: `kubectl get cluster -n url-shortener` -- Verify Valkey connectivity: `kubectl exec -n url-shortener -- redis-cli -h valkey ping` - Roll back deployment if needed: `kubectl rollout undo deployment/url-shortener -n url-shortener` -- Check pod resource usage: `kubectl top pods -n url-shortener` +- Restart app services after the root cause is fixed --- ## High Latency Alert -**Symptoms**: `HighLatency` fired +**Symptoms**: `HighLatency` fired (95th percentile request latency > 2 seconds for 2 minutes) **What it means**: The 95th percentile request latency is above the configured threshold, indicating slow responses. **Immediate Actions**: -1. Check app performance metrics in Grafana. -2. Review database query latency and slow requests. -3. Check container resource usage: - - `docker stats pe-hackathon-app1-1 pe-hackathon-app2-1 pe-hackathon-db-1` -4. Confirm if latency is caused by CPU, memory, or I/O bottlenecks. +1. Check pod resource usage: `kubectl top pods -n url-shortener` +2. Check node resources: `kubectl top nodes` +3. Review app performance metrics in Grafana (`http://192.168.1.112:30030`) +4. Check database performance: `kubectl exec -n url-shortener postgres-cluster-1 -- pg_stat_activity` +5. Review Valkey performance: `kubectl exec -n url-shortener valkey-node-0 -- redis-cli info stats` **Diagnosis**: - Slow database queries @@ -138,20 +110,12 @@ When an alert fires, follow this checklist in order: **Resolution**: - Optimize the slow path or query -- Add caching if appropriate -- Scale resources or reduce load - -**Kubernetes Production Resolution**: -- Check pod resource usage: `kubectl top pods -n url-shortener` -- Check node resources: `kubectl top nodes` - Scale deployment if needed: `kubectl scale deployment url-shortener -n url-shortener --replicas=4` -- Check database performance: `kubectl exec -n url-shortener postgres-cluster-1 -- pg_stat_activity` -- Review Valkey performance: `kubectl exec -n url-shortener valkey-node-0 -- redis-cli info stats` -- Check network policies or service mesh if applicable +- Add caching if appropriate --- -## Kubernetes HA Scenarios +## HA Scenarios **Node Failure**: - Symptoms: Multiple pods become unavailable, alerts fire for affected services @@ -176,35 +140,16 @@ When an alert fires, follow this checklist in order: ## Common Emergency Commands -### Docker Development -- Restart app service: `docker-compose restart app` -- Check logs for all app instances: - - `docker logs pe-hackathon-app1-1 --tail 100` - - `docker logs pe-hackathon-app2-1 --tail 100` -- Check Prometheus targets: `http://localhost:9090/targets` -- Check Alertmanager: `http://localhost:9093` -- Verify app metrics endpoint: `http://localhost:5000/metrics` - -### Kubernetes Production - Check pod status: `kubectl get pods -n url-shortener -o wide` - Check logs: `kubectl logs -n url-shortener -l app=url-shortener --tail 100` - Restart deployment: `kubectl rollout restart deployment/url-shortener -n url-shortener` - Check node status: `kubectl get nodes` +- Check database cluster: `kubectl get cluster -n url-shortener` - Access monitoring services via NodePorts: - Prometheus: `http://192.168.1.112:30090` - Grafana: `http://192.168.1.112:30030` - App: `http://192.168.1.112:30080` - Alertmanager: `kubectl port-forward svc/alertmanager-svc -n monitoring 9093:9093` then `http://localhost:9093` -- Check database cluster: `kubectl get cluster -n url-shortener` - -## Kubernetes Production Endpoints - -If using the Kubernetes production stack, use the node port endpoints: - -- App: `http://:30080` -- Prometheus: `http://:30090` -- Grafana: `http://:30030` -- Alertmanager: port-forward with `kubectl port-forward svc/alertmanager-svc -n monitoring 9093:9093` then open `http://localhost:9093` To find the node IP: - `kubectl get nodes -o wide` @@ -219,6 +164,8 @@ After the incident: - Note any policy, deployment, or dependency changes - Review whether the alert produced a useful signal or noise +--- + ## Troubleshooting Access Issues **Cannot access NodePort services:** @@ -231,16 +178,11 @@ After the incident: - Check if services exist: `kubectl get svc -n monitoring` - Verify pods are running: `kubectl get pods -n monitoring` - Check service selectors match pod labels -- Ensure you're running the command on a machine with kubectl access to the cluster -**SSH access issues:** -- Verify SSH key or password authentication -- Check if SSH service is running on target nodes -- Ensure firewall allows SSH (port 22) +--- ## Existing Alert Summaries -Keep the following current as reference: - `ServiceDown`: app instance not reachable - `HighErrorRate`: > 5% 5xx error rate for 2 minutes - `HighLatency`: 95th percentile request latency > 2 seconds for 2 minutes From b5a2dafa4c23b963e3c4cba16221226fae396771 Mon Sep 17 00:00:00 2001 From: Tarit Witworrasakul <60599564+wtarit@users.noreply.github.com> Date: Sun, 5 Apr 2026 05:53:33 -0700 Subject: [PATCH 2/3] Update run book --- RUNBOOK.md | 211 ++++++++++++++++++++--------------------------------- 1 file changed, 78 insertions(+), 133 deletions(-) diff --git a/RUNBOOK.md b/RUNBOOK.md index 750b245..888f4e4 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -1,188 +1,133 @@ # Incident Response Runbook -## In Case of Emergency +## Service Endpoints -When an alert fires, follow this checklist in order: +| Service | URL | +|---|---| +| App | `http://192.168.1.112:30080` (or .110, .111) | +| Prometheus | `http://192.168.1.112:30090` (or .110, .111) | +| Grafana | `http://192.168.1.112:30030` (or .110, .111) | +| Alertmanager | `kubectl port-forward svc/alertmanager-svc -n monitoring 9093:9093` → `http://localhost:9093` | -1. **Access the environment** - - Access services via NodePorts: - - Prometheus: `http://192.168.1.112:30090` (or .110, .111) - - Grafana: `http://192.168.1.112:30030` (or .110, .111) - - App: `http://192.168.1.112:30080` (or .110, .111) - - Alertmanager: Run `kubectl port-forward svc/alertmanager-svc -n monitoring 9093:9093`, then access `http://localhost:9093` +Find node IPs: `kubectl get nodes -o wide` -2. **Confirm the alert** - - Check Prometheus targets: `http://192.168.1.112:30090/targets` - - Check Grafana dashboards: `http://192.168.1.112:30030` - - For Alertmanager: Run `kubectl port-forward svc/alertmanager-svc -n monitoring 9093:9093`, then access `http://localhost:9093` - - Identify alert name, severity, affected target, and status - -3. **Check current service state** - - Check pods: `kubectl get pods -n url-shortener -o wide` - - Confirm Prometheus target status: `http://192.168.1.112:30090/targets` - -4. **Capture evidence** - - Save the alert details and timestamps - - Collect relevant logs and metrics screenshots +--- -5. **Triage by alert type** - - ServiceDown → check pod availability - - HighErrorRate → inspect app errors and traffic patterns - - HighLatency → inspect resource usage and slow queries +## When an Alert Fires -6. **Resolve and document** - - Apply fix, then verify the alert returns to normal - - Note root cause, resolution, and next preventative action +1. **Confirm** — Open Alertmanager and identify the alert name, severity, and affected target +2. **Assess** — Check pod state and Prometheus targets: + - `kubectl get pods -n url-shortener -o wide` + - Prometheus targets page: `/targets` +3. **Triage** — Jump to the relevant section below +4. **Resolve** — Apply fix, verify alert clears, document root cause --- ## Service Down Alert -**Symptoms**: Prometheus alert `ServiceDown` fired (app instance not reachable) +> `ServiceDown` — app instance not reachable -**What it means**: Prometheus cannot scrape a Flask app target. This usually means the pod is crashed, unreachable, or the service is not listening. +Prometheus cannot scrape the app. Pod is likely crashed, pending, or not listening. -**Immediate Actions**: -1. Check pod status: `kubectl get pods -n url-shortener -o wide` -2. If pod is in CrashLoopBackOff, check logs: `kubectl logs -n url-shortener --previous` -3. Check deployment events: `kubectl describe deployment url-shortener -n url-shortener` -4. Verify database connectivity: `kubectl exec -n url-shortener -- pg_isready -h postgres-cluster-rw -U postgres` -5. Check node resources if pods can't schedule: `kubectl describe node ` +**Investigate**: +1. `kubectl get pods -n url-shortener -o wide` +2. `kubectl logs -n url-shortener --previous` (if CrashLoopBackOff) +3. `kubectl describe deployment url-shortener -n url-shortener` +4. `kubectl exec -n url-shortener -- pg_isready -h postgres-cluster-rw -U postgres` +5. `kubectl describe node ` (if pods stuck in Pending) -**Diagnosis**: -- CrashLoopBackOff: app is crashing on startup — check logs for config or dependency errors -- Pending pods: insufficient resources or scheduling constraints -- OOMKilled: pod exceeded memory limits -- DNS failure: target name not resolvable inside the cluster +**Common causes**: +- CrashLoopBackOff → config or dependency error in logs +- Pending → insufficient node resources +- OOMKilled → pod exceeded memory limits -**Resolution**: -- Restart the pod: `kubectl delete pod -n url-shortener` (deployment will recreate it) -- Restart the deployment: `kubectl rollout restart deployment/url-shortener -n url-shortener` -- If the app repeatedly crashes, fix the root cause from logs and redeploy +**Fix**: +- Delete pod to let deployment recreate it: `kubectl delete pod -n url-shortener` +- Or restart deployment: `kubectl rollout restart deployment/url-shortener -n url-shortener` +- If recurring, fix root cause from logs and redeploy --- ## High Error Rate Alert -**Symptoms**: `HighErrorRate` fired (> 5% 5xx error rate for 2 minutes) - -**What it means**: A high percentage of requests returned 5xx responses over the last 5 minutes. +> `HighErrorRate` — > 5% 5xx error rate for 2 minutes -**Immediate Actions**: -1. Check logs from all app pods: `kubectl logs -n url-shortener -l app=url-shortener --tail 100 | grep -i error` -2. Identify failing request patterns in Grafana (`http://192.168.1.112:30030`) -3. Confirm if the issue is isolated to a single endpoint or system-wide -4. Check database cluster health: `kubectl get cluster -n url-shortener` -5. Verify Valkey connectivity: `kubectl exec -n url-shortener -- redis-cli -h valkey ping` -6. Check pod resource usage: `kubectl top pods -n url-shortener` +**Investigate**: +1. `kubectl logs -n url-shortener -l app=url-shortener --tail 100 | grep -i error` +2. Check Grafana for failing endpoint patterns +3. `kubectl get cluster -n url-shortener` (database health) +4. `kubectl exec -n url-shortener -- redis-cli -h valkey ping` (Valkey health) +5. `kubectl top pods -n url-shortener` -**Diagnosis**: +**Common causes**: - Database connection failures -- API or code exceptions -- Bad request payloads or validation issues -- Dependency timeouts / upstream failures +- Code exceptions or bad deployments +- Dependency timeouts -**Resolution**: -- Fix the failing code or configuration -- Roll back deployment if needed: `kubectl rollout undo deployment/url-shortener -n url-shortener` -- Restart app services after the root cause is fixed +**Fix**: +- Roll back if caused by recent deploy: `kubectl rollout undo deployment/url-shortener -n url-shortener` +- Fix failing code or configuration and redeploy --- ## High Latency Alert -**Symptoms**: `HighLatency` fired (95th percentile request latency > 2 seconds for 2 minutes) +> `HighLatency` — p95 request latency > 2 seconds for 2 minutes -**What it means**: The 95th percentile request latency is above the configured threshold, indicating slow responses. +**Investigate**: +1. `kubectl top pods -n url-shortener` +2. `kubectl top nodes` +3. Check Grafana latency panels +4. `kubectl exec -n url-shortener postgres-cluster-1 -- pg_stat_activity` (slow queries) +5. `kubectl exec -n url-shortener valkey-node-0 -- redis-cli info stats` -**Immediate Actions**: -1. Check pod resource usage: `kubectl top pods -n url-shortener` -2. Check node resources: `kubectl top nodes` -3. Review app performance metrics in Grafana (`http://192.168.1.112:30030`) -4. Check database performance: `kubectl exec -n url-shortener postgres-cluster-1 -- pg_stat_activity` -5. Review Valkey performance: `kubectl exec -n url-shortener valkey-node-0 -- redis-cli info stats` - -**Diagnosis**: +**Common causes**: - Slow database queries -- High CPU or memory usage -- Network latency or I/O waits -- Backpressure from downstream services +- High CPU/memory usage +- Network or I/O bottlenecks -**Resolution**: -- Optimize the slow path or query -- Scale deployment if needed: `kubectl scale deployment url-shortener -n url-shortener --replicas=4` -- Add caching if appropriate +**Fix**: +- Scale up to reduce load: `kubectl scale deployment url-shortener -n url-shortener --replicas=4` +- Document slow queries or bottlenecks found for follow-up fix after the incident --- ## HA Scenarios **Node Failure**: -- Symptoms: Multiple pods become unavailable, alerts fire for affected services -- Check node status: `kubectl get nodes` -- If node is NotReady, investigate: `kubectl describe node ` -- Pods will automatically reschedule to healthy nodes (if anti-affinity allows) -- Monitor cluster recovery: `kubectl get pods -n url-shortener -w` +- `kubectl get nodes` — look for NotReady +- `kubectl describe node ` to investigate +- Pods auto-reschedule to healthy nodes; monitor with `kubectl get pods -n url-shortener -w` **Database Failover**: -- Symptoms: PostgreSQL cluster shows degraded status -- Check cluster health: `kubectl get cluster -n url-shortener` -- Monitor failover: `kubectl logs -n cnpg-system -l app.kubernetes.io/name=cloudnative-pg --follow` +- `kubectl get cluster -n url-shortener` — check cluster health +- `kubectl logs -n cnpg-system -l app.kubernetes.io/name=cloudnative-pg --follow` — monitor failover - Verify new primary: `kubectl get pods -n url-shortener -l cnpg.io/cluster=postgres-cluster -o custom-columns=NAME:.metadata.name,ROLE:.metadata.labels.role` **Valkey Sentinel Failover**: -- Symptoms: Cache operations fail, Valkey pods restart -- Check Valkey cluster: `kubectl get pods -n url-shortener -l app.kubernetes.io/name=valkey` -- Monitor Sentinel logs: `kubectl logs -n url-shortener valkey-node-0 -c sentinel` -- Verify master election: `kubectl exec -n url-shortener valkey-node-0 -- redis-cli -p 26379 sentinel masters` - ---- - -## Common Emergency Commands - -- Check pod status: `kubectl get pods -n url-shortener -o wide` -- Check logs: `kubectl logs -n url-shortener -l app=url-shortener --tail 100` -- Restart deployment: `kubectl rollout restart deployment/url-shortener -n url-shortener` -- Check node status: `kubectl get nodes` -- Check database cluster: `kubectl get cluster -n url-shortener` -- Access monitoring services via NodePorts: - - Prometheus: `http://192.168.1.112:30090` - - Grafana: `http://192.168.1.112:30030` - - App: `http://192.168.1.112:30080` - - Alertmanager: `kubectl port-forward svc/alertmanager-svc -n monitoring 9093:9093` then `http://localhost:9093` - -To find the node IP: -- `kubectl get nodes -o wide` +- `kubectl get pods -n url-shortener -l app.kubernetes.io/name=valkey` +- `kubectl logs -n url-shortener valkey-node-0 -c sentinel` +- Verify master: `kubectl exec -n url-shortener valkey-node-0 -- redis-cli -p 26379 sentinel masters` --- -## Post-Incident Notes +## Post-Incident -After the incident: -- Document the root cause and fix +- Document root cause and fix - Update alert thresholds if needed -- Note any policy, deployment, or dependency changes -- Review whether the alert produced a useful signal or noise +- Review whether the alert was useful signal or noise --- -## Troubleshooting Access Issues - -**Cannot access NodePort services:** -- Verify firewall allows access to node ports (30080, 30090, 30030) -- Check if nodes are reachable: `ping 192.168.1.112` -- Confirm services are exposed: `kubectl get svc -n monitoring -o wide` -- Try different node IPs (.110, .111) if one node is down - -**kubectl port-forward not working for Alertmanager:** -- Check if services exist: `kubectl get svc -n monitoring` -- Verify pods are running: `kubectl get pods -n monitoring` -- Check service selectors match pod labels - ---- +## Troubleshooting Access -## Existing Alert Summaries +**Cannot reach NodePort services:** +- Check firewall allows ports 30080, 30090, 30030 +- `ping 192.168.1.112` — verify node is reachable +- `kubectl get svc -n monitoring -o wide` — confirm services are exposed +- Try other node IPs (.110, .111) if one is down -- `ServiceDown`: app instance not reachable -- `HighErrorRate`: > 5% 5xx error rate for 2 minutes -- `HighLatency`: 95th percentile request latency > 2 seconds for 2 minutes +**Alertmanager port-forward not working:** +- `kubectl get svc -n monitoring` — verify service exists +- `kubectl get pods -n monitoring` — verify pod is running From c778f3691228daecb69b2fcdfb13b60f53a95ccb Mon Sep 17 00:00:00 2001 From: Tarit Witworrasakul <60599564+wtarit@users.noreply.github.com> Date: Sun, 5 Apr 2026 05:54:40 -0700 Subject: [PATCH 3/3] move RUNBOOK.md to docs --- RUNBOOK.md => docs/IncidentReportQuest/RUNBOOK.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename RUNBOOK.md => docs/IncidentReportQuest/RUNBOOK.md (100%) diff --git a/RUNBOOK.md b/docs/IncidentReportQuest/RUNBOOK.md similarity index 100% rename from RUNBOOK.md rename to docs/IncidentReportQuest/RUNBOOK.md