various fixes after another round of discord solving simulation

skills/crowdsec/SKILL.md

@@ -99,7 +99,8 @@ Hub. If it's far behind `version.crowdsec.net/latest`, pull a newer tag
 | "ban duration", "captcha", "decisions", "simulation", "alerts but no bans" | [references/configure/profiles.md](./references/configure/profiles.md) |
 | "allowlist my office / CDN / monitoring IP", "I'm getting blocked by CAPI", "exclude IP from any ban" | [references/configure/allowlists.md](./references/configure/allowlists.md) |
 | "whitelist vs allowlist vs postoverflow", "which suppression layer should I use" | [references/configure/allowlists.md](./references/configure/allowlists.md) § Suppression mechanisms |
-| "alert me on slack/email/webhook" | [references/configure/notifications.md](./references/configure/notifications.md) *(TODO — stub)* |
+| "test my whitelist works", "is my postoverflow / dynamic-IP whitelist actually firing" | [references/configure/allowlists.md](./references/configure/allowlists.md) § Verification — does a whitelist actually work? |
+| "notifications", "alert me on slack/email/webhook", "notification not firing" | [references/configure/notifications.md](./references/configure/notifications.md) |
 | "block at the firewall", "iptables", "nftables", "ipset" | [references/configure/bouncers/firewall.md](./references/configure/bouncers/firewall.md) |
 | "nginx bouncer", "lua / openresty module" | [references/configure/bouncers/web-servers.md](./references/configure/bouncers/web-servers.md) § nginx |
 | "haproxy bouncer", "SPOA / SPOE" | [references/configure/bouncers/web-servers.md](./references/configure/bouncers/web-servers.md) § haproxy |
@@ -111,14 +112,16 @@ Hub. If it's far behind `version.crowdsec.net/latest`, pull a newer tag
 | "Console", "enroll", "share signals" | [references/install/console.md](./references/install/console.md) |
 | "upgrade", "back up", "roll back", "new version", "tainted items after upgrade" | [references/operate/upgrades.md](./references/operate/upgrades.md) |
 | "old/outdated version", "`cscli` command or flag missing", "hub item won't install", "behavior doesn't match the docs", "installed from the distro package" | [references/operate/upgrades.md](./references/operate/upgrades.md) § Detect & fix an outdated / distro-packaged install (see **Step 1.5** above) |
-| "multiple agents", "remote LAPI", "mTLS", "postgres backend" | [references/operate/multi-server.md](./references/operate/multi-server.md) *(TODO — stub)* |
+| "multiple agents", "remote LAPI", "mTLS", "postgres backend" | [references/operate/multi-server.md](./references/operate/multi-server.md) *(partial — machine cleanup done, rest stub)* |
+| "stale machines / log processors in `cscli machines list`", "prune dead agents", "ephemeral k8s pods piling up" | [references/operate/multi-server.md](./references/operate/multi-server.md) § Decommissioning stale machines |
 | "is it working?", "smoke test", "validate install", "verify setup", "did detection / WAF / blocking actually wire up?" | [references/operate/health-check.md](./references/operate/health-check.md) |
 | **Debug — common** · "it's broken" / "not working" / general diagnosis | [references/debug/common/triage.md](./references/debug/common/triage.md) → run `bash ${CLAUDE_SKILL_DIR}/scripts/diagnose.sh` |
 | **Debug — common** · specific error string | [references/debug/common/errors.md](./references/debug/common/errors.md) |
 | **Debug — common** · "container can't see logs", "mount", "SELinux/AppArmor denied", "k8s RBAC / DaemonSet" | [references/debug/common/platform-gotchas.md](./references/debug/common/platform-gotchas.md) |
 | **Debug — by symptom** · "logs not parsed", "0 parsed" | [references/debug/symptoms/parsing.md](./references/debug/symptoms/parsing.md) |
 | **Debug — by symptom** · "no alerts firing" | [references/debug/symptoms/no-alerts.md](./references/debug/symptoms/no-alerts.md) |
 | **Debug — by symptom** · "decision exists but not blocked" | [references/debug/symptoms/not-blocked.md](./references/debug/symptoms/not-blocked.md) |
+| **Debug — by symptom** · "bouncer blocks everything", "locked out of all services", "every request 403 after adding the bouncer" | [references/debug/symptoms/not-blocked.md](./references/debug/symptoms/not-blocked.md) § 7 — Inverse symptom |
 | **Debug — by feature** · AppSec/WAF not blocking, false positives, captcha | [references/appsec/troubleshoot.md](./references/appsec/troubleshoot.md) |
 | "switch from fail2ban" | [references/migrate/from-fail2ban.md](./references/migrate/from-fail2ban.md) *(TODO — stub)* |
 

skills/crowdsec/references/configure/allowlists.md

@@ -4,6 +4,10 @@ verified:
     version: "1.7.8"
     env: systemd
     notes: "allowlists add/check/list"
+  - date: 2026-05-26
+    version: "1.7.8"
+    env: systemd
+    notes: "parser whitelist via cscli explain --verbose (ignored by whitelist, private-IP); postoverflow via crowdsec -dsn ... -no-api replay (whitelisted-skip lines)"
 ---
 
 # Configure — Allowlists, whitelist parsers, and postoverflows
@@ -200,6 +204,70 @@ sudo cscli decisions add -i 192.0.2.99 --duration 5m --reason "retest"
 sudo cscli allowlists delete test
 ```
 
+## Verification — does a whitelist actually work?
+
+Allowlists are tested at the LAPI (above). The two **whitelist** layers run at
+different phases and are tested with different tools. *Authoring* either is out of
+scope (canonical docs cover syntax); confirming a deployed one fires is operational.
+
+### Whitelist parser (`parsers/s02-enrich/`) — use `cscli explain`
+
+A whitelist parser drops the **event** during enrichment, so `cscli explain` shows
+it directly — no traffic or bucket needed. Replay a line from a whitelisted source
+(the shipped `crowdsecurity/whitelists` covers RFC1918, so any private IP works):
+
+```bash
+LINE='May 26 10:00:00 host sshd[111]: Failed password for invalid user admin from 10.0.0.5 port 2222 ssh2'
+sudo cscli explain --log "$LINE" --type syslog --verbose
+```
+
+A matching whitelist tags the parser and ends the line with `ignored by whitelist`:
+
+```
+	|	└ 🟢 crowdsecurity/whitelists (~2 [whitelisted])
+	|		└ update evt.Whitelisted : false -> true
+	|		└ update evt.WhitelistReason :  -> private ipv4/ipv6 ip/ranges
+	└-------- parser success, ignored by whitelist (private ipv4/ipv6 ip/ranges) 🟢
+```
+
+No `[whitelisted]` tag / no `ignored by whitelist` line → your parser isn't
+matching (wrong expression, file not in `s02-enrich`, or not reloaded).
+
+### Postoverflow (`postoverflows/s01-whitelist/`) — replay through the engine
+
+A postoverflow runs **after** a bucket overflows, which `cscli explain` does not
+simulate (it stops at scenario routing). The dynamic-home-IP whitelist
+(<https://docs.crowdsec.net/u/getting_started/post_installation/whitelists/>) is this
+kind. Test it by replaying enough log lines to overflow, in one-shot mode:
+
+```bash
+# build a log that would trip the scenario from the whitelisted IP
+for i in $(seq 1 20); do
+  printf 'May 26 10:00:%02d host sshd[%d]: Failed password for invalid user x from 203.0.113.77 port 22 ssh2\n' "$i" "$i"
+done | sudo tee /tmp/bf.log >/dev/null
+
+sudo crowdsec -dsn file:///tmp/bf.log -type syslog -no-api
+```
+
+> **`-no-api` is mandatory while the service is running.** Without it the one-shot
+> tries to start its own LAPI and dies: `bind: address already in use` (fatal on
+> `:8080`). With `-no-api` it reuses the running LAPI.
+
+Read the output:
+
+- **Not matching** → the overflow is reported:
+  `Ip 203.0.113.77 performed 'crowdsecurity/ssh-bf' (6 events over 5s)`
+- **Matching** → the overflow is suppressed and *named*:
+  ```
+  Ban for 203.0.113.77 whitelisted, reason [home IP] name=test/home-ip-wl stage=s01-whitelist
+  [Ip 203.0.113.77 performed 'crowdsecurity/ssh-bf' ...] is whitelisted, skip.
+  ```
+  The `name=` / `stage=` confirm **your** rule matched (not a shipped one like
+  `crowdsecurity/seo-bots-whitelist`). For a dynamic-DNS whitelist
+  (`... in LookupHost("me.ddns.net")`) the same replay applies — resolution happens
+  at evaluation time, so this also catches a stale/typo'd hostname (no `whitelisted`
+  line = the name didn't resolve to the source IP).
+
 ## When allowlists silently fail to apply
 
 - **Per-request AppSec blocks bypass allowlists.** AppSec inband rules return 403 before any decision is written, so allowlisting the source IP doesn't help with inband false positives. Fix at the appsec-config (`exclude_path:` or per-rule disable). See [../appsec/troubleshoot.md](../appsec/troubleshoot.md).

skills/crowdsec/references/configure/bouncers/firewall.md

@@ -1,9 +1,9 @@
 ---
 verified:
-  - date: 2026-05-22
+  - date: 2026-05-26
     version: "1.7.8"
     env: systemd
-    notes: "nftables bouncer v0.0.34 install+register, nft sets/chains (priority filter-10), ban→cscli set→drop; added stale-key recovery note"
+    notes: "re-verified nftables bouncer v0.0.34 install+register, per-origin/family sets, stale ${API_KEY}→stream-halted, flush-table does not evict set elements; added wrong-set inverse-symptom pitfall"
 ---
 
 # Bouncers — Firewall (nftables / iptables / ipset)
@@ -141,6 +141,25 @@ sudo cscli decisions delete -i 192.0.2.66
 - **"Banned but still reachable"** → almost always `update_frequency` not
   elapsed, `disable_ipv6` masking a v6 client, or the bouncer service stopped.
   Full decision tree: [../../debug/symptoms/not-blocked.md](../../debug/symptoms/not-blocked.md).
+- **"Decision in `cscli decisions list` but not in the set"** (nft, ipset, all
+  backends) → you are almost always looking in the **wrong set**. The bouncer splits
+  decisions into multiple sets by *origin* (`-cscli` manual, `-CAPI` community,
+  `-crowdsec` console push, `-lists` blocklists) **and** by *family* (IPv4 vs IPv6).
+  Don't guess the name — **list the sets first**, then look inside the right one:
+  ```bash
+  sudo nft list sets        | grep crowdsec    # nftables backend: set names per table
+  sudo ipset list -name     | grep crowdsec    # iptables/ipset backend: set names
+  # then, e.g.:
+  sudo nft list set ip crowdsec crowdsec-blacklists-cscli | grep <ip>
+  sudo ipset test crowdsec-blacklists-cscli <ip>
+  ```
+  (A `nft flush table` / external ruleset reload does **not** evict the bouncer's
+  set elements on 1.7.8/v0.0.34 — if a restart is what "makes the IP appear",
+  suspect wrong-set/family or an unhealthy bouncer next, not a flush race.)
+- **Bouncer unhealthy = nothing enforced.** A bad/placeholder `api_key` makes LAPI
+  return `access forbidden` and the service exits with `bouncer stream halted`
+  (see §2). `systemctl is-active crowdsec-firewall-bouncer` plus the **Last API
+  pull** column in `cscli bouncers list` are the fast health checks.
 
 ## Teardown
 

skills/crowdsec/references/configure/hub.md

@@ -4,6 +4,10 @@ verified:
     version: "1.7.8"
     env: systemd
     notes: "update/upgrade, list -o raw, collections install+remove, inspect tainted fields"
+  - date: 2026-05-26
+    version: "1.7.8"
+    env: systemd
+    notes: "reproduced dangling enable-symlink → item silently not loaded → parser failure; find -xtype l fix"
 ---
 
 # Configure — Hub management
@@ -124,6 +128,20 @@ sudo systemctl reload crowdsec
   recreate / `helm upgrade`) to take effect.
 - **`upgrade` silently skips your local/tainted items** — by design. Reconcile them
   deliberately with `--force` (after saving any edits to an override).
+- **Dangling enable-symlinks silently drop items.** If `hub_dir` is changed in
+  `config.yaml`, or a hub dir is partially restored, the enable-symlinks under
+  `/etc/crowdsec/{parsers,scenarios,postoverflows,collections,...}/` can point at a
+  target that no longer exists. CrowdSec logs `Ignoring file …: lstat …: no such file
+  or directory` (a *warning*, easy to miss) and **does not load that item** — so you
+  can have `crowdsecurity/sshd-success-logs` loaded while `sshd-logs` is dangling,
+  and `cscli explain` then shows `🔴 parser failure` with no alerts ever raised.
+  Detect and fix:
+  ```bash
+  sudo find /etc/crowdsec -xtype l            # list broken (dangling) symlinks
+  sudo find /etc/crowdsec -xtype l -delete     # remove them
+  sudo cscli collections install crowdsecurity/linux --force   # re-enable cleanly
+  sudo systemctl reload crowdsec               # warnings should be gone
+  ```
 
 ## Per-environment notes
 

skills/crowdsec/references/configure/notifications.md

@@ -1,12 +1,109 @@
+---
+verified:
+  - date: 2026-05-26
+    version: "1.7.8"
+    env: systemd
+    notes: "cscli notifications test/list; http plugin 400 on bad template; render-to-sink via nc+jq (catches missing-comma)"
+---
+
 # Configure — Notifications (http/email/slack)
 
 Canonical docs: <https://docs.crowdsec.net/docs/next/local_api/notification_plugins/intro>
 
-> STUB. To cover:
-> - Plugin layout: `/etc/crowdsec/notifications/*.yaml` + binary in `/usr/lib/crowdsec/plugins/`
-> - Wiring a plugin into a profile (`notifications:`)
-> - Templating with go-template (alert/decision fields)
-> - HTTP: webhook patterns (Slack incoming, Discord, generic)
-> - Email: SMTP, auth, TLS
-> - Testing a plugin: `cscli notifications test`
-> - Pitfalls: plugin binary missing exec bit; templating errors swallow notification silently
+Notifications fire when a **profile** matches an alert and routes it to a named
+plugin. Two moving parts: the plugin **config** (`/etc/crowdsec/notifications/*.yaml`)
+and the **profile** wiring (`/etc/crowdsec/profiles.yaml`). The plugin binaries
+live in `/usr/lib/crowdsec/plugins/` (`notification-http`, `-slack`, `-email`,
+`-splunk`, `-sentinel`, `-file`).
+
+## Wire a plugin into a profile
+
+A plugin only fires if a profile names it. In `profiles.yaml`:
+
+```yaml
+notifications:
+  - http_default      # must equal the `name:` field in the plugin yaml
+```
+
+The `name:` in the plugin config and the string in `profiles.yaml` must match
+exactly, or the alert is generated but nothing is sent (no error). After editing
+either file: `sudo systemctl reload crowdsec`. See [profiles.md](profiles.md).
+
+## Test without waiting for a real attack
+
+Don't trigger a live ban to check a webhook. Two cscli paths:
+
+```bash
+sudo cscli notifications test http_default     # send a synthetic alert to ONE plugin
+sudo cscli notifications list                  # plugins cscli can see
+sudo cscli notifications reinject <alert-id>   # replay a REAL past alert through profiles
+```
+
+`test` spawns the plugin with its on-disk config and pushes a generic alert —
+the fastest loop for iterating on a template or URL.
+
+## The #1 failure: a broken template fails silently-ish
+
+A template that renders **invalid JSON** (or a payload the endpoint rejects) does
+**not** crash CrowdSec. You get one line and nothing delivered:
+
+```
+level=warning msg="HTTP server returned non 200 status code: 400" @module=http-plugin
+```
+
+Critically, **CrowdSec never logs the rendered body** — only the status code. So
+"why is my Discord/Slack/webhook silent" can't be answered from the engine logs
+alone. The fix is to **see what was actually sent**:
+
+### Render-to-sink: the debugging technique
+
+Point the plugin at a local listener that captures the request, then fire a test.
+`nc` + `jq` is enough — no server to write. Set `url: http://127.0.0.1:9999/` in
+the plugin yaml, then in **one shell** capture a single request, keep only the body
+(everything after the blank line), and validate it:
+
+```bash
+nc -l 9999 | sed '1,/^\r\{0,1\}$/d' | jq .
+```
+
+In **another shell** trigger it:
+
+```bash
+sudo cscli notifications test http_default
+```
+
+`jq` pretty-prints the rendered payload if it's valid JSON, or pinpoints the bug if
+not — e.g. a **missing comma** between two keys yields
+`jq: parse error: Expected separator between values` (the same defect makes the real
+endpoint return 400). `nc` serves the one request and exits; the plugin logs a
+transport error since it gets no proper HTTP reply, but you already have the body.
+
+(For a persistent capture or to also exercise the response code, the `file` plugin
+— `notification-file` writing the rendered output to a path — is a no-network
+alternative for the same "what did it render" question.)
+
+Common template bugs:
+
+- Missing comma / trailing comma between JSON fields.
+- A value typed as a string where the API wants a literal — e.g. Discord rejects
+  `"inline": "false"` (string); it must be `"inline": false` (boolean).
+- Redefining a variable already bound at the top of the template (`{{$alert := .}}`
+  inside a `range` that already set context) — harmless to render but a frequent
+  red herring; the real break is almost always the JSON shape.
+
+## Per-environment notes
+
+| Env | What changes |
+|---|---|
+| **systemd / bare-metal** | Recipes as-is. Plugin binaries from the package; `notification_dir` is `/etc/crowdsec/notifications/`. |
+| **Docker / compose** | `docker compose exec crowdsec cscli notifications test <name>`. Mount or bake the plugin yaml; the `crowdsecurity/crowdsec` image ships the same plugin binaries. |
+| **Kubernetes / Helm** | Configure via the chart's `config.notifications` / extra volumes; `kubectl exec -n <ns> <lapi-pod> -- cscli notifications test <name>`. Notifications fire from the **LAPI** pod (where profiles run), not the agents. |
+
+## Pitfalls
+
+- **Plugin binary not executable** → plugin never loads. The package sets the bit;
+  hand-copied binaries may not. Check `/usr/lib/crowdsec/plugins/` perms.
+- **Name mismatch** between `profiles.yaml` and the plugin `name:` → silent no-send.
+- **`group_wait`/`group_threshold`/`max_retry`** in the plugin yaml batch and delay
+  sends — a "missing" notification may just be batched. For dedup of repeated bans,
+  tune these rather than expecting one message per event.