From 70fae3761086edff95b8e8bc5ff7ee29886a9ca5 Mon Sep 17 00:00:00 2001 From: Thibault Koechlin Date: Fri, 29 May 2026 14:51:17 +0200 Subject: [PATCH] various fixes after another round of discord solving simulation --- skills/crowdsec/SKILL.md | 7 +- .../references/configure/allowlists.md | 68 +++++++++++ .../references/configure/bouncers/firewall.md | 23 +++- skills/crowdsec/references/configure/hub.md | 18 +++ .../references/configure/notifications.md | 113 ++++++++++++++++-- .../references/debug/symptoms/not-blocked.md | 41 +++++++ .../references/operate/multi-server.md | 48 ++++++++ 7 files changed, 306 insertions(+), 12 deletions(-) diff --git a/skills/crowdsec/SKILL.md b/skills/crowdsec/SKILL.md index 7e7ede3..e35b8b3 100644 --- a/skills/crowdsec/SKILL.md +++ b/skills/crowdsec/SKILL.md @@ -99,7 +99,8 @@ Hub. If it's far behind `version.crowdsec.net/latest`, pull a newer tag | "ban duration", "captcha", "decisions", "simulation", "alerts but no bans" | [references/configure/profiles.md](./references/configure/profiles.md) | | "allowlist my office / CDN / monitoring IP", "I'm getting blocked by CAPI", "exclude IP from any ban" | [references/configure/allowlists.md](./references/configure/allowlists.md) | | "whitelist vs allowlist vs postoverflow", "which suppression layer should I use" | [references/configure/allowlists.md](./references/configure/allowlists.md) § Suppression mechanisms | -| "alert me on slack/email/webhook" | [references/configure/notifications.md](./references/configure/notifications.md) *(TODO — stub)* | +| "test my whitelist works", "is my postoverflow / dynamic-IP whitelist actually firing" | [references/configure/allowlists.md](./references/configure/allowlists.md) § Verification — does a whitelist actually work? | +| "notifications", "alert me on slack/email/webhook", "notification not firing" | [references/configure/notifications.md](./references/configure/notifications.md) | | "block at the firewall", "iptables", "nftables", "ipset" | [references/configure/bouncers/firewall.md](./references/configure/bouncers/firewall.md) | | "nginx bouncer", "lua / openresty module" | [references/configure/bouncers/web-servers.md](./references/configure/bouncers/web-servers.md) § nginx | | "haproxy bouncer", "SPOA / SPOE" | [references/configure/bouncers/web-servers.md](./references/configure/bouncers/web-servers.md) § haproxy | @@ -111,7 +112,8 @@ Hub. If it's far behind `version.crowdsec.net/latest`, pull a newer tag | "Console", "enroll", "share signals" | [references/install/console.md](./references/install/console.md) | | "upgrade", "back up", "roll back", "new version", "tainted items after upgrade" | [references/operate/upgrades.md](./references/operate/upgrades.md) | | "old/outdated version", "`cscli` command or flag missing", "hub item won't install", "behavior doesn't match the docs", "installed from the distro package" | [references/operate/upgrades.md](./references/operate/upgrades.md) § Detect & fix an outdated / distro-packaged install (see **Step 1.5** above) | -| "multiple agents", "remote LAPI", "mTLS", "postgres backend" | [references/operate/multi-server.md](./references/operate/multi-server.md) *(TODO — stub)* | +| "multiple agents", "remote LAPI", "mTLS", "postgres backend" | [references/operate/multi-server.md](./references/operate/multi-server.md) *(partial — machine cleanup done, rest stub)* | +| "stale machines / log processors in `cscli machines list`", "prune dead agents", "ephemeral k8s pods piling up" | [references/operate/multi-server.md](./references/operate/multi-server.md) § Decommissioning stale machines | | "is it working?", "smoke test", "validate install", "verify setup", "did detection / WAF / blocking actually wire up?" | [references/operate/health-check.md](./references/operate/health-check.md) | | **Debug — common** · "it's broken" / "not working" / general diagnosis | [references/debug/common/triage.md](./references/debug/common/triage.md) → run `bash ${CLAUDE_SKILL_DIR}/scripts/diagnose.sh` | | **Debug — common** · specific error string | [references/debug/common/errors.md](./references/debug/common/errors.md) | @@ -119,6 +121,7 @@ Hub. If it's far behind `version.crowdsec.net/latest`, pull a newer tag | **Debug — by symptom** · "logs not parsed", "0 parsed" | [references/debug/symptoms/parsing.md](./references/debug/symptoms/parsing.md) | | **Debug — by symptom** · "no alerts firing" | [references/debug/symptoms/no-alerts.md](./references/debug/symptoms/no-alerts.md) | | **Debug — by symptom** · "decision exists but not blocked" | [references/debug/symptoms/not-blocked.md](./references/debug/symptoms/not-blocked.md) | +| **Debug — by symptom** · "bouncer blocks everything", "locked out of all services", "every request 403 after adding the bouncer" | [references/debug/symptoms/not-blocked.md](./references/debug/symptoms/not-blocked.md) § 7 — Inverse symptom | | **Debug — by feature** · AppSec/WAF not blocking, false positives, captcha | [references/appsec/troubleshoot.md](./references/appsec/troubleshoot.md) | | "switch from fail2ban" | [references/migrate/from-fail2ban.md](./references/migrate/from-fail2ban.md) *(TODO — stub)* | diff --git a/skills/crowdsec/references/configure/allowlists.md b/skills/crowdsec/references/configure/allowlists.md index abe888e..9f15f49 100644 --- a/skills/crowdsec/references/configure/allowlists.md +++ b/skills/crowdsec/references/configure/allowlists.md @@ -4,6 +4,10 @@ verified: version: "1.7.8" env: systemd notes: "allowlists add/check/list" + - date: 2026-05-26 + version: "1.7.8" + env: systemd + notes: "parser whitelist via cscli explain --verbose (ignored by whitelist, private-IP); postoverflow via crowdsec -dsn ... -no-api replay (whitelisted-skip lines)" --- # Configure — Allowlists, whitelist parsers, and postoverflows @@ -200,6 +204,70 @@ sudo cscli decisions add -i 192.0.2.99 --duration 5m --reason "retest" sudo cscli allowlists delete test ``` +## Verification — does a whitelist actually work? + +Allowlists are tested at the LAPI (above). The two **whitelist** layers run at +different phases and are tested with different tools. *Authoring* either is out of +scope (canonical docs cover syntax); confirming a deployed one fires is operational. + +### Whitelist parser (`parsers/s02-enrich/`) — use `cscli explain` + +A whitelist parser drops the **event** during enrichment, so `cscli explain` shows +it directly — no traffic or bucket needed. Replay a line from a whitelisted source +(the shipped `crowdsecurity/whitelists` covers RFC1918, so any private IP works): + +```bash +LINE='May 26 10:00:00 host sshd[111]: Failed password for invalid user admin from 10.0.0.5 port 2222 ssh2' +sudo cscli explain --log "$LINE" --type syslog --verbose +``` + +A matching whitelist tags the parser and ends the line with `ignored by whitelist`: + +``` + | └ 🟢 crowdsecurity/whitelists (~2 [whitelisted]) + | └ update evt.Whitelisted : false -> true + | └ update evt.WhitelistReason : -> private ipv4/ipv6 ip/ranges + └-------- parser success, ignored by whitelist (private ipv4/ipv6 ip/ranges) 🟢 +``` + +No `[whitelisted]` tag / no `ignored by whitelist` line → your parser isn't +matching (wrong expression, file not in `s02-enrich`, or not reloaded). + +### Postoverflow (`postoverflows/s01-whitelist/`) — replay through the engine + +A postoverflow runs **after** a bucket overflows, which `cscli explain` does not +simulate (it stops at scenario routing). The dynamic-home-IP whitelist +() is this +kind. Test it by replaying enough log lines to overflow, in one-shot mode: + +```bash +# build a log that would trip the scenario from the whitelisted IP +for i in $(seq 1 20); do + printf 'May 26 10:00:%02d host sshd[%d]: Failed password for invalid user x from 203.0.113.77 port 22 ssh2\n' "$i" "$i" +done | sudo tee /tmp/bf.log >/dev/null + +sudo crowdsec -dsn file:///tmp/bf.log -type syslog -no-api +``` + +> **`-no-api` is mandatory while the service is running.** Without it the one-shot +> tries to start its own LAPI and dies: `bind: address already in use` (fatal on +> `:8080`). With `-no-api` it reuses the running LAPI. + +Read the output: + +- **Not matching** → the overflow is reported: + `Ip 203.0.113.77 performed 'crowdsecurity/ssh-bf' (6 events over 5s)` +- **Matching** → the overflow is suppressed and *named*: + ``` + Ban for 203.0.113.77 whitelisted, reason [home IP] name=test/home-ip-wl stage=s01-whitelist + [Ip 203.0.113.77 performed 'crowdsecurity/ssh-bf' ...] is whitelisted, skip. + ``` + The `name=` / `stage=` confirm **your** rule matched (not a shipped one like + `crowdsecurity/seo-bots-whitelist`). For a dynamic-DNS whitelist + (`... in LookupHost("me.ddns.net")`) the same replay applies — resolution happens + at evaluation time, so this also catches a stale/typo'd hostname (no `whitelisted` + line = the name didn't resolve to the source IP). + ## When allowlists silently fail to apply - **Per-request AppSec blocks bypass allowlists.** AppSec inband rules return 403 before any decision is written, so allowlisting the source IP doesn't help with inband false positives. Fix at the appsec-config (`exclude_path:` or per-rule disable). See [../appsec/troubleshoot.md](../appsec/troubleshoot.md). diff --git a/skills/crowdsec/references/configure/bouncers/firewall.md b/skills/crowdsec/references/configure/bouncers/firewall.md index 33ef479..975d41c 100644 --- a/skills/crowdsec/references/configure/bouncers/firewall.md +++ b/skills/crowdsec/references/configure/bouncers/firewall.md @@ -1,9 +1,9 @@ --- verified: - - date: 2026-05-22 + - date: 2026-05-26 version: "1.7.8" env: systemd - notes: "nftables bouncer v0.0.34 install+register, nft sets/chains (priority filter-10), ban→cscli set→drop; added stale-key recovery note" + notes: "re-verified nftables bouncer v0.0.34 install+register, per-origin/family sets, stale ${API_KEY}→stream-halted, flush-table does not evict set elements; added wrong-set inverse-symptom pitfall" --- # Bouncers — Firewall (nftables / iptables / ipset) @@ -141,6 +141,25 @@ sudo cscli decisions delete -i 192.0.2.66 - **"Banned but still reachable"** → almost always `update_frequency` not elapsed, `disable_ipv6` masking a v6 client, or the bouncer service stopped. Full decision tree: [../../debug/symptoms/not-blocked.md](../../debug/symptoms/not-blocked.md). +- **"Decision in `cscli decisions list` but not in the set"** (nft, ipset, all + backends) → you are almost always looking in the **wrong set**. The bouncer splits + decisions into multiple sets by *origin* (`-cscli` manual, `-CAPI` community, + `-crowdsec` console push, `-lists` blocklists) **and** by *family* (IPv4 vs IPv6). + Don't guess the name — **list the sets first**, then look inside the right one: + ```bash + sudo nft list sets | grep crowdsec # nftables backend: set names per table + sudo ipset list -name | grep crowdsec # iptables/ipset backend: set names + # then, e.g.: + sudo nft list set ip crowdsec crowdsec-blacklists-cscli | grep + sudo ipset test crowdsec-blacklists-cscli + ``` + (A `nft flush table` / external ruleset reload does **not** evict the bouncer's + set elements on 1.7.8/v0.0.34 — if a restart is what "makes the IP appear", + suspect wrong-set/family or an unhealthy bouncer next, not a flush race.) +- **Bouncer unhealthy = nothing enforced.** A bad/placeholder `api_key` makes LAPI + return `access forbidden` and the service exits with `bouncer stream halted` + (see §2). `systemctl is-active crowdsec-firewall-bouncer` plus the **Last API + pull** column in `cscli bouncers list` are the fast health checks. ## Teardown diff --git a/skills/crowdsec/references/configure/hub.md b/skills/crowdsec/references/configure/hub.md index 00b8d41..9028e48 100644 --- a/skills/crowdsec/references/configure/hub.md +++ b/skills/crowdsec/references/configure/hub.md @@ -4,6 +4,10 @@ verified: version: "1.7.8" env: systemd notes: "update/upgrade, list -o raw, collections install+remove, inspect tainted fields" + - date: 2026-05-26 + version: "1.7.8" + env: systemd + notes: "reproduced dangling enable-symlink → item silently not loaded → parser failure; find -xtype l fix" --- # Configure — Hub management @@ -124,6 +128,20 @@ sudo systemctl reload crowdsec recreate / `helm upgrade`) to take effect. - **`upgrade` silently skips your local/tainted items** — by design. Reconcile them deliberately with `--force` (after saving any edits to an override). +- **Dangling enable-symlinks silently drop items.** If `hub_dir` is changed in + `config.yaml`, or a hub dir is partially restored, the enable-symlinks under + `/etc/crowdsec/{parsers,scenarios,postoverflows,collections,...}/` can point at a + target that no longer exists. CrowdSec logs `Ignoring file …: lstat …: no such file + or directory` (a *warning*, easy to miss) and **does not load that item** — so you + can have `crowdsecurity/sshd-success-logs` loaded while `sshd-logs` is dangling, + and `cscli explain` then shows `🔴 parser failure` with no alerts ever raised. + Detect and fix: + ```bash + sudo find /etc/crowdsec -xtype l # list broken (dangling) symlinks + sudo find /etc/crowdsec -xtype l -delete # remove them + sudo cscli collections install crowdsecurity/linux --force # re-enable cleanly + sudo systemctl reload crowdsec # warnings should be gone + ``` ## Per-environment notes diff --git a/skills/crowdsec/references/configure/notifications.md b/skills/crowdsec/references/configure/notifications.md index a5b6012..316fd71 100644 --- a/skills/crowdsec/references/configure/notifications.md +++ b/skills/crowdsec/references/configure/notifications.md @@ -1,12 +1,109 @@ +--- +verified: + - date: 2026-05-26 + version: "1.7.8" + env: systemd + notes: "cscli notifications test/list; http plugin 400 on bad template; render-to-sink via nc+jq (catches missing-comma)" +--- + # Configure — Notifications (http/email/slack) Canonical docs: -> STUB. To cover: -> - Plugin layout: `/etc/crowdsec/notifications/*.yaml` + binary in `/usr/lib/crowdsec/plugins/` -> - Wiring a plugin into a profile (`notifications:`) -> - Templating with go-template (alert/decision fields) -> - HTTP: webhook patterns (Slack incoming, Discord, generic) -> - Email: SMTP, auth, TLS -> - Testing a plugin: `cscli notifications test` -> - Pitfalls: plugin binary missing exec bit; templating errors swallow notification silently +Notifications fire when a **profile** matches an alert and routes it to a named +plugin. Two moving parts: the plugin **config** (`/etc/crowdsec/notifications/*.yaml`) +and the **profile** wiring (`/etc/crowdsec/profiles.yaml`). The plugin binaries +live in `/usr/lib/crowdsec/plugins/` (`notification-http`, `-slack`, `-email`, +`-splunk`, `-sentinel`, `-file`). + +## Wire a plugin into a profile + +A plugin only fires if a profile names it. In `profiles.yaml`: + +```yaml +notifications: + - http_default # must equal the `name:` field in the plugin yaml +``` + +The `name:` in the plugin config and the string in `profiles.yaml` must match +exactly, or the alert is generated but nothing is sent (no error). After editing +either file: `sudo systemctl reload crowdsec`. See [profiles.md](profiles.md). + +## Test without waiting for a real attack + +Don't trigger a live ban to check a webhook. Two cscli paths: + +```bash +sudo cscli notifications test http_default # send a synthetic alert to ONE plugin +sudo cscli notifications list # plugins cscli can see +sudo cscli notifications reinject # replay a REAL past alert through profiles +``` + +`test` spawns the plugin with its on-disk config and pushes a generic alert — +the fastest loop for iterating on a template or URL. + +## The #1 failure: a broken template fails silently-ish + +A template that renders **invalid JSON** (or a payload the endpoint rejects) does +**not** crash CrowdSec. You get one line and nothing delivered: + +``` +level=warning msg="HTTP server returned non 200 status code: 400" @module=http-plugin +``` + +Critically, **CrowdSec never logs the rendered body** — only the status code. So +"why is my Discord/Slack/webhook silent" can't be answered from the engine logs +alone. The fix is to **see what was actually sent**: + +### Render-to-sink: the debugging technique + +Point the plugin at a local listener that captures the request, then fire a test. +`nc` + `jq` is enough — no server to write. Set `url: http://127.0.0.1:9999/` in +the plugin yaml, then in **one shell** capture a single request, keep only the body +(everything after the blank line), and validate it: + +```bash +nc -l 9999 | sed '1,/^\r\{0,1\}$/d' | jq . +``` + +In **another shell** trigger it: + +```bash +sudo cscli notifications test http_default +``` + +`jq` pretty-prints the rendered payload if it's valid JSON, or pinpoints the bug if +not — e.g. a **missing comma** between two keys yields +`jq: parse error: Expected separator between values` (the same defect makes the real +endpoint return 400). `nc` serves the one request and exits; the plugin logs a +transport error since it gets no proper HTTP reply, but you already have the body. + +(For a persistent capture or to also exercise the response code, the `file` plugin +— `notification-file` writing the rendered output to a path — is a no-network +alternative for the same "what did it render" question.) + +Common template bugs: + +- Missing comma / trailing comma between JSON fields. +- A value typed as a string where the API wants a literal — e.g. Discord rejects + `"inline": "false"` (string); it must be `"inline": false` (boolean). +- Redefining a variable already bound at the top of the template (`{{$alert := .}}` + inside a `range` that already set context) — harmless to render but a frequent + red herring; the real break is almost always the JSON shape. + +## Per-environment notes + +| Env | What changes | +|---|---| +| **systemd / bare-metal** | Recipes as-is. Plugin binaries from the package; `notification_dir` is `/etc/crowdsec/notifications/`. | +| **Docker / compose** | `docker compose exec crowdsec cscli notifications test `. Mount or bake the plugin yaml; the `crowdsecurity/crowdsec` image ships the same plugin binaries. | +| **Kubernetes / Helm** | Configure via the chart's `config.notifications` / extra volumes; `kubectl exec -n -- cscli notifications test `. Notifications fire from the **LAPI** pod (where profiles run), not the agents. | + +## Pitfalls + +- **Plugin binary not executable** → plugin never loads. The package sets the bit; + hand-copied binaries may not. Check `/usr/lib/crowdsec/plugins/` perms. +- **Name mismatch** between `profiles.yaml` and the plugin `name:` → silent no-send. +- **`group_wait`/`group_threshold`/`max_retry`** in the plugin yaml batch and delay + sends — a "missing" notification may just be batched. For dedup of repeated bans, + tune these rather than expecting one message per event. diff --git a/skills/crowdsec/references/debug/symptoms/not-blocked.md b/skills/crowdsec/references/debug/symptoms/not-blocked.md index d05f104..2aa6431 100644 --- a/skills/crowdsec/references/debug/symptoms/not-blocked.md +++ b/skills/crowdsec/references/debug/symptoms/not-blocked.md @@ -4,6 +4,10 @@ verified: version: "1.7.8" env: systemd notes: "allowlists/decisions/bouncers ladder, LAPI curl 200, firewall nft sets/chains/counter" + - date: 2026-05-26 + version: "1.7.8" + env: systemd + notes: "added §7 inverse symptom (bouncer blocks everything); LAPI 403 access-forbidden on bad/missing key reproduced via curl" --- # Debug — Decisions exist but bouncer not blocking @@ -122,3 +126,40 @@ sudo cscli decisions list -o raw | grep -c CAPI # community decisions present A fresh engine takes a pull cycle to import the community blocklist; the firewall bouncer then needs one more `update_frequency` to materialise it into `crowdsec-blacklists-CAPI`. + +## 7 — Inverse symptom: the bouncer blocks *everything* + +The opposite report — "after setting up the bouncer I'm locked out of all my +services / every request is 403, even my own IP" — usually is **not** a decision +at all. Two causes: + +**Bouncer can't read decisions and fails closed.** A web-server bouncer +(traefik/nginx/caddy) that gets a non-200 from LAPI may, depending on its +fallback setting, deny *all* traffic rather than allow it. The trigger is almost +always a bad/expired/placeholder `api_key` — LAPI then answers every decision +query with **HTTP 403** and a body `{"message":"access forbidden"}`, distinct +from the *connection-refused* "LAPI unreachable" case the fail-open setting is +named for. Confirm which one you have: + +```bash +# from the bouncer host — substitute the key from the bouncer's config +curl -s -o /dev/null -w '%{http_code}\n' -H "X-Api-Key: " http://:8080/v1/decisions +# 403 → bad/missing/rotated key (fix the key) ← the "blocks everything" cause +# 200 → key is fine; look at trust config below +# refused/timeout → network path (fail-open governs THIS case, not 403) +``` + +`cscli bouncers list` corroborates: a bouncer with an **empty or stale "Last API +pull"** never authenticated. Fix = re-mint and paste the key: +`KEY=$(sudo cscli bouncers add -o raw)` → write into the bouncer config → +restart. (Same root cause, engine-side, as the firewall bouncer's +`bouncer stream halted` — see [../../configure/bouncers/firewall.md](../../configure/bouncers/firewall.md) §2.) +If service availability must survive a dead LAPI, set the bouncer's fallback to +fail-open (`CrowdsecFallback: allow` / `enable_hard_fails` off / `appsec_fail_open: true`, +per bouncer) — see [../../configure/bouncers/web-servers.md](../../configure/bouncers/web-servers.md). + +**Trust config inverted.** Putting your proxy/CDN/Docker range in the bouncer's +*blocking* path instead of its trusted-header list (or vice-versa) makes it judge +every request by the wrong IP. See the `clientTrustedIPs` / +`forwardedHeadersTrustedIPs` notes in +[../../configure/bouncers/web-servers.md](../../configure/bouncers/web-servers.md). diff --git a/skills/crowdsec/references/operate/multi-server.md b/skills/crowdsec/references/operate/multi-server.md index 5d35fff..ce49114 100644 --- a/skills/crowdsec/references/operate/multi-server.md +++ b/skills/crowdsec/references/operate/multi-server.md @@ -1,7 +1,55 @@ +--- +verified: + - date: 2026-05-26 + version: "1.7.8" + env: systemd + notes: "machines add -f -, machines prune (duration/force/<2min guard), prune semantics" +--- + # Operate — Multi-server / distributed LAPI Canonical docs: +## Decommissioning stale machines (log processors) + +Every agent registers as a **machine** in `cscli machines list`. Ephemeral agents +— Kubernetes pods, autoscaled VMs, re-imaged hosts — leave behind dead entries +each time they roll. They're cosmetic (a dead machine can't push) but clutter the +list and the Console. + +```bash +sudo cscli machines list # Last Heartbeat ⚠️ - / old = stale +sudo cscli machines prune # default: validated machines idle > 10m +sudo cscli machines prune --duration 1h # widen the idle window +sudo cscli machines prune --not-validated-only # only never-validated registrations +sudo cscli machines delete # remove one explicitly +``` + +Behavior confirmed on 1.7.8: + +- **Prune is idle-time based**, compared against last heartbeat/update — default + `--duration 10m`. A machine that was active seconds ago is **not** pruned even + with `--force`; "No machines to prune." That's correct, not a bug. +- **`--force` skips the confirmation prompt but NOT the sub-2-minute safety guard.** + `cscli machines prune --duration 1s --force` still prompts ("less than 2 minutes… + Continue?") and, non-interactively, dies with `Error: ... EOF`. Use a duration + ≥ 2m for unattended runs. +- **There is no built-in auto-prune scheduler.** Run `cscli machines prune` from + cron/systemd-timer if you want it automatic. `unregister_on_exit` + () + only covers **graceful** shutdown — a killed/OOM'd pod or a hard node failure + never unregisters, so prune is still needed. + +> **Gotcha — `cscli machines add` clobbers local creds.** Plain +> `cscli machines add ` writes to `/etc/crowdsec/local_api_credentials.yaml` +> and refuses if it exists. To register an extra machine (e.g. a remote agent's +> key) without touching the local file, send the creds to stdout: `-f -` +> (`cscli machines add -a -f -`). + +Per-env: Docker `docker compose exec crowdsec cscli machines prune`; k8s +`kubectl exec -n -- cscli machines prune` — run against the +**LAPI** pod (the registry), and prefer a CronJob over per-replica logic. + ## Sections to fill > STUB. To cover: