From cc826e8bcd7602e5dd2b4e6f0ecd7df516ec0532 Mon Sep 17 00:00:00 2001 From: Richard Shade Date: Tue, 29 Sep 2026 12:17:27 -0500 Subject: [PATCH 1/2] cmd/tail-watch,openspec: add notification sinks and incident delivery Alert transitions reached nobody. Turn each one into a versioned JSON event and deliver it to webhook and Slack sinks without a probe ever waiting on a sink. Each sink has its own worker and bounded queue, in-order delivery, a retry schedule with Retry-After and max_age, and a late flag. Webhooks are HMAC signed and can batch. Slack renders Block Kit with grouping, a one message per second limit, and page-only mentions. An open incident book keeps a recovery owed for every page, including across restarts: with state_dir one atomic notification-state.json holds incidents, per-sink delivery state and the pending events of queue: disk sinks, and is committed before any worker can send from it. A dropped intake slot is repaired from the alerter's dirty snapshots. Also add the heartbeat dead-man's-switch ping and the notify-test subcommand, and export the tailwatch_notify_* metrics. The alert state machine now lets a DOWN implicit target be removed, so authoritative topology reconciliation can close a restored learned router's incident as removed. Configured targets are still refused. Add a live test and compose fixtures with one tailnet node per check, and scripts/live-stack.sh to bring up only those nodes. Live verification, on a real tailnet with a Slack incoming webhook and a receiver that verifies every signature: - icmp, tsping, tcp, http and a LAN target behind a subnet router were each failed and healed: one down and one recovery apiece, with the original down_event_id, 15s outages, layers service, tunnel, service, service and lan. All requests verified; 28 heartbeats arrived. - Restarting the collector mid-outage with queue: disk did not page again and delivered the original recovery. - notify-test returned 200 for both sinks. The Slack channel showed the start, DOWN and RECOVERED messages. - The Slack sink uses a memory queue, so an alert still inside its group_wait window when the collector restarts is lost and counted as dropped{reason="restart"}. Use queue: disk to keep it. - With a tag:monitor collector, the tailnet policy must grant tcp:9000 and tcp:8080 or those fixtures are down from the start; validated with the policy test API, not applied. testdata/snippet.hujson has the grants. Updates tailscale/tailscale#5967 Signed-off-by: Richard Shade --- CLAUDE.md | 74 +-- Dockerfile | 4 +- README.md | 337 ++++++++++ cmd/tail-watch/alert_target.go | 6 +- cmd/tail-watch/alert_test.go | 20 +- cmd/tail-watch/checks_test.go | 2 +- cmd/tail-watch/config.go | 16 +- cmd/tail-watch/config_sinks.go | 411 ++++++++++++ cmd/tail-watch/config_sinks_test.go | 286 ++++++++ cmd/tail-watch/config_test.go | 9 +- cmd/tail-watch/deliver.go | 183 ++++++ cmd/tail-watch/event.go | 293 +++++++++ cmd/tail-watch/event_test.go | 405 ++++++++++++ cmd/tail-watch/heartbeat.go | 113 ++++ cmd/tail-watch/heartbeat_test.go | 95 +++ cmd/tail-watch/incidents.go | 488 ++++++++++++++ cmd/tail-watch/live_notify_test.go | 493 ++++++++++++++ cmd/tail-watch/live_tailnet_test.go | 4 +- cmd/tail-watch/main.go | 88 ++- cmd/tail-watch/main_test.go | 20 + cmd/tail-watch/notify.go | 594 +++++++++++++++++ cmd/tail-watch/notify_helpers_test.go | 352 ++++++++++ cmd/tail-watch/notify_integration_test.go | 179 +++++ cmd/tail-watch/notify_metrics.go | 132 ++++ cmd/tail-watch/notify_metrics_test.go | 136 ++++ cmd/tail-watch/notify_test.go | 565 ++++++++++++++++ cmd/tail-watch/notifytest.go | 181 +++++ cmd/tail-watch/notifytest_test.go | 179 +++++ cmd/tail-watch/presence_test.go | 17 + cmd/tail-watch/queue.go | 347 ++++++++++ cmd/tail-watch/queue_test.go | 619 ++++++++++++++++++ cmd/tail-watch/recheck_test.go | 2 +- cmd/tail-watch/restore_test.go | 381 +++++++++++ cmd/tail-watch/sink.go | 158 +++++ cmd/tail-watch/slack.go | 268 ++++++++ cmd/tail-watch/slack_test.go | 415 ++++++++++++ .../testdata/compose.live-notify.yaml | 107 +++ .../testdata/payload/collector.started.json | 11 + .../payload/target.down.implicit.json | 31 + .../testdata/payload/target.down.json | 28 + .../testdata/payload/target.down.warn.json | 28 + .../testdata/payload/target.flapping.json | 27 + .../testdata/payload/target.recovered.json | 31 + .../payload/target.recovered.removed.json | 30 + cmd/tail-watch/testdata/payload/test.json | 10 + .../testdata/slack/down-mention.json | 28 + .../testdata/slack/down-status-link.json | 21 + cmd/tail-watch/testdata/slack/down.json | 21 + .../testdata/slack/recovered-removed.json | 21 + cmd/tail-watch/testdata/slack/recovered.json | 21 + .../testdata/slack/router-14-deps.json | 21 + .../testdata/slack/warn-with-mention.json | 21 + cmd/tail-watch/testdata/snippet.hujson | 4 +- cmd/tail-watch/webhook.go | 71 ++ cmd/tail-watch/webhook_test.go | 490 ++++++++++++++ examples/tail-watch.yaml | 26 + .../.openspec.yaml | 0 .../2026-09-29-add-notifications}/design.md | 13 +- .../2026-09-29-add-notifications}/proposal.md | 10 +- .../specs/alert-state-machine/spec.md | 56 ++ .../specs/collector-heartbeat/spec.md | 0 .../specs/notification-delivery/spec.md | 0 .../specs/notification-events/spec.md | 5 +- .../specs/notify-test/spec.md | 0 .../specs/slack-sink/spec.md | 0 .../specs/webhook-sink/spec.md | 0 .../2026-09-29-add-notifications}/tasks.md | 80 +-- openspec/specs/alert-state-machine/spec.md | 19 +- openspec/specs/collector-heartbeat/spec.md | 40 ++ openspec/specs/notification-delivery/spec.md | 326 +++++++++ openspec/specs/notification-events/spec.md | 199 ++++++ openspec/specs/notify-test/spec.md | 45 ++ openspec/specs/slack-sink/spec.md | 97 +++ openspec/specs/webhook-sink/spec.md | 137 ++++ scripts/live-stack.sh | 109 +++ 75 files changed, 9946 insertions(+), 110 deletions(-) create mode 100644 cmd/tail-watch/config_sinks.go create mode 100644 cmd/tail-watch/config_sinks_test.go create mode 100644 cmd/tail-watch/deliver.go create mode 100644 cmd/tail-watch/event.go create mode 100644 cmd/tail-watch/event_test.go create mode 100644 cmd/tail-watch/heartbeat.go create mode 100644 cmd/tail-watch/heartbeat_test.go create mode 100644 cmd/tail-watch/incidents.go create mode 100644 cmd/tail-watch/live_notify_test.go create mode 100644 cmd/tail-watch/notify.go create mode 100644 cmd/tail-watch/notify_helpers_test.go create mode 100644 cmd/tail-watch/notify_integration_test.go create mode 100644 cmd/tail-watch/notify_metrics.go create mode 100644 cmd/tail-watch/notify_metrics_test.go create mode 100644 cmd/tail-watch/notify_test.go create mode 100644 cmd/tail-watch/notifytest.go create mode 100644 cmd/tail-watch/notifytest_test.go create mode 100644 cmd/tail-watch/queue.go create mode 100644 cmd/tail-watch/queue_test.go create mode 100644 cmd/tail-watch/restore_test.go create mode 100644 cmd/tail-watch/sink.go create mode 100644 cmd/tail-watch/slack.go create mode 100644 cmd/tail-watch/slack_test.go create mode 100644 cmd/tail-watch/testdata/compose.live-notify.yaml create mode 100644 cmd/tail-watch/testdata/payload/collector.started.json create mode 100644 cmd/tail-watch/testdata/payload/target.down.implicit.json create mode 100644 cmd/tail-watch/testdata/payload/target.down.json create mode 100644 cmd/tail-watch/testdata/payload/target.down.warn.json create mode 100644 cmd/tail-watch/testdata/payload/target.flapping.json create mode 100644 cmd/tail-watch/testdata/payload/target.recovered.json create mode 100644 cmd/tail-watch/testdata/payload/target.recovered.removed.json create mode 100644 cmd/tail-watch/testdata/payload/test.json create mode 100644 cmd/tail-watch/testdata/slack/down-mention.json create mode 100644 cmd/tail-watch/testdata/slack/down-status-link.json create mode 100644 cmd/tail-watch/testdata/slack/down.json create mode 100644 cmd/tail-watch/testdata/slack/recovered-removed.json create mode 100644 cmd/tail-watch/testdata/slack/recovered.json create mode 100644 cmd/tail-watch/testdata/slack/router-14-deps.json create mode 100644 cmd/tail-watch/testdata/slack/warn-with-mention.json create mode 100644 cmd/tail-watch/webhook.go create mode 100644 cmd/tail-watch/webhook_test.go rename openspec/changes/{add-notifications => archive/2026-09-29-add-notifications}/.openspec.yaml (100%) rename openspec/changes/{add-notifications => archive/2026-09-29-add-notifications}/design.md (96%) rename openspec/changes/{add-notifications => archive/2026-09-29-add-notifications}/proposal.md (91%) create mode 100644 openspec/changes/archive/2026-09-29-add-notifications/specs/alert-state-machine/spec.md rename openspec/changes/{add-notifications => archive/2026-09-29-add-notifications}/specs/collector-heartbeat/spec.md (100%) rename openspec/changes/{add-notifications => archive/2026-09-29-add-notifications}/specs/notification-delivery/spec.md (100%) rename openspec/changes/{add-notifications => archive/2026-09-29-add-notifications}/specs/notification-events/spec.md (97%) rename openspec/changes/{add-notifications => archive/2026-09-29-add-notifications}/specs/notify-test/spec.md (100%) rename openspec/changes/{add-notifications => archive/2026-09-29-add-notifications}/specs/slack-sink/spec.md (100%) rename openspec/changes/{add-notifications => archive/2026-09-29-add-notifications}/specs/webhook-sink/spec.md (100%) rename openspec/changes/{add-notifications => archive/2026-09-29-add-notifications}/tasks.md (75%) create mode 100644 openspec/specs/collector-heartbeat/spec.md create mode 100644 openspec/specs/notification-delivery/spec.md create mode 100644 openspec/specs/notification-events/spec.md create mode 100644 openspec/specs/notify-test/spec.md create mode 100644 openspec/specs/slack-sink/spec.md create mode 100644 openspec/specs/webhook-sink/spec.md create mode 100755 scripts/live-stack.sh diff --git a/CLAUDE.md b/CLAUDE.md index c446aa0..2662c12 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -27,44 +27,32 @@ `recheck.go` runs at most eight immediate rechecks. `Result.Generation` is the per-target release counter; `execute` stamps the value from when the check acquired its lock. -- Later phase-1 changes add the notification files in the same directory. -- Create only one `prober.Prober` per process. `prober.New()` registers - metrics on the default Prometheus registry and panics if called again. - Unit tests call check methods directly and do not construct a `Prober`. -- Every network access goes through `netEnv`. Do not add a package-level - dialer. `tsping` name resolution may fall back to the host resolver. -- `TestLiveTailnetDocker` stays skipped unless `TAIL_WATCH_LIVE_TAILNET=1`. It - reads `tailscale-phase-2/tailscale` from `pulumi env` (`TAIL_WATCH_PULUMI_ENV` - overrides that) and drives `examples/compose.yaml` through - `docker compose`, so it tests the shipped non-root image. It builds - `tail-watch`, starts `tailscale`, and runs `compose run tail-watch --once`. The - collector key is tagged `TAIL_WATCH_LIVE_TAG` (default `tag:monitor`; set it - empty for an untagged member node). The tailnet policy must own that tag - and grant it the peer and `10.10.0.0/16`: merge - `cmd/tail-watch/testdata/snippet.hujson` into the lab policy (it lives in - `tailscale-utils-lab`, `lab/policy/lab.acl.hujson`). Until then, run with - `TAIL_WATCH_LIVE_TAG=`. If no connected device has an - enabled subnet route, the test merges - `cmd/tail-watch/testdata/compose.live-router.yaml`, which adds a router for - `10.10.50.0/24` that answers `10.10.50.1`. `TAIL_WATCH_LIVE_PEER` and - `TAIL_WATCH_LIVE_SUBNET` override the discovered targets. Cleanup runs - `compose down --volumes` and deletes only the `tail-watch-live-` devices - that run created. Its Tailscale API calls are test setup, not collector - traffic, so they do not go through `netEnv`. -- `TestLiveTailnetClassification` (same gate and credentials) always adds its - own router on `10.10.51.0/24`, a separate prefix so the two live tests never - form an HA pair. It runs the collector as a daemon, scrapes `/metrics` - from the `tailscale` service, removes the router's LAN address (expects - `lan`), then stops the router (expects `control`). Control marks a stopped - node disconnected in seconds, but peers' netmaps have taken 28s to more than - 3 min to get `Online=false`. Until then `tunnel` is the correct layer, so the - control wait is 8 min. -- `TestLiveTailnetAlerts` uses its own router on `10.10.52.0/24`. It stops - and restarts that router, checks target-state gauges, and verifies exactly - one router DOWN and recovery in the transition logs, with silent dependents. - It uses the same live-test gate and credentials. -- Keep the compose `tailscale` service's `ping_group_range` sysctl. tail-watch - runs as uid 65532 in that network namespace and opens ICMP as `udp4`. +- Notifications: `event.go` (payload, ids, transition mapping, routing), + `incidents.go` (incident and per-sink delivery state, `restoreOpen`), `queue.go` + (bounded queues, atomic `notification-state.json`, startup replay), `notify.go` (the + coordinator and per-sink workers), `deliver.go` (shared retry loop), `sink.go`, + `webhook.go`, `slack.go`, `heartbeat.go`, `notifytest.go`, `notify_metrics.go`, and + sink schema in `config_sinks.go`. +- One `notifier` mutex guards the whole `book` (incidents plus queues), and every mutation + commits the snapshot under it, so workers only see durable intents after the commit. + `ObserveTransition` only writes the 4096-slot intake channel; never add I/O to it. + A dropped transition is repaired by `reconcile()` from the alerter's dirty snapshots + (1s ticker); target removal reaches the notifier only that way. +- Notification tests use `notify_helpers_test.go`: `countingClock` wraps `tstest.Clock` + and counts timers and tickers. Wait with `waitTimers`/`tickers` before `Advance`, or a + worker that has not armed its timer yet misses the advance. Retry timers and group-wait + timers both come from `NewTimer`, so count them when scripting a sequence. +- Golden files live in `cmd/tail-watch/testdata/{payload,slack}`; regenerate with + `go test -run 'TestPayloadGolden|TestSlackGolden' -update ./cmd/tail-watch`, then + review the diff. +- Startup order matters: `newNotifier` (corrupt state or unset secrets stop startup) runs + in `run()` before the prober; `restoreOpen` runs in `startCollectors` after the alerter + and classifier exist and before any target is scheduled. `--once` never builds a notifier, + resolves secrets, or sends a heartbeat. +- `alerter.RemoveTarget` refuses a configured target with an open incident but allows a + DOWN implicit one. The learner removes a restored learned router only after an + authoritative reconciliation, and the removed dirty snapshot makes the notifier close + the incident with `reason: removed`. ## Commits @@ -95,3 +83,13 @@ - Validate with `openspec validate --strict` after editing artifacts. - `.claude/` is generated by OpenSpec and excluded from markdownlint. MD013 is off under `openspec/` because apply tracks one task per checkbox line. + +## Live notification test + +- `TestLiveNotifications` (same gate as the other live tests) uses + `cmd/tail-watch/testdata/compose.live-notify.yaml`; `scripts/live-stack.sh up|down` + brings up just the nodes for manual work. The tailscale image's busybox has no `httpd`, + so the http fixture is an `nc` responder that must read the request before replying. +- The Slack webhook is read from `SLACK_WEBHOOK_URL` or from the pulumi env named by + `TAIL_WATCH_SLACK_PULUMI_ENV`; never write it to a file or log it (`redactSecrets` + strips `hooks.slack.com` URLs). diff --git a/Dockerfile b/Dockerfile index 167f0ff..ed71cbe 100644 --- a/Dockerfile +++ b/Dockerfile @@ -9,7 +9,9 @@ RUN go build -o /out/tail-watch ./cmd/tail-watch FROM alpine:3.24 -RUN adduser -D -H -u 65532 tail-watch +RUN adduser -D -H -u 65532 tail-watch \ + && mkdir -p /var/lib/tail-watch \ + && chown tail-watch /var/lib/tail-watch COPY --from=build /out/tail-watch /usr/local/bin/tail-watch USER tail-watch ENTRYPOINT ["/usr/local/bin/tail-watch"] diff --git a/README.md b/README.md index 7c7fcac..ad55794 100644 --- a/README.md +++ b/README.md @@ -255,6 +255,343 @@ Prometheus series use the `tailwatch_` prefix: | `tailwatch_target_parent{target,parent}` | gauge, 1 for the current parent | | `tailwatch_target_state{target}` | gauge, 0 unknown through 4 flapping | +## Notifications + +Every alert transition becomes a versioned JSON event that tail-watch delivers to +the sinks in the `sinks` list. A probe never waits on a sink: each sink has its own +worker and bounded queue, and a slow receiver cannot delay a check or another sink. +Sinks are `webhook` and `slack`. Secrets come only from `*_env` or `*_file` keys, +never from the YAML itself, and are read again on every send, so a rotated secret +needs no restart. Logs show a URL only as scheme and host (`https://hooks.slack.com/…`). + +| Event | When | +| --- | --- | +| `target.down` | a target goes `DOWN` and has no open incident | +| `target.recovered` | a target with an open incident reaches `UP` | +| `target.flapping` | a target crosses `flap_limit` | +| `collector.started` | once, after every target and check has a first result | +| `test` | only from `tail-watch notify-test` | + +A start with everything `UP` is silent apart from `collector.started`. A target +that is already failing at startup pages once. + +Severity is `info`, `warn`, or `page`. `target.down` uses the target's `severity` +(default `page`), a recovery uses the severity of the down it resolves, flapping is +`warn`, and `collector.started` and `test` are `info`. + +### Sink options + +```yaml +sinks: + - name: ops-webhook + type: webhook + url_env: OPS_WEBHOOK_URL # or url_file + secret_env: OPS_WEBHOOK_SECRET # HMAC key; without it requests are unsigned + headers_env: + Authorization: OPS_WEBHOOK_AUTH # header name: env var holding the value + min_severity: page # info < warn < page + events: [target.down, target.recovered] + max_age: 1h + queue: disk # needs collector.state_dir + queue_size: 1000 + batch: true + group_wait: 10s + max_batch: 100 + - name: ops-slack + type: slack + url_env: SLACK_WEBHOOK_URL + mention_on_page: "" + status_url: https://monitor-nyc.example.ts.net:9109 + group_wait: 10s + match: + sites: [nyc, sfo] # also targets (globs) and tags +``` + +Every sink accepts `events` (default all), `min_severity` (default `info`), `match` +with `sites`, `targets` and `tags`, `max_age` (default `1h`), `queue` (`memory` or +`disk`, default `memory`), and `queue_size` (default `1000`). An event reaches a sink +only when its type is listed, its severity is high enough, and every `match` key +matches. `tags` compares the Tailscale tags of the node the target resolves to. The +names `incidents`, `intake` and `state` are reserved because they appear as metric +labels. `tail-watch check-config` prints each sink's name, type, filters and queue mode, +never a secret. + +A recovery ignores the current filters: it goes to every sink that was selected for its +down, so a sink with `events: [target.down]` still hears when the target recovers. A +sink that never saw the down never sees the recovery. + +### Payload + +Every event is one JSON object. Within `version: 1` fields are only ever added. + +```json +{ + "version": 1, + "id": "evt_01JAB3K9T6M4X2Q8R7N5W0YZCD", + "type": "target.down", + "severity": "page", + "time": "2026-04-02T14:02:11Z", + "collector": { "name": "monitor-nyc", "tailscale_ip": "100.101.102.103" }, + "target": { "name": "nyc-nas", "addr": "10.20.0.50", "site": "nyc", "parent": "nyc-router" }, + "check": "tcp:445", + "layer": "service", + "error": "connection refused", + "since": "2026-04-02T14:00:41Z", + "failures": 3, + "path": { "kind": "direct", "endpoint": "203.0.113.5:41641", "latency_ms": 12 }, + "dependents": [] +} +``` + +`layer` is `unrouted`, `control`, `tunnel`, `lan`, or `service` and appears only on +`target.down` and `target.flapping`. `target.implicit: true` marks a router tail-watch +learned rather than one from your config. `target.recovered` adds `duration_seconds` +and `down_event_id`, and `reason: "removed"` when the target disappeared instead of +recovering. Fields that do not apply are omitted, not sent as `null`. `late: true` +marks an event first attempted more than a minute after its `time`; it keeps its +original `time`. + +`id` is `evt_` plus a 26 character Crockford base32 string that sorts by creation time. +It is the same across every retry and every sink. Delivery is at least once, so +receivers must deduplicate on `id` and must accept a `target.recovered` whose +`down_event_id` they never saw. + +### Webhook sink + +Each event is one `POST` with `Content-Type: application/json`, +`User-Agent: tail-watch/`, `Tail-Watch-Event-Id`, and any `headers_env` +headers. Redirects are not followed and each request times out after 10s. + +| Response | What tail-watch does | +| --- | --- | +| `2xx` | delivered | +| `408`, `5xx`, timeout, network error | retry with growing delay | +| `429` | retry, waiting at least `Retry-After` | +| anything else, including `3xx` and other `4xx` | drop, log once, count `permanent` | + +The delay is 1s, 2s, 4s and so on with 0.5–1.5× jitter, never more than 5 minutes. An +event older than `max_age` when an attempt is due is dropped and counted as `expired`. + +With a secret, each request carries `Tail-Watch-Signature: t=,v1=`, +where `v1` is the lowercase hex HMAC-SHA256 of `.` under the secret. A retry +sends identical body bytes with a fresh `t`. Verify it, and reject stale timestamps: + +```go +func verify(secret, header string, body []byte, tolerance time.Duration) bool { + var ts, sig string + for _, part := range strings.Split(header, ",") { + k, v, _ := strings.Cut(part, "=") + switch k { + case "t": + ts = v + case "v1": + sig = v + } + } + secs, err := strconv.ParseInt(ts, 10, 64) + if err != nil || time.Since(time.Unix(secs, 0)).Abs() > tolerance { + return false + } + mac := hmac.New(sha256.New, []byte(secret)) + fmt.Fprintf(mac, "%s.%s", ts, body) + return hmac.Equal([]byte(hex.EncodeToString(mac.Sum(nil))), []byte(sig)) +} +``` + +With `batch: true` the sink waits `group_wait` after the first queued event, or until +`max_batch` events are waiting, and sends +`{"version": 1, "batch_id": "bat_…", "events": [...]}` with `Tail-Watch-Batch-Id` in +place of `Tail-Watch-Event-Id`. The signature covers the whole body. A retry resends the +same bytes and `batch_id`; events that arrive meanwhile go into the next batch, and a +permanent failure or expiry drops the whole batch and counts every event in it. + +### Slack sink + +A Slack sink posts to an incoming-webhook URL. Each message carries Block Kit blocks +and a plain-text fallback: + +```text +:red_circle: DOWN nyc-nas tcp:445 (service) +addr 10.20.0.50 · site nyc · parent nyc-router · path direct 12 ms +error: connection refused +down since 14:00 UTC · 3 failures · collector monitor-nyc +``` + +A router learned at runtime shows `(learned)` after its name. A down with dependents adds +`and N targets behind it` and names up to 10. `mention_on_page` is prepended only to a +`target.down` of severity `page`, never to recoveries, flapping notices or warnings. With +`status_url` each target line links to `/#`. Events that arrive +within `group_wait` of the first become one message of at most 10 lines followed by +`and N more`, and the sink sends at most one message per second. Retry, `Retry-After`, +permanent-failure and `max_age` rules match the webhook sink. Slack sinks reject `batch`, +`max_batch`, `secret_*` and `headers_env`. + +### Queues, `state_dir` and open incidents + +`queue: memory` (the default) loses pending events when tail-watch restarts. `queue: disk` +needs `collector.state_dir` and keeps them. With `state_dir`, tail-watch keeps one file, +`notification-state.json`, holding open incidents, per-sink delivery state and the +pending events of disk sinks. Each change is written to a temporary file, fsynced, +renamed over the file, and the directory is fsynced, and only then can a worker send +the new events. The directory is created `0700` and the file `0600`; tail-watch warns if +the directory is readable by group or others. The file holds hostnames, addresses and +event payloads, never URLs, secrets or header values. If the file cannot be decoded, +startup fails with the path so you can repair it; tail-watch never silently forgets +open incidents. + +Open incidents are tracked even without `state_dir`. With it, a restart during an outage +does not page again, and the eventual recovery names the original `down_event_id`. A +configured target deleted from the config while an incident was open closes with +`reason: removed`. If the snapshot cannot be written, tail-watch keeps delivering from +memory, logs the loss of durability once, counts `tailwatch_notify_persist_errors_total`, +and retries the whole state every second. + +Delivery is at least once after a successful commit, and the limits are explicit and +counted: an event dropped because the queue is full, expired, permanently rejected, or +lost on a memory queue at restart shows up in `tailwatch_notify_dropped_total`. A `down` +dropped before any request was made takes its unsent `recovered` with it; if a request +was ever made, the `recovered` is still sent, because the receiver may have accepted a +request whose response was lost. Each sink keeps strict order. + +Disk commits are serialized. A slow disk can delay every durable enqueue and any worker +waiting to record an attempt, but it never delays a check, and the intake that receives +transitions from the alert state machine never blocks: if its 4096 slots fill, the +transition is dropped, counted as `dropped{sink="intake"}`, and the target's state is +reconciled within a second once there is room, so a lost recovery is still sent. + +| Metric | Kind | +| --- | --- | +| `tailwatch_notify_sent_total{sink,type}` | counter | +| `tailwatch_notify_failed_total{sink,reason}` | counter, reason `timeout`, `http_5xx`, `http_429`, `http_4xx_permanent`, `network` | +| `tailwatch_notify_dropped_total{sink,reason}` | counter, reason `queue_full`, `expired`, `permanent`, `restart` | +| `tailwatch_notify_persist_errors_total{sink}` | counter, sink is always `state` | +| `tailwatch_notify_queue_len{sink}` | gauge | +| `tailwatch_notify_latency_seconds{sink}` | histogram, event creation to confirmed delivery | +| `tailwatch_notify_batch_size{sink}` | histogram, batched webhook sinks only | +| `tailwatch_open_incidents` | gauge, when `state_dir` is set | +| `tailwatch_heartbeat_failures_total` | counter | + +Alert on `increase(tailwatch_notify_dropped_total[15m]) > 0` and on a +`tailwatch_notify_queue_len` that stays above zero: both mean an alert did not get out. + +### Watching the watcher + +A dead collector cannot report itself, so point `heartbeat` at a dead-man's-switch +service: + +```yaml +heartbeat: + url_env: HEARTBEAT_URL # or url_file + interval: 1m +``` + +tail-watch sends a `GET` every `interval`, starting once every target and check has a +first result, with a 10s timeout. A failed ping is logged, counted in +`tailwatch_heartbeat_failures_total`, and not retried before the next interval. When the +process exits the pings stop and the service alerts. + +### Trying it out + +```bash +./bin/tail-watch notify-test --config examples/tail-watch.yaml +./bin/tail-watch notify-test --sink ops-slack --timeout 10s +``` + +`notify-test` sends one `test` event through each sink's real signing, rendering and +retry code, bypassing filters, queues and `state_dir`. It prints one line per sink with +the HTTP status or error and exits 0 only when every selected sink answered `2xx` within +`--timeout` (default 30s). It works when tailscaled is down, using the OS hostname as +the collector name. Run it after every config change. + +### Egress path + +Sinks use the host's ordinary routing table, so the collector's node must reach them. +tail-watch warns at startup when an exit node is active, because notifications would +then leave through the exit node. A receiver on your own tailnet needs a grant from +`tag:monitor`: + +```jsonc +{ "src": ["tag:monitor"], "dst": ["tag:ops-receiver"], "ip": ["tcp:443"] } +``` + +### Live notification test + +`TestLiveNotifications` runs the whole notification path on a real tailnet. It is off +unless `TAIL_WATCH_LIVE_TAILNET=1`. It starts the collector and one fixture node per +check (`icmp`, `tsping`, `tcp`, `http`, and a LAN target behind a subnet router), fails and +heals each one, restarts the collector mid-outage with `queue: disk`, and runs +`notify-test`. A signature-verifying webhook receiver in the test process checks every +event and heartbeat. The compose fixtures are in +`cmd/tail-watch/testdata/compose.live-notify.yaml`. Devices are ephemeral, named +`tail-watch-live-notify-*`, and deleted afterwards. + +```bash +TAIL_WATCH_LIVE_TAILNET=1 TAIL_WATCH_LIVE_TAG= \ + TAIL_WATCH_SLACK_PULUMI_ENV=/tail-watch-live/slack \ + go test -count=1 -timeout 18m -run TestLiveNotifications -v ./cmd/tail-watch +``` + +With `TAIL_WATCH_LIVE_TAG` unset the collector joins as `tag:monitor`, so the tailnet +policy must own that tag and grant it the fixtures: merge +`cmd/tail-watch/testdata/snippet.hujson`. The grants that matter are `tcp:9000` and +`tcp:8080` to members (the `tcp` and `http` fixtures) and `icmp` to `10.10.0.0/16` (the LAN +target). Without the two TCP grants those fixtures are down from the start, with +`i/o timeout` errors because the packets are dropped. Setting `TAIL_WATCH_LIVE_TAG=` to +empty runs the collector as an untagged member node, which needs no policy change. + +Tailnet credentials come from the `tailscale-phase-2/tailscale` pulumi env +(`TAIL_WATCH_PULUMI_ENV` overrides it). Slack is optional: without a webhook the test +runs the webhook sink only. Slack is one-way, so the test checks tail-watch's own sent and +failed counters for that sink; look in the channel for the messages. +`TAIL_WATCH_LIVE_MENTION` sets `mention_on_page` for the Slack sink. + +The Slack sink uses a memory queue, so an event still inside its `group_wait` window when +the test restarts the collector is lost and counted as `dropped{reason="restart"}`. The +test accepts that reason and fails on any other drop or failure. The webhook sink uses +`queue: disk` and must survive the restart without paging again. + +To bring up only the tailnet side, without the collector process or any scenario, use +`scripts/live-stack.sh up`. It mints ephemeral keys from the pulumi env, starts the +collector's tailscale node and the five fixture nodes, approves the LAN route, and prints +each node's address. `scripts/live-stack.sh down` removes the containers and deletes +that stack's devices. The webhook can live in a pulumi env such as +`richard-shade-gmail-com/tail-watch-live/slack`; pass its name as +`TAIL_WATCH_SLACK_PULUMI_ENV`. To rotate it, run +`pulumi env set environmentVariables.SLACK_WEBHOOK_URL '' --secret`. + +Set `queue: disk` on the Slack sink if an alert must survive a collector restart. The +default `queue: memory` loses events still waiting in the group window. + +#### Getting a Slack webhook URL + +tail-watch posts to an incoming webhook URL. It does not use bot or app-level tokens +(`xoxb-`, `xapp-`), and a bot token cannot produce one. Slack's own guide, +[Sending messages using incoming webhooks](https://docs.slack.dev/messaging/sending-messages-using-incoming-webhooks/), +is the reference; its steps are: + +1. Create an app at , or open an existing one. +2. In the app's settings sidebar choose **Incoming Webhooks** and switch + **Activate Incoming Webhooks** on. +3. Click **Add New Webhook to Workspace**, pick the channel, and click **Allow**. +4. Copy the URL from **Webhook URLs for Your Workspace**. It starts with + `https://hooks.slack.com/services/`. + +Slack has also documented the `incoming-webhook` scope, which produces a webhook during +an OAuth install and shows a channel picker. If your app's sidebar has no +**Incoming Webhooks** entry, that is the alternative to try. The settings pages change +over time, so if these steps do not match what you see, treat this section as out of date +and check Slack's guide. + +Check the URL with +`curl -X POST -H 'Content-type: application/json' --data '{"text":"hello"}' "$SLACK_WEBHOOK_URL"`, +which answers `ok`. Store it as a secret, not in a file: + +```bash +pulumi env init /tail-watch-live/slack +pulumi env set /tail-watch-live/slack \ + environmentVariables.SLACK_WEBHOOK_URL 'https://hooks.slack.com/services/...' --secret +``` + ## Development and releases `mise run check` builds, tests, and lints with the pinned tools. The lint diff --git a/cmd/tail-watch/alert_target.go b/cmd/tail-watch/alert_target.go index 0013544..6b6e21c 100644 --- a/cmd/tail-watch/alert_target.go +++ b/cmd/tail-watch/alert_target.go @@ -63,7 +63,9 @@ func (a *alerter) addLocked(t *Target) error { return nil } -// RemoveTarget drops a target whose last emitted state is not DOWN. +// RemoveTarget drops a target whose last emitted state is not DOWN. An implicit +// target may be removed while DOWN: the learner asks only after authoritative +// reconciliation, and the removed snapshot lets the notifier close its incident. // Suppressed dependents are released the way a parent recovery releases them. func (a *alerter) RemoveTarget(name string) error { if a == nil { @@ -84,7 +86,7 @@ func (a *alerter) removeLocked(name string) ([]string, error) { if ts == nil { return nil, nil } - if ts.incidentOpen() { + if ts.incidentOpen() && !ts.target.Implicit { return nil, fmt.Errorf("target %s is down", name) } a.disarmHold(ts) diff --git a/cmd/tail-watch/alert_test.go b/cmd/tail-watch/alert_test.go index 78d08e5..6b694b7 100644 --- a/cmd/tail-watch/alert_test.go +++ b/cmd/tail-watch/alert_test.go @@ -974,7 +974,7 @@ func TestAlerterWiring(t *testing.T) { probes: nil, watch: func(context.Context, ipn.NotifyWatchOpt) (ipnWatch, error) { return nil, errors.New("stopped") - }, + }, notify: nil, collector: CollectorInfo{Name: "", TailscaleIP: ""}, } runtime := app.startCollectors(ctx, nil) t.Cleanup(func() { @@ -1234,3 +1234,21 @@ func advanceAlertClock(al *alerter, clk *tstest.Clock, d time.Duration) { clk.Advance(d) al.timerCallbacks.Wait() } + +func TestRemoveDownImplicitTarget(t *testing.T) { + t.Parallel() + implicit := alertTarget("nyc-router", "tsping") + implicit.Implicit = true + al, _, _ := testAlerter(t, []*Target{implicit}, nil) + failN(al, implicit, "tsping", layerService, 3) + if !al.IsDown("nyc-router") { + t.Fatal("setup: implicit target should be down") + } + if err := al.RemoveTarget("nyc-router"); err != nil { + t.Fatalf("a down implicit target must be removable: %v", err) + } + snaps := al.Snapshots() + if len(snaps) != 1 || !snaps[0].Removed || !snaps[0].Dirty || snaps[0].State != StateDown { + t.Fatalf("removal must leave a removed dirty snapshot of the open incident: %+v", snaps) + } +} diff --git a/cmd/tail-watch/checks_test.go b/cmd/tail-watch/checks_test.go index 9f61366..3dfab8a 100644 --- a/cmd/tail-watch/checks_test.go +++ b/cmd/tail-watch/checks_test.go @@ -390,7 +390,7 @@ func TestRunCheckRecordsTimeout(t *testing.T) { metrics: nil, watch: nil, probeMu: sync.Mutex{}, - probes: nil, + probes: nil, notify: nil, collector: CollectorInfo{Name: "", TailscaleIP: ""}, } ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond) defer cancel() diff --git a/cmd/tail-watch/config.go b/cmd/tail-watch/config.go index 66f9996..c29b491 100644 --- a/cmd/tail-watch/config.go +++ b/cmd/tail-watch/config.go @@ -48,6 +48,8 @@ const ( type Config struct { Collector CollectorConfig Targets []Target + Sinks []SinkConfig + Heartbeat *HeartbeatConfig } // CollectorConfig is the process-level section of a config file. @@ -115,9 +117,11 @@ func loadConfig(path string) (*Config, error) { } type rawConfig struct { - Collector rawCollector `yaml:"collector"` - Defaults rawDefaults `yaml:"defaults"` - Targets []rawTarget `yaml:"targets"` + Collector rawCollector `yaml:"collector"` + Defaults rawDefaults `yaml:"defaults"` + Targets []rawTarget `yaml:"targets"` + Sinks []rawSink `yaml:"sinks"` + Heartbeat *rawHeartbeat `yaml:"heartbeat"` } type rawCollector struct { @@ -237,7 +241,9 @@ func resolveConfig(raw rawConfig) (*Config, []error) { Listen: raw.Collector.Listen, StateDir: raw.Collector.StateDir, }, - Targets: make([]Target, len(raw.Targets)), + Targets: make([]Target, len(raw.Targets)), + Sinks: nil, + Heartbeat: nil, } seen := map[string]int{} checksOK := make([]bool, len(raw.Targets)) @@ -285,6 +291,8 @@ func resolveConfig(raw rawConfig) (*Config, []error) { } } errs = append(errs, parentCycles(cfg.Targets, byName)...) + cfg.Sinks = resolveSinks(&errs, raw.Sinks, raw.Collector.StateDir) + cfg.Heartbeat = resolveHeartbeat(&errs, raw.Heartbeat) return cfg, errs } diff --git a/cmd/tail-watch/config_sinks.go b/cmd/tail-watch/config_sinks.go new file mode 100644 index 0000000..96f8c86 --- /dev/null +++ b/cmd/tail-watch/config_sinks.go @@ -0,0 +1,411 @@ +package main + +import ( + "errors" + "fmt" + "net/url" + "os" + "path" + "regexp" + "slices" + "strings" + "time" +) + +const ( + SeverityInfo Severity = "info" + + rankInfo = 0 + rankWarn = 1 + rankPage = 2 + + sinkWebhook = "webhook" + sinkSlack = "slack" + + queueMemory = "memory" + queueDisk = "disk" + + defaultSinkMaxAge = time.Hour + defaultQueueSize = 1000 + defaultGroupWait = 10 * time.Second + defaultMaxBatch = 100 + defaultHeartbeatEvery = time.Minute + + sinkIncidents = "incidents" + sinkIntake = "intake" + sinkState = "state" +) + +// EventType is the type field of a notification payload. +type EventType string + +const ( + EventDown EventType = "target.down" + EventRecovered EventType = "target.recovered" + EventFlapping EventType = "target.flapping" + EventDegraded EventType = "target.degraded" + EventStarted EventType = "collector.started" + EventTest EventType = "test" +) + +func knownEventTypes() []EventType { + return []EventType{EventDown, EventRecovered, EventFlapping, EventDegraded, EventStarted, EventTest} +} + +// rank orders severities: info < warn < page. +func (s Severity) rank() int { + switch s { + case SeverityPage: + return rankPage + case SeverityWarn: + return rankWarn + case SeverityInfo: + return rankInfo + default: + return rankInfo + } +} + +// secretRef names where a secret value lives. Exactly one field is set. +type secretRef struct { + Env string + File string +} + +func (r secretRef) isSet() bool { return r.Env != "" || r.File != "" } + +// resolve reads the current secret value. File values are whitespace trimmed. +func (r secretRef) resolve() (string, error) { + switch { + case r.Env != "": + v := os.Getenv(r.Env) + if v == "" { + return "", fmt.Errorf("environment variable %s is unset or empty", r.Env) + } + return v, nil + case r.File != "": + b, err := os.ReadFile(r.File) + if err != nil { + return "", fmt.Errorf("secret file: %w", err) + } + v := strings.TrimSpace(string(b)) + if v == "" { + return "", fmt.Errorf("secret file %s is empty", r.File) + } + return v, nil + default: + return "", errors.New("no secret source configured") + } +} + +// redactURL keeps only the scheme and host of a URL. +func redactURL(raw string) string { + u, err := url.Parse(raw) + if err != nil || u.Scheme == "" || u.Host == "" { + return "" + } + return u.Scheme + "://" + u.Host + "/…" +} + +// MatchConfig narrows which targets a sink hears about. +type MatchConfig struct { + Sites []string + Targets []string + Tags []string +} + +// SinkConfig is one resolved notification sink. +type SinkConfig struct { + Name string + Type string + URL secretRef + Secret secretRef + HeadersEnv map[string]string + MinSeverity Severity + Events []EventType + Match MatchConfig + MaxAge time.Duration + Queue string + QueueSize int + Batch bool + GroupWait time.Duration + MaxBatch int + MentionOnPage string + StatusURL string +} + +// HeartbeatConfig is the dead-man's-switch ping. +type HeartbeatConfig struct { + URL secretRef + Interval time.Duration +} + +type rawMatch struct { + Sites []string `yaml:"sites"` + Targets []string `yaml:"targets"` + Tags []string `yaml:"tags"` +} + +type rawSink struct { + Name string `yaml:"name"` + Type string `yaml:"type"` + URLEnv string `yaml:"url_env"` + URLFile string `yaml:"url_file"` + SecretEnv string `yaml:"secret_env"` + SecretFile string `yaml:"secret_file"` + HeadersEnv map[string]string `yaml:"headers_env"` + MinSeverity string `yaml:"min_severity"` + Events []string `yaml:"events"` + Match rawMatch `yaml:"match"` + MaxAge yamlDuration `yaml:"max_age"` + Queue string `yaml:"queue"` + QueueSize *int `yaml:"queue_size"` + Batch *bool `yaml:"batch"` + GroupWait yamlDuration `yaml:"group_wait"` + MaxBatch *int `yaml:"max_batch"` + MentionOnPage string `yaml:"mention_on_page"` + StatusURL string `yaml:"status_url"` +} + +type rawHeartbeat struct { + URLEnv string `yaml:"url_env"` + URLFile string `yaml:"url_file"` + Interval yamlDuration `yaml:"interval"` +} + +var sinkNameRE = regexp.MustCompile(`^[A-Za-z0-9._-]+$`) + +func resolveSinks(errs *[]error, raws []rawSink, stateDir string) []SinkConfig { + out := make([]SinkConfig, len(raws)) + seen := map[string]int{} + for i, rs := range raws { + out[i] = resolveSink(errs, i, rs, stateDir, seen) + } + return out +} + +func resolveSink(errs *[]error, i int, rs rawSink, stateDir string, seen map[string]int) SinkConfig { + loc := fmt.Sprintf("sinks[%d]", i) + label := rs.Name + if label == "" { + label = "(unnamed)" + } + fail := func(key, format string, args ...any) { + *errs = append(*errs, fmt.Errorf("%s.%s: sink %s: %s", loc, key, label, fmt.Sprintf(format, args...))) + } + s := SinkConfig{ + Name: rs.Name, + Type: rs.Type, + URL: secretRef{Env: rs.URLEnv, File: rs.URLFile}, + Secret: secretRef{Env: rs.SecretEnv, File: rs.SecretFile}, + HeadersEnv: rs.HeadersEnv, + MinSeverity: SeverityInfo, + Events: nil, + Match: MatchConfig{Sites: rs.Match.Sites, Targets: rs.Match.Targets, Tags: rs.Match.Tags}, + MaxAge: defaultSinkMaxAge, + Queue: queueMemory, + QueueSize: defaultQueueSize, + Batch: false, + GroupWait: defaultGroupWait, + MaxBatch: defaultMaxBatch, + MentionOnPage: rs.MentionOnPage, + StatusURL: rs.StatusURL, + } + switch { + case rs.Name == "": + fail("name", "name is required") + case !sinkNameRE.MatchString(rs.Name): + fail("name", "name %q must match [A-Za-z0-9._-]", rs.Name) + case rs.Name == sinkIncidents || rs.Name == sinkIntake || rs.Name == sinkState: + fail("name", "name %q is reserved", rs.Name) + default: + if prev, dup := seen[rs.Name]; dup { + fail("name", "duplicate sink name %q (also sinks[%d])", rs.Name, prev) + } else { + seen[rs.Name] = i + } + } + switch rs.Type { + case sinkWebhook, sinkSlack: + default: + fail("type", "type %q must be webhook or slack", rs.Type) + } + if (rs.URLEnv == "") == (rs.URLFile == "") { + fail("url", "exactly one of url_env or url_file is required") + } + if rs.SecretEnv != "" && rs.SecretFile != "" { + fail("secret", "set only one of secret_env or secret_file") + } + resolveSinkTypeKeys(fail, rs) + resolveSinkFilters(fail, rs, &s) + resolveSinkLimits(errs, fail, loc, rs, &s, stateDir) + return s +} + +func resolveSinkTypeKeys(fail func(key, format string, args ...any), rs rawSink) { + switch rs.Type { + case sinkSlack: + for _, k := range []struct { + key string + set bool + }{ + {"batch", rs.Batch != nil}, + {"max_batch", rs.MaxBatch != nil}, + {"secret_env", rs.SecretEnv != ""}, + {"secret_file", rs.SecretFile != ""}, + {"headers_env", len(rs.HeadersEnv) > 0}, + } { + if k.set { + fail(k.key, "%s is not valid for a slack sink", k.key) + } + } + case sinkWebhook: + if rs.MentionOnPage != "" { + fail("mention_on_page", "mention_on_page is only valid for a slack sink") + } + if rs.StatusURL != "" { + fail("status_url", "status_url is only valid for a slack sink") + } + } +} + +func resolveSinkFilters(fail func(key, format string, args ...any), rs rawSink, s *SinkConfig) { + switch Severity(rs.MinSeverity) { + case "": + case SeverityInfo, SeverityWarn, SeverityPage: + s.MinSeverity = Severity(rs.MinSeverity) + default: + fail("min_severity", "min_severity %q must be info, warn or page", rs.MinSeverity) + } + for _, e := range rs.Events { + if !slices.Contains(knownEventTypes(), EventType(e)) { + fail("events", "unknown event type %q", e) + continue + } + s.Events = append(s.Events, EventType(e)) + } + for _, g := range rs.Match.Targets { + if _, err := path.Match(g, "x"); err != nil { + fail("match.targets", "invalid glob %q", g) + } + } +} + +func resolveSinkLimits( + errs *[]error, + fail func(key, format string, args ...any), + loc string, + rs rawSink, + s *SinkConfig, + stateDir string, +) { + switch rs.Queue { + case "": + case queueMemory: + case queueDisk: + s.Queue = queueDisk + if stateDir == "" { + fail("queue", "queue: disk requires collector.state_dir") + } + default: + fail("queue", "queue %q must be memory or disk", rs.Queue) + } + applyDur(errs, loc+".max_age", rs.MaxAge, &s.MaxAge) + applyDur(errs, loc+".group_wait", rs.GroupWait, &s.GroupWait) + applyInt(rs.QueueSize, &s.QueueSize) + applyInt(rs.MaxBatch, &s.MaxBatch) + if rs.Batch != nil { + s.Batch = *rs.Batch + } + if s.MaxAge <= 0 { + fail("max_age", "max_age must be positive") + } + if s.GroupWait <= 0 { + fail("group_wait", "group_wait must be positive") + } + if s.QueueSize < 1 { + fail("queue_size", "queue_size must be positive") + } + if s.MaxBatch < 1 { + fail("max_batch", "max_batch must be positive") + } +} + +func resolveHeartbeat(errs *[]error, rh *rawHeartbeat) *HeartbeatConfig { + if rh == nil { + return nil + } + hb := &HeartbeatConfig{ + URL: secretRef{Env: rh.URLEnv, File: rh.URLFile}, + Interval: defaultHeartbeatEvery, + } + if (rh.URLEnv == "") == (rh.URLFile == "") { + *errs = append(*errs, errors.New("heartbeat.url: exactly one of url_env or url_file is required")) + } + applyDur(errs, "heartbeat.interval", rh.Interval, &hb.Interval) + if hb.Interval <= 0 { + *errs = append(*errs, errors.New("heartbeat.interval: interval must be positive")) + } + return hb +} + +// resolveSecrets fails fast when any referenced secret cannot be read now. +func (c *Config) resolveSecrets() error { + var errs []error + for _, s := range c.Sinks { + if err := s.checkSecrets(); err != nil { + errs = append(errs, err) + } + } + if c.Heartbeat != nil { + if _, err := c.Heartbeat.URL.resolve(); err != nil { + errs = append(errs, fmt.Errorf("heartbeat: %w", err)) + } + } + return errors.Join(errs...) +} + +func (s SinkConfig) checkSecrets() error { + var errs []error + if _, err := s.URL.resolve(); err != nil { + errs = append(errs, fmt.Errorf("sink %s: url: %w", s.Name, err)) + } + if s.Secret.isSet() { + if _, err := s.Secret.resolve(); err != nil { + errs = append(errs, fmt.Errorf("sink %s: secret: %w", s.Name, err)) + } + } + for header, env := range s.HeadersEnv { + if _, err := (secretRef{Env: env, File: ""}).resolve(); err != nil { + errs = append(errs, fmt.Errorf("sink %s: header %s: %w", s.Name, header, err)) + } + } + return errors.Join(errs...) +} + +// describeSink is the check-config line for a sink. It never prints secrets. +func describeSink(s SinkConfig) string { + events := "all" + if len(s.Events) > 0 { + parts := make([]string, len(s.Events)) + for i, e := range s.Events { + parts[i] = string(e) + } + events = strings.Join(parts, ",") + } + line := fmt.Sprintf("sink %s type=%s min_severity=%s events=%s queue=%s", + s.Name, s.Type, s.MinSeverity, events, s.Queue) + var match []string + for _, kv := range []struct { + key string + vals []string + }{{"sites", s.Match.Sites}, {"targets", s.Match.Targets}, {"tags", s.Match.Tags}} { + if len(kv.vals) > 0 { + match = append(match, kv.key+":"+strings.Join(kv.vals, ",")) + } + } + if len(match) > 0 { + line += " match=" + strings.Join(match, " ") + } + return line +} diff --git a/cmd/tail-watch/config_sinks_test.go b/cmd/tail-watch/config_sinks_test.go new file mode 100644 index 0000000..3a68fee --- /dev/null +++ b/cmd/tail-watch/config_sinks_test.go @@ -0,0 +1,286 @@ +package main + +import ( + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +const sinkBase = ` +collector: + state_dir: /var/lib/tail-watch +targets: + - name: nas + addr: 10.0.0.1 + checks: [icmp] +` + +const designSinks = sinkBase + ` +sinks: + - name: ops-webhook + type: webhook + url_env: OPS_WEBHOOK_URL + secret_env: OPS_WEBHOOK_SECRET + headers_env: + Authorization: OPS_WEBHOOK_AUTH + min_severity: page + events: [target.down, target.recovered] + max_age: 1h + queue: disk + queue_size: 1000 + batch: true + group_wait: 10s + max_batch: 100 + - name: ops-slack + type: slack + url_env: SLACK_WEBHOOK_URL + min_severity: warn + mention_on_page: "" + group_wait: 10s + queue: memory + status_url: https://monitor-nyc.example.ts.net:9109 + match: + sites: [nyc, sfo] +heartbeat: + url_env: HEARTBEAT_URL + interval: 1m +` + +func TestParseDesignSinks(t *testing.T) { + t.Parallel() + cfg, err := parseConfig([]byte(designSinks)) + if err != nil { + t.Fatal(err) + } + if len(cfg.Sinks) != 2 || cfg.Heartbeat == nil { + t.Fatalf("sinks=%d heartbeat=%v", len(cfg.Sinks), cfg.Heartbeat) + } + wh := cfg.Sinks[0] + if wh.Queue != queueDisk || !wh.Batch || wh.MinSeverity != SeverityPage || wh.URL.Env != "OPS_WEBHOOK_URL" || + wh.HeadersEnv["Authorization"] != "OPS_WEBHOOK_AUTH" || len(wh.Events) != 2 { + t.Fatalf("webhook: %+v", wh) + } + sl := cfg.Sinks[1] + if sl.MentionOnPage != "" || len(sl.Match.Sites) != 2 || sl.Queue != queueMemory { + t.Fatalf("slack: %+v", sl) + } +} + +func TestSinkDefaults(t *testing.T) { + t.Parallel() + cfg, err := parseConfig([]byte(sinkBase + ` +sinks: + - {name: a, type: webhook, url_env: U} +heartbeat: + url_file: /run/hb +`)) + if err != nil { + t.Fatal(err) + } + s := cfg.Sinks[0] + if s.MinSeverity != SeverityInfo || len(s.Events) != 0 || s.MaxAge != time.Hour || s.Queue != queueMemory || + s.QueueSize != 1000 || s.GroupWait != 10*time.Second || s.MaxBatch != 100 || s.Batch { + t.Fatalf("defaults: %+v", s) + } + if cfg.Heartbeat.Interval != time.Minute { + t.Fatalf("heartbeat interval %s", cfg.Heartbeat.Interval) + } +} + +func TestSinkValidation(t *testing.T) { + t.Parallel() + tests := []struct { + name string + yaml string + want string + }{ + { + "duplicate name", + "sinks:\n - {name: a, type: slack, url_env: U}\n - {name: a, type: slack, url_env: U}", + "duplicate sink name", + }, + {"invalid name", "sinks:\n - {name: 'a b', type: slack, url_env: U}", "must match"}, + {"reserved incidents", "sinks:\n - {name: incidents, type: slack, url_env: U}", "reserved"}, + {"reserved intake", "sinks:\n - {name: intake, type: slack, url_env: U}", "reserved"}, + {"reserved state", "sinks:\n - {name: state, type: slack, url_env: U}", "reserved"}, + {"unknown type", "sinks:\n - {name: a, type: pagerduty, url_env: U}", "must be webhook or slack"}, + {"no url", "sinks:\n - {name: a, type: slack}", "exactly one of url_env or url_file"}, + { + "two urls", + "sinks:\n - {name: a, type: slack, url_env: U, url_file: /f}", + "exactly one of url_env or url_file", + }, + { + "slack batch", + "sinks:\n - {name: a, type: slack, url_env: U, batch: true}", + "batch is not valid for a slack sink", + }, + {"slack secret", "sinks:\n - {name: a, type: slack, url_env: U, secret_env: S}", "secret_env is not valid"}, + { + "webhook mention", + "sinks:\n - {name: a, type: webhook, url_env: U, mention_on_page: x}", + "only valid for a slack sink", + }, + { + "disk without state_dir", + "sinks:\n - {name: a, type: webhook, url_env: U, queue: disk}", + "requires collector.state_dir", + }, + {"bad queue", "sinks:\n - {name: a, type: webhook, url_env: U, queue: tape}", "must be memory or disk"}, + { + "queue_size", + "sinks:\n - {name: a, type: webhook, url_env: U, queue_size: 0}", + "queue_size must be positive", + }, + {"max_batch", "sinks:\n - {name: a, type: webhook, url_env: U, max_batch: -1}", "max_batch must be positive"}, + { + "group_wait", + "sinks:\n - {name: a, type: webhook, url_env: U, group_wait: 0s}", + "group_wait must be positive", + }, + {"max_age", "sinks:\n - {name: a, type: webhook, url_env: U, max_age: -1s}", "max_age must be positive"}, + { + "bad event", + "sinks:\n - {name: a, type: webhook, url_env: U, events: [target.exploded]}", + "unknown event type", + }, + {"bad glob", "sinks:\n - {name: a, type: webhook, url_env: U, match: {targets: ['[']}}", "invalid glob"}, + {"bad severity", "sinks:\n - {name: a, type: webhook, url_env: U, min_severity: loud}", "min_severity"}, + {"heartbeat no url", "heartbeat:\n interval: 1m", "exactly one of url_env or url_file"}, + {"heartbeat interval", "heartbeat:\n url_env: H\n interval: 0s", "interval must be positive"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + src := sinkBase + if tt.name == "disk without state_dir" { + src = strings.Replace(src, " state_dir: /var/lib/tail-watch\n", " listen: 127.0.0.1:1\n", 1) + } + _, err := parseConfig([]byte(src + tt.yaml)) + if err == nil || !strings.Contains(err.Error(), tt.want) { + t.Fatalf("want %q, got %v", tt.want, err) + } + }) + } +} + +func TestSinkValidationCollectsAllErrors(t *testing.T) { + t.Parallel() + _, err := parseConfig([]byte(sinkBase + ` +sinks: + - {name: state, type: slack, url_env: U} + - {name: b, type: nope} + - {name: c, type: webhook, url_env: U, queue: disk, queue_size: 0} +`)) + if err == nil { + t.Fatal("expected errors") + } + for _, want := range []string{"reserved", "must be webhook or slack", "queue_size must be positive"} { + if !strings.Contains(err.Error(), want) { + t.Fatalf("missing %q in %v", want, err) + } + } +} + +func TestSecretRefResolve(t *testing.T) { + t.Setenv("TW_SECRET", "s3cret") + t.Setenv("TW_EMPTY", "") + dir := t.TempDir() + file := filepath.Join(dir, "secret") + if err := os.WriteFile(file, []byte(" from-file\n"), 0o600); err != nil { + t.Fatal(err) + } + empty := filepath.Join(dir, "empty") + if err := os.WriteFile(empty, []byte(" \n"), 0o600); err != nil { + t.Fatal(err) + } + tests := []struct { + name string + ref secretRef + want string + err string + }{ + {"env", secretRef{Env: "TW_SECRET", File: ""}, "s3cret", ""}, + {"file trimmed", secretRef{Env: "", File: file}, "from-file", ""}, + {"unset env", secretRef{Env: "TW_UNSET_NOPE", File: ""}, "", "TW_UNSET_NOPE is unset or empty"}, + {"empty env", secretRef{Env: "TW_EMPTY", File: ""}, "", "TW_EMPTY is unset or empty"}, + {"unreadable file", secretRef{Env: "", File: filepath.Join(dir, "missing")}, "", "secret file"}, + {"empty file", secretRef{Env: "", File: empty}, "", "is empty"}, + } + for _, tt := range tests { + got, err := tt.ref.resolve() + if tt.err != "" { + if err == nil || !strings.Contains(err.Error(), tt.err) { + t.Fatalf("%s: want %q, got %v", tt.name, tt.err, err) + } + continue + } + if err != nil || got != tt.want { + t.Fatalf("%s: got %q, %v", tt.name, got, err) + } + } +} + +func TestResolveSecretsNamesSinkAndVariable(t *testing.T) { + t.Parallel() + cfg, err := parseConfig([]byte(sinkBase + "sinks:\n - {name: ops, type: slack, url_env: TW_NOPE_URL}\n")) + if err != nil { + t.Fatal(err) + } + err = cfg.resolveSecrets() + if err == nil || !strings.Contains(err.Error(), "ops") || !strings.Contains(err.Error(), "TW_NOPE_URL") { + t.Fatalf("got %v", err) + } +} + +func TestRedactURL(t *testing.T) { + t.Parallel() + tests := map[string]string{ + "https://hooks.slack.com/services/T0/B0/x": "https://hooks.slack.com/…", + "http://10.0.0.1:8080/a?k=v": "http://10.0.0.1:8080/…", + "not a url": "", + } + for in, want := range tests { + if got := redactURL(in); got != want { + t.Fatalf("redactURL(%q) = %q, want %q", in, got, want) + } + } +} + +func TestCheckConfigPrintsSinksWithoutSecrets(t *testing.T) { + t.Setenv("TW_HOOK_URL", "https://hooks.example.com/SECRETPATH") + t.Setenv("TW_HOOK_SECRET", "hmac-value-xyz") + t.Setenv("TW_HOOK_AUTH", "Bearer token-value") + path := filepath.Join(t.TempDir(), "c.yaml") + src := sinkBase + ` +sinks: + - name: ops + type: webhook + url_env: TW_HOOK_URL + secret_env: TW_HOOK_SECRET + headers_env: {Authorization: TW_HOOK_AUTH} + events: [target.down] + match: {sites: [nyc]} +heartbeat: + url_env: TW_HOOK_URL +` + if err := os.WriteFile(path, []byte(src), 0o600); err != nil { + t.Fatal(err) + } + var stdout, stderr strings.Builder + if code := run([]string{"check-config", "--config", path}, &stdout, &stderr); code != 0 { + t.Fatalf("exit %d: %s", code, stderr.String()) + } + out := stdout.String() + if !strings.Contains(out, "sink ops type=webhook") || !strings.Contains(out, "queue=memory") || + !strings.Contains(out, "target.down") || !strings.Contains(out, "sites:nyc") { + t.Fatalf("missing sink description:\n%s", out) + } + for _, secret := range []string{"SECRETPATH", "hmac-value-xyz", "token-value", "TW_HOOK_SECRET"} { + if strings.Contains(out, secret) { + t.Fatalf("output leaks %q:\n%s", secret, out) + } + } +} diff --git a/cmd/tail-watch/config_test.go b/cmd/tail-watch/config_test.go index dc17a74..991701c 100644 --- a/cmd/tail-watch/config_test.go +++ b/cmd/tail-watch/config_test.go @@ -328,10 +328,11 @@ func TestExampleConfig(t *testing.T) { func TestListenFlagOverride(t *testing.T) { t.Parallel() - cfg := &Config{Collector: CollectorConfig{Listen: "127.0.0.1:1", - StateDir: "", - }, - Targets: nil, + cfg := &Config{ + Collector: CollectorConfig{Listen: "127.0.0.1:1", StateDir: ""}, + Targets: nil, + Sinks: nil, + Heartbeat: nil, } if got := cfg.ListenAddr("127.0.0.1:2"); got != "127.0.0.1:2" { t.Fatal(got) diff --git a/cmd/tail-watch/deliver.go b/cmd/tail-watch/deliver.go new file mode 100644 index 0000000..89b87ec --- /dev/null +++ b/cmd/tail-watch/deliver.go @@ -0,0 +1,183 @@ +package main + +import ( + "context" + "strconv" + "time" + + "tailscale.com/tstime" +) + +const ( + maxBackoffShift = 20 + jitterFloor = 0.5 + maxRetryDelay = 5 * time.Minute + lateAfter = time.Minute +) + +type resultClass int + +const ( + resultOK resultClass = iota + resultRetry + resultPermanent +) + +// deliveryResult is the classified outcome of one request. +type deliveryResult struct { + Class resultClass + Status int + Reason string + RetryAfter time.Duration + Err error +} + +func (r deliveryResult) String() string { + switch { + case r.Status != 0: + return strconv.Itoa(r.Status) + case r.Err != nil: + return r.Err.Error() + default: + return r.Reason + } +} + +// sendMeta identifies one request. ID is the event id, or the batch id. +type sendMeta struct { + ID string + At time.Time +} + +// sink renders and sends requests for one destination. +type sink interface { + Name() string + // Render builds the first-attempt body for evs. unitID is the event id of a + // single event or the batch id of a group. + Render(evs []Event, unitID string) ([]byte, error) + // Send makes one request with body and classifies the result. + Send(ctx context.Context, body []byte, m sendMeta) deliveryResult + // MinInterval is the smallest spacing between requests, or zero. + MinInterval() time.Duration +} + +// retryDelay is min(2^(attempt-1) s, 5m) scaled by 0.5-1.5x jitter, raised to +// retryAfter, and capped at 5m. +func retryDelay(attempt int, jitter float64, retryAfter time.Duration) time.Duration { + shift := min(max(attempt-1, 0), maxBackoffShift) + base := min(time.Second<> (bitsPerByte * (idTimeBytes - 1 - i))) & byteMask) + } + if src == nil { + src = rand.Reader + } + if _, err := io.ReadFull(src, b[idTimeBytes:]); err != nil { + return "", fmt.Errorf("event id: %w", err) + } + out := make([]byte, idChars) + for i := range out { + v := 0 + for j := range 5 { + v <<= 1 + if p := i*5 + j - idPadBits; p >= 0 && b[p/bitsPerByte]>>(bitsPerByte-1-p%bitsPerByte)&1 == 1 { + v |= 1 + } + } + out[i] = idAlphabet[v] + } + return prefix + string(out), nil +} + +// stamp marshals as RFC 3339 UTC with second precision. +type stamp time.Time + +func (s stamp) MarshalJSON() ([]byte, error) { + return json.Marshal(time.Time(s).UTC().Format(time.RFC3339)) +} + +func (s *stamp) UnmarshalJSON(b []byte) error { + var str string + if err := json.Unmarshal(b, &str); err != nil { + return err + } + t, err := time.Parse(time.RFC3339, str) + if err != nil { + return err + } + *s = stamp(t) + return nil +} + +// Time returns s as a [time.Time]. +func (s stamp) Time() time.Time { return time.Time(s) } + +// CollectorInfo identifies the collector in every payload. +type CollectorInfo struct { + Name string `json:"name"` + TailscaleIP string `json:"tailscale_ip,omitempty"` +} + +// EventTarget is the target object of a payload. +type EventTarget struct { + Name string `json:"name"` + Addr string `json:"addr"` + Site string `json:"site,omitempty"` + Parent string `json:"parent,omitempty"` + Implicit bool `json:"implicit,omitempty"` +} + +// EventPath is how the latest probe reached the target. +type EventPath struct { + Kind string `json:"kind,omitempty"` + Endpoint string `json:"endpoint,omitempty"` + DERPRegion string `json:"derp_region,omitempty"` + LatencyMS *int64 `json:"latency_ms,omitempty"` +} + +// Event is the version 1 notification payload. Changes within version 1 only +// add fields. +type Event struct { + Version int `json:"version"` + ID string `json:"id"` + Type EventType `json:"type"` + Severity Severity `json:"severity"` + Time stamp `json:"time"` + Late bool `json:"late,omitempty"` + Collector CollectorInfo `json:"collector"` + Target *EventTarget `json:"target,omitempty"` + Check string `json:"check,omitempty"` + Layer string `json:"layer,omitempty"` + Error string `json:"error,omitempty"` + Since *stamp `json:"since,omitempty"` + Failures int `json:"failures,omitempty"` + Path *EventPath `json:"path,omitempty"` + Dependents *[]string `json:"dependents,omitempty"` + DurationSeconds *int64 `json:"duration_seconds,omitempty"` + DownEventID string `json:"down_event_id,omitempty"` + Reason string `json:"reason,omitempty"` +} + +func depsPtr(d []string) *[]string { + out := slices.Clone(d) + if out == nil { + out = []string{} + } + return &out +} + +func eventTargetOf(t Target) *EventTarget { + return &EventTarget{Name: t.Name, Addr: t.Addr, Site: t.Site, Parent: t.Parent, Implicit: t.Implicit} +} + +func eventPathOf(p pathInfo) *EventPath { + if p == (pathInfo{Kind: "", Endpoint: "", DERP: "", Latency: 0}) { + return nil + } + out := &EventPath{Kind: p.Kind, Endpoint: p.Endpoint, DERPRegion: p.DERP, LatencyMS: nil} + if p.Latency > 0 { + ms := p.Latency.Milliseconds() + out.LatencyMS = &ms + } + return out +} + +func stampPtr(t time.Time) *stamp { + if t.IsZero() { + return nil + } + s := stamp(t) + return &s +} + +func errString(err error) string { + if err == nil { + return "" + } + return err.Error() +} + +// baseEvent fills the fields every payload carries. +func baseEvent(id string, typ EventType, sev Severity, at time.Time, col CollectorInfo) Event { + return Event{ + Version: eventVersion, ID: id, Type: typ, Severity: sev, Time: stamp(at), Late: false, + Collector: col, Target: nil, Check: "", Layer: "", Error: "", Since: nil, Failures: 0, + Path: nil, Dependents: nil, DurationSeconds: nil, DownEventID: "", Reason: "", + } +} + +func newDownEvent(id string, col CollectorInfo, tr Transition) Event { + sev := tr.Target.Severity + if sev == "" { + sev = SeverityPage + } + ev := baseEvent(id, EventDown, sev, tr.At, col) + ev.Target = eventTargetOf(tr.Target) + ev.Check = tr.Check + ev.Layer = string(tr.Layer) + ev.Error = errString(tr.Err) + ev.Since = stampPtr(downSinceOf(tr)) + ev.Failures = tr.Failures + ev.Path = eventPathOf(tr.Path) + ev.Dependents = depsPtr(tr.Dependents) + return ev +} + +func downSinceOf(tr Transition) time.Time { + if !tr.Since.IsZero() { + return tr.Since + } + return tr.DownSince +} + +func newFlappingEvent(id string, col CollectorInfo, tr Transition) Event { + ev := baseEvent(id, EventFlapping, SeverityWarn, tr.At, col) + ev.Target = eventTargetOf(tr.Target) + ev.Check = tr.Check + ev.Layer = string(tr.Layer) + ev.Error = errString(tr.Err) + ev.Since = stampPtr(downSinceOf(tr)) + ev.Failures = tr.Failures + ev.Path = eventPathOf(tr.Path) + return ev +} + +// newRecoveredEvent resolves the down event held by inc. Reason is "removed" +// when the target disappeared rather than recovered. +func newRecoveredEvent(id string, col CollectorInfo, inc *incident, at time.Time, deps []string, reason string) Event { + down := inc.Down + ev := baseEvent(id, EventRecovered, down.Severity, at, col) + ev.Target = down.Target + ev.Check = down.Check + ev.Error = down.Error + ev.Since = down.Since + ev.Failures = down.Failures + ev.Path = down.Path + ev.Dependents = depsPtr(deps) + ev.DownEventID = down.ID + ev.Reason = reason + begin := down.Time.Time() + if down.Since != nil { + begin = down.Since.Time() + } + secs := max(int64(at.Sub(begin).Seconds()), 0) + ev.DurationSeconds = &secs + return ev +} + +func newStartedEvent(id string, col CollectorInfo, at time.Time) Event { + return baseEvent(id, EventStarted, SeverityInfo, at, col) +} + +func newTestEvent(id string, col CollectorInfo, at time.Time) Event { + return baseEvent(id, EventTest, SeverityInfo, at, col) +} + +// mapKind is what a transition means for the incident book. +type mapKind int + +const ( + mapNone mapKind = iota + mapDown + mapRecovered + mapFlapping +) + +// mapTransition applies the transition-to-event table. open reports whether +// the target already has an open incident. +func mapTransition(tr Transition, open bool) mapKind { + switch { + case tr.Flapping: + return mapFlapping + case tr.To == StateDown && !open: + return mapDown + case tr.To == StateUp && open: + return mapRecovered + default: + return mapNone + } +} + +// accepts reports whether the sink's routing filters pass ev. tags are the +// tailnet tags of the node the event's target resolves to. +func (s SinkConfig) accepts(ev Event, tags []string) bool { + if ev.Type == EventTest { + return true + } + if len(s.Events) > 0 && !slices.Contains(s.Events, ev.Type) { + return false + } + if ev.Severity.rank() < s.MinSeverity.rank() { + return false + } + if ev.Target == nil { + return true + } + m := s.Match + if len(m.Sites) > 0 && !slices.Contains(m.Sites, ev.Target.Site) { + return false + } + if len(m.Targets) > 0 && !slices.ContainsFunc(m.Targets, func(g string) bool { + ok, err := path.Match(g, ev.Target.Name) + return err == nil && ok + }) { + return false + } + if len(m.Tags) > 0 && !slices.ContainsFunc(m.Tags, func(t string) bool { return slices.Contains(tags, t) }) { + return false + } + return true +} diff --git a/cmd/tail-watch/event_test.go b/cmd/tail-watch/event_test.go new file mode 100644 index 0000000..57e7ac8 --- /dev/null +++ b/cmd/tail-watch/event_test.go @@ -0,0 +1,405 @@ +package main + +import ( + "bytes" + "crypto/rand" + "encoding/json" + "errors" + "flag" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +//nolint:gochecknoglobals // Flag must be a package variable. +var update = flag.Bool("update", false, "rewrite golden files") + +func assertGolden(t *testing.T, path string, raw []byte) { + t.Helper() + var buf bytes.Buffer + if err := json.Indent(&buf, raw, "", " "); err != nil { + t.Fatalf("golden %s: %v", path, err) + } + buf.WriteByte('\n') + if *update { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, buf.Bytes(), 0o644); err != nil { + t.Fatal(err) + } + return + } + want, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read golden (run with -update): %v", err) + } + if !bytes.Equal(want, buf.Bytes()) { + t.Fatalf("golden %s differs:\n--- want\n%s\n--- got\n%s", path, want, buf.Bytes()) + } +} + +func TestNewIDShapeAndOrder(t *testing.T) { + t.Parallel() + clk := newCountingClock() + a, err := newID(prefixEvent, clk.Now(), rand.Reader) + if err != nil { + t.Fatal(err) + } + clk.Advance(time.Millisecond) + b, err := newID(prefixEvent, clk.Now(), rand.Reader) + if err != nil { + t.Fatal(err) + } + for _, id := range []string{a, b} { + body, ok := strings.CutPrefix(id, prefixEvent) + if !ok || len(body) != 26 { + t.Fatalf("bad id %q", id) + } + for _, r := range body { + if !strings.ContainsRune(idAlphabet, r) { + t.Fatalf("id %q has character %q outside Crockford base32", id, r) + } + } + } + if a >= b { + t.Fatalf("ids do not sort by time: %s >= %s", a, b) + } + batch, err := newID(prefixBatch, clk.Now(), rand.Reader) + if err != nil || !strings.HasPrefix(batch, "bat_") { + t.Fatalf("batch id %q, %v", batch, err) + } +} + +func TestNewIDKnownVector(t *testing.T) { + t.Parallel() + zero := bytes.NewReader(make([]byte, 10)) + id, err := newID(prefixEvent, time.UnixMilli(0), zero) + if err != nil { + t.Fatal(err) + } + if id != "evt_00000000000000000000000000" { + t.Fatalf("zero id %q", id) + } + ones := bytes.NewReader(bytes.Repeat([]byte{0xff}, 10)) + id, err = newID(prefixEvent, time.UnixMilli(1<<48-1), ones) + if err != nil { + t.Fatal(err) + } + if id != "evt_7ZZZZZZZZZZZZZZZZZZZZZZZZZ" { + t.Fatalf("max id %q", id) + } +} + +func goldenCollector() CollectorInfo { + return CollectorInfo{Name: "monitor-nyc", TailscaleIP: "100.101.102.103"} +} + +func goldenTransition() Transition { + at := time.Date(2026, 4, 2, 14, 2, 11, 0, time.UTC) + return Transition{ + Sequence: 7, + Target: nasTarget(), + From: StateUp, + To: StateDown, + Flapping: false, + Check: "tcp:445", + Layer: layerService, + Err: errors.New("connection refused"), + Since: at.Add(-90 * time.Second), + At: at, + Failures: 3, + Path: pathInfo{Kind: "direct", Endpoint: "203.0.113.5:41641", DERP: "", Latency: 12 * time.Millisecond}, + Dependents: nil, + DownSince: at.Add(-90 * time.Second), + } +} + +func marshalEvent(t *testing.T, ev Event) []byte { + t.Helper() + b, err := marshalJSON(ev) + if err != nil { + t.Fatal(err) + } + return b +} + +func TestPayloadGolden(t *testing.T) { + t.Parallel() + tr := goldenTransition() + down := newDownEvent("evt_01JAB3K9T6M4X2Q8R7N5W0YZCD", goldenCollector(), tr) + inc := &incident{ + ID: "inc_test", Status: incidentOpen, Target: targetDefOf(nasTarget()), Down: down, + DownSince: tr.At.Add(-90 * time.Second), Recovery: nil, Sinks: map[string]*sinkDelivery{}, + } + recAt := tr.At.Add(12 * time.Minute) + + implicit := goldenTransition() + implicit.Target = Target{ + Name: "nyc-router", Addr: "100.64.0.9", Site: "nyc", Severity: SeverityPage, Implicit: true, + Parent: "", InsecureTLS: false, Interval: 0, Timeout: 0, FailThreshold: 0, RecoverThreshold: 0, + FlapLimit: 0, FlapWindow: 0, Network: "", Checks: nil, StableID: "", NodeID: 0, + } + implicit.Layer = layerUnrouted + implicit.Dependents = []string{"nyc-nas", "nyc-cam"} + + flap := goldenTransition() + flap.Flapping = true + + warn := goldenTransition() + warn.Target.Severity = SeverityWarn + warn.Layer = layerLAN + + cases := map[string]Event{ + "target.down": down, + "target.down.implicit": newDownEvent("evt_01JAB3K9T6M4X2Q8R7N5W0YZCE", goldenCollector(), implicit), + "target.down.warn": newDownEvent("evt_01JAB3K9T6M4X2Q8R7N5W0YZCF", goldenCollector(), warn), + "target.recovered": newRecoveredEvent( + "evt_01JAB3K9T6M4X2Q8R7N5W0YZCG", + goldenCollector(), + inc, + recAt, + []string{"nyc-cam"}, + "", + ), + "target.recovered.removed": newRecoveredEvent( + "evt_01JAB3K9T6M4X2Q8R7N5W0YZCH", + goldenCollector(), + inc, + recAt, + nil, + reasonRemoved, + ), + "target.flapping": newFlappingEvent("evt_01JAB3K9T6M4X2Q8R7N5W0YZCJ", goldenCollector(), flap), + "collector.started": newStartedEvent("evt_01JAB3K9T6M4X2Q8R7N5W0YZCK", goldenCollector(), tr.At), + "test": newTestEvent( + "evt_01JAB3K9T6M4X2Q8R7N5W0YZCM", + CollectorInfo{Name: "monitor-nyc", TailscaleIP: ""}, + tr.At, + ), + } + for name, ev := range cases { + assertGolden(t, filepath.Join("testdata", "payload", name+".json"), marshalEvent(t, ev)) + } +} + +func TestPayloadContract(t *testing.T) { + t.Parallel() + tr := goldenTransition() + down := newDownEvent("evt_A", goldenCollector(), tr) + decode := func(ev Event) map[string]any { + var m map[string]any + if err := json.Unmarshal(marshalEvent(t, ev), &m); err != nil { + t.Fatal(err) + } + return m + } + m := decode(down) + if m["dependents"] == nil { + t.Fatal("down payload must carry a non-nil dependents array") + } + if _, ok := m["duration_seconds"]; ok { + t.Fatal("down payload must omit duration_seconds") + } + rec := newRecoveredEvent("evt_B", goldenCollector(), &incident{ + ID: "inc_test2", Status: incidentOpen, Target: targetDefOf(nasTarget()), Down: down, + DownSince: tr.At.Add(-90 * time.Second), Recovery: nil, Sinks: map[string]*sinkDelivery{}, + }, tr.At.Add(12*time.Minute), nil, "") + if _, ok := decode(rec)["layer"]; ok { + t.Fatal("recovered payload must omit layer") + } + if got := *rec.DurationSeconds; got != 12*60+90 { + t.Fatalf("duration_seconds = %d", got) + } + if rec.DownEventID != "evt_A" || rec.Severity != down.Severity { + t.Fatalf("recovered does not reference its down: %+v", rec) + } + started := newStartedEvent("evt_C", goldenCollector(), tr.At) + m = decode(started) + for _, k := range []string{"target", "layer", "dependents", "check"} { + if _, ok := m[k]; ok { + t.Fatalf("collector.started must omit %s", k) + } + } + if m["version"].(float64) != 1 { + t.Fatal("version must be 1") + } +} + +func TestMapTransition(t *testing.T) { + t.Parallel() + tests := []struct { + name string + from State + to State + flap bool + open bool + want mapKind + }{ + {"UNKNOWN to DOWN pages", StateUnknown, StateDown, false, false, mapDown}, + {"UP to DOWN", StateUp, StateDown, false, false, mapDown}, + {"SUPPRESSED to DOWN", StateSuppressed, StateDown, false, false, mapDown}, + {"DOWN again with open incident dedupes", StateSuppressed, StateDown, false, true, mapNone}, + {"DOWN to UP with open incident", StateDown, StateUp, false, true, mapRecovered}, + {"SUPPRESSED to UP with open incident", StateSuppressed, StateUp, false, true, mapRecovered}, + {"UNKNOWN to UP is silent", StateUnknown, StateUp, false, false, mapNone}, + {"UP without incident is silent", StateDown, StateUp, false, false, mapNone}, + {"to SUPPRESSED is silent", StateUp, StateSuppressed, false, false, mapNone}, + {"to SUPPRESSED with open incident is silent", StateDown, StateSuppressed, false, true, mapNone}, + {"flapping warns", StateUp, StateDown, true, false, mapFlapping}, + {"flapping warns with open incident", StateDown, StateUp, true, true, mapFlapping}, + } + for _, tt := range tests { + got := mapTransition(Transition{ + Sequence: 0, + Target: Target{ + Name: "", + Addr: "", + Site: "", + Severity: SeverityPage, + Parent: "", + InsecureTLS: false, + Interval: 0, + Timeout: 0, + FailThreshold: 0, + RecoverThreshold: 0, + FlapLimit: 0, + FlapWindow: 0, + Network: NetworkTailnet, + Checks: nil, + Implicit: false, + StableID: "", + NodeID: 0, + }, + From: tt.from, + To: tt.to, + Flapping: tt.flap, + Check: "", + Layer: "", + Err: nil, + Since: time.Time{}, + At: time.Time{}, + Failures: 0, + Path: pathInfo{Kind: "", Endpoint: "", DERP: "", Latency: 0}, + Dependents: nil, + DownSince: time.Time{}, + }, tt.open) + if got != tt.want { + t.Errorf("%s: got %v want %v", tt.name, got, tt.want) + } + } +} + +func TestDownLayers(t *testing.T) { + t.Parallel() + for _, l := range []layer{layerUnrouted, layerControl, layerTunnel, layerLAN, layerService} { + tr := goldenTransition() + tr.Layer = l + if got := newDownEvent("evt_A", goldenCollector(), tr).Layer; got != string(l) { + t.Fatalf("layer %q became %q", l, got) + } + } +} + +func TestSeverityRules(t *testing.T) { + t.Parallel() + tr := goldenTransition() + if newDownEvent("evt_A", goldenCollector(), tr).Severity != SeverityPage { + t.Fatal("default severity is page") + } + tr.Target.Severity = SeverityWarn + down := newDownEvent("evt_A", goldenCollector(), tr) + if down.Severity != SeverityWarn { + t.Fatal("warn target should produce warn down") + } + if newRecoveredEvent("evt_B", goldenCollector(), &incident{ + ID: "inc_test3", Status: incidentOpen, Target: targetDefOf(nasTarget()), Down: down, + DownSince: tr.At.Add(-90 * time.Second), Recovery: nil, Sinks: map[string]*sinkDelivery{}, + }, tr.At, nil, "").Severity != SeverityWarn { + t.Fatal("recovery takes the down's severity") + } + if newFlappingEvent("evt_C", goldenCollector(), tr).Severity != SeverityWarn { + t.Fatal("flapping is warn") + } + if newStartedEvent("evt_D", goldenCollector(), tr.At).Severity != SeverityInfo || + newTestEvent("evt_E", goldenCollector(), tr.At).Severity != SeverityInfo { + t.Fatal("started and test are info") + } +} + +func TestSinkAccepts(t *testing.T) { + t.Parallel() + tr := goldenTransition() + down := newDownEvent("evt_A", goldenCollector(), tr) + flap := newFlappingEvent("evt_B", goldenCollector(), tr) + started := newStartedEvent("evt_C", goldenCollector(), tr.At) + test := newTestEvent("evt_D", goldenCollector(), tr.At) + base := testSink("s") + with := func(mut func(*SinkConfig)) SinkConfig { + s := base + mut(&s) + return s + } + tests := []struct { + name string + sink SinkConfig + ev Event + tags []string + want bool + }{ + {"defaults accept all", base, down, nil, true}, + {"severity floor rejects warn", with(func(s *SinkConfig) { s.MinSeverity = SeverityPage }), flap, nil, false}, + {"severity floor accepts page", with(func(s *SinkConfig) { s.MinSeverity = SeverityPage }), down, nil, true}, + {"events list rejects", with(func(s *SinkConfig) { s.Events = []EventType{EventRecovered} }), down, nil, false}, + {"events list accepts", with(func(s *SinkConfig) { s.Events = []EventType{EventDown} }), down, nil, true}, + {"site rejects", with(func(s *SinkConfig) { s.Match.Sites = []string{"sfo", "lon"} }), down, nil, false}, + {"site accepts", with(func(s *SinkConfig) { s.Match.Sites = []string{"nyc", "sfo"} }), down, nil, true}, + {"target glob accepts", with(func(s *SinkConfig) { s.Match.Targets = []string{"nyc-*"} }), down, nil, true}, + {"target glob rejects", with(func(s *SinkConfig) { s.Match.Targets = []string{"sfo-*"} }), down, nil, false}, + { + "tag accepts", + with(func(s *SinkConfig) { s.Match.Tags = []string{"tag:nas"} }), + down, + []string{"tag:nas"}, + true, + }, + { + "tag rejects", + with(func(s *SinkConfig) { s.Match.Tags = []string{"tag:nas"} }), + down, + []string{"tag:other"}, + false, + }, + { + "untagged never matches tags", + with(func(s *SinkConfig) { s.Match.Tags = []string{"tag:nas"} }), + down, + nil, + false, + }, + {"every match key must hold", with(func(s *SinkConfig) { + s.Match.Sites = []string{"nyc"} + s.Match.Targets = []string{"sfo-*"} + }), down, nil, false}, + {"started ignores match", with(func(s *SinkConfig) { s.Match.Sites = []string{"sfo"} }), started, nil, true}, + { + "started still honours events", + with(func(s *SinkConfig) { s.Events = []EventType{EventDown} }), + started, + nil, + false, + }, + {"test bypasses everything", with(func(s *SinkConfig) { + s.MinSeverity = SeverityPage + s.Events = []EventType{EventDown} + s.Match.Sites = []string{"sfo"} + }), test, nil, true}, + } + for _, tt := range tests { + if got := tt.sink.accepts(tt.ev, tt.tags); got != tt.want { + t.Errorf("%s: got %v want %v", tt.name, got, tt.want) + } + } +} diff --git a/cmd/tail-watch/heartbeat.go b/cmd/tail-watch/heartbeat.go new file mode 100644 index 0000000..b110766 --- /dev/null +++ b/cmd/tail-watch/heartbeat.go @@ -0,0 +1,113 @@ +package main + +import ( + "context" + "log/slog" + "net/http" + "time" + + "github.com/prometheus/client_golang/prometheus" + "tailscale.com/tstime" +) + +const heartbeatTimeout = 10 * time.Second + +// heartbeat pings a dead-man's-switch URL every interval once the first round +// of checks is complete. A failed ping is counted and not retried early. +type heartbeat struct { + cfg HeartbeatConfig + clock tstime.Clock + log *slog.Logger + client *http.Client + first <-chan struct{} + failures prometheus.Counter +} + +func newHeartbeat( + cfg HeartbeatConfig, + clock tstime.Clock, + log *slog.Logger, + env netEnv, + first <-chan struct{}, + reg prometheus.Registerer, +) *heartbeat { + tr := &http.Transport{Proxy: http.ProxyFromEnvironment, DisableKeepAlives: true, ForceAttemptHTTP2: true} + if env != nil { + tr.DialContext = env.Dial + } + failures := prometheus.NewCounter(prometheus.CounterOpts{ + Namespace: metricsPrefix, Name: "heartbeat_failures_total", + Help: "Heartbeat pings that failed.", + }) + if reg != nil { + reg.MustRegister(failures) + } + return &heartbeat{ + cfg: cfg, clock: clock, log: log, first: first, failures: failures, + client: &http.Client{Transport: tr, Timeout: heartbeatTimeout}, + } +} + +// run blocks until ctx ends. It pings on every tick after first closes. +func (h *heartbeat) run(ctx context.Context) { + select { + case <-h.first: + case <-ctx.Done(): + return + } + tk, tick := h.clock.NewTicker(h.cfg.Interval) + defer tk.Stop() + for { + select { + case <-tick: + h.ping(ctx) + case <-ctx.Done(): + return + } + } +} + +func (h *heartbeat) ping(ctx context.Context) { + url, err := h.cfg.URL.resolve() + if err != nil { + h.fail("heartbeat url unavailable", err) + return + } + req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil) + if err != nil { + h.fail("heartbeat request", errBadURL(url)) + return + } + req.Header.Set("User-Agent", "tail-watch/"+versionString()) + resp, err := h.client.Do(req) + if err != nil { + if ctx.Err() != nil { + return + } + h.fail("heartbeat ping failed", classifyError(err, url).Err) + return + } + defer resp.Body.Close() + if resp.StatusCode < 200 || resp.StatusCode > 299 { + h.failures.Inc() + h.log.WarnContext( + context.Background(), + "heartbeat ping rejected", + "url", + redactURL(url), + "status", + resp.StatusCode, + ) + } +} + +func (h *heartbeat) fail(msg string, err error) { + h.failures.Inc() + h.log.Warn(msg, "err", err) +} + +type badURLError string + +func (e badURLError) Error() string { return "invalid URL " + string(e) } + +func errBadURL(raw string) error { return badURLError(redactURL(raw)) } diff --git a/cmd/tail-watch/heartbeat_test.go b/cmd/tail-watch/heartbeat_test.go new file mode 100644 index 0000000..91c38c6 --- /dev/null +++ b/cmd/tail-watch/heartbeat_test.go @@ -0,0 +1,95 @@ +package main + +import ( + "context" + "net/http" + "net/http/httptest" + "sync/atomic" + "testing" + "time" + + "github.com/prometheus/client_golang/prometheus" + "github.com/prometheus/client_golang/prometheus/testutil" +) + +func startHeartbeat(t *testing.T, url string, first chan struct{}) (*heartbeat, *countingClock) { + t.Helper() + clk := newCountingClock() + hb := newHeartbeat( + HeartbeatConfig{URL: urlFile(t, url), Interval: time.Minute}, + clk, discardLog(), nil, first, prometheus.NewRegistry(), + ) + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan struct{}) + go func() { hb.run(ctx); close(done) }() + t.Cleanup(func() { cancel(); <-done }) + return hb, clk +} + +func TestHeartbeatPingsEveryIntervalAfterFirstRound(t *testing.T) { + t.Parallel() + var hits atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodGet { + t.Errorf("method %s", r.Method) + } + hits.Add(1) + })) + defer srv.Close() + first := make(chan struct{}) + _, clk := startHeartbeat(t, srv.URL+"/ping/secret", first) + time.Sleep(30 * time.Millisecond) + if clk.tickers.Load() != 0 { + t.Fatal("heartbeat must wait for the first round") + } + close(first) + eventually(t, func() bool { return clk.tickers.Load() == 1 }) + for i := 1; i <= 5; i++ { + clk.Advance(time.Minute) + want := int32(i) + eventually(t, func() bool { return hits.Load() == want }) + } +} + +func TestHeartbeatFailureCountedAndNotRetriedEarly(t *testing.T) { + t.Parallel() + var hits atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + hits.Add(1) + w.WriteHeader(http.StatusInternalServerError) + })) + defer srv.Close() + first := make(chan struct{}) + close(first) + hb, clk := startHeartbeat(t, srv.URL, first) + eventually(t, func() bool { return clk.tickers.Load() == 1 }) + clk.Advance(time.Minute) + eventually(t, func() bool { return testutil.ToFloat64(hb.failures) == 1 }) + time.Sleep(30 * time.Millisecond) + if hits.Load() != 1 { + t.Fatalf("failed ping was retried early: %d requests", hits.Load()) + } +} + +func TestHeartbeatStopsOnShutdown(t *testing.T) { + t.Parallel() + var hits atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) { hits.Add(1) })) + defer srv.Close() + clk := newCountingClock() + first := make(chan struct{}) + close(first) + hb := newHeartbeat(HeartbeatConfig{URL: urlFile(t, srv.URL), Interval: time.Minute}, + clk, discardLog(), nil, first, prometheus.NewRegistry()) + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan struct{}) + go func() { hb.run(ctx); close(done) }() + eventually(t, func() bool { return clk.tickers.Load() == 1 }) + cancel() + <-done + clk.Advance(time.Hour) + time.Sleep(30 * time.Millisecond) + if hits.Load() != 0 { + t.Fatal("heartbeat pinged after shutdown") + } +} diff --git a/cmd/tail-watch/incidents.go b/cmd/tail-watch/incidents.go new file mode 100644 index 0000000..bd507c4 --- /dev/null +++ b/cmd/tail-watch/incidents.go @@ -0,0 +1,488 @@ +package main + +import ( + "log/slog" + "slices" + "strings" + "time" + + "tailscale.com/tailcfg" +) + +// phase is where one event stands for one sink. +type phase string + +const ( + phaseNone phase = "" + phasePending phase = "pending" + phaseAttempted phase = "attempted" + phaseAcked phase = "acknowledged" + phaseDropped phase = "dropped" +) + +func (p phase) terminal() bool { + return p == phaseNone || p == phaseAcked || p == phaseDropped +} + +// sinkDelivery is one sink's delivery state for one incident. The *Tried flags +// remember that a request was made even after the phase became dropped: the +// receiver may have accepted a request whose response was lost. +type sinkDelivery struct { + Down phase `json:"down"` + DownTried bool `json:"down_tried,omitempty"` + Rec phase `json:"recovery,omitempty"` + RecTried bool `json:"recovery_tried,omitempty"` +} + +type incidentStatus string + +const ( + incidentOpen incidentStatus = "open" + incidentClosing incidentStatus = "closing" +) + +// targetDef is the persisted identity of an incident's target. It carries +// enough to recreate a learned router before checks are scheduled. +type targetDef struct { + Name string `json:"name"` + Addr string `json:"addr"` + Site string `json:"site,omitempty"` + Parent string `json:"parent,omitempty"` + Severity Severity `json:"severity"` + Implicit bool `json:"implicit,omitempty"` + StableID string `json:"stable_id,omitempty"` + NodeID int64 `json:"node_id,omitempty"` +} + +func targetDefOf(t Target) targetDef { + return targetDef{ + Name: t.Name, Addr: t.Addr, Site: t.Site, Parent: t.Parent, Severity: t.Severity, + Implicit: t.Implicit, StableID: string(t.StableID), NodeID: int64(t.NodeID), + } +} + +// implicitTarget rebuilds a learned reachability target from its record. +func (d targetDef) implicitTarget() *Target { + base := builtinTiming() + return &Target{ + Name: d.Name, Addr: d.Addr, Site: d.Site, Parent: d.Parent, Severity: d.Severity, + InsecureTLS: false, Interval: base.Interval, Timeout: base.Timeout, + FailThreshold: base.FailThreshold, RecoverThreshold: base.RecoverThreshold, + FlapLimit: base.FlapLimit, FlapWindow: base.FlapWindow, Network: NetworkTailnet, + Checks: []check{tspingCheck{resolver: nil}}, + Implicit: true, StableID: tailcfg.StableNodeID(d.StableID), NodeID: tailcfg.NodeID(d.NodeID), + } +} + +// incident is one outage with its delivery bookkeeping. It lives until every +// sink's intents are terminal. +type incident struct { + ID string `json:"id"` + Status incidentStatus `json:"status"` + Target targetDef `json:"target"` + Down Event `json:"down"` + DownSince time.Time `json:"down_since"` + Recovery *Event `json:"recovery,omitempty"` + Sinks map[string]*sinkDelivery `json:"sinks"` +} + +// queued is one delivery intent for one sink. +type queued struct { + Seq uint64 `json:"seq"` + IncidentID string `json:"incident_id,omitempty"` + Event Event `json:"event"` + Attempted bool `json:"attempted,omitempty"` + UnitID string `json:"unit_id,omitempty"` +} + +// unit is one request: a single event, a webhook batch or a Slack group. +// Body is fixed at the first attempt and reused byte for byte. +type unit struct { + ID string `json:"id"` + Sink string `json:"sink"` + Members []uint64 `json:"members"` + Body []byte `json:"body"` +} + +// unitKey identifies a unit. A request id is only unique within one sink. +func unitKey(sink, id string) string { return sink + "/" + id } + +func (u *unit) key() string { return unitKey(u.Sink, u.ID) } + +// book is the notifier's authoritative in-memory state. It is guarded by the +// notifier mutex; a snapshot of it is what queue.go persists. +type book struct { + log *slog.Logger + m *notifyMetrics + sinks map[string]SinkConfig + incidents map[string]*incident + open map[string]string + queues map[string]*memQueue + units map[string]*unit + nextSeq uint64 +} + +func newBook(log *slog.Logger, m *notifyMetrics, sinks []SinkConfig) *book { + b := &book{ + log: log, m: m, + sinks: map[string]SinkConfig{}, + incidents: map[string]*incident{}, + open: map[string]string{}, + queues: map[string]*memQueue{}, + units: map[string]*unit{}, + nextSeq: 1, + } + for _, s := range sinks { + b.sinks[s.Name] = s + b.queues[s.Name] = newMemQueue(s.QueueSize) + } + return b +} + +func (b *book) seq() uint64 { + s := b.nextSeq + b.nextSeq++ + return s +} + +func (b *book) openIncidentFor(target string) *incident { + if id, ok := b.open[target]; ok { + return b.incidents[id] + } + return nil +} + +func (b *book) delivery(it *queued, sink string) (*incident, *sinkDelivery) { + if it.IncidentID == "" { + return nil, nil + } + inc := b.incidents[it.IncidentID] + if inc == nil { + return nil, nil + } + return inc, inc.Sinks[sink] +} + +func (b *book) refreshQueueLen(sink string) { + if q := b.queues[sink]; q != nil { + b.m.queueLen(sink, q.Len()) + } +} + +// push queues it for sink, dropping the oldest unattempted event when the +// queue is full. An attempted event is in flight and is never a victim. +func (b *book) push(sink string, it *queued) { + q := b.queues[sink] + if q == nil { + return + } + for q.Len() >= q.limit { + victim := q.oldestPending() + if victim == nil { + b.dropItem(sink, it, reasonQueueFull) + return + } + q.Remove(victim.Seq) + b.dropItem(sink, victim, reasonQueueFull) + } + if it.Event.Type == EventRecovered { + if _, d := b.delivery(it, sink); d == nil || d.Rec != phasePending { + return + } + } + q.push(it) + b.refreshQueueLen(sink) +} + +// openIncident records a new outage and queues its down event for sinks. +func (b *book) openIncident(ev Event, def targetDef, downSince time.Time, sinks []string) *incident { + inc := &incident{ + ID: ev.ID, Status: incidentOpen, Target: def, Down: ev, DownSince: downSince, + Recovery: nil, Sinks: map[string]*sinkDelivery{}, + } + b.incidents[inc.ID] = inc + b.open[def.Name] = inc.ID + for _, name := range sinks { + inc.Sinks[name] = &sinkDelivery{Down: phasePending, DownTried: false, Rec: phaseNone, RecTried: false} + } + for _, name := range sinks { + b.push(name, &queued{Seq: b.seq(), IncidentID: inc.ID, Event: ev, Attempted: false, UnitID: ""}) + } + return inc +} + +// closeIncident moves inc to closing and queues rec for every sink whose down +// state still makes a recovery meaningful. +func (b *book) closeIncident(inc *incident, rec Event) { + inc.Status = incidentClosing + inc.Recovery = &rec + delete(b.open, inc.Target.Name) + for _, name := range slices.Sorted(mapsKeys(inc.Sinks)) { + d := inc.Sinks[name] + if _, ok := b.sinks[name]; !ok { + b.log.Warn("sink removed; skipping recovery", "sink", name, "incident", inc.ID) + d.Rec = phaseDropped + continue + } + if d.Down == phaseDropped && !d.DownTried { + d.Rec = phaseDropped + continue + } + d.Rec = phasePending + b.push(name, &queued{Seq: b.seq(), IncidentID: inc.ID, Event: rec, Attempted: false, UnitID: ""}) + } + b.gc(inc) +} + +func mapsKeys[V any](m map[string]V) func(yield func(string) bool) { + return func(yield func(string) bool) { + for k := range m { + if !yield(k) { + return + } + } + } +} + +// gc forgets a closing incident once every intent is terminal. +func (b *book) gc(inc *incident) { + if inc == nil || inc.Status != incidentClosing { + return + } + for _, d := range inc.Sinks { + if !d.Down.terminal() || !d.Rec.terminal() { + return + } + } + delete(b.incidents, inc.ID) +} + +// beginUnit marks items attempted, together, and records the first-attempt body. +func (b *book) beginUnit(sink, id string, items []*queued, body []byte) *unit { + u := &unit{ID: id, Sink: sink, Members: make([]uint64, 0, len(items)), Body: body} + for _, it := range items { + it.Attempted = true + it.UnitID = id + u.Members = append(u.Members, it.Seq) + _, d := b.delivery(it, sink) + if d == nil { + continue + } + switch it.Event.Type { + case EventFlapping, EventDegraded, EventStarted, EventTest: + // carries no incident delivery state + case EventDown: + d.Down, d.DownTried = phaseAttempted, true + case EventRecovered: + d.Rec, d.RecTried = phaseAttempted, true + } + } + b.units[u.key()] = u + return u +} + +func (b *book) unitItems(u *unit) []*queued { + q := b.queues[u.Sink] + if q == nil { + return nil + } + items := make([]*queued, 0, len(u.Members)) + for _, seq := range u.Members { + if it := q.Get(seq); it != nil { + items = append(items, it) + } + } + return items +} + +// ackUnit records a 2xx for every member of u. +func (b *book) ackUnit(u *unit, now time.Time) { + items := b.unitItems(u) + q := b.queues[u.Sink] + for _, it := range items { + q.Remove(it.Seq) + b.m.sent(u.Sink, it.Event.Type, now.Sub(it.Event.Time.Time())) + inc, d := b.delivery(it, u.Sink) + if d == nil { + continue + } + switch it.Event.Type { + case EventFlapping, EventDegraded, EventStarted, EventTest: + // carries no incident delivery state + case EventDown: + d.Down = phaseAcked + case EventRecovered: + d.Rec = phaseAcked + } + b.gc(inc) + } + delete(b.units, u.key()) + b.refreshQueueLen(u.Sink) +} + +// dropUnit drops every member of u, counting each event. +func (b *book) dropUnit(u *unit, reason string) { + items := b.unitItems(u) + q := b.queues[u.Sink] + for _, it := range items { + q.Remove(it.Seq) + } + for _, it := range items { + b.dropItem(u.Sink, it, reason) + } + delete(b.units, u.key()) + b.refreshQueueLen(u.Sink) +} + +// dropItem counts a lost event and updates the incident's delivery state. A +// never-attempted down takes its unsent recovery with it; an attempted one +// keeps the recovery eligible. +func (b *book) dropItem(sink string, it *queued, reason string) { + b.m.dropped(sink, reason) + b.log.Warn( + "notification dropped", + "sink", + sink, + "event", + it.Event.ID, + "type", + string(it.Event.Type), + "reason", + reason, + ) + inc, d := b.delivery(it, sink) + if d == nil { + return + } + switch it.Event.Type { + case EventFlapping, EventDegraded, EventStarted, EventTest: + // carries no incident delivery state + case EventDown: + d.Down = phaseDropped + if !d.DownTried && d.Rec == phasePending { + if q := b.queues[sink]; q != nil { + if rec := q.findRecovery(inc.ID); rec != nil { + q.Remove(rec.Seq) + } + } + d.Rec = phaseDropped + } + case EventRecovered: + d.Rec = phaseDropped + } + b.gc(inc) +} + +// expireStale drops unattempted events older than the sink's max_age. +func (b *book) expireStale(sink string, now time.Time) bool { + q := b.queues[sink] + cfg, ok := b.sinks[sink] + if q == nil || !ok { + return false + } + changed := false + for _, it := range slices.Clone(q.items) { + if it.Attempted || now.Sub(it.Event.Time.Time()) <= cfg.MaxAge { + continue + } + q.Remove(it.Seq) + b.dropItem(sink, it, reasonExpired) + changed = true + } + b.refreshQueueLen(sink) + return changed +} + +func (b *book) pendingItems(sink string) []*queued { + q := b.queues[sink] + if q == nil { + return nil + } + var out []*queued + for _, it := range q.items { + if !it.Attempted { + out = append(out, it) + } + } + return out +} + +// sinkUnits returns sink's in-flight units, oldest first. +func (b *book) sinkUnits(sink string) []*unit { + var out []*unit + for _, u := range b.units { + if u.Sink == sink { + out = append(out, u) + } + } + slices.SortFunc(out, func(x, y *unit) int { + switch { + case len(x.Members) == 0 || len(y.Members) == 0: + return 0 + case x.Members[0] < y.Members[0]: + return -1 + case x.Members[0] > y.Members[0]: + return 1 + default: + return 0 + } + }) + return out +} + +// unitExpired reports whether any member of u is older than the sink's max_age. +func (b *book) unitExpired(u *unit, now time.Time) bool { + cfg, ok := b.sinks[u.Sink] + if !ok { + return false + } + return slices.ContainsFunc(b.unitItems(u), func(it *queued) bool { + return now.Sub(it.Event.Time.Time()) > cfg.MaxAge + }) +} + +func (b *book) openCount() int { + return len(b.open) +} + +// incidentRestorer seeds a target DOWN without a second page. +type incidentRestorer interface { + Restore(name string, since time.Time) error +} + +// implicitRestorer recreates a learned reachability target by node identity. +type implicitRestorer interface { + RestoreImplicit(t *Target) +} + +// restoreOpen re-establishes every open incident before any check is scheduled: +// configured targets are seeded DOWN, learned routers are recreated from their +// recorded identity and seeded, and a configured target that no longer exists +// closes as removed. A restore refusal is logged and the incident retained; a +// learned router that cannot be rediscovered stays open until authoritative +// topology reconciliation removes it. +func (n *notifier) restoreOpen(configured map[string]bool, alerts incidentRestorer, implicit implicitRestorer) { + open := n.openIncidents() + slices.SortFunc(open, func(a, b incident) int { return strings.Compare(a.Target.Name, b.Target.Name) }) + for _, inc := range open { + name := inc.Target.Name + switch { + case inc.Target.Implicit: + if implicit == nil { + continue + } + if err := alerts.Restore(name, inc.DownSince); err != nil { + n.log.Error("cannot restore open incident; keeping it", "target", name, "incident", inc.ID, "err", err) + continue + } + implicit.RestoreImplicit(inc.Target.implicitTarget()) + case configured[name]: + if err := alerts.Restore(name, inc.DownSince); err != nil { + n.log.Error("cannot restore open incident; keeping it", "target", name, "incident", inc.ID, "err", err) + } + default: + n.log.Info("configured target removed while an incident was open", "target", name, "incident", inc.ID) + n.closeRemoved(name) + } + } +} diff --git a/cmd/tail-watch/live_notify_test.go b/cmd/tail-watch/live_notify_test.go new file mode 100644 index 0000000..7bd76dc --- /dev/null +++ b/cmd/tail-watch/live_notify_test.go @@ -0,0 +1,493 @@ +package main + +import ( + "crypto/rand" + "encoding/hex" + "encoding/json" + "fmt" + "io" + "net" + "net/http" + "os" + "os/exec" + "path/filepath" + "slices" + "strconv" + "strings" + "sync" + "testing" + "time" +) + +const ( + liveNotifyWait = 2 * time.Minute + liveLANPrefix = "10.10.53.0/24" + liveLANAddr = "10.10.53.1" + liveSlackEnvVar = "SLACK_WEBHOOK_URL" +) + +// TestLiveNotifications runs the whole notification path on a real tailnet. +// It brings up the collector plus one fixture node per check (icmp, tsping, +// tcp, http, and a LAN target behind a subnet router), fails and heals each +// one, and asserts on what a signature-verifying webhook receiver in this +// process saw. It is off unless TAIL_WATCH_LIVE_TAILNET=1. +// +// Credentials and cleanup work as in TestLiveTailnetDocker. Slack is added +// when SLACK_WEBHOOK_URL is set, or when TAIL_WATCH_SLACK_PULUMI_ENV names a +// pulumi env with environmentVariables.SLACK_WEBHOOK_URL. Slack is one-way, so +// the test checks tail-watch's own sent and failed counters for that sink; +// look in the channel for the messages. TAIL_WATCH_LIVE_MENTION sets +// mention_on_page for that sink. +func TestLiveNotifications(t *testing.T) { + t.Parallel() + if os.Getenv("TAIL_WATCH_LIVE_TAILNET") == "" { + t.Skip("set TAIL_WATCH_LIVE_TAILNET=1 to run the Docker tailnet test") + } + requireLiveTools(t) + api := liveCreds(t) + runID := liveHostPrefix + "-notify-" + strconv.FormatInt(time.Now().UnixNano()%1_000_000, 10) + t.Cleanup(func() { api.deleteRun(t, runID) }) + + slack := liveSlackURL(t) + rcv := newLiveReceiver(t) + stack := newLiveStack(t, runID, api.authKey(t, liveTags())) + stack.files = append(stack.files, filepath.Join(packageDir(t), "testdata", "compose.live-notify.yaml")) + stack.env["TS_FX_PREFIX"] = runID + fixtures := map[string]string{ + "fx-icmp": "TS_KEY_ICMP", "fx-tsping": "TS_KEY_TSPING", "fx-tcp": "TS_KEY_TCP", + "fx-http": "TS_KEY_HTTP", "lan-router": "TS_KEY_LAN", + } + for service, keyVar := range fixtures { + stack.env[keyVar] = api.authKey(t, nil) + stack.services = append(stack.services, service) + } + stack.env["OPS_WEBHOOK_URL"] = rcv.hookURL + stack.env["OPS_WEBHOOK_SECRET"] = rcv.secret + stack.env["HEARTBEAT_URL"] = rcv.beatURL + if slack != "" { + stack.env[liveSlackEnvVar] = slack + } + + stack.start(t) + enableLiveRoute(t, api, runID+"-lan", liveLANPrefix) + requireKernelMode(t, stack) + waitLiveRoute(t, stack, liveLANAddr) + ips := map[string]string{} + for _, service := range []string{"fx-icmp", "fx-tsping", "fx-tcp", "fx-http"} { + ips[service] = liveFixtureIP(t, stack, service) + } + stack.env["TAIL_WATCH_CONFIG_FILE"] = writeNotifyConfig(t, ips, slack != "") + stack.output(t, "up", "--detach", "--no-deps", "tail-watch") + + rcv.waitFor(t, "collector.started", func(e Event) bool { return e.Type == EventStarted }) + rcv.waitBeats(t, 2) + + scenarios := []liveScenario{ + { + target: "fx-icmp", check: "icmp", + fail: func() { + stack.output( + t, + "exec", + "--no-TTY", + "fx-icmp", + "iptables", + "-I", + "INPUT", + "1", + "-p", + "icmp", + "-j", + "DROP", + ) + }, + heal: func() { + stack.output(t, "exec", "--no-TTY", "fx-icmp", "iptables", "-D", "INPUT", "-p", "icmp", "-j", "DROP") + }, + }, + { + target: "fx-tsping", check: "tsping", + fail: func() { stack.output(t, "pause", "fx-tsping") }, + heal: func() { stack.output(t, "unpause", "fx-tsping") }, + }, + { + target: "fx-tcp", + check: "tcp:9000", + fail: func() { stack.output(t, "exec", "--no-TTY", "fx-tcp", "sh", "-c", "touch /tmp/down; pkill nc || true") }, + heal: func() { stack.output(t, "exec", "--no-TTY", "fx-tcp", "rm", "-f", "/tmp/down") }, + }, + { + target: "fx-http", check: "http:", + fail: func() { + stack.output(t, "exec", "--no-TTY", "fx-http", "sh", "-c", "touch /tmp/down; pkill nc || true") + }, + heal: func() { stack.output(t, "exec", "--no-TTY", "fx-http", "rm", "-f", "/tmp/down") }, + }, + { + target: "fx-lan", check: "icmp", + fail: func() { + stack.output( + t, + "exec", + "--no-TTY", + "lan-router", + "iptables", + "-I", + "INPUT", + "1", + "-p", + "icmp", + "-j", + "DROP", + ) + }, + heal: func() { + stack.output(t, "exec", "--no-TTY", "lan-router", "iptables", "-D", "INPUT", "-p", "icmp", "-j", "DROP") + }, + }, + } + for _, sc := range scenarios { + down, rec := sc.run(t, rcv, runID) + t.Logf("%s: down layer=%q check=%q error=%q; recovered after %ds", + sc.target, down.Layer, down.Check, down.Error, *rec.DurationSeconds) + } + + restartMidOutage(t, stack, rcv, scenarios[2]) + + out := stack.output(t, "run", "--rm", "--no-deps", "--no-TTY", "tail-watch", "notify-test") + t.Logf("notify-test:\n%s", strings.TrimSpace(redactSecrets(string(out)))) + if !strings.Contains(string(out), "ops-webhook webhook 200") { + t.Fatalf("notify-test did not report the webhook sink as 200:\n%s", out) + } + if slack != "" && !strings.Contains(string(out), "ops-slack slack 200") { + t.Fatalf("notify-test did not report the Slack sink as 200:\n%s", out) + } + + metrics := string(stack.output(t, "exec", "--no-TTY", "tailscale", "wget", "-qO-", "http://127.0.0.1:9109/metrics")) + assertSinkCounters(t, metrics, "ops-webhook") + if slack != "" { + assertSinkCounters(t, metrics, "ops-slack") + } + if bad := rcv.badSignatures(); bad != 0 { + t.Fatalf("%d webhook requests failed signature verification", bad) + } + t.Logf("verified %d webhook events, %d heartbeats; slack=%v", rcv.eventCount(), rcv.beatCount(), slack != "") +} + +// liveScenario fails one fixture, expects exactly one down for its target, heals +// it, and expects the matching recovery. +type liveScenario struct { + target string + check string + fail func() + heal func() +} + +func (sc liveScenario) run(t *testing.T, rcv *liveReceiver, runID string) (Event, Event) { + t.Helper() + sc.fail() + down := rcv.waitFor(t, sc.target+" down", func(e Event) bool { + return e.Type == EventDown && e.targetLabel() == sc.target && !rcv.hasRecovery(e.ID) + }) + if !strings.HasPrefix(down.Check, sc.check) || down.Severity != SeverityPage || down.Dependents == nil || + down.Collector.Name != runID || down.Collector.TailscaleIP == "" { + t.Fatalf("%s: unexpected down payload %+v", sc.target, down) + } + sc.heal() + rec := rcv.waitFor(t, sc.target+" recovered", func(e Event) bool { + return e.Type == EventRecovered && e.DownEventID == down.ID + }) + if rec.DurationSeconds == nil || rec.Layer != "" { + t.Fatalf("%s: unexpected recovery payload %+v", sc.target, rec) + } + if n := rcv.countDowns(sc.target); n != 1 { + t.Fatalf("%s: %d down events, want 1", sc.target, n) + } + return down, rec +} + +func (e Event) targetLabel() string { + if e.Target == nil { + return "" + } + return e.Target.Name +} + +// restartMidOutage checks that a restart during an outage neither pages again +// nor loses the recovery: the disk queue and open incident survive. +func restartMidOutage(t *testing.T, stack *liveStack, rcv *liveReceiver, sc liveScenario) { + t.Helper() + before := rcv.countDowns(sc.target) + sc.fail() + down := rcv.waitFor(t, "restart scenario down", func(e Event) bool { + return e.Type == EventDown && e.targetLabel() == sc.target && !rcv.hasRecovery(e.ID) + }) + starts := rcv.countType(EventStarted) + stack.output(t, "restart", "tail-watch") + rcv.waitFor(t, "collector.started after restart", func(Event) bool { return rcv.countType(EventStarted) > starts }) + time.Sleep(20 * time.Second) + if n := rcv.countDowns(sc.target); n != before+1 { + t.Fatalf("restart paged again: %d downs for %s, want %d", n, sc.target, before+1) + } + sc.heal() + rcv.waitFor(t, "recovery after restart", func(e Event) bool { + return e.Type == EventRecovered && e.DownEventID == down.ID + }) +} + +func assertSinkCounters(t *testing.T, metrics, sink string) { + t.Helper() + sent, unexpected := 0, 0 + for line := range strings.SplitSeq(metrics, "\n") { + if !strings.Contains(line, `sink="`+sink+`"`) { + continue + } + switch { + case strings.HasPrefix(line, "tailwatch_notify_sent_total{"): + sent += metricValue(line) + case strings.HasPrefix(line, "tailwatch_notify_failed_total{"), + strings.HasPrefix(line, "tailwatch_notify_dropped_total{"): + t.Logf("sink %s: %s", sink, line) + // A memory queue loses events still waiting when the collector restarts. + if metricValue(line) > 0 && !strings.Contains(line, `reason="restart"`) { + unexpected += metricValue(line) + } + } + } + t.Logf("sink %s: sent=%d unexpected failures or drops=%d", sink, sent, unexpected) + if sent == 0 || unexpected != 0 { + t.Fatalf("sink %s: sent=%d unexpected=%d", sink, sent, unexpected) + } +} + +func metricValue(line string) int { + f := strings.Fields(line) + v, _ := strconv.ParseFloat(f[len(f)-1], 64) + return int(v) +} + +func liveFixtureIP(t *testing.T, stack *liveStack, service string) string { + t.Helper() + var ip string + ok := pollUntil(60*time.Second, func() bool { + out, err := stack.command(t.Context(), "exec", "--no-TTY", service, "tailscale", "ip", "-4").Output() + ip = strings.TrimSpace(string(out)) + return err == nil && ip != "" + }) + if !ok { + t.Fatalf("%s has no tailnet address", service) + } + return firstIPv4(strings.Fields(ip)) +} + +func writeNotifyConfig(t *testing.T, ips map[string]string, withSlack bool) string { + t.Helper() + var b strings.Builder + b.WriteString(`collector: + listen: 127.0.0.1:9109 + state_dir: /var/lib/tail-watch +defaults: + interval: 5s + timeout: 2s + fail_threshold: 2 + recover_threshold: 2 +targets: +`) + fmt.Fprintf(&b, " - {name: fx-icmp, addr: %s, checks: [icmp]}\n", ips["fx-icmp"]) + fmt.Fprintf(&b, " - {name: fx-tsping, addr: %s, checks: [tsping]}\n", ips["fx-tsping"]) + fmt.Fprintf(&b, " - {name: fx-tcp, addr: %s, checks: [\"tcp:9000\"]}\n", ips["fx-tcp"]) + fmt.Fprintf(&b, " - {name: fx-http, addr: %s, checks: [\"http:http://%s:8080/|tail-watch-ok\"]}\n", + ips["fx-http"], ips["fx-http"]) + fmt.Fprintf(&b, " - {name: fx-lan, addr: %s, checks: [icmp]}\n", liveLANAddr) + b.WriteString(`sinks: + - name: ops-webhook + type: webhook + url_env: OPS_WEBHOOK_URL + secret_env: OPS_WEBHOOK_SECRET + queue: disk + group_wait: 2s +`) + if withSlack { + b.WriteString(" - name: ops-slack\n type: slack\n url_env: SLACK_WEBHOOK_URL\n group_wait: 2s\n") + if mention := os.Getenv("TAIL_WATCH_LIVE_MENTION"); mention != "" { + fmt.Fprintf(&b, " mention_on_page: %q\n", mention) + } + } + b.WriteString("heartbeat:\n url_env: HEARTBEAT_URL\n interval: 5s\n") + path := filepath.Join(t.TempDir(), "tail-watch.yaml") + if err := os.WriteFile(path, []byte(b.String()), 0o644); err != nil { + t.Fatal(err) + } + return path +} + +// liveSlackURL returns the Slack webhook URL from the environment or a pulumi +// env, or "" to run without Slack. It is never logged. +func liveSlackURL(t *testing.T) string { + t.Helper() + if v := os.Getenv(liveSlackEnvVar); v != "" { + return v + } + name := os.Getenv("TAIL_WATCH_SLACK_PULUMI_ENV") + if name == "" { + t.Log("no Slack webhook configured; running the webhook sink only") + return "" + } + out, err := exec.CommandContext(t.Context(), "pulumi", "env", "open", name, "--format", "json").Output() + if err != nil { + t.Fatalf("pulumi env open %s: %v", name, err) + } + var env struct { + Vars map[string]string `json:"environmentVariables"` + } + if uerr := json.Unmarshal(out, &env); uerr != nil || env.Vars[liveSlackEnvVar] == "" { + t.Fatalf("%s has no environmentVariables.%s", name, liveSlackEnvVar) + } + return env.Vars[liveSlackEnvVar] +} + +// liveReceiver is the webhook and heartbeat receiver the collector posts to. It +// runs in the test process and verifies every signature. +type liveReceiver struct { + hookURL, beatURL, secret string + + mu sync.Mutex + events []Event + beats int + bad int +} + +func newLiveReceiver(t *testing.T) *liveReceiver { + t.Helper() + raw := make([]byte, 16) + if _, err := rand.Read(raw); err != nil { + t.Fatal(err) + } + r := &liveReceiver{ + hookURL: "", beatURL: "", secret: hex.EncodeToString(raw), + mu: sync.Mutex{}, events: nil, beats: 0, bad: 0, + } + mux := http.NewServeMux() + mux.HandleFunc("/hook", r.handleHook) + mux.HandleFunc("/beat", func(w http.ResponseWriter, _ *http.Request) { + r.mu.Lock() + r.beats++ + r.mu.Unlock() + w.WriteHeader(http.StatusOK) + }) + ln, err := net.Listen("tcp", "0.0.0.0:0") + if err != nil { + t.Fatal(err) + } + srv := &http.Server{Handler: mux, ReadHeaderTimeout: 5 * time.Second} + go func() { _ = srv.Serve(ln) }() + t.Cleanup(func() { _ = srv.Close() }) + port := ln.Addr().(*net.TCPAddr).Port + r.hookURL = fmt.Sprintf("http://host.docker.internal:%d/hook", port) + r.beatURL = fmt.Sprintf("http://host.docker.internal:%d/beat", port) + return r +} + +func (r *liveReceiver) handleHook(w http.ResponseWriter, req *http.Request) { + body, err := io.ReadAll(req.Body) + if err != nil { + w.WriteHeader(http.StatusBadRequest) + return + } + r.mu.Lock() + defer r.mu.Unlock() + if !verifySignature(r.secret, req.Header.Get(signatureHeader), body, time.Now(), 5*time.Minute) { + r.bad++ + w.WriteHeader(http.StatusUnauthorized) + return + } + var ev Event + if jerr := json.Unmarshal(body, &ev); jerr != nil { + w.WriteHeader(http.StatusBadRequest) + return + } + r.events = append(r.events, ev) + w.WriteHeader(http.StatusOK) +} + +func (r *liveReceiver) snapshot() []Event { + r.mu.Lock() + defer r.mu.Unlock() + return slices.Clone(r.events) +} + +func (r *liveReceiver) waitFor(t *testing.T, what string, match func(Event) bool) Event { + t.Helper() + var found Event + if !pollUntil(liveNotifyWait, func() bool { + for _, e := range r.snapshot() { + if match(e) { + found = e + return true + } + } + return false + }) { + t.Fatalf("timed out waiting for %s; saw %s", what, r.summary()) + } + return found +} + +func (r *liveReceiver) waitBeats(t *testing.T, n int) { + t.Helper() + if !pollUntil(liveNotifyWait, func() bool { return r.beatCount() >= n }) { + t.Fatalf("saw %d heartbeats, want %d", r.beatCount(), n) + } +} + +func (r *liveReceiver) summary() string { + var parts []string + for _, e := range r.snapshot() { + part := string(e.Type) + ":" + e.targetLabel() + if e.Type == EventDown { + part += fmt.Sprintf("[layer=%s error=%q]", e.Layer, truncate(e.Error, 90)) + } + parts = append(parts, part) + } + return strings.Join(parts, " ") +} + +func (r *liveReceiver) hasRecovery(downID string) bool { + return slices.ContainsFunc(r.snapshot(), func(e Event) bool { + return e.Type == EventRecovered && e.DownEventID == downID + }) +} + +func (r *liveReceiver) countDowns(target string) int { + n := 0 + for _, e := range r.snapshot() { + if e.Type == EventDown && e.targetLabel() == target { + n++ + } + } + return n +} + +func (r *liveReceiver) countType(typ EventType) int { + n := 0 + for _, e := range r.snapshot() { + if e.Type == typ { + n++ + } + } + return n +} + +func (r *liveReceiver) eventCount() int { return len(r.snapshot()) } + +func (r *liveReceiver) beatCount() int { + r.mu.Lock() + defer r.mu.Unlock() + return r.beats +} + +func (r *liveReceiver) badSignatures() int { + r.mu.Lock() + defer r.mu.Unlock() + return r.bad +} diff --git a/cmd/tail-watch/live_tailnet_test.go b/cmd/tail-watch/live_tailnet_test.go index b6a79c3..192efe0 100644 --- a/cmd/tail-watch/live_tailnet_test.go +++ b/cmd/tail-watch/live_tailnet_test.go @@ -638,8 +638,8 @@ func sanitizeTarget(name string) string { return b.String() } -var secretRE = regexp.MustCompile(`tskey-[A-Za-z0-9_-]+`) +var secretRE = regexp.MustCompile(`tskey-[A-Za-z0-9_-]+|https://hooks\.slack\.com/\S+`) func redactSecrets(s string) string { - return secretRE.ReplaceAllString(s, "tskey-redacted") + return secretRE.ReplaceAllString(s, "redacted") } diff --git a/cmd/tail-watch/main.go b/cmd/tail-watch/main.go index 90434d6..4c6e4bb 100644 --- a/cmd/tail-watch/main.go +++ b/cmd/tail-watch/main.go @@ -8,6 +8,7 @@ import ( "io" "log/slog" "net/http" + "net/netip" "os" "os/signal" "runtime" @@ -51,10 +52,13 @@ type app struct { class *classifier learner *learner alerts *alerter + notify *notifier metrics prometheus.Registerer - watch func(context.Context, ipn.NotifyWatchOpt) (ipnWatch, error) - probeMu sync.Mutex - probes map[string]*sync.Mutex + // collector identifies this node in notification payloads. + collector CollectorInfo + watch func(context.Context, ipn.NotifyWatchOpt) (ipnWatch, error) + probeMu sync.Mutex + probes map[string]*sync.Mutex } func main() { @@ -65,6 +69,9 @@ func run(args []string, stdout, stderr io.Writer) int { if len(args) > 0 && args[0] == "check-config" { return runCheckConfig(args[1:], stdout, stderr) } + if len(args) > 0 && args[0] == "notify-test" { + return runNotifyTest(args[1:], stdout, stderr) + } fs := flag.NewFlagSet("tail-watch", flag.ContinueOnError) fs.SetOutput(stderr) configFlag := fs.String("config", "", "config file path (overrides TAIL_WATCH_CONFIG)") @@ -111,10 +118,13 @@ func run(args []string, stdout, stderr io.Writer) int { class: nil, learner: nil, alerts: nil, + notify: nil, metrics: nil, watch: nil, probeMu: sync.Mutex{}, probes: nil, + + collector: CollectorInfo{Name: "", TailscaleIP: ""}, } a.pub.add(debugConsumer{log: log}) a.bindResolver() @@ -125,9 +135,39 @@ func run(args []string, stdout, stderr io.Writer) int { if *onceFlag { return a.runOnce() } + if notifyErr := a.buildNotifier(); notifyErr != nil { + _, _ = fmt.Fprintf(stderr, "%v\n", notifyErr) + return 1 + } return a.runServer() } +// buildNotifier resolves every sink secret and opens the notification state. +// It runs before the prober so a bad secret or a corrupt snapshot stops startup. +func (a *app) buildNotifier() error { + if err := a.cfg.resolveSecrets(); err != nil { + return err + } + for _, sc := range a.cfg.Sinks { + if sc.Type == sinkWebhook && !sc.Secret.isSet() { + a.log.Warn("webhook sink is unsigned: no secret_env or secret_file", "sink", sc.Name) + } + } + if len(a.cfg.Sinks) == 0 { + return nil + } + n, err := newNotifier(notifierConfig{ + Clock: a.clock, Log: a.log, Sinks: a.cfg.Sinks, Collector: a.collector, + StateDir: a.cfg.Collector.StateDir, Alerts: nil, Tags: nil, Registry: a.metricRegisterer(), + NewSink: nil, Env: a.env, Jitter: nil, IntakeCap: 0, + }) + if err != nil { + return err + } + a.notify = n + return nil +} + func runCheckConfig(args []string, stdout, stderr io.Writer) int { fs := flag.NewFlagSet("check-config", flag.ContinueOnError) fs.SetOutput(stderr) @@ -148,6 +188,12 @@ func runCheckConfig(args []string, stdout, stderr io.Writer) int { _, _ = fmt.Fprintf(stdout, "%s interval=%s timeout=%s network=%s checks=%s\n", t.Name, t.Interval, t.Timeout, t.Network, formatChecks(t.Checks)) } + for _, s := range cfg.Sinks { + _, _ = fmt.Fprintln(stdout, describeSink(s)) + } + if cfg.Heartbeat != nil { + _, _ = fmt.Fprintf(stdout, "heartbeat interval=%s\n", cfg.Heartbeat.Interval) + } return 0 } @@ -238,6 +284,10 @@ func (a *app) startCollectors(ctx context.Context, p *prober.Prober) *runtimeReg a.alerts = al context.AfterFunc(ctx, al.stopDrain) al.addConsumer(transitionLog{log: a.log}) + if a.notify != nil { + a.notify.attach(al, a.presence.TagsFor) + al.addConsumer(a.notify) + } reg.alert = al reg.seed(targets) a.class = newClassifier(a.presence, a.env, a.log, a.clock, a.metricRegisterer(), targets, reg) @@ -246,6 +296,14 @@ func (a *app) startCollectors(ctx context.Context, p *prober.Prober) *runtimeReg a.class.addConsumer(al) a.pub.add(a.class) a.learner = newLearner(a.class) + if a.notify != nil { + names := make(map[string]bool, len(targets)) + for _, t := range targets { + names[t.Name] = true + } + a.notify.restoreOpen(names, al, a.class) + a.notify.Start(ctx) + } go a.presence.run(ctx) go a.learner.run(ctx) for _, t := range targets { @@ -295,6 +353,10 @@ func (a *app) runServer() int { defer stop() p := prober.New().WithSpread(true).WithMetricNamespace("tailwatch") probes := a.schedule(ctx, p) + if hb := a.cfg.Heartbeat; hb != nil { + beat := newHeartbeat(*hb, a.clock, a.log, a.env, a.alerts.FirstRoundDone(), a.metricRegisterer()) + go beat.run(ctx) + } mux := http.NewServeMux() mux.Handle("/", tsweb.StdHandler( p.StatusHandler(prober.WithTitle("tail-watch")), @@ -342,6 +404,9 @@ func (a *app) runServer() int { for _, stop := range probes { _ = stop() } + if a.notify != nil { + a.notify.Stop() + } if err := srv.Shutdown(shutCtx); err != nil { a.log.Error("shutdown", "err", err) } @@ -371,6 +436,7 @@ func (a *app) preflight(ctx context.Context) error { GOOS: runtime.GOOS, Socket: a.socket, }) + a.collector = collectorFrom(res) if res.Hostname != "" || len(res.IPs) > 0 || len(res.Tags) > 0 { a.log.InfoContext(ctx, "collector identity", "hostname", res.Hostname, "ips", res.IPs, "tags", res.Tags) } @@ -642,3 +708,19 @@ func versionString() string { } return "dev" } + +// collectorFrom picks the payload identity: the node's hostname and its first +// Tailscale IPv4 address, or IPv6 when it has none. +func collectorFrom(res preflightResult) CollectorInfo { + col := CollectorInfo{Name: res.Hostname, TailscaleIP: ""} + for _, ip := range res.IPs { + if addr, err := netip.ParseAddr(ip); err == nil && addr.Is4() { + col.TailscaleIP = ip + return col + } + } + if len(res.IPs) > 0 { + col.TailscaleIP = res.IPs[0] + } + return col +} diff --git a/cmd/tail-watch/main_test.go b/cmd/tail-watch/main_test.go index bd79906..ad3767f 100644 --- a/cmd/tail-watch/main_test.go +++ b/cmd/tail-watch/main_test.go @@ -742,3 +742,23 @@ func httpGet(t *testing.T, url string) string { } return string(b) } + +func TestCollectorFromPrefersIPv4(t *testing.T) { + t.Parallel() + tests := []struct { + name string + ips []string + want string + }{ + {"v4 first", []string{"100.64.0.8", "fd7a:115c:a1e0::1"}, "100.64.0.8"}, + {"v4 after v6", []string{"fd7a:115c:a1e0::1", "100.64.0.8"}, "100.64.0.8"}, + {"v6 only", []string{"fd7a:115c:a1e0::1"}, "fd7a:115c:a1e0::1"}, + {"none", nil, ""}, + } + for _, tt := range tests { + got := collectorFrom(preflightResult{Hostname: "monitor", IPs: tt.ips, Tags: nil, Warnings: nil}) + if got.Name != "monitor" || got.TailscaleIP != tt.want { + t.Errorf("%s: got %+v want ip %q", tt.name, got, tt.want) + } + } +} diff --git a/cmd/tail-watch/notify.go b/cmd/tail-watch/notify.go new file mode 100644 index 0000000..46f93ba --- /dev/null +++ b/cmd/tail-watch/notify.go @@ -0,0 +1,594 @@ +package main + +import ( + "context" + "log/slog" + "math/rand/v2" + "sync" + "time" + + "github.com/prometheus/client_golang/prometheus" + "tailscale.com/tstime" +) + +const ( + defaultIntakeCap = 4096 + reconcileInterval = time.Second +) + +// alertBridge is the part of the alert state machine the notifier uses. +type alertBridge interface { + Snapshots() []alertSnapshot + Acknowledge(name string, seq uint64) + MarkDirty(name string) + FirstRoundDone() <-chan struct{} +} + +type notifierConfig struct { + Clock tstime.Clock + Log *slog.Logger + Sinks []SinkConfig + Collector CollectorInfo + StateDir string + Alerts alertBridge + // Tags reports the tailnet tags of the node a target resolves to. + Tags func(target string) []string + Registry prometheus.Registerer + NewSink func(SinkConfig) (sink, error) + // Env carries sink connections. Nil uses the default dialer. + Env netEnv + Jitter func() float64 + // IntakeCap overrides the 4096-slot intake for tests. + IntakeCap int +} + +// notifier turns alert transitions into incidents and sink deliveries. One +// mutex guards the book; every mutation and its durable commit happen under it, +// so workers only see new intents after they are committed. Checks never touch +// that mutex: ObserveTransition only writes to the intake channel. +type notifier struct { + cfg notifierConfig + log *slog.Logger + clock tstime.Clock + m *notifyMetrics + intake chan Transition + + mu sync.Mutex + b *book + store *stateStore + sinks map[string]sink + reconciled map[string]uint64 + wake map[string]chan struct{} + persistBad bool + + cancel context.CancelFunc + wg sync.WaitGroup +} + +func newNotifier(cfg notifierConfig) (*notifier, error) { + if cfg.Clock == nil { + cfg.Clock = tstime.StdClock{} + } + if cfg.Log == nil { + cfg.Log = slog.New(slog.DiscardHandler) + } + if cfg.Jitter == nil { + cfg.Jitter = rand.Float64 + } + if cfg.IntakeCap <= 0 { + cfg.IntakeCap = defaultIntakeCap + } + if cfg.NewSink == nil { + cfg.NewSink = func(sc SinkConfig) (sink, error) { return newSinkWithEnv(sc, cfg.Collector, cfg.Env) } + } + m := newNotifyMetrics(cfg.Registry, cfg.StateDir != "") + n := ¬ifier{ + cfg: cfg, log: cfg.Log, clock: cfg.Clock, m: m, + intake: make(chan Transition, cfg.IntakeCap), + mu: sync.Mutex{}, + b: newBook(cfg.Log, m, cfg.Sinks), + store: nil, + sinks: map[string]sink{}, + reconciled: map[string]uint64{}, + wake: map[string]chan struct{}{}, + persistBad: false, + cancel: nil, + wg: sync.WaitGroup{}, + } + for _, sc := range cfg.Sinks { + s, err := cfg.NewSink(sc) + if err != nil { + return nil, err + } + n.sinks[sc.Name] = s + n.wake[sc.Name] = make(chan struct{}, 1) + m.queueLen(sc.Name, 0) + } + if cfg.StateDir != "" { + store, err := newStateStore(cfg.StateDir, cfg.Log) + if err != nil { + return nil, err + } + st, err := store.load() + if err != nil { + return nil, err + } + n.store = store + if st != nil { + n.b.restore(st, cfg.Clock.Now()) + } + n.commitLocked() + } + return n, nil +} + +// attach connects the notifier to the alert state machine and presence tags +// before Start. +func (n *notifier) attach(al alertBridge, tags func(string) []string) { + n.mu.Lock() + defer n.mu.Unlock() + n.cfg.Alerts = al + n.cfg.Tags = tags +} + +// ObserveTransition queues tr without I/O or waiting. A full intake drops the +// transition and marks its target dirty for reconciliation. +func (n *notifier) ObserveTransition(tr Transition) { + select { + case n.intake <- tr: + default: + n.m.dropped(sinkIntake, reasonQueueFull) + n.log.Warn("notification intake full; transition dropped", "target", tr.Target.Name, "seq", tr.Sequence) + if n.cfg.Alerts != nil { + n.cfg.Alerts.MarkDirty(tr.Target.Name) + } + } +} + +// Start runs the coordinator and one worker per sink until ctx ends or Stop. +func (n *notifier) Start(ctx context.Context) { + ctx, n.cancel = context.WithCancel(ctx) + n.wg.Add(1) + go n.loop(ctx) + for _, sc := range n.cfg.Sinks { + w := &worker{ + n: n, cfg: sc, s: n.sinks[sc.Name], wake: n.wake[sc.Name], + loop: retryLoop{clock: n.clock, jitter: n.cfg.Jitter, sink: n.sinks[sc.Name], last: time.Time{}}, + resume: nil, windowStart: time.Time{}, + } + n.wg.Add(1) + go w.run(ctx) + } + if n.cfg.Alerts != nil { + n.wg.Add(1) + go n.watchFirstRound(ctx) + } +} + +// Stop ends the workers and waits for them. Attempted deliveries stay attempted. +func (n *notifier) Stop() { + if n.cancel != nil { + n.cancel() + } + n.wg.Wait() +} + +func (n *notifier) loop(ctx context.Context) { + defer n.wg.Done() + tk, tick := n.clock.NewTicker(reconcileInterval) + defer tk.Stop() + for { + select { + case tr := <-n.intake: + n.handle(tr) + case <-tick: + n.reconcile() + n.retryPersist() + case <-ctx.Done(): + return + } + } +} + +func (n *notifier) handle(tr Transition) { + n.mu.Lock() + defer n.mu.Unlock() + if n.applyLocked(tr, false) { + n.commitLocked() + n.wakeLocked() + } +} + +// reconcile applies every dirty alert snapshot, so a dropped transition or a +// removed target still gets its event without another state change. +func (n *notifier) reconcile() { + if n.cfg.Alerts == nil { + return + } + snaps := n.cfg.Alerts.Snapshots() + n.mu.Lock() + defer n.mu.Unlock() + changed := false + for _, s := range snaps { + if !s.Dirty { + continue + } + if n.applyLocked(transitionOfSnapshot(s), s.Removed) { + changed = true + } + } + if changed { + n.commitLocked() + n.wakeLocked() + } +} + +func transitionOfSnapshot(s alertSnapshot) Transition { + return Transition{ + Sequence: s.Sequence, Target: s.Target, From: s.State, To: s.State, Flapping: false, + Check: s.Check, Layer: s.Layer, Err: s.Err, Since: s.Since, At: s.At, Failures: s.Failures, + Path: s.Path, Dependents: nil, DownSince: s.DownSince, + } +} + +// applyLocked maps one transition (or reconciled snapshot) onto the incident +// book and reports whether anything changed. Sequences at or below the +// target's reconciled sequence are stale and ignored. +func (n *notifier) applyLocked(tr Transition, removed bool) bool { + name := tr.Target.Name + if tr.Sequence != 0 && tr.Sequence <= n.reconciled[name] { + return false + } + n.reconciled[name] = max(n.reconciled[name], tr.Sequence) + if n.cfg.Alerts != nil { + defer n.cfg.Alerts.Acknowledge(name, tr.Sequence) + } + inc := n.b.openIncidentFor(name) + if removed { + if inc == nil { + return false + } + return n.closeLocked(inc, n.clock.Now(), nil, reasonRemoved) + } + switch mapTransition(tr, inc != nil) { + case mapDown: + id, ok := n.newIDLocked(prefixEvent) + if !ok { + return false + } + ev := newDownEvent(id, n.cfg.Collector, tr) + n.b.openIncident(ev, targetDefOf(tr.Target), downSinceOf(tr), n.routeLocked(ev)) + n.log.Info("incident opened", "target", name, "event", id, "check", tr.Check, "layer", string(tr.Layer)) + return true + case mapRecovered: + return n.closeLocked(inc, tr.At, tr.Dependents, "") + case mapFlapping: + id, ok := n.newIDLocked(prefixEvent) + if !ok { + return false + } + n.enqueueLocked(newFlappingEvent(id, n.cfg.Collector, tr)) + return true + case mapNone: + return false + } + return false +} + +func (n *notifier) closeLocked(inc *incident, at time.Time, deps []string, reason string) bool { + id, ok := n.newIDLocked(prefixEvent) + if !ok { + return false + } + n.b.closeIncident(inc, newRecoveredEvent(id, n.cfg.Collector, inc, at, deps, reason)) + n.log.Info("incident closed", "target", inc.Target.Name, "event", id, "reason", reason) + return true +} + +func (n *notifier) newIDLocked(prefix string) (string, bool) { + id, err := newID(prefix, n.clock.Now(), nil) + if err != nil { + n.log.Error("cannot allocate event id", "err", err) + return "", false + } + return id, true +} + +// routeLocked lists the sinks whose filters accept ev, in config order. +func (n *notifier) routeLocked(ev Event) []string { + var tags []string + if ev.Target != nil && n.cfg.Tags != nil { + tags = n.cfg.Tags(ev.Target.Name) + } + var out []string + for _, sc := range n.cfg.Sinks { + if sc.accepts(ev, tags) { + out = append(out, sc.Name) + } + } + return out +} + +// enqueueLocked queues an event that opens no incident, such as a flap notice. +func (n *notifier) enqueueLocked(ev Event) { + for _, name := range n.routeLocked(ev) { + n.b.push(name, &queued{Seq: n.b.seq(), IncidentID: "", Event: ev, Attempted: false, UnitID: ""}) + } +} + +func (n *notifier) watchFirstRound(ctx context.Context) { + defer n.wg.Done() + select { + case <-n.cfg.Alerts.FirstRoundDone(): + case <-ctx.Done(): + return + } + n.mu.Lock() + defer n.mu.Unlock() + id, ok := n.newIDLocked(prefixEvent) + if !ok { + return + } + n.enqueueLocked(newStartedEvent(id, n.cfg.Collector, n.clock.Now())) + n.commitLocked() + n.wakeLocked() +} + +// commitLocked persists the book. On failure the in-memory state stays +// authoritative, delivery continues, and the full state is retried later. +func (n *notifier) commitLocked() { + n.m.openIncidents(n.b.openCount()) + if n.store == nil { + return + } + if err := n.store.commit(n.b.snapshot()); err != nil { + n.m.persistFailed() + if !n.persistBad { + n.log.Error("notification state not persisted; durability degraded", "err", err) + } + n.persistBad = true + return + } + if n.persistBad { + n.log.Info("notification state persistence restored") + } + n.persistBad = false +} + +func (n *notifier) retryPersist() { + n.mu.Lock() + defer n.mu.Unlock() + if n.persistBad { + n.commitLocked() + } +} + +func (n *notifier) wakeLocked() { + for _, ch := range n.wake { + select { + case ch <- struct{}{}: + default: + } + } +} + +// openIncidents returns copies of the open incident records. +func (n *notifier) openIncidents() []incident { + n.mu.Lock() + defer n.mu.Unlock() + var out []incident + for _, id := range n.b.open { + if inc := n.b.incidents[id]; inc != nil { + out = append(out, *inc) + } + } + return out +} + +// closeRemoved closes name's open incident with reason removed, as when a +// configured target was deleted from the config while tail-watch was stopped. +func (n *notifier) closeRemoved(name string) { + n.mu.Lock() + defer n.mu.Unlock() + if inc := n.b.openIncidentFor(name); inc != nil && n.closeLocked(inc, n.clock.Now(), nil, reasonRemoved) { + n.commitLocked() + n.wakeLocked() + } +} + +// worker delivers one sink's queue in order. +type worker struct { + n *notifier + cfg SinkConfig + s sink + wake chan struct{} + loop retryLoop + resume []*unit + windowStart time.Time +} + +func (w *worker) grouped() bool { + return w.cfg.Type == sinkSlack || w.cfg.Batch +} + +func (w *worker) groupCap() int { + switch { + case w.cfg.Type == sinkSlack: + return w.cfg.QueueSize + case w.cfg.Batch: + return w.cfg.MaxBatch + default: + return 1 + } +} + +func (w *worker) run(ctx context.Context) { + defer w.n.wg.Done() + w.n.mu.Lock() + w.resume = w.n.b.sinkUnits(w.cfg.Name) + w.n.mu.Unlock() + for ctx.Err() == nil { + u := w.next(ctx) + if u == nil { + return + } + w.deliver(ctx, u) + } +} + +// next returns the next request to send, resuming attempted units before +// forming a new one. It blocks until there is work or ctx ends. +func (w *worker) next(ctx context.Context) *unit { + for { + if len(w.resume) > 0 { + u := w.resume[0] + w.resume = w.resume[1:] + return u + } + if ctx.Err() != nil { + return nil + } + u, delay := w.poll() + if u != nil { + return u + } + if delay >= 0 && !w.wait(ctx, delay) { + return nil + } + } +} + +// poll decides, under the notifier lock, whether a request is ready. It returns +// a formed unit, or how long to wait (zero: until woken; negative: poll again). +func (w *worker) poll() (*unit, time.Duration) { + n := w.n + n.mu.Lock() + defer n.mu.Unlock() + now := n.clock.Now() + changed := n.b.expireStale(w.cfg.Name, now) + pending := n.b.pendingItems(w.cfg.Name) + if len(pending) == 0 { + if changed { + n.commitLocked() + } + w.windowStart = time.Time{} + return nil, 0 + } + if remaining := w.groupRemaining(now, len(pending)); remaining > 0 { + if changed { + n.commitLocked() + } + return nil, remaining + } + u := w.formLocked(pending[:min(len(pending), w.groupCap())], now) + n.commitLocked() + w.windowStart = time.Time{} + if u == nil { + return nil, -1 + } + return u, 0 +} + +// groupRemaining is how much of the group window is left. Zero means send now. +func (w *worker) groupRemaining(now time.Time, pending int) time.Duration { + if !w.grouped() { + return 0 + } + if w.windowStart.IsZero() { + w.windowStart = now + } + if pending >= w.groupCap() { + return 0 + } + return max(w.cfg.GroupWait-now.Sub(w.windowStart), 0) +} + +// formLocked fixes the first-attempt body of a request and marks its members +// attempted. The commit that follows makes that durable before any send. +func (w *worker) formLocked(items []*queued, now time.Time) *unit { + n := w.n + evs := make([]Event, len(items)) + for i, it := range items { + it.Event.Late = now.Sub(it.Event.Time.Time()) > lateAfter + evs[i] = it.Event + } + id := items[0].Event.ID + if w.grouped() { + bid, ok := n.newIDLocked(prefixBatch) + if !ok { + return nil + } + id = bid + } + body, err := w.s.Render(evs, id) + u := n.b.beginUnit(w.cfg.Name, id, items, body) + if err != nil { + n.log.Error("cannot render notification", "sink", w.cfg.Name, "err", err) + n.b.dropUnit(u, reasonPermanent) + return nil + } + if w.cfg.Type == sinkWebhook && w.cfg.Batch { + n.m.batch(w.cfg.Name, len(items)) + } + return u +} + +// wait blocks until a wake-up, d elapsing (when d > 0) or ctx ending. +func (w *worker) wait(ctx context.Context, d time.Duration) bool { + var timeout <-chan time.Time + if d > 0 { + t, ch := w.n.clock.NewTimer(d) + defer t.Stop() + timeout = ch + } + select { + case <-w.wake: + return true + case <-timeout: + return true + case <-ctx.Done(): + return false + } +} + +func (w *worker) deliver(ctx context.Context, u *unit) { + n := w.n + expired := func() bool { + n.mu.Lock() + defer n.mu.Unlock() + return n.b.unitExpired(u, n.clock.Now()) + } + onFail := func(res deliveryResult) { + n.m.failed(w.cfg.Name, res.Reason) + n.log.WarnContext(ctx, "notification attempt failed", "sink", w.cfg.Name, "id", u.ID, "reason", res.Reason, + "status", res.Status, "err", res.Err) + } + out := w.loop.run(ctx, u.Body, u.ID, expired, onFail) + if out.Kind == outcomeCancelled { + return + } + n.mu.Lock() + defer n.mu.Unlock() + switch out.Kind { + case outcomeDelivered: + n.b.ackUnit(u, n.clock.Now()) + case outcomePermanent: + n.log.ErrorContext( + ctx, + "notification rejected permanently", + "sink", + w.cfg.Name, + "id", + u.ID, + "status", + out.Result.Status, + ) + n.b.dropUnit(u, reasonPermanent) + case outcomeExpired: + n.b.dropUnit(u, reasonExpired) + case outcomeCancelled: + return + } + n.commitLocked() + n.wakeLocked() +} diff --git a/cmd/tail-watch/notify_helpers_test.go b/cmd/tail-watch/notify_helpers_test.go new file mode 100644 index 0000000..22e45b5 --- /dev/null +++ b/cmd/tail-watch/notify_helpers_test.go @@ -0,0 +1,352 @@ +package main + +import ( + "context" + "encoding/json" + "errors" + "log/slog" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/prometheus/client_golang/prometheus" + "tailscale.com/tstest" + "tailscale.com/tstime" +) + +func testNow() time.Time { return time.Date(2026, 4, 2, 14, 2, 11, 0, time.UTC) } + +// countingClock wraps a tstest.Clock and counts timers so a test can wait for a +// goroutine to arm its timer before advancing time. +type countingClock struct { + *tstest.Clock + + timers atomic.Int64 + tickers atomic.Int64 +} + +func newCountingClock() *countingClock { + return &countingClock{Clock: tstest.NewClock(tstest.ClockOpts{ + Start: testNow(), Step: 0, TimerChannelSize: 0, FollowRealTime: false, + }), timers: atomic.Int64{}, tickers: atomic.Int64{}} +} + +func (c *countingClock) NewTimer(d time.Duration) (tstime.TimerController, <-chan time.Time) { + c.timers.Add(1) + return c.Clock.NewTimer(d) +} + +func (c *countingClock) NewTicker(d time.Duration) (tstime.TickerController, <-chan time.Time) { + c.tickers.Add(1) + return c.Clock.NewTicker(d) +} + +func (c *countingClock) AfterFunc(d time.Duration, f func()) tstime.TimerController { + return c.Clock.AfterFunc(d, f) +} + +// waitTimers blocks until the clock has handed out more than prev timers. +func (c *countingClock) waitTimers(t *testing.T, prev int64) { + t.Helper() + eventually(t, func() bool { return c.timers.Load() > prev }) +} + +func eventually(t *testing.T, cond func() bool) { + t.Helper() + deadline := time.Now().Add(5 * time.Second) + for time.Now().Before(deadline) { + if cond() { + return + } + time.Sleep(2 * time.Millisecond) + } + t.Fatal("condition not met in time") +} + +// fakeSink is a scripted receiver. +type fakeSink struct { + name string + interval time.Duration + mu sync.Mutex + script []deliveryResult + calls []fakeCall + block chan struct{} +} + +type fakeCall struct { + ID string + Body []byte + At time.Time +} + +func (f *fakeSink) Name() string { return f.name } + +func (f *fakeSink) MinInterval() time.Duration { return f.interval } + +func (f *fakeSink) Render(evs []Event, id string) ([]byte, error) { + return marshalJSON(map[string]any{"id": id, "events": evs}) +} + +func (f *fakeSink) Send(ctx context.Context, body []byte, m sendMeta) deliveryResult { + if f.block != nil { + select { + case <-f.block: + case <-ctx.Done(): + return deliveryResult{Class: resultRetry, Status: 0, Reason: failTimeout, RetryAfter: 0, Err: ctx.Err()} + } + } + f.mu.Lock() + defer f.mu.Unlock() + f.calls = append(f.calls, fakeCall{ID: m.ID, Body: append([]byte(nil), body...), At: m.At}) + if len(f.script) > 0 { + res := f.script[0] + f.script = f.script[1:] + return res + } + return deliveryResult{Class: resultOK, Status: 200, Reason: "", RetryAfter: 0, Err: nil} +} + +func (f *fakeSink) callCount() int { + f.mu.Lock() + defer f.mu.Unlock() + return len(f.calls) +} + +func (f *fakeSink) snapshot() []fakeCall { + f.mu.Lock() + defer f.mu.Unlock() + return append([]fakeCall(nil), f.calls...) +} + +// eventTypes decodes the event types of every delivered call in order. +func (f *fakeSink) eventTypes(t *testing.T) []EventType { + t.Helper() + var out []EventType + for _, c := range f.snapshot() { + var body struct { + Events []Event `json:"events"` + } + if err := json.Unmarshal(c.Body, &body); err != nil { + t.Fatal(err) + } + for _, e := range body.Events { + out = append(out, e.Type) + } + } + return out +} + +func statusRes(status int) deliveryResult { return classifyStatus(status, nil, testNow()) } + +type notifyHarness struct { + n *notifier + clock *countingClock + sinks map[string]*fakeSink + reg *prometheus.Registry + alert *fakeAlerts +} + +type fakeAlerts struct { + mu sync.Mutex + snaps map[string]alertSnapshot + acked map[string]uint64 + dirty map[string]bool + first chan struct{} +} + +func newFakeAlerts() *fakeAlerts { + return &fakeAlerts{ + mu: sync.Mutex{}, + snaps: map[string]alertSnapshot{}, + acked: map[string]uint64{}, + dirty: map[string]bool{}, + first: make(chan struct{}), + } +} + +func (f *fakeAlerts) Snapshots() []alertSnapshot { + f.mu.Lock() + defer f.mu.Unlock() + var out []alertSnapshot + for name, s := range f.snaps { + s.Dirty = f.dirty[name] + out = append(out, s) + } + return out +} + +func (f *fakeAlerts) Acknowledge(name string, seq uint64) { + f.mu.Lock() + defer f.mu.Unlock() + if s, ok := f.snaps[name]; ok && seq < s.Sequence { + return + } + f.acked[name] = seq + f.dirty[name] = false + if s, ok := f.snaps[name]; ok && s.Removed { + delete(f.snaps, name) + } +} + +func (f *fakeAlerts) MarkDirty(name string) { + f.mu.Lock() + defer f.mu.Unlock() + f.dirty[name] = true +} + +func (f *fakeAlerts) FirstRoundDone() <-chan struct{} { return f.first } + +func (f *fakeAlerts) setSnapshot(s alertSnapshot) { + f.mu.Lock() + defer f.mu.Unlock() + f.snaps[s.Name] = s + f.dirty[s.Name] = true +} + +type harnessOpts struct { + sinks []SinkConfig + stateDir string + tags func(string) []string + intake int + scripts map[string][]deliveryResult + clock *countingClock +} + +func testSink(name string) SinkConfig { + return SinkConfig{ + Name: name, + Type: sinkWebhook, + URL: secretRef{Env: "", File: ""}, + Secret: secretRef{Env: "", File: ""}, + HeadersEnv: nil, + MinSeverity: SeverityInfo, + Events: nil, + Match: MatchConfig{Sites: nil, Targets: nil, Tags: nil}, + MaxAge: time.Hour, + Queue: queueMemory, + QueueSize: 1000, + Batch: false, + GroupWait: 10 * time.Second, + MaxBatch: 100, + MentionOnPage: "", + StatusURL: "", + } +} + +func newHarness(t *testing.T, o harnessOpts) *notifyHarness { + t.Helper() + h := ¬ifyHarness{ + n: nil, clock: o.clock, sinks: map[string]*fakeSink{}, reg: prometheus.NewRegistry(), alert: newFakeAlerts(), + } + if h.clock == nil { + h.clock = newCountingClock() + } + for _, sc := range o.sinks { + h.sinks[sc.Name] = &fakeSink{ + name: sc.Name, + script: o.scripts[sc.Name], + interval: 0, + mu: sync.Mutex{}, + calls: nil, + block: nil, + } + } + n, err := newNotifier(notifierConfig{ + Clock: h.clock, Log: slog.New(slog.DiscardHandler), Sinks: o.sinks, + Collector: CollectorInfo{Name: "monitor-nyc", TailscaleIP: "100.101.102.103"}, + StateDir: o.stateDir, Alerts: h.alert, Tags: o.tags, Registry: h.reg, + NewSink: func(sc SinkConfig) (sink, error) { + f, ok := h.sinks[sc.Name] + if !ok { + return nil, errors.New("no fake sink " + sc.Name) + } + return f, nil + }, + Env: nil, + Jitter: func() float64 { return 0.5 }, + IntakeCap: o.intake, + }) + if err != nil { + t.Fatal(err) + } + h.n = n + return h +} + +func (h *notifyHarness) start(t *testing.T) { + t.Helper() + ctx, cancel := context.WithCancel(context.Background()) + h.n.Start(ctx) + eventually(t, func() bool { return h.clock.tickers.Load() > 0 }) + t.Cleanup(func() { cancel(); h.n.Stop() }) +} + +func nasTarget() Target { + return Target{ + Name: "nyc-nas", Addr: "10.20.0.50", Site: "nyc", Severity: SeverityPage, Parent: "nyc-router", + InsecureTLS: false, Interval: 0, Timeout: 0, FailThreshold: 0, RecoverThreshold: 0, + FlapLimit: 0, FlapWindow: 0, Network: "", Checks: nil, Implicit: false, StableID: "", NodeID: 0, + } +} + +func downTransition(seq uint64, at time.Time) Transition { + return Transition{ + Sequence: seq, + Target: nasTarget(), + From: StateUp, + To: StateDown, + Flapping: false, + Check: "tcp:445", + Layer: layerService, + Err: errors.New("connection refused"), + Since: at.Add(-90 * time.Second), + At: at, + Failures: 3, + Path: pathInfo{Kind: "direct", Endpoint: "", DERP: "", Latency: 12 * time.Millisecond}, + Dependents: nil, + DownSince: at.Add(-90 * time.Second), + } +} + +func upTransition(seq uint64, at time.Time) Transition { + return Transition{ + Sequence: seq, + Target: nasTarget(), + From: StateDown, + To: StateUp, + Flapping: false, + Check: "", + Layer: "", + Err: nil, + Since: time.Time{}, + At: at, + Failures: 0, + Path: pathInfo{Kind: "", Endpoint: "", DERP: "", Latency: 0}, + Dependents: nil, + DownSince: time.Time{}, + } +} + +// realSinkHarness builds the notifier around the real webhook and Slack sinks +// instead of fakes. +func realSinkHarness(t *testing.T, o harnessOpts) *notifyHarness { + t.Helper() + h := ¬ifyHarness{ + n: nil, clock: o.clock, sinks: map[string]*fakeSink{}, reg: prometheus.NewRegistry(), alert: newFakeAlerts(), + } + if h.clock == nil { + h.clock = newCountingClock() + } + col := CollectorInfo{Name: "monitor-nyc", TailscaleIP: "100.101.102.103"} + n, err := newNotifier(notifierConfig{ + Clock: h.clock, Log: slog.New(slog.DiscardHandler), Sinks: o.sinks, Collector: col, + StateDir: o.stateDir, Alerts: h.alert, Tags: o.tags, Registry: h.reg, + NewSink: nil, Env: nil, Jitter: func() float64 { return 0.5 }, IntakeCap: 0, + }) + if err != nil { + t.Fatal(err) + } + h.n = n + return h +} diff --git a/cmd/tail-watch/notify_integration_test.go b/cmd/tail-watch/notify_integration_test.go new file mode 100644 index 0000000..fd0c913 --- /dev/null +++ b/cmd/tail-watch/notify_integration_test.go @@ -0,0 +1,179 @@ +package main + +import ( + "bytes" + "fmt" + "net" + "os" + "os/exec" + "strings" + "syscall" + "testing" + "time" +) + +func closedPort(t *testing.T) int { + t.Helper() + ln, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + port := ln.Addr().(*net.TCPAddr).Port + _ = ln.Close() + return port +} + +func sinkConfig(port, listen int, extra string) string { + return fmt.Sprintf(` +collector: + listen: 127.0.0.1:%d +%s +defaults: + fail_threshold: 1 + recover_threshold: 1 +targets: + - name: loop + addr: 127.0.0.1 + interval: 200ms + timeout: 100ms + checks: ["tcp:%d"] +sinks: + - name: hook + type: webhook + url_env: TW_IT_HOOK + secret_env: TW_IT_SECRET +heartbeat: + url_env: TW_IT_BEAT + interval: 1s +`, listen, extra, port) +} + +func TestServerDeliversNotificationsAndHeartbeat(t *testing.T) { + t.Parallel() + bin := buildTailWatch(t) + sock, closeAPI := fakeTailscaled(t, runningStatus(), nil) + t.Cleanup(closeAPI) + hook := newReceiver(t) + beat := newReceiver(t) + listen := freePort(t) + path := writeCfg(t, sinkConfig(closedPort(t), listen, "")) + + cmd := exec.Command(bin, "--config", path, "--socket", sock) + cmd.Env = append(os.Environ(), + "TW_IT_HOOK="+hook.srv.URL, "TW_IT_SECRET=integration-secret", "TW_IT_BEAT="+beat.srv.URL) + var stderr bytes.Buffer + cmd.Stderr = &stderr + if err := cmd.Start(); err != nil { + t.Fatal(err) + } + done := make(chan error, 1) + go func() { done <- cmd.Wait() }() + t.Cleanup(func() { _ = cmd.Process.Kill() }) + + eventually(t, func() bool { return hook.count() >= 2 }) + types := map[string]bool{} + for i := range hook.count() { + body := string(hook.request(i).Body) + switch { + case strings.Contains(body, `"type":"target.down"`): + types["down"] = true + if !strings.Contains(body, `"name":"monitor"`) || !strings.Contains(body, `"tailscale_ip":"100.64.0.8"`) { + t.Fatalf("collector identity missing: %s", body) + } + case strings.Contains(body, `"type":"collector.started"`): + types["started"] = true + } + if hook.request(i).Header.Get(signatureHeader) == "" { + t.Fatal("configured secret must sign every request") + } + } + if !types["down"] || !types["started"] { + t.Fatalf("got %v\nstderr:\n%s", types, stderr.String()) + } + eventually(t, func() bool { return beat.count() >= 1 }) + + if err := cmd.Process.Signal(syscall.SIGTERM); err != nil { + t.Fatal(err) + } + select { + case err := <-done: + if err != nil { + t.Fatalf("exit %v\n%s", err, stderr.String()) + } + case <-time.After(10 * time.Second): + t.Fatal("SIGTERM did not stop the collector") + } + settled := beat.count() + time.Sleep(1500 * time.Millisecond) + if beat.count() != settled { + t.Fatal("heartbeat continued after shutdown") + } +} + +func TestOnceWithSinksSendsNothing(t *testing.T) { + t.Parallel() + bin := buildTailWatch(t) + sock, closeAPI := fakeTailscaled(t, runningStatus(), nil) + t.Cleanup(closeAPI) + hook := newReceiver(t) + beat := newReceiver(t) + path := writeCfg(t, sinkConfig(closedPort(t), freePort(t), "")) + out, errOut, code := runBin(t, bin, + []string{"TW_IT_HOOK=" + hook.srv.URL, "TW_IT_SECRET=x", "TW_IT_BEAT=" + beat.srv.URL}, + "--once", "--config", path, "--socket", sock) + if code != 1 || !strings.Contains(out, "loop") { + t.Fatalf("--once against a down target: exit %d\n%s\n%s", code, out, errOut) + } + time.Sleep(300 * time.Millisecond) + if hook.count() != 0 || beat.count() != 0 { + t.Fatalf("--once must not notify or heartbeat: %d %d", hook.count(), beat.count()) + } +} + +func TestOnceDoesNotNeedSecrets(t *testing.T) { + t.Parallel() + bin := buildTailWatch(t) + sock, closeAPI := fakeTailscaled(t, runningStatus(), nil) + t.Cleanup(closeAPI) + path := writeCfg(t, sinkConfig(closedPort(t), freePort(t), "")) + _, errOut, code := runBin(t, bin, nil, "--once", "--config", path, "--socket", sock) + if code != 1 || strings.Contains(errOut, "TW_IT_HOOK") { + t.Fatalf("exit %d\n%s", code, errOut) + } +} + +func TestServerFailsFastOnUnsetSecret(t *testing.T) { + t.Parallel() + bin := buildTailWatch(t) + sock, closeAPI := fakeTailscaled(t, runningStatus(), nil) + t.Cleanup(closeAPI) + path := writeCfg(t, sinkConfig(closedPort(t), freePort(t), "")) + _, errOut, code := runBin(t, bin, nil, "--config", path, "--socket", sock) + if code == 0 || !strings.Contains(errOut, "hook") || !strings.Contains(errOut, "TW_IT_HOOK") { + t.Fatalf("exit %d, stderr must name the sink and variable:\n%s", code, errOut) + } +} + +func TestServerRefusesCorruptState(t *testing.T) { + t.Parallel() + bin := buildTailWatch(t) + sock, closeAPI := fakeTailscaled(t, runningStatus(), nil) + t.Cleanup(closeAPI) + dir := t.TempDir() + if err := os.WriteFile(dir+"/"+stateFileName, []byte("garbage"), 0o600); err != nil { + t.Fatal(err) + } + path := writeCfg(t, sinkConfig(closedPort(t), freePort(t), " state_dir: "+dir)) + _, errOut, code := runBin( + t, + bin, + []string{"TW_IT_HOOK=http://127.0.0.1:1", "TW_IT_SECRET=x", "TW_IT_BEAT=http://127.0.0.1:1"}, + "--config", + path, + "--socket", + sock, + ) + if code == 0 || !strings.Contains(errOut, "corrupt") { + t.Fatalf("exit %d\n%s", code, errOut) + } +} diff --git a/cmd/tail-watch/notify_metrics.go b/cmd/tail-watch/notify_metrics.go new file mode 100644 index 0000000..2c26293 --- /dev/null +++ b/cmd/tail-watch/notify_metrics.go @@ -0,0 +1,132 @@ +package main + +import ( + "time" + + "github.com/prometheus/client_golang/prometheus" +) + +const ( + reasonQueueFull = "queue_full" + reasonExpired = "expired" + reasonPermanent = "permanent" + reasonRestart = "restart" + + failTimeout = "timeout" + failHTTP5xx = "http_5xx" + failHTTP429 = "http_429" + failHTTP4xx = "http_4xx_permanent" + failNetwork = "network" + labelSink = "sink" + metricsPrefix = "tailwatch" +) + +func latencyBuckets() []float64 { + return []float64{0.1, 0.5, 1, 5, 10, 30, 60, 300, 900, 3600} +} + +// notifyMetrics holds the tailwatch_notify_* series. A nil value is a no-op. +type notifyMetrics struct { + sentC *prometheus.CounterVec + failedC *prometheus.CounterVec + droppedC *prometheus.CounterVec + persistErr *prometheus.CounterVec + batchSizeH *prometheus.HistogramVec + queueLenG *prometheus.GaugeVec + latencyH *prometheus.HistogramVec + openG prometheus.Gauge +} + +func newNotifyMetrics(reg prometheus.Registerer, withOpen bool) *notifyMetrics { + m := ¬ifyMetrics{ + sentC: prometheus.NewCounterVec(prometheus.CounterOpts{ + Namespace: metricsPrefix, Name: "notify_sent_total", + Help: "Notifications delivered, by sink and event type.", + }, []string{labelSink, "type"}), + failedC: prometheus.NewCounterVec(prometheus.CounterOpts{ + Namespace: metricsPrefix, Name: "notify_failed_total", + Help: "Failed delivery attempts, by sink and reason.", + }, []string{labelSink, "reason"}), + droppedC: prometheus.NewCounterVec(prometheus.CounterOpts{ + Namespace: metricsPrefix, Name: "notify_dropped_total", + Help: "Notifications dropped without delivery, by sink and reason.", + }, []string{labelSink, "reason"}), + persistErr: prometheus.NewCounterVec(prometheus.CounterOpts{ + Namespace: metricsPrefix, Name: "notify_persist_errors_total", + Help: "Failed writes of notification state.", + }, []string{labelSink}), + batchSizeH: prometheus.NewHistogramVec(prometheus.HistogramOpts{ + Namespace: metricsPrefix, Name: "notify_batch_size", + Help: "Events per batched webhook request.", + Buckets: []float64{1, 2, 5, 10, 25, 50, 100, 250, 500}, + }, []string{labelSink}), + queueLenG: prometheus.NewGaugeVec(prometheus.GaugeOpts{ + Namespace: metricsPrefix, Name: "notify_queue_len", + Help: "Events waiting in a sink queue.", + }, []string{labelSink}), + latencyH: prometheus.NewHistogramVec(prometheus.HistogramOpts{ + Namespace: metricsPrefix, Name: "notify_latency_seconds", + Help: "Seconds from event creation to confirmed delivery.", + Buckets: latencyBuckets(), + }, []string{labelSink}), + openG: nil, + } + cols := []prometheus.Collector{m.sentC, m.failedC, m.droppedC, m.persistErr, m.batchSizeH, m.queueLenG, m.latencyH} + if withOpen { + m.openG = prometheus.NewGauge(prometheus.GaugeOpts{ + Namespace: metricsPrefix, Name: "open_incidents", + Help: "Incidents opened and not yet closed.", + }) + cols = append(cols, m.openG) + } + if reg != nil { + for _, c := range cols { + reg.MustRegister(c) + } + } + return m +} + +func (m *notifyMetrics) sent(sink string, typ EventType, latency time.Duration) { + if m == nil { + return + } + m.sentC.WithLabelValues(sink, string(typ)).Inc() + m.latencyH.WithLabelValues(sink).Observe(max(latency, 0).Seconds()) +} + +func (m *notifyMetrics) failed(sink, reason string) { + if m != nil { + m.failedC.WithLabelValues(sink, reason).Inc() + } +} + +func (m *notifyMetrics) dropped(sink, reason string) { + if m != nil { + m.droppedC.WithLabelValues(sink, reason).Inc() + } +} + +func (m *notifyMetrics) persistFailed() { + if m != nil { + m.persistErr.WithLabelValues(sinkState).Inc() + } +} + +func (m *notifyMetrics) batch(sink string, n int) { + if m != nil { + m.batchSizeH.WithLabelValues(sink).Observe(float64(n)) + } +} + +func (m *notifyMetrics) queueLen(sink string, n int) { + if m != nil { + m.queueLenG.WithLabelValues(sink).Set(float64(n)) + } +} + +func (m *notifyMetrics) openIncidents(n int) { + if m != nil && m.openG != nil { + m.openG.Set(float64(n)) + } +} diff --git a/cmd/tail-watch/notify_metrics_test.go b/cmd/tail-watch/notify_metrics_test.go new file mode 100644 index 0000000..619f491 --- /dev/null +++ b/cmd/tail-watch/notify_metrics_test.go @@ -0,0 +1,136 @@ +package main + +import ( + "errors" + "path/filepath" + "sync/atomic" + "testing" + "time" + + "github.com/prometheus/client_golang/prometheus/testutil" +) + +func TestNotifyMetricsAfterScriptedRun(t *testing.T) { + t.Parallel() + plain := testSink("plain") + batch := testSink("batch") + batch.Batch = true + h := newHarness(t, harnessOpts{ + sinks: []SinkConfig{plain, batch}, + stateDir: filepath.Join(t.TempDir(), "state"), + tags: nil, + intake: 0, + scripts: map[string][]deliveryResult{"plain": {statusRes(503)}}, + clock: nil, + }) + var failed atomic.Bool + h.n.store.setInject(func() error { + if failed.CompareAndSwap(false, true) { + return errors.New("disk full") + } + return nil + }) + h.start(t) + before := h.clock.timers.Load() + h.n.ObserveTransition(downTransition(1, h.clock.Now())) + eventually(t, func() bool { return h.sinks["plain"].callCount() == 1 }) + h.clock.waitTimers(t, before) + h.clock.Advance(batch.GroupWait) + eventually(t, func() bool { + return h.sinks["plain"].callCount() == 2 && h.sinks["batch"].callCount() == 1 + }) + eventually(t, func() bool { + return testutil.ToFloat64(h.n.m.sentC.WithLabelValues("plain", "target.down")) == 1 + }) + + if n := testutil.ToFloat64(h.n.m.failedC.WithLabelValues("plain", failHTTP5xx)); n != 1 { + t.Fatalf("failed_total = %v", n) + } + if n := testutil.ToFloat64(h.n.m.persistErr.WithLabelValues(sinkState)); n != 1 { + t.Fatalf("persist_errors_total = %v", n) + } + if n := testutil.ToFloat64(h.n.m.queueLenG.WithLabelValues("plain")); n != 0 { + t.Fatalf("queue_len = %v", n) + } + if n := testutil.ToFloat64(h.n.m.openG); n != 1 { + t.Fatalf("open_incidents = %v", n) + } + if n := testutil.CollectAndCount(h.n.m.latencyH); n != 2 { + t.Fatalf("latency series = %d", n) + } + if n := testutil.CollectAndCount(h.n.m.batchSizeH); n != 1 { + t.Fatalf("batch_size series = %d (batched sinks only)", n) + } + fams, err := h.reg.Gather() + if err != nil { + t.Fatal(err) + } + have := map[string]bool{} + for _, f := range fams { + have[f.GetName()] = true + } + for _, name := range []string{ + "tailwatch_notify_sent_total", "tailwatch_notify_failed_total", "tailwatch_notify_queue_len", + "tailwatch_notify_persist_errors_total", "tailwatch_notify_batch_size", + "tailwatch_notify_latency_seconds", "tailwatch_open_incidents", + } { + if !have[name] { + t.Errorf("metric %s not registered", name) + } + } +} + +func TestQueueOverflowCountsQueueFull(t *testing.T) { + t.Parallel() + sc := testSink("a") + sc.QueueSize = 2 + h := newHarness( + t, + harnessOpts{sinks: []SinkConfig{sc}, stateDir: "", tags: nil, intake: 0, scripts: nil, clock: nil}, + ) + h.n.mu.Lock() + for i := range 5 { + h.n.enqueueLocked(newFlappingEvent("evt_"+string(rune('A'+i)), goldenCollector(), goldenTransition())) + } + h.n.mu.Unlock() + if n := testutil.ToFloat64(h.n.m.droppedC.WithLabelValues("a", reasonQueueFull)); n != 3 { + t.Fatalf("dropped queue_full = %v", n) + } + if n := testutil.ToFloat64(h.n.m.queueLenG.WithLabelValues("a")); n != 2 { + t.Fatalf("queue_len = %v", n) + } +} + +func TestSlowCommitDelaysDurableEnqueuesButNotIntake(t *testing.T) { + t.Parallel() + h := newHarness(t, harnessOpts{ + sinks: []SinkConfig{testSink("a")}, + stateDir: filepath.Join(t.TempDir(), "state"), + tags: nil, + intake: 0, + scripts: nil, + clock: nil, + }) + gate := make(chan struct{}) + h.n.store.setInject(func() error { <-gate; return nil }) + h.start(t) + done := make(chan struct{}) + go func() { + for i := range 20 { + h.n.ObserveTransition(downTransition(uint64(i+1), h.clock.Now())) + } + close(done) + }() + select { + case <-done: + case <-time.After(time.Second): + t.Fatal("intake blocked behind a slow commit") + } + time.Sleep(30 * time.Millisecond) + if h.sinks["a"].callCount() != 0 { + t.Fatal("nothing may be sent while the commit is pending") + } + h.n.store.setInject(nil) + close(gate) + eventually(t, func() bool { return h.sinks["a"].callCount() == 1 }) +} diff --git a/cmd/tail-watch/notify_test.go b/cmd/tail-watch/notify_test.go new file mode 100644 index 0000000..7ec2a10 --- /dev/null +++ b/cmd/tail-watch/notify_test.go @@ -0,0 +1,565 @@ +package main + +import ( + "encoding/json" + "testing" + "time" + + "github.com/prometheus/client_golang/prometheus/testutil" +) + +func idsOf(t *testing.T, f *fakeSink) []string { + t.Helper() + var out []string + for _, c := range f.snapshot() { + var body struct { + Events []Event `json:"events"` + } + if err := json.Unmarshal(c.Body, &body); err != nil { + t.Fatal(err) + } + for _, e := range body.Events { + out = append(out, e.ID) + } + } + return out +} + +func TestDownThenRecoveredDelivered(t *testing.T) { + t.Parallel() + h := newHarness( + t, + harnessOpts{ + sinks: []SinkConfig{testSink("a"), testSink("b")}, + stateDir: "", + tags: nil, + intake: 0, + scripts: nil, + clock: nil, + }, + ) + h.start(t) + h.n.ObserveTransition(downTransition(1, h.clock.Now())) + h.n.ObserveTransition(upTransition(2, h.clock.Now())) + for _, name := range []string{"a", "b"} { + f := h.sinks[name] + eventually(t, func() bool { return f.callCount() == 2 }) + got := f.eventTypes(t) + if got[0] != EventDown || got[1] != EventRecovered { + t.Fatalf("%s got %v", name, got) + } + } + if n := testutil.ToFloat64(h.n.m.sentC.WithLabelValues("a", "target.down")); n != 1 { + t.Fatalf("sent down = %v", n) + } +} + +func TestSilentStartupAndSuppression(t *testing.T) { + t.Parallel() + h := newHarness( + t, + harnessOpts{sinks: []SinkConfig{testSink("a")}, stateDir: "", tags: nil, intake: 0, scripts: nil, clock: nil}, + ) + h.start(t) + tr := upTransition(1, h.clock.Now()) + tr.From = StateUnknown + h.n.ObserveTransition(tr) + sup := downTransition(2, h.clock.Now()) + sup.To = StateSuppressed + h.n.ObserveTransition(sup) + close(h.alert.first) + eventually(t, func() bool { return h.sinks["a"].callCount() == 1 }) + if got := h.sinks["a"].eventTypes(t); len(got) != 1 || got[0] != EventStarted { + t.Fatalf("got %v, want only collector.started", got) + } +} + +func TestCollectorStartedOnce(t *testing.T) { + t.Parallel() + h := newHarness( + t, + harnessOpts{sinks: []SinkConfig{testSink("a")}, stateDir: "", tags: nil, intake: 0, scripts: nil, clock: nil}, + ) + h.start(t) + close(h.alert.first) + eventually(t, func() bool { return h.sinks["a"].callCount() == 1 }) + h.clock.Advance(time.Minute) + time.Sleep(30 * time.Millisecond) + if got := h.sinks["a"].callCount(); got != 1 { + t.Fatalf("collector.started emitted %d times", got) + } +} + +func TestRetryScheduleUnderClock(t *testing.T) { + t.Parallel() + h := newHarness(t, harnessOpts{ + sinks: []SinkConfig{testSink("a")}, + scripts: map[string][]deliveryResult{"a": { + statusRes(500), statusRes(500), statusRes(503), + }}, stateDir: "", tags: nil, intake: 0, clock: nil, + }) + h.start(t) + h.n.ObserveTransition(downTransition(1, h.clock.Now())) + f := h.sinks["a"] + for i, wait := range []time.Duration{time.Second, 2 * time.Second, 4 * time.Second} { + before := h.clock.timers.Load() + eventually(t, func() bool { return f.callCount() == i+1 }) + h.clock.waitTimers(t, before) + h.clock.Advance(wait - time.Millisecond) + time.Sleep(20 * time.Millisecond) + if f.callCount() != i+1 { + t.Fatalf("retry %d fired before %s", i+1, wait) + } + h.clock.Advance(time.Millisecond) + } + eventually(t, func() bool { return f.callCount() == 4 }) + calls := f.snapshot() + for i, want := range []time.Duration{time.Second, 2 * time.Second, 4 * time.Second} { + if got := calls[i+1].At.Sub(calls[i].At); got != want { + t.Fatalf("attempt %d delay %s want %s", i+2, got, want) + } + } + if calls[0].ID != calls[3].ID || string(calls[0].Body) != string(calls[3].Body) { + t.Fatal("retries must resend identical bytes") + } + if n := testutil.ToFloat64(h.n.m.failedC.WithLabelValues("a", failHTTP5xx)); n != 3 { + t.Fatalf("failed_total = %v", n) + } +} + +func TestRetryAfterHonoured(t *testing.T) { + t.Parallel() + limited := deliveryResult{ + Class: resultRetry, + Status: 429, + Reason: failHTTP429, + RetryAfter: 30 * time.Second, + Err: nil, + } + h := newHarness(t, harnessOpts{ + sinks: []SinkConfig{ + testSink("a"), + }, + scripts: map[string][]deliveryResult{"a": {limited}}, + stateDir: "", + tags: nil, + intake: 0, + clock: nil, + }) + h.start(t) + before := h.clock.timers.Load() + h.n.ObserveTransition(downTransition(1, h.clock.Now())) + f := h.sinks["a"] + eventually(t, func() bool { return f.callCount() == 1 }) + h.clock.waitTimers(t, before) + h.clock.Advance(29 * time.Second) + time.Sleep(20 * time.Millisecond) + if f.callCount() != 1 { + t.Fatal("retried before Retry-After") + } + h.clock.Advance(time.Second) + eventually(t, func() bool { return f.callCount() == 2 }) +} + +func TestRetryDelayCap(t *testing.T) { + t.Parallel() + want := []int64{1, 2, 4, 8, 16, 32, 64, 128, 256, 300, 300, 300} + for i, w := range want { + if got := retryDelay(i+1, 0.5, 0); got != time.Duration(w)*time.Second { + t.Fatalf("attempt %d: %s want %s", i+1, got, time.Duration(w)*time.Second) + } + } + if got := retryDelay(1, 0.999, 0); got < 1400*time.Millisecond || got > 1500*time.Millisecond { + t.Fatalf("max jitter %s", got) + } + if got := retryDelay(1, 0, 0); got != 500*time.Millisecond { + t.Fatalf("min jitter %s", got) + } + if got := retryDelay(20, 1.4, 0); got != maxRetryDelay { + t.Fatalf("jittered delay must stay capped, got %s", got) + } + if got := retryDelay(1, 0.5, time.Hour); got != maxRetryDelay { + t.Fatalf("Retry-After must be capped, got %s", got) + } +} + +func TestOrderingDownRetriedRecoveryQueued(t *testing.T) { + t.Parallel() + h := newHarness(t, harnessOpts{ + sinks: []SinkConfig{ + testSink("a"), + }, + scripts: map[string][]deliveryResult{"a": {statusRes(503)}}, + stateDir: "", + tags: nil, + intake: 0, + clock: nil, + }) + h.start(t) + before := h.clock.timers.Load() + h.n.ObserveTransition(downTransition(1, h.clock.Now())) + f := h.sinks["a"] + eventually(t, func() bool { return f.callCount() == 1 }) + h.clock.waitTimers(t, before) + h.n.ObserveTransition(upTransition(2, h.clock.Now())) + time.Sleep(30 * time.Millisecond) + if f.callCount() != 1 { + t.Fatal("recovery must wait behind the retrying down") + } + h.clock.Advance(time.Second) + eventually(t, func() bool { return f.callCount() == 3 }) + got := f.eventTypes(t) + if got[0] != EventDown || got[1] != EventDown || got[2] != EventRecovered { + t.Fatalf("order %v", got) + } +} + +func TestBlockedReceiverDoesNotBlockOtherSink(t *testing.T) { + t.Parallel() + h := newHarness( + t, + harnessOpts{ + sinks: []SinkConfig{testSink("slow"), testSink("fast")}, + stateDir: "", + tags: nil, + intake: 0, + scripts: nil, + clock: nil, + }, + ) + release := make(chan struct{}) + h.sinks["slow"].block = release + h.start(t) + h.n.ObserveTransition(downTransition(1, h.clock.Now())) + eventually(t, func() bool { return h.sinks["fast"].callCount() == 1 }) + if h.sinks["slow"].callCount() != 0 { + t.Fatal("slow sink should still be blocked") + } + h.n.ObserveTransition(upTransition(2, h.clock.Now())) + eventually(t, func() bool { return h.sinks["fast"].callCount() == 2 }) + close(release) + eventually(t, func() bool { return h.sinks["slow"].callCount() == 2 }) +} + +func TestIntakeNeverBlocks(t *testing.T) { + t.Parallel() + h := newHarness( + t, + harnessOpts{sinks: []SinkConfig{testSink("a")}, stateDir: "", tags: nil, scripts: nil, clock: nil, intake: 2}, + ) + done := make(chan struct{}) + go func() { + for i := range 50 { + h.n.ObserveTransition(downTransition(uint64(i+1), h.clock.Now())) + } + close(done) + }() + select { + case <-done: + case <-time.After(2 * time.Second): + t.Fatal("ObserveTransition blocked with no coordinator running") + } + if n := testutil.ToFloat64(h.n.m.droppedC.WithLabelValues(sinkIntake, reasonQueueFull)); n != 48 { + t.Fatalf("intake drops = %v", n) + } + h.alert.mu.Lock() + defer h.alert.mu.Unlock() + if !h.alert.dirty["nyc-nas"] { + t.Fatal("dropped transition must mark the target dirty") + } +} + +func TestDroppedRecoveryReconciledFromSnapshot(t *testing.T) { + t.Parallel() + h := newHarness( + t, + harnessOpts{sinks: []SinkConfig{testSink("a")}, stateDir: "", tags: nil, intake: 0, scripts: nil, clock: nil}, + ) + h.start(t) + h.n.ObserveTransition(downTransition(1, h.clock.Now())) + f := h.sinks["a"] + eventually(t, func() bool { return f.callCount() == 1 }) + + snap := alertSnapshot{ + Sequence: 5, + Name: "nyc-nas", + State: StateUp, + At: h.clock.Now(), + Target: nasTarget(), + DownSince: h.clock.Now(), + Flapping: false, + Check: "", + Layer: layerNone, + Err: nil, + Since: time.Time{}, + Failures: 0, + Path: pathInfo{Kind: "", Endpoint: "", DERP: "", Latency: 0}, + Removed: false, + Dirty: false, + } + h.alert.setSnapshot(snap) + h.clock.Advance(reconcileInterval) + eventually(t, func() bool { return f.callCount() == 2 }) + if got := f.eventTypes(t); got[1] != EventRecovered { + t.Fatalf("got %v", got) + } + h.alert.mu.Lock() + acked := h.alert.acked["nyc-nas"] + dirty := h.alert.dirty["nyc-nas"] + h.alert.mu.Unlock() + if acked != 5 || dirty { + t.Fatalf("snapshot not acknowledged: acked=%d dirty=%v", acked, dirty) + } + // A queued transition at or below the reconciled sequence is ignored. + h.n.ObserveTransition(downTransition(4, h.clock.Now())) + time.Sleep(30 * time.Millisecond) + if f.callCount() != 2 { + t.Fatal("stale transition must not open an incident") + } +} + +func TestDroppedDownReconciledFromSnapshot(t *testing.T) { + t.Parallel() + h := newHarness( + t, + harnessOpts{sinks: []SinkConfig{testSink("a")}, stateDir: "", tags: nil, intake: 0, scripts: nil, clock: nil}, + ) + h.start(t) + snap := alertSnapshot{ + Sequence: 3, + Name: "nyc-nas", + State: StateDown, + Check: "tcp:445", + Layer: layerService, + At: h.clock.Now(), + Target: nasTarget(), + DownSince: h.clock.Now(), + Flapping: false, + Err: nil, + Since: time.Time{}, + Failures: 0, + Path: pathInfo{Kind: "", Endpoint: "", DERP: "", Latency: 0}, + Removed: false, + Dirty: false, + } + h.alert.setSnapshot(snap) + h.clock.Advance(reconcileInterval) + f := h.sinks["a"] + eventually(t, func() bool { return f.callCount() == 1 }) + if got := f.eventTypes(t); got[0] != EventDown { + t.Fatalf("got %v", got) + } +} + +func TestNewerDirtyStatePreserved(t *testing.T) { + t.Parallel() + h := newHarness( + t, + harnessOpts{sinks: []SinkConfig{testSink("a")}, stateDir: "", tags: nil, intake: 0, scripts: nil, clock: nil}, + ) + h.start(t) + h.alert.setSnapshot( + alertSnapshot{ + Sequence: 9, + Name: "nyc-nas", + State: StateUp, + Target: nasTarget(), + Flapping: false, + Check: "", + Layer: layerNone, + Err: nil, + Since: time.Time{}, + At: time.Time{}, + Failures: 0, + Path: pathInfo{Kind: "", Endpoint: "", DERP: "", Latency: 0}, + DownSince: time.Time{}, + Removed: false, + Dirty: false, + }) + h.n.ObserveTransition(downTransition(4, h.clock.Now())) + time.Sleep(30 * time.Millisecond) + h.alert.mu.Lock() + dirty := h.alert.dirty["nyc-nas"] + h.alert.mu.Unlock() + if !dirty { + t.Fatal("acknowledging seq 4 must leave the newer seq 9 mark dirty") + } +} + +func TestRemovedTargetClosesIncident(t *testing.T) { + t.Parallel() + h := newHarness( + t, + harnessOpts{sinks: []SinkConfig{testSink("a")}, stateDir: "", tags: nil, intake: 0, scripts: nil, clock: nil}, + ) + h.start(t) + h.n.ObserveTransition(downTransition(1, h.clock.Now())) + f := h.sinks["a"] + eventually(t, func() bool { return f.callCount() == 1 }) + h.alert.setSnapshot(alertSnapshot{ + Sequence: 6, + Name: "nyc-nas", + State: StateDown, + Removed: true, + Target: nasTarget(), + Flapping: false, + Check: "", + Layer: layerNone, + Err: nil, + Since: time.Time{}, + At: time.Time{}, + Failures: 0, + Path: pathInfo{Kind: "", Endpoint: "", DERP: "", Latency: 0}, + DownSince: time.Time{}, + Dirty: false, + }) + h.clock.Advance(reconcileInterval) + eventually(t, func() bool { return f.callCount() == 2 }) + var body struct { + Events []Event `json:"events"` + } + if err := json.Unmarshal(f.snapshot()[1].Body, &body); err != nil { + t.Fatal(err) + } + if body.Events[0].Type != EventRecovered || body.Events[0].Reason != reasonRemoved { + t.Fatalf("got %+v", body.Events[0]) + } +} + +func TestFlappingNoticeDoesNotTouchIncident(t *testing.T) { + t.Parallel() + h := newHarness( + t, + harnessOpts{sinks: []SinkConfig{testSink("a")}, stateDir: "", tags: nil, intake: 0, scripts: nil, clock: nil}, + ) + h.start(t) + h.n.ObserveTransition(downTransition(1, h.clock.Now())) + flap := downTransition(2, h.clock.Now()) + flap.Flapping = true + h.n.ObserveTransition(flap) + f := h.sinks["a"] + eventually(t, func() bool { return f.callCount() == 2 }) + if got := f.eventTypes(t); got[1] != EventFlapping { + t.Fatalf("got %v", got) + } + h.n.mu.Lock() + defer h.n.mu.Unlock() + if h.n.b.openIncidentFor("nyc-nas") == nil { + t.Fatal("flapping notice must leave the incident open") + } +} + +func TestRoutingUsesTags(t *testing.T) { + t.Parallel() + tagged := testSink("tagged") + tagged.Match.Tags = []string{"tag:nas"} + other := testSink("other") + other.Match.Tags = []string{"tag:printer"} + h := newHarness(t, harnessOpts{ + sinks: []SinkConfig{tagged, other}, + tags: func(string) []string { return []string{"tag:nas"} }, stateDir: "", intake: 0, scripts: nil, clock: nil, + }) + h.start(t) + h.n.ObserveTransition(downTransition(1, h.clock.Now())) + eventually(t, func() bool { return h.sinks["tagged"].callCount() == 1 }) + time.Sleep(30 * time.Millisecond) + if h.sinks["other"].callCount() != 0 { + t.Fatal("tag filter should exclude the printer sink") + } +} + +func TestLateFlagFixedAtFirstAttempt(t *testing.T) { + t.Parallel() + h := newHarness(t, harnessOpts{ + sinks: []SinkConfig{ + testSink("a"), + }, + scripts: map[string][]deliveryResult{"a": {statusRes(503)}}, + stateDir: "", + tags: nil, + intake: 0, + clock: nil, + }) + tr := downTransition(1, h.clock.Now()) + h.clock.Advance(5 * time.Minute) + h.start(t) + before := h.clock.timers.Load() + h.n.ObserveTransition(tr) + f := h.sinks["a"] + eventually(t, func() bool { return f.callCount() == 1 }) + h.clock.waitTimers(t, before) + h.clock.Advance(time.Second) + eventually(t, func() bool { return f.callCount() == 2 }) + calls := f.snapshot() + var body struct { + Events []Event `json:"events"` + } + if err := json.Unmarshal(calls[0].Body, &body); err != nil { + t.Fatal(err) + } + if !body.Events[0].Late { + t.Fatal("event first attempted 5 minutes after creation must be late") + } + if string(calls[0].Body) != string(calls[1].Body) { + t.Fatal("retry body changed") + } + if time.Time(body.Events[0].Time).Equal(h.clock.Now()) { + t.Fatal("late event must keep its original time") + } +} + +func TestExpiredWhileRetrying(t *testing.T) { + t.Parallel() + sc := testSink("a") + sc.MaxAge = time.Minute + fail := make([]deliveryResult, 20) + for i := range fail { + fail[i] = statusRes(503) + } + h := newHarness( + t, + harnessOpts{ + sinks: []SinkConfig{sc}, + scripts: map[string][]deliveryResult{"a": fail}, + stateDir: "", + tags: nil, + intake: 0, + clock: nil, + }, + ) + h.start(t) + before := h.clock.timers.Load() + h.n.ObserveTransition(downTransition(1, h.clock.Now())) + eventually(t, func() bool { return h.sinks["a"].callCount() == 1 }) + h.clock.waitTimers(t, before) + h.clock.Advance(2 * time.Minute) + eventually(t, func() bool { + return testutil.ToFloat64(h.n.m.droppedC.WithLabelValues("a", reasonExpired)) == 1 + }) +} + +func TestPermanentFailureDropsOnce(t *testing.T) { + t.Parallel() + h := newHarness(t, harnessOpts{ + sinks: []SinkConfig{ + testSink("a"), + }, + scripts: map[string][]deliveryResult{"a": {statusRes(400)}}, + stateDir: "", + tags: nil, + intake: 0, + clock: nil, + }) + h.start(t) + h.n.ObserveTransition(downTransition(1, h.clock.Now())) + eventually(t, func() bool { + return testutil.ToFloat64(h.n.m.droppedC.WithLabelValues("a", reasonPermanent)) == 1 + }) + time.Sleep(30 * time.Millisecond) + if h.sinks["a"].callCount() != 1 { + t.Fatal("permanent failure must not retry") + } + if testutil.ToFloat64(h.n.m.failedC.WithLabelValues("a", failHTTP4xx)) != 1 { + t.Fatal("permanent failure should count as failed") + } +} diff --git a/cmd/tail-watch/notifytest.go b/cmd/tail-watch/notifytest.go new file mode 100644 index 0000000..7898512 --- /dev/null +++ b/cmd/tail-watch/notifytest.go @@ -0,0 +1,181 @@ +package main + +import ( + "context" + "errors" + "flag" + "fmt" + "io" + "net/netip" + "os" + "strings" + "sync" + "time" + + "tailscale.com/tstime" +) + +const ( + defaultNotifyTimeout = 30 * time.Second + identityTimeout = 3 * time.Second +) + +// notifyTestDeps are the seams of notify-test. Nil fields use the real thing. +type notifyTestDeps struct { + identity func(ctx context.Context) (CollectorInfo, error) + env netEnv + clock tstime.Clock +} + +func runNotifyTest(args []string, stdout, stderr io.Writer) int { + return runNotifyTestWith(args, stdout, stderr, notifyTestDeps{identity: nil, env: nil, clock: nil}) +} + +// runNotifyTestWith sends one test event through each selected sink's real +// signing, rendering and retry code. It bypasses routing filters, queues and +// state_dir, and works when tailscaled is unreachable. +func runNotifyTestWith(args []string, stdout, stderr io.Writer, deps notifyTestDeps) int { + fs := flag.NewFlagSet("notify-test", flag.ContinueOnError) + fs.SetOutput(stderr) + sinkFlag := fs.String("sink", "", "send only to the sink with this name (default: every sink)") + configFlag := fs.String("config", "", "config file path (overrides TAIL_WATCH_CONFIG)") + socketFlag := fs.String("socket", "", "tailscaled LocalAPI socket") + timeoutFlag := fs.Duration("timeout", defaultNotifyTimeout, "give up on a sink after this long") + if err := fs.Parse(args); err != nil { + if errors.Is(err, flag.ErrHelp) { + return 0 + } + return exitUsage + } + cfg, err := loadConfig(resolveConfigPath(*configFlag, os.Getenv("TAIL_WATCH_CONFIG"))) + if err != nil { + _, _ = fmt.Fprintf(stderr, "%v\n", err) + return 1 + } + selected, err := selectSinks(cfg.Sinks, *sinkFlag) + if err != nil { + _, _ = fmt.Fprintf(stderr, "%v\n", err) + return exitUsage + } + if deps.clock == nil { + deps.clock = tstime.StdClock{} + } + if deps.env == nil { + deps.env = newKernelEnv(socketPath(*socketFlag)) + } + if deps.identity == nil { + deps.identity = func(ctx context.Context) (CollectorInfo, error) { return localIdentity(ctx, deps.env) } + } + col := collectorOrHostname(deps.identity) + ctx, cancel := context.WithTimeout(context.Background(), *timeoutFlag) + defer cancel() + id, err := newID(prefixEvent, deps.clock.Now(), nil) + if err != nil { + _, _ = fmt.Fprintf(stderr, "%v\n", err) + return 1 + } + ev := newTestEvent(id, col, deps.clock.Now()) + lines := make([]string, len(selected)) + oks := make([]bool, len(selected)) + var wg sync.WaitGroup + for i, sc := range selected { + wg.Go(func() { + lines[i], oks[i] = sendTest(ctx, sc, col, ev, deps) + }) + } + wg.Wait() + code := 0 + for i, line := range lines { + _, _ = fmt.Fprintln(stdout, line) + if !oks[i] { + code = 1 + } + } + return code +} + +func selectSinks(all []SinkConfig, name string) ([]SinkConfig, error) { + if len(all) == 0 { + return nil, errors.New("no sinks configured") + } + if name == "" { + return all, nil + } + names := make([]string, len(all)) + for i, sc := range all { + if sc.Name == name { + return []SinkConfig{sc}, nil + } + names[i] = sc.Name + } + return nil, fmt.Errorf("unknown sink %q; configured sinks: %s", name, strings.Join(names, ", ")) +} + +// collectorOrHostname asks tailscaled for the collector identity and falls back +// to the OS hostname, without an address, when it is unreachable. +func collectorOrHostname(identity func(context.Context) (CollectorInfo, error)) CollectorInfo { + ctx, cancel := context.WithTimeout(context.Background(), identityTimeout) + defer cancel() + if col, err := identity(ctx); err == nil && col.Name != "" { + return col + } + host, err := os.Hostname() + if err != nil { + host = "unknown" + } + return CollectorInfo{Name: host, TailscaleIP: ""} +} + +func localIdentity(ctx context.Context, env netEnv) (CollectorInfo, error) { + st, err := env.LocalClient().Status(ctx) + if err != nil { + return CollectorInfo{}, err + } + return identityOf(st.Self.HostName, st.TailscaleIPs), nil +} + +func identityOf(host string, ips []netip.Addr) CollectorInfo { + col := CollectorInfo{Name: host, TailscaleIP: ""} + for _, ip := range ips { + if ip.Is4() { + col.TailscaleIP = ip.String() + return col + } + } + if len(ips) > 0 { + col.TailscaleIP = ips[0].String() + } + return col +} + +// sendTest delivers ev through one real sink and returns its report line. +func sendTest(ctx context.Context, sc SinkConfig, col CollectorInfo, ev Event, deps notifyTestDeps) (string, bool) { + prefix := fmt.Sprintf("%s %s", sc.Name, sc.Type) + if err := sc.checkSecrets(); err != nil { + return fmt.Sprintf("%s error: %v", prefix, err), false + } + s, err := newSinkWithEnv(sc, col, deps.env) + if err != nil { + return fmt.Sprintf("%s error: %v", prefix, err), false + } + id := ev.ID + if sc.Type == sinkSlack || sc.Batch { + if id, err = newID(prefixBatch, deps.clock.Now(), nil); err != nil { + return fmt.Sprintf("%s error: %v", prefix, err), false + } + } + body, err := s.Render([]Event{ev}, id) + if err != nil { + return fmt.Sprintf("%s error: %v", prefix, err), false + } + loop := retryLoop{clock: deps.clock, jitter: func() float64 { return jitterFloor }, sink: s, last: time.Time{}} + out := loop.run(ctx, body, id, nil, nil) + switch out.Kind { + case outcomeDelivered: + return fmt.Sprintf("%s %d", prefix, out.Result.Status), true + case outcomeCancelled: + return fmt.Sprintf("%s error: timed out after retries (last: %s)", prefix, out.Result), false + case outcomePermanent, outcomeExpired: + } + return fmt.Sprintf("%s %s", prefix, out.Result), false +} diff --git a/cmd/tail-watch/notifytest_test.go b/cmd/tail-watch/notifytest_test.go new file mode 100644 index 0000000..ceb12db --- /dev/null +++ b/cmd/tail-watch/notifytest_test.go @@ -0,0 +1,179 @@ +package main + +import ( + "context" + "errors" + "net/http" + "strings" + "testing" + "time" +) + +func notifyTestConfig(t *testing.T, sinks string) string { + t.Helper() + return writeCfg(t, "targets:\n - name: nas\n addr: 10.0.0.1\n checks: [icmp]\n"+sinks) +} + +func staticIdentity(context.Context) (CollectorInfo, error) { + return CollectorInfo{Name: "monitor-nyc", TailscaleIP: "100.101.102.103"}, nil +} + +func TestNotifyTestAllSinksSucceed(t *testing.T) { + a := newReceiver(t) + b := newReceiver(t) + t.Setenv("TW_NT_A", a.srv.URL) + t.Setenv("TW_NT_B", b.srv.URL) + path := notifyTestConfig(t, ` +sinks: + - {name: hook, type: webhook, url_env: TW_NT_A, min_severity: page, events: [target.down]} + - {name: chat, type: slack, url_env: TW_NT_B, match: {sites: [nowhere]}} +`) + var out, errOut strings.Builder + code := runNotifyTestWith( + []string{"--config", path}, + &out, + &errOut, + notifyTestDeps{identity: staticIdentity, env: nil, clock: nil}, + ) + if code != 0 { + t.Fatalf("exit %d\n%s\n%s", code, out.String(), errOut.String()) + } + for _, want := range []string{"hook webhook 200", "chat slack 200"} { + if !strings.Contains(out.String(), want) { + t.Fatalf("missing %q in\n%s", want, out.String()) + } + } + if a.count() != 1 || b.count() != 1 { + t.Fatalf("filters must be bypassed: %d %d", a.count(), b.count()) + } + if !strings.Contains(string(a.request(0).Body), `"type":"test"`) { + t.Fatalf("body %s", a.request(0).Body) + } +} + +func TestNotifyTestReportsFailingSink(t *testing.T) { + good := newReceiver(t) + bad := newReceiver(t, http.StatusUnauthorized) + t.Setenv("TW_NT_A", good.srv.URL) + t.Setenv("TW_NT_B", bad.srv.URL) + path := notifyTestConfig(t, ` +sinks: + - {name: hook, type: webhook, url_env: TW_NT_A} + - {name: strict, type: webhook, url_env: TW_NT_B} +`) + var out, errOut strings.Builder + code := runNotifyTestWith( + []string{"--config", path}, + &out, + &errOut, + notifyTestDeps{identity: staticIdentity, env: nil, clock: nil}, + ) + if code != 1 { + t.Fatalf("exit %d\n%s", code, out.String()) + } + if !strings.Contains(out.String(), "strict webhook 401") || !strings.Contains(out.String(), "hook webhook 200") { + t.Fatalf("output\n%s", out.String()) + } +} + +func TestNotifyTestSelectsOneSink(t *testing.T) { + a := newReceiver(t) + b := newReceiver(t) + t.Setenv("TW_NT_A", a.srv.URL) + t.Setenv("TW_NT_B", b.srv.URL) + path := notifyTestConfig(t, ` +sinks: + - {name: one, type: webhook, url_env: TW_NT_A} + - {name: two, type: webhook, url_env: TW_NT_B} +`) + var out, errOut strings.Builder + if code := runNotifyTestWith([]string{"--config", path, "--sink", "two"}, &out, &errOut, + notifyTestDeps{identity: staticIdentity, env: nil, clock: nil}); code != 0 { + t.Fatalf("exit %d %s", code, errOut.String()) + } + if a.count() != 0 || b.count() != 1 { + t.Fatalf("counts %d %d", a.count(), b.count()) + } +} + +func TestNotifyTestUnknownSinkListsNames(t *testing.T) { + t.Setenv("TW_NT_A", "http://127.0.0.1:1") + path := notifyTestConfig(t, ` +sinks: + - {name: one, type: webhook, url_env: TW_NT_A} + - {name: two, type: webhook, url_env: TW_NT_A} +`) + var out, errOut strings.Builder + code := runNotifyTestWith([]string{"--config", path, "--sink", "nope"}, &out, &errOut, + notifyTestDeps{identity: staticIdentity, env: nil, clock: nil}) + if code == 0 { + t.Fatal("unknown sink must fail") + } + if !strings.Contains(errOut.String(), "one") || !strings.Contains(errOut.String(), "two") { + t.Fatalf("stderr %q must list the configured sinks", errOut.String()) + } +} + +func TestNotifyTestWorksWithoutTailscaled(t *testing.T) { + rc := newReceiver(t) + t.Setenv("TW_NT_A", rc.srv.URL) + path := notifyTestConfig(t, "sinks:\n - {name: hook, type: webhook, url_env: TW_NT_A}\n") + var out, errOut strings.Builder + down := func(context.Context) (CollectorInfo, error) { return CollectorInfo{}, errors.New("no socket") } + if code := runNotifyTestWith( + []string{"--config", path}, + &out, + &errOut, + notifyTestDeps{identity: down, env: nil, clock: nil}, + ); code != 0 { + t.Fatalf("exit %d %s", code, errOut.String()) + } + body := string(rc.request(0).Body) + if !strings.Contains(body, `"collector":{"name":"`) || strings.Contains(body, "tailscale_ip") { + t.Fatalf("hostname fallback expected, body %s", body) + } +} + +func TestNotifyTestUnsetSecretIsReported(t *testing.T) { + t.Parallel() + path := notifyTestConfig(t, "sinks:\n - {name: hook, type: webhook, url_env: TW_NT_NEVER_SET}\n") + var out, errOut strings.Builder + code := runNotifyTestWith( + []string{"--config", path}, + &out, + &errOut, + notifyTestDeps{identity: staticIdentity, env: nil, clock: nil}, + ) + if code != 1 || !strings.Contains(out.String(), "TW_NT_NEVER_SET") { + t.Fatalf("exit %d\n%s", code, out.String()) + } +} + +func TestNotifyTestTimeoutStopsRetrying(t *testing.T) { + rc := newReceiver(t, 503, 503, 503, 503, 503, 503) + t.Setenv("TW_NT_A", rc.srv.URL) + path := notifyTestConfig(t, "sinks:\n - {name: flaky, type: webhook, url_env: TW_NT_A}\n") + var out, errOut strings.Builder + start := time.Now() + code := runNotifyTestWith([]string{"--config", path, "--timeout", "300ms"}, &out, &errOut, + notifyTestDeps{identity: staticIdentity, env: nil, clock: nil}) + if code != 1 || time.Since(start) > 3*time.Second { + t.Fatalf("exit %d after %s\n%s", code, time.Since(start), out.String()) + } + if !strings.Contains(out.String(), "flaky webhook") { + t.Fatalf("output %q", out.String()) + } +} + +func TestNotifyTestHelpPrintsFlags(t *testing.T) { + t.Parallel() + var out, errOut strings.Builder + if code := run([]string{"notify-test", "--help"}, &out, &errOut); code != 0 { + t.Fatalf("exit %d", code) + } + for _, flagName := range []string{"-sink", "-config", "-timeout"} { + if !strings.Contains(errOut.String(), flagName) { + t.Fatalf("help lacks %s:\n%s", flagName, errOut.String()) + } + } +} diff --git a/cmd/tail-watch/presence_test.go b/cmd/tail-watch/presence_test.go index f404788..1fd1811 100644 --- a/cmd/tail-watch/presence_test.go +++ b/cmd/tail-watch/presence_test.go @@ -387,3 +387,20 @@ func nycNodeUnsetOnline() *tailcfg.Node { n.Online = nil return n } + +func TestTagsForIgnoresLocalTargets(t *testing.T) { + t.Parallel() + remote := alertTarget("nyc-nas", "icmp") + remote.Addr = "10.20.0.50" + local := alertTarget("lab-nas", "icmp") + local.Addr = "10.20.0.50" + local.Network = NetworkLocal + _, pres := newTestClassifier(t, new(fakeEnv), remote, local) + pres.apply(ipn.Notify{InitialStatus: presenceStatus()}) + if got := pres.TagsFor("nyc-nas"); len(got) != 1 || got[0] != "tag:router" { + t.Fatalf("tailnet target tags %v", got) + } + if got := pres.TagsFor("lab-nas"); got != nil { + t.Fatalf("a local target must not inherit tailnet tags, got %v", got) + } +} diff --git a/cmd/tail-watch/queue.go b/cmd/tail-watch/queue.go new file mode 100644 index 0000000..256c40c --- /dev/null +++ b/cmd/tail-watch/queue.go @@ -0,0 +1,347 @@ +package main + +import ( + "encoding/json" + "errors" + "fmt" + "io/fs" + "log/slog" + "os" + "path/filepath" + "slices" + "sync" + "time" +) + +// memQueue is one sink's bounded FIFO of delivery intents. When it is full the +// oldest event that has not been attempted is dropped to make room. +type memQueue struct { + limit int + items []*queued +} + +func newMemQueue(limit int) *memQueue { + return &memQueue{limit: max(limit, 1), items: nil} +} + +// Len is the number of queued events, including those in flight. +func (q *memQueue) Len() int { return len(q.items) } + +func (q *memQueue) push(it *queued) { + q.items = append(q.items, it) +} + +// Peek returns up to n events from the head without removing them. +func (q *memQueue) Peek(n int) []*queued { + if n > len(q.items) { + n = len(q.items) + } + return append([]*queued(nil), q.items[:n]...) +} + +// Ack removes the events with the given sequence numbers. +func (q *memQueue) Ack(seqs ...uint64) { + for _, s := range seqs { + q.Remove(s) + } +} + +// Get returns the event with sequence seq, or nil. +func (q *memQueue) Get(seq uint64) *queued { + for _, it := range q.items { + if it.Seq == seq { + return it + } + } + return nil +} + +// Remove deletes the event with sequence seq, if present. +func (q *memQueue) Remove(seq uint64) { + for i, it := range q.items { + if it.Seq == seq { + q.items = append(q.items[:i], q.items[i+1:]...) + return + } + } +} + +func (q *memQueue) oldestPending() *queued { + for _, it := range q.items { + if !it.Attempted { + return it + } + } + return nil +} + +func (q *memQueue) findRecovery(incidentID string) *queued { + for _, it := range q.items { + if it.IncidentID == incidentID && it.Event.Type == EventRecovered { + return it + } + } + return nil +} + +const ( + stateFileName = "notification-state.json" + stateVersion = 1 + stateDirMode = 0o700 + stateFileMode = 0o600 +) + +// notifyState is the versioned on-disk snapshot: incident records, per-sink +// delivery state, sequence allocation and the pending events of disk sinks. It +// never holds URLs, secrets or headers. +type notifyState struct { + Version int `json:"version"` + NextSeq uint64 `json:"next_seq"` + Incidents []*incident `json:"incidents"` + Queues map[string][]*queued `json:"queues,omitempty"` + Units []*unit `json:"units,omitempty"` +} + +// stateStore writes and reads the snapshot under one directory. +type stateStore struct { + dir string + path string + injectMu sync.Mutex + inject func() error +} + +// setInject makes commits call fn first and fail when it errors. Tests use it. +func (s *stateStore) setInject(fn func() error) { + s.injectMu.Lock() + s.inject = fn + s.injectMu.Unlock() +} + +func newStateStore(dir string, log *slog.Logger) (*stateStore, error) { + if err := os.MkdirAll(dir, stateDirMode); err != nil { + return nil, fmt.Errorf("state_dir %s: %w", dir, err) + } + if fi, err := os.Stat(dir); err == nil && fi.Mode().Perm()&0o077 != 0 { + log.Warn("state_dir is readable by group or others", "dir", dir, "mode", fi.Mode().Perm().String()) + } + s := &stateStore{dir: dir, path: filepath.Join(dir, stateFileName), injectMu: sync.Mutex{}, inject: nil} + stale, _ := filepath.Glob(filepath.Join(dir, stateFileName+".*.tmp")) + for _, f := range stale { + _ = os.Remove(f) + } + return s, nil +} + +// load reads the committed snapshot. A missing file is an empty state; a file +// that cannot be decoded is an error that leaves the file untouched. +func (s *stateStore) load() (*notifyState, error) { + b, err := os.ReadFile(s.path) + if errors.Is(err, fs.ErrNotExist) { + return nil, nil //nolint:nilnil // No snapshot yet. + } + if err != nil { + return nil, fmt.Errorf("read %s: %w", s.path, err) + } + var st notifyState + if unmarshalErr := json.Unmarshal(b, &st); unmarshalErr != nil { + return nil, fmt.Errorf( + "notification state %s is corrupt (%w); repair or move the file aside and restart, "+ + "but note open incidents are forgotten if it is removed", s.path, unmarshalErr) + } + if st.Version != stateVersion { + return nil, fmt.Errorf("notification state %s has unsupported version %d", s.path, st.Version) + } + return &st, nil +} + +// commit atomically replaces the snapshot: write a temporary file, fsync it, +// rename it over the snapshot, then fsync the directory. +func (s *stateStore) commit(st *notifyState) error { + s.injectMu.Lock() + inject := s.inject + s.injectMu.Unlock() + if inject != nil { + if err := inject(); err != nil { + return err + } + } + data, err := json.Marshal(st) + if err != nil { + return err + } + tmp, err := os.CreateTemp(s.dir, stateFileName+".*.tmp") + if err != nil { + return err + } + name := tmp.Name() + if chmodErr := os.Chmod(name, stateFileMode); chmodErr != nil { + _ = tmp.Close() + _ = os.Remove(name) + return chmodErr + } + if _, writeErr := tmp.Write(data); writeErr != nil { + _ = tmp.Close() + _ = os.Remove(name) + return writeErr + } + if syncErr := tmp.Sync(); syncErr != nil { + _ = tmp.Close() + _ = os.Remove(name) + return syncErr + } + if closeErr := tmp.Close(); closeErr != nil { + _ = os.Remove(name) + return closeErr + } + if renameErr := os.Rename(name, s.path); renameErr != nil { + _ = os.Remove(name) + return renameErr + } + d, err := os.Open(s.dir) + if err != nil { + return err + } + defer d.Close() + return d.Sync() +} + +// snapshot captures the book for persistence. Memory sinks keep no pending +// payloads on disk, only the incident delivery state. +func (b *book) snapshot() *notifyState { + st := ¬ifyState{ + Version: stateVersion, NextSeq: b.nextSeq, Incidents: nil, + Queues: map[string][]*queued{}, Units: nil, + } + for _, id := range slices.Sorted(mapsKeys(b.incidents)) { + st.Incidents = append(st.Incidents, b.incidents[id]) + } + for name, cfg := range b.sinks { + if cfg.Queue == queueDisk { + st.Queues[name] = append([]*queued{}, b.queues[name].items...) + } + } + for _, key := range slices.Sorted(mapsKeys(b.units)) { + if u := b.units[key]; b.sinks[u.Sink].Queue == queueDisk { + st.Units = append(st.Units, u) + } + } + return st +} + +// restore replays a committed snapshot: incidents first, then the disk queues, +// then the policies that apply at startup. Memory-sink payloads are gone, so +// they are marked dropped with reason restart. Expiry and queue limits update +// delivery state in the same pass. now decides expiry. +func (b *book) restore(st *notifyState, now time.Time) { + b.nextSeq = max(st.NextSeq, 1) + b.restoreIncidents(st.Incidents) + b.restoreQueues(st.Queues) + for _, u := range st.Units { + if q := b.queues[u.Sink]; q != nil { + b.units[u.key()] = u + } + } + b.clearOrphanAttempts() + b.loseMemoryIntents() + for _, name := range slices.Sorted(mapsKeys(b.sinks)) { + if b.sinks[name].Queue != queueDisk { + continue + } + b.trim(name) + b.expireStale(name, now) + b.expireUnits(name, now) + } + for _, id := range slices.Sorted(mapsKeys(b.incidents)) { + b.gc(b.incidents[id]) + } + for name := range b.queues { + b.refreshQueueLen(name) + } +} + +func (b *book) restoreIncidents(incs []*incident) { + for _, inc := range incs { + if inc.Sinks == nil { + inc.Sinks = map[string]*sinkDelivery{} + } + b.incidents[inc.ID] = inc + if inc.Status == incidentOpen { + b.open[inc.Target.Name] = inc.ID + } + } +} + +// restoreQueues reloads disk-sink queues. A queue whose sink is gone, or is no +// longer a disk sink, is lost and its events are dropped with reason restart. +func (b *book) restoreQueues(queues map[string][]*queued) { + for _, name := range slices.Sorted(mapsKeys(queues)) { + cfg, ok := b.sinks[name] + if !ok || cfg.Queue != queueDisk { + for _, it := range queues[name] { + b.dropItem(name, it, reasonRestart) + } + continue + } + for _, it := range queues[name] { + b.queues[name].push(it) + } + } +} + +// clearOrphanAttempts forgets an attempted flag whose unit did not survive. +func (b *book) clearOrphanAttempts() { + for name, q := range b.queues { + for _, it := range q.items { + if it.Attempted && b.units[unitKey(name, it.UnitID)] == nil { + it.Attempted, it.UnitID = false, "" + } + } + } +} + +// loseMemoryIntents marks every delivery whose sink keeps no pending payloads +// across restarts as dropped. Recoveries go first so each lost event counts once. +func (b *book) loseMemoryIntents() { + for _, id := range slices.Sorted(mapsKeys(b.incidents)) { + inc := b.incidents[id] + for name, d := range inc.Sinks { + if cfg, ok := b.sinks[name]; ok && cfg.Queue == queueDisk { + continue + } + if inc.Recovery != nil && (d.Rec == phasePending || d.Rec == phaseAttempted) { + b.dropItem( + name, + &queued{Seq: 0, IncidentID: inc.ID, Event: *inc.Recovery, Attempted: false, UnitID: ""}, + reasonRestart, + ) + } + if d.Down == phasePending || d.Down == phaseAttempted { + b.dropItem( + name, + &queued{Seq: 0, IncidentID: inc.ID, Event: inc.Down, Attempted: false, UnitID: ""}, + reasonRestart, + ) + } + } + } +} + +func (b *book) trim(sink string) { + q := b.queues[sink] + for q.Len() > q.limit { + victim := q.oldestPending() + if victim == nil { + return + } + q.Remove(victim.Seq) + b.dropItem(sink, victim, reasonQueueFull) + } +} + +func (b *book) expireUnits(sink string, now time.Time) { + for _, u := range b.sinkUnits(sink) { + if b.unitExpired(u, now) { + b.dropUnit(u, reasonExpired) + } + } +} diff --git a/cmd/tail-watch/queue_test.go b/cmd/tail-watch/queue_test.go new file mode 100644 index 0000000..81151cf --- /dev/null +++ b/cmd/tail-watch/queue_test.go @@ -0,0 +1,619 @@ +package main + +import ( + "bytes" + "encoding/json" + "errors" + "log/slog" + "os" + "path/filepath" + "strings" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/prometheus/client_golang/prometheus" + "github.com/prometheus/client_golang/prometheus/testutil" +) + +func discardLog() *slog.Logger { return slog.New(slog.DiscardHandler) } + +func testBook(sinks ...SinkConfig) (*book, *notifyMetrics) { + m := newNotifyMetrics(prometheus.NewRegistry(), true) + return newBook(discardLog(), m, sinks), m +} + +func plainEvent(id string, at time.Time) Event { + return baseEvent(id, EventFlapping, SeverityPage, at, goldenCollector()) +} + +func TestMemQueueOrderAndOverflow(t *testing.T) { + t.Parallel() + sc := testSink("a") + sc.QueueSize = 3 + b, m := testBook(sc) + for i := range 5 { + b.push( + "a", + &queued{ + Seq: b.seq(), + Event: plainEvent("evt_"+string(rune('A'+i)), testNow()), + IncidentID: "", + Attempted: false, + UnitID: "", + }, + ) + } + q := b.queues["a"] + if q.Len() != 3 { + t.Fatalf("len %d", q.Len()) + } + head := q.Peek(10) + if head[0].Event.ID != "evt_C" || head[2].Event.ID != "evt_E" { + t.Fatalf("drop-oldest broke ordering: %s..%s", head[0].Event.ID, head[2].Event.ID) + } + if n := testutil.ToFloat64(m.droppedC.WithLabelValues("a", reasonQueueFull)); n != 2 { + t.Fatalf("queue_full = %v", n) + } + q.Ack(head[0].Seq) + if q.Len() != 2 || q.Peek(1)[0].Event.ID != "evt_D" { + t.Fatal("ack did not remove the head") + } +} + +func TestOverflowSparesInFlight(t *testing.T) { + t.Parallel() + sc := testSink("a") + sc.QueueSize = 2 + b, _ := testBook(sc) + first := &queued{ + Seq: b.seq(), + Event: plainEvent("evt_A", testNow()), + IncidentID: "", + Attempted: false, + UnitID: "", + } + second := &queued{ + Seq: b.seq(), + Event: plainEvent("evt_B", testNow()), + IncidentID: "", + Attempted: false, + UnitID: "", + } + b.push("a", first) + b.push("a", second) + b.beginUnit("a", "evt_A", []*queued{first}, []byte("x")) + b.push( + "a", + &queued{ + Seq: b.seq(), + Event: plainEvent("evt_C", testNow()), + IncidentID: "", + Attempted: false, + UnitID: "", + }, + ) + ids := []string{} + for _, it := range b.queues["a"].items { + ids = append(ids, it.Event.ID) + } + if strings.Join(ids, ",") != "evt_A,evt_C" { + t.Fatalf("in-flight event must survive overflow, got %v", ids) + } +} + +func downFor(id string) Event { + tr := goldenTransition() + tr.Target.Name = "nas" + return newDownEvent(id, goldenCollector(), tr) +} + +func recFor(b *book) Event { + const incID = "evt_D1" + return newRecoveredEvent("evt_R"+incID, goldenCollector(), b.incidents[incID], testNow().Add(time.Minute), nil, "") +} + +func TestNeverAttemptedDownDropTakesRecovery(t *testing.T) { + t.Parallel() + b, m := testBook(testSink("a")) + inc := b.openIncident( + downFor("evt_D1"), + targetDef{ + Name: "nas", + Addr: "", + Site: "", + Parent: "", + Severity: SeverityPage, + Implicit: false, + StableID: "", + NodeID: 0, + }, + testNow(), + []string{"a"}, + ) + b.closeIncident(inc, recFor(b)) + q := b.queues["a"] + if q.Len() != 2 { + t.Fatalf("down and recovery should be queued, got %d", q.Len()) + } + down := q.Peek(1)[0] + q.Remove(down.Seq) + b.dropItem("a", down, reasonQueueFull) + if q.Len() != 0 { + t.Fatal("recovery of a never-attempted dropped down must be discarded") + } + if _, ok := b.incidents["evt_D1"]; ok { + t.Fatal("incident should be garbage collected once every intent is terminal") + } + if n := testutil.ToFloat64(m.droppedC.WithLabelValues("a", reasonQueueFull)); n != 1 { + t.Fatalf("only the down drop is counted, got %v", n) + } +} + +func TestAttemptedDownKeepsRecoveryEligible(t *testing.T) { + t.Parallel() + for _, reason := range []string{reasonExpired, reasonPermanent} { + b, _ := testBook(testSink("a")) + inc := b.openIncident( + downFor("evt_D1"), + targetDef{ + Name: "nas", + Addr: "", + Site: "", + Parent: "", + Severity: SeverityPage, + Implicit: false, + StableID: "", + NodeID: 0, + }, + testNow(), + []string{"a"}, + ) + down := b.queues["a"].Peek(1)[0] + u := b.beginUnit("a", down.Event.ID, []*queued{down}, []byte("body")) + b.closeIncident(inc, recFor(b)) + b.dropUnit(u, reason) + d := b.incidents["evt_D1"].Sinks["a"] + if d.Down != phaseDropped || !d.DownTried || d.Rec != phasePending { + t.Fatalf("%s: down=%s tried=%v rec=%s", reason, d.Down, d.DownTried, d.Rec) + } + if q := b.queues["a"]; q.Len() != 1 || q.Peek(1)[0].Event.Type != EventRecovered { + t.Fatalf("%s: recovery must remain queued", reason) + } + } +} + +func TestRecoveryAfterDropStillEligibleWhenAttempted(t *testing.T) { + t.Parallel() + b, _ := testBook(testSink("a")) + inc := b.openIncident( + downFor("evt_D1"), + targetDef{ + Name: "nas", + Addr: "", + Site: "", + Parent: "", + Severity: SeverityPage, + Implicit: false, + StableID: "", + NodeID: 0, + }, + testNow(), + []string{"a"}, + ) + down := b.queues["a"].Peek(1)[0] + u := b.beginUnit("a", down.Event.ID, []*queued{down}, []byte("body")) + b.dropUnit(u, reasonExpired) + b.closeIncident(inc, recFor(b)) + if d := b.incidents["evt_D1"].Sinks["a"]; d.Rec != phasePending { + t.Fatalf("recovery after an attempted-then-dropped down must be pending, got %s", d.Rec) + } +} + +func TestRecoveryIneligibleAfterUnattemptedDrop(t *testing.T) { + t.Parallel() + b, _ := testBook(testSink("a")) + inc := b.openIncident( + downFor("evt_D1"), + targetDef{ + Name: "nas", + Addr: "", + Site: "", + Parent: "", + Severity: SeverityPage, + Implicit: false, + StableID: "", + NodeID: 0, + }, + testNow(), + []string{"a"}, + ) + down := b.queues["a"].Peek(1)[0] + b.queues["a"].Remove(down.Seq) + b.dropItem("a", down, reasonExpired) + b.closeIncident(inc, recFor(b)) + if b.queues["a"].Len() != 0 { + t.Fatal("no orphan recovery may be queued") + } +} + +func TestFilteredSinkNeverGetsRecovery(t *testing.T) { + t.Parallel() + b, _ := testBook(testSink("a"), testSink("b")) + inc := b.openIncident( + downFor("evt_D1"), + targetDef{ + Name: "nas", + Addr: "", + Site: "", + Parent: "", + Severity: SeverityPage, + Implicit: false, + StableID: "", + NodeID: 0, + }, + testNow(), + []string{"a"}, + ) + b.closeIncident(inc, recFor(b)) + if b.queues["b"].Len() != 0 { + t.Fatal("sink filtered out of the down must not see the recovery") + } + if b.queues["a"].Len() != 2 { + t.Fatal("selected sink must hold down and recovery") + } +} + +func TestRecoveryBypassesFiltersAndSkipsRemovedSink(t *testing.T) { + t.Parallel() + b, _ := testBook(testSink("a")) + inc := b.openIncident( + downFor("evt_D1"), + targetDef{ + Name: "nas", + Addr: "", + Site: "", + Parent: "", + Severity: SeverityPage, + Implicit: false, + StableID: "", + NodeID: 0, + }, + testNow(), + []string{"a", "gone"}, + ) + b.closeIncident(inc, recFor(b)) + if got := b.incidents["evt_D1"].Sinks["gone"].Rec; got != phaseDropped { + t.Fatalf("removed sink recovery = %q, want dropped", got) + } + if b.queues["a"].Len() != 2 { + t.Fatal("configured sink keeps its recovery") + } +} + +func TestAckReleasesClosingIncident(t *testing.T) { + t.Parallel() + b, m := testBook(testSink("a")) + inc := b.openIncident( + downFor("evt_D1"), + targetDef{ + Name: "nas", + Addr: "", + Site: "", + Parent: "", + Severity: SeverityPage, + Implicit: false, + StableID: "", + NodeID: 0, + }, + testNow(), + []string{"a"}, + ) + b.closeIncident(inc, recFor(b)) + items := b.queues["a"].Peek(2) + u := b.beginUnit("a", "bat_1", items, []byte("b")) + d := b.incidents["evt_D1"].Sinks["a"] + if d.Down != phaseAttempted || d.Rec != phaseAttempted || !d.DownTried || !d.RecTried { + t.Fatalf("unattempted down and recovery must be marked together: %+v", d) + } + b.ackUnit(u, testNow().Add(2*time.Second)) + if len(b.incidents) != 0 { + t.Fatal("closed incident with every intent acknowledged must be forgotten") + } + if n := testutil.ToFloat64(m.sentC.WithLabelValues("a", "target.recovered")); n != 1 { + t.Fatalf("sent recovered = %v", n) + } +} + +func TestNewOutageOpensWhileRecoveryPending(t *testing.T) { + t.Parallel() + b, _ := testBook(testSink("a")) + first := b.openIncident( + downFor("evt_D1"), + targetDef{ + Name: "nas", + Addr: "", + Site: "", + Parent: "", + Severity: SeverityPage, + Implicit: false, + StableID: "", + NodeID: 0, + }, + testNow(), + []string{"a"}, + ) + b.closeIncident(first, recFor(b)) + second := b.openIncident( + downFor("evt_D2"), + targetDef{ + Name: "nas", + Addr: "", + Site: "", + Parent: "", + Severity: SeverityPage, + Implicit: false, + StableID: "", + NodeID: 0, + }, + testNow(), + []string{"a"}, + ) + if second.ID == first.ID || b.openIncidentFor("nas") != second { + t.Fatal("a fresh outage needs its own incident") + } + var order []string + for _, it := range b.queues["a"].items { + order = append(order, it.Event.ID) + } + if strings.Join(order, ",") != "evt_D1,evt_Revt_D1,evt_D2" { + t.Fatalf("new down must queue behind the old recovery: %v", order) + } +} + +func newDiskNotifier(t *testing.T, dir string, sinks ...SinkConfig) *notifyHarness { + t.Helper() + for i := range sinks { + sinks[i].Queue = queueDisk + } + return newHarness(t, harnessOpts{sinks: sinks, stateDir: dir, tags: nil, intake: 0, scripts: nil, clock: nil}) +} + +func readState(t *testing.T, dir string) notifyState { + t.Helper() + raw, err := os.ReadFile(filepath.Join(dir, stateFileName)) + if err != nil { + t.Fatal(err) + } + var st notifyState + if unmarshalErr := json.Unmarshal(raw, &st); unmarshalErr != nil { + t.Fatal(unmarshalErr) + } + return st +} + +func TestCommitHidesIntentsFromWorkersAndCommitsSinksTogether(t *testing.T) { + t.Parallel() + dir := t.TempDir() + h := newDiskNotifier(t, dir, testSink("a"), testSink("b")) + gate := make(chan struct{}) + h.n.store.setInject(func() error { <-gate; return nil }) + h.start(t) + h.n.ObserveTransition(downTransition(1, h.clock.Now())) + time.Sleep(50 * time.Millisecond) + if h.sinks["a"].callCount()+h.sinks["b"].callCount() != 0 { + t.Fatal("workers sent a disk intent before its commit") + } + if _, err := os.Stat(filepath.Join(dir, stateFileName)); err == nil { + st := readState(t, dir) + if len(st.Incidents) != 0 { + t.Fatal("snapshot changed before commit") + } + } + h.n.store.setInject(nil) + close(gate) + eventually(t, func() bool { return h.sinks["a"].callCount() == 1 && h.sinks["b"].callCount() == 1 }) + eventually(t, func() bool { + st := readState(t, dir) + return len(st.Incidents) == 1 && len(st.Queues["a"]) == 0 && len(st.Queues["b"]) == 0 && + st.Incidents[0].Sinks["a"].Down == phaseAcked + }) +} + +func TestSnapshotHoldsIncidentAndIntentsTogether(t *testing.T) { + t.Parallel() + dir := filepath.Join(t.TempDir(), "state") + h := newDiskNotifier(t, dir, testSink("a"), testSink("b")) + h.n.mu.Lock() + h.n.applyLocked(downTransition(1, h.clock.Now()), false) + h.n.commitLocked() + h.n.mu.Unlock() + st := readState(t, dir) + if len(st.Incidents) != 1 || st.Incidents[0].Status != incidentOpen || len(st.Queues["a"]) != 1 || + len(st.Queues["b"]) != 1 { + t.Fatalf("snapshot %+v", st) + } + fi, err := os.Stat(filepath.Join(dir, stateFileName)) + if err != nil || fi.Mode().Perm() != 0o600 { + t.Fatalf("state file mode %v, %v", fi.Mode(), err) + } + di, err := os.Stat(dir) + if err != nil || di.Mode().Perm()&0o077 != 0 { + t.Fatalf("state dir mode %v, %v", di.Mode(), err) + } +} + +func TestReplayCrashAfterCommitBeforeDistribution(t *testing.T) { + t.Parallel() + dir := t.TempDir() + first := newDiskNotifier(t, dir, testSink("a")) + first.n.mu.Lock() + first.n.applyLocked(downTransition(1, first.clock.Now()), false) + first.n.applyLocked(upTransition(2, first.clock.Now()), false) + first.n.commitLocked() + first.n.mu.Unlock() + var downID string + for _, inc := range readState(t, dir).Incidents { + downID = inc.ID + if inc.Status != incidentClosing || inc.Recovery == nil || inc.Recovery.DownEventID != inc.ID { + t.Fatalf("closing incident lost its recovery: %+v", inc) + } + } + second := newDiskNotifier(t, dir, testSink("a")) + second.start(t) + f := second.sinks["a"] + eventually(t, func() bool { return f.callCount() == 2 }) + got := idsOf(t, f) + if got[0] != downID { + t.Fatalf("replay must keep the original down id %s, got %v", downID, got) + } + var body struct { + Events []Event `json:"events"` + } + if err := json.Unmarshal(f.snapshot()[1].Body, &body); err != nil || body.Events[0].DownEventID != downID { + t.Fatalf("recovery must reference the original down: %v %+v", err, body) + } +} + +func TestReplayCrashBeforeRenameKeepsOldSnapshot(t *testing.T) { + t.Parallel() + dir := t.TempDir() + first := newDiskNotifier(t, dir, testSink("a")) + first.n.mu.Lock() + first.n.applyLocked(downTransition(1, first.clock.Now()), false) + first.n.commitLocked() + first.n.mu.Unlock() + stale := filepath.Join(dir, stateFileName+".12345.tmp") + if err := os.WriteFile(stale, []byte(`{"version":1,"incidents":[]}`), 0o600); err != nil { + t.Fatal(err) + } + second := newDiskNotifier(t, dir, testSink("a")) + if _, err := os.Stat(stale); !errors.Is(err, os.ErrNotExist) { + t.Fatal("abandoned temporary file must be removed") + } + if len(second.n.openIncidents()) != 1 { + t.Fatal("old snapshot's open incident must survive") + } +} + +func TestReplayCrashAfterSendBeforeAckResendsSameBody(t *testing.T) { + t.Parallel() + dir := t.TempDir() + first := newDiskNotifier(t, dir, testSink("a")) + var commits atomic.Int32 + first.n.store.setInject(func() error { + if commits.Add(1) > 2 { + return errors.New("crash before acknowledgment commit") + } + return nil + }) + first.start(t) + first.n.ObserveTransition(downTransition(1, first.clock.Now())) + eventually(t, func() bool { return first.sinks["a"].callCount() == 1 }) + eventually(t, func() bool { return testutil.ToFloat64(first.n.m.persistErr.WithLabelValues(sinkState)) >= 1 }) + sent := first.sinks["a"].snapshot()[0] + first.n.Stop() + + second := newDiskNotifier(t, dir, testSink("a")) + second.start(t) + eventually(t, func() bool { return second.sinks["a"].callCount() == 1 }) + again := second.sinks["a"].snapshot()[0] + if again.ID != sent.ID || !bytes.Equal(again.Body, sent.Body) { + t.Fatal("replay after an unacknowledged send must reuse the id and body bytes") + } +} + +func TestCorruptSnapshotStopsStartup(t *testing.T) { + t.Parallel() + dir := t.TempDir() + path := filepath.Join(dir, stateFileName) + if err := os.WriteFile(path, []byte("{not json"), 0o600); err != nil { + t.Fatal(err) + } + sc := testSink("a") + sc.Queue = queueDisk + _, err := newNotifier(notifierConfig{ + Sinks: []SinkConfig{sc}, StateDir: dir, Registry: prometheus.NewRegistry(), + NewSink: func(SinkConfig) (sink, error) { + return &fakeSink{name: "a", interval: 0, mu: sync.Mutex{}, script: nil, calls: nil, block: nil}, nil + }, Clock: nil, Log: nil, Collector: CollectorInfo{Name: "", TailscaleIP: ""}, Alerts: nil, Tags: nil, Env: nil, Jitter: nil, IntakeCap: 0, + }) + if err == nil || !strings.Contains(err.Error(), "corrupt") || !strings.Contains(err.Error(), path) { + t.Fatalf("want an actionable corruption error, got %v", err) + } + if b, _ := os.ReadFile(path); string(b) != "{not json" { + t.Fatal("corrupt file must be preserved for repair") + } +} + +func TestPersistFailureFallsBackToMemoryAndRecovers(t *testing.T) { + t.Parallel() + dir := t.TempDir() + h := newDiskNotifier(t, dir, testSink("a")) + h.n.store.setInject(func() error { return errors.New("read-only file system") }) + h.start(t) + h.n.ObserveTransition(downTransition(1, h.clock.Now())) + eventually(t, func() bool { return h.sinks["a"].callCount() == 1 }) + if n := testutil.ToFloat64(h.n.m.persistErr.WithLabelValues(sinkState)); n < 1 { + t.Fatalf("persist_errors_total = %v", n) + } + h.n.store.setInject(nil) + h.clock.Advance(reconcileInterval) + eventually(t, func() bool { + raw, err := os.ReadFile(filepath.Join(dir, stateFileName)) + return err == nil && bytes.Contains(raw, []byte(`"incidents"`)) && bytes.Contains(raw, []byte("nyc-nas")) + }) +} + +func TestWarnsWhenStateDirReadableByOthers(t *testing.T) { + t.Parallel() + dir := filepath.Join(t.TempDir(), "state") + if err := os.Mkdir(dir, 0o755); err != nil { + t.Fatal(err) + } + if err := os.Chmod(dir, 0o755); err != nil { + t.Fatal(err) + } + var buf bytes.Buffer + if _, err := newStateStore(dir, slog.New(slog.NewTextHandler(&buf, nil))); err != nil { + t.Fatal(err) + } + if !strings.Contains(buf.String(), "readable by group or others") { + t.Fatalf("no warning logged: %q", buf.String()) + } + var quiet bytes.Buffer + private := filepath.Join(t.TempDir(), "private") + if _, err := newStateStore(private, slog.New(slog.NewTextHandler(&quiet, nil))); err != nil { + t.Fatal(err) + } + if quiet.Len() != 0 { + t.Fatalf("private dir should not warn: %q", quiet.String()) + } +} + +func TestExpiredMemberDropsWholeAttemptedBatch(t *testing.T) { + t.Parallel() + sc := testSink("a") + sc.Batch = true + sc.MaxAge = time.Hour + b, m := testBook(sc) + old := testNow().Add(-2 * time.Hour) + for i, at := range []time.Time{testNow(), old, testNow()} { + b.push("a", &queued{ + Seq: b.seq(), IncidentID: "", Attempted: false, UnitID: "", + Event: plainEvent("evt_"+string(rune('A'+i)), at), + }) + } + u := b.beginUnit("a", "bat_1", b.queues["a"].Peek(3), []byte("body")) + if !b.unitExpired(u, testNow()) { + t.Fatal("a unit holding an old event must count as expired") + } + b.dropUnit(u, reasonExpired) + if n := testutil.ToFloat64(m.droppedC.WithLabelValues("a", reasonExpired)); n != 3 { + t.Fatalf("every event in the batch must be counted, got %v", n) + } + if b.queues["a"].Len() != 0 || len(b.units) != 0 { + t.Fatal("the batch and its members must be gone") + } +} diff --git a/cmd/tail-watch/recheck_test.go b/cmd/tail-watch/recheck_test.go index 1f0ef39..6a2da27 100644 --- a/cmd/tail-watch/recheck_test.go +++ b/cmd/tail-watch/recheck_test.go @@ -78,7 +78,7 @@ func TestInFlightTimeoutUsesStartGeneration(t *testing.T) { metrics: nil, watch: nil, probeMu: sync.Mutex{}, - probes: nil, + probes: nil, notify: nil, collector: CollectorInfo{Name: "", TailscaleIP: ""}, } app.pub.add(bridge{al: al}) done := make(chan struct{}) diff --git a/cmd/tail-watch/restore_test.go b/cmd/tail-watch/restore_test.go new file mode 100644 index 0000000..b23cabd --- /dev/null +++ b/cmd/tail-watch/restore_test.go @@ -0,0 +1,381 @@ +package main + +import ( + "bytes" + "encoding/json" + "strings" + "testing" + "time" + + "github.com/prometheus/client_golang/prometheus/testutil" + + "github.com/prometheus/client_golang/prometheus" + "tailscale.com/tailcfg" +) + +// seedOpenIncident runs a first notifier until its down is acknowledged, then +// stops it, leaving an open incident in dir. +func seedOpenIncident(t *testing.T, dir string, tr Transition) { + t.Helper() + first := newDiskNotifier(t, dir, testSink("a")) + first.start(t) + first.n.ObserveTransition(tr) + eventually(t, func() bool { return first.sinks["a"].callCount() == 1 }) + eventually(t, func() bool { + st := readState(t, dir) + return len(st.Incidents) == 1 && st.Incidents[0].Sinks["a"].Down == phaseAcked + }) + first.n.Stop() +} + +type alertedNotifier struct { + *notifyHarness + + al *alerter +} + +// newAlertedNotifier wires a notifier to a real alerter, as main does. +func newAlertedNotifier(t *testing.T, dir string, targets []*Target) *alertedNotifier { + t.Helper() + return newAlertedNotifierParents(t, dir, targets, nil) +} + +func newAlertedNotifierParents( + t *testing.T, + dir string, + targets []*Target, + parents map[string]string, +) *alertedNotifier { + t.Helper() + clk := newCountingClock() + al, err := newAlerter(alertConfig{ + Clock: clk, Targets: targets, Parents: parents, Recheck: nil, Registerer: prometheus.NewRegistry(), QueueCap: 0, + }) + if err != nil { + t.Fatal(err) + } + t.Cleanup(al.stopDrain) + sc := testSink("a") + sc.Queue = queueDisk + h := newHarness( + t, + harnessOpts{sinks: []SinkConfig{sc}, stateDir: dir, clock: clk, tags: nil, intake: 0, scripts: nil}, + ) + h.n.cfg.Alerts = al + al.addConsumer(h.n) + return &alertedNotifier{notifyHarness: h, al: al} +} + +type fakeImplicit struct { + al *alerter + restored []*Target +} + +func (f *fakeImplicit) RestoreImplicit(t *Target) { + f.restored = append(f.restored, t) + _ = f.al.AddTarget(t) +} + +func downIDs(t *testing.T, f *fakeSink) ([]Event, []Event) { + t.Helper() + var downs, recoveries []Event + for _, c := range f.snapshot() { + var body struct { + Events []Event `json:"events"` + } + if err := json.Unmarshal(c.Body, &body); err != nil { + t.Fatal(err) + } + for _, e := range body.Events { + switch e.Type { + case EventDown: + downs = append(downs, e) + case EventRecovered: + recoveries = append(recoveries, e) + case EventFlapping, EventDegraded, EventStarted, EventTest: + } + } + } + return downs, recoveries +} + +func TestRestartDuringOutageRecoversWithOriginalID(t *testing.T) { + t.Parallel() + dir := t.TempDir() + clk := newCountingClock() + seedOpenIncident(t, dir, downTransition(1, clk.Now())) + var origID string + for _, inc := range readState(t, dir).Incidents { + origID = inc.ID + } + + nas := alertTarget("nyc-nas", "icmp") + an := newAlertedNotifier(t, dir, []*Target{nas}) + an.n.restoreOpen(map[string]bool{"nyc-nas": true}, an.al, nil) + if !an.al.IsDown("nyc-nas") { + t.Fatal("open incident must restore the target as DOWN") + } + an.start(t) + failN(an.al, nas, "icmp", layerService, 3) + time.Sleep(50 * time.Millisecond) + if downs, _ := downIDs(t, an.sinks["a"]); len(downs) != 0 { + t.Fatal("a restored outage must not page again") + } + recoverTarget(an.al, nas) + f := an.sinks["a"] + eventually(t, func() bool { _, r := downIDs(t, f); return len(r) == 1 }) + downs, recs := downIDs(t, f) + if len(downs) != 0 || len(recs) != 1 || recs[0].DownEventID != origID { + t.Fatalf("downs=%d recoveries=%+v want recovery of %s", len(downs), recs, origID) + } +} + +func TestImplicitIncidentSurvivesFailedDiscovery(t *testing.T) { + t.Parallel() + dir := t.TempDir() + clk := newCountingClock() + tr := downTransition(1, clk.Now()) + tr.Target = Target{ + Name: "nyc-router", + Addr: "100.64.0.9", + Site: "nyc", + Severity: SeverityPage, + Implicit: true, + StableID: tailcfg.StableNodeID( + "nSTABLE1", + ), + NodeID: 42, + Parent: "", + InsecureTLS: false, + Interval: 0, + Timeout: 0, + FailThreshold: 0, + RecoverThreshold: 0, + FlapLimit: 0, + FlapWindow: 0, + Network: NetworkTailnet, + Checks: nil, + } + seedOpenIncident(t, dir, tr) + + an := newAlertedNotifier(t, dir, nil) + fi := &fakeImplicit{al: an.al, restored: nil} + an.n.restoreOpen(map[string]bool{}, an.al, fi) + if len(fi.restored) != 1 { + t.Fatalf("restored %d implicit targets", len(fi.restored)) + } + got := fi.restored[0] + if got.Name != "nyc-router" || !got.Implicit || got.StableID != "nSTABLE1" || got.NodeID != 42 || + got.Addr != "100.64.0.9" { + t.Fatalf("implicit identity not preserved: %+v", got) + } + if !an.al.IsDown("nyc-router") { + t.Fatal("recreated router must start DOWN") + } + an.start(t) + an.clock.Advance(3 * reconcileInterval) + time.Sleep(50 * time.Millisecond) + if _, recs := downIDs(t, an.sinks["a"]); len(recs) != 0 || len(an.n.openIncidents()) != 1 { + t.Fatal("failed discovery must keep the incident without a removed recovery") + } +} + +func TestRestoreRefusalKeepsIncidentAndDedupes(t *testing.T) { + t.Parallel() + dir := t.TempDir() + clk := newCountingClock() + seedOpenIncident(t, dir, downTransition(1, clk.Now())) + var origID string + for _, inc := range readState(t, dir).Incidents { + origID = inc.ID + } + + nas := alertTarget("nyc-nas", "icmp") + an := newAlertedNotifier(t, dir, []*Target{nas}) + pass(an.al, nas) + an.n.restoreOpen(map[string]bool{"nyc-nas": true}, an.al, nil) + if len(an.n.openIncidents()) != 1 { + t.Fatal("a refused restore must keep the incident") + } + an.start(t) + failN(an.al, nas, "icmp", layerService, 3) + time.Sleep(50 * time.Millisecond) + if downs, _ := downIDs(t, an.sinks["a"]); len(downs) != 0 { + t.Fatal("another down while the incident is open must be deduplicated") + } + recoverTarget(an.al, nas) + f := an.sinks["a"] + eventually(t, func() bool { _, r := downIDs(t, f); return len(r) == 1 }) + _, recs := downIDs(t, f) + if len(recs) != 1 || recs[0].DownEventID != origID { + t.Fatalf("recovery must use the original metadata: %+v", recs) + } +} + +func TestConfiguredTargetRemovedWhileOpenClosesRemoved(t *testing.T) { + t.Parallel() + dir := t.TempDir() + clk := newCountingClock() + seedOpenIncident(t, dir, downTransition(1, clk.Now())) + an := newAlertedNotifier(t, dir, nil) + an.n.restoreOpen(map[string]bool{}, an.al, nil) + an.start(t) + f := an.sinks["a"] + eventually(t, func() bool { _, r := downIDs(t, f); return len(r) == 1 }) + _, recs := downIDs(t, f) + if len(recs) != 1 || recs[0].Reason != reasonRemoved { + t.Fatalf("got %+v", recs) + } + eventually(t, func() bool { + return len(readState(t, dir).Incidents) == 0 + }) +} + +func TestClosingIncidentNotRestoredAsDown(t *testing.T) { + t.Parallel() + dir := t.TempDir() + first := newDiskNotifier(t, dir, testSink("a")) + first.n.mu.Lock() + first.n.applyLocked(downTransition(1, first.clock.Now()), false) + first.n.applyLocked(upTransition(2, first.clock.Now()), false) + first.n.commitLocked() + first.n.mu.Unlock() + second := newDiskNotifier(t, dir, testSink("a")) + if len(second.n.openIncidents()) != 0 { + t.Fatal("a closing incident must not be restored as open") + } + second.n.Start(t.Context()) + defer second.n.Stop() + eventually(t, func() bool { return second.sinks["a"].callCount() == 2 }) +} + +func TestRestoredImplicitIncidentClosesWhenRemoved(t *testing.T) { + t.Parallel() + dir := t.TempDir() + clk := newCountingClock() + tr := downTransition(1, clk.Now()) + tr.Target = Target{ + Name: "nyc-router", Addr: "100.64.0.9", Site: "nyc", Severity: SeverityPage, Implicit: true, + StableID: tailcfg.StableNodeID("nSTABLE1"), NodeID: 42, Parent: "", InsecureTLS: false, Interval: 0, + Timeout: 0, FailThreshold: 0, RecoverThreshold: 0, FlapLimit: 0, FlapWindow: 0, + Network: NetworkTailnet, Checks: nil, + } + seedOpenIncident(t, dir, tr) + var origID string + for _, inc := range readState(t, dir).Incidents { + origID = inc.ID + } + + an := newAlertedNotifier(t, dir, nil) + fi := &fakeImplicit{al: an.al, restored: nil} + an.n.restoreOpen(map[string]bool{}, an.al, fi) + an.start(t) + if err := an.al.RemoveTarget("nyc-router"); err != nil { + t.Fatalf("authoritative removal of a down implicit target must succeed: %v", err) + } + an.clock.Advance(reconcileInterval) + f := an.sinks["a"] + eventually(t, func() bool { _, r := downIDs(t, f); return len(r) == 1 }) + _, recs := downIDs(t, f) + if recs[0].Reason != reasonRemoved || recs[0].DownEventID != origID { + t.Fatalf("got %+v, want a removed recovery of %s", recs[0], origID) + } + if len(an.n.openIncidents()) != 0 { + t.Fatal("incident must be closed after removal") + } +} + +func TestPagedTargetKeepsIncidentWhenParentFails(t *testing.T) { + t.Parallel() + router := alertTarget("router", "tsping") + nas := alertTarget("nas", "icmp") + an := newAlertedNotifierParents(t, t.TempDir(), []*Target{router, nas}, map[string]string{"nas": "router"}) + an.start(t) + pass(an.al, router) + pass(an.al, nas) + failN(an.al, nas, "icmp", layerService, 3) + f := an.sinks["a"] + eventually(t, func() bool { d, _ := downIDs(t, f); return len(d) == 1 }) + failN(an.al, router, "tsping", layerService, 3) + eventually(t, func() bool { d, _ := downIDs(t, f); return len(d) == 2 }) + recoverTarget(an.al, nas) + eventually(t, func() bool { _, r := downIDs(t, f); return len(r) == 1 }) + downs, recs := downIDs(t, f) + if recs[0].Target.Name != "nas" || recs[0].DownEventID != downs[0].ID { + t.Fatalf("nas must recover exactly once against its own down: %+v", recs) + } + nasDowns := 0 + for _, d := range downs { + if d.Target.Name == "nas" { + nasDowns++ + } + } + if nasDowns != 1 { + t.Fatalf("nas paged %d times, want 1", nasDowns) + } +} + +func TestReplayExpiresStaleDownAndItsRecovery(t *testing.T) { + t.Parallel() + dir := t.TempDir() + first := newDiskNotifier(t, dir, testSink("a")) + first.n.mu.Lock() + first.n.applyLocked(downTransition(1, first.clock.Now()), false) + first.clock.Advance(50 * time.Minute) + first.n.applyLocked(upTransition(2, first.clock.Now()), false) + first.n.commitLocked() + first.n.mu.Unlock() + + late := newCountingClock() + late.Advance(70 * time.Minute) + sc := testSink("a") + sc.Queue = queueDisk + second := newHarness(t, harnessOpts{ + sinks: []SinkConfig{sc}, stateDir: dir, tags: nil, intake: 0, scripts: nil, clock: late, + }) + if n := testutil.ToFloat64(second.n.m.droppedC.WithLabelValues("a", reasonExpired)); n != 1 { + t.Fatalf("expired drops = %v, want only the stale down", n) + } + st := readState(t, dir) + if len(st.Incidents) != 0 || len(st.Queues["a"]) != 0 { + t.Fatalf("the down and its dependent recovery must both be gone in one commit: %+v", st) + } + second.start(t) + time.Sleep(50 * time.Millisecond) + if second.sinks["a"].callCount() != 0 { + t.Fatal("an orphan recovery was sent for a down that never went out") + } +} + +func TestReplayBatchRetryKeepsIDAndBody(t *testing.T) { + t.Parallel() + dir := t.TempDir() + sc := testSink("a") + sc.Queue = queueDisk + sc.Batch = true + first := newHarness(t, harnessOpts{ + sinks: []SinkConfig{sc}, stateDir: dir, tags: nil, intake: 0, scripts: nil, clock: nil, + }) + first.sinks["a"].block = make(chan struct{}) + first.start(t) + first.n.ObserveTransition(downTransition(1, first.clock.Now())) + first.clock.waitTimers(t, 0) + first.clock.Advance(sc.GroupWait) + eventually(t, func() bool { return len(readState(t, dir).Units) == 1 }) + unit := readState(t, dir).Units[0] + first.n.Stop() + + second := newHarness(t, harnessOpts{ + sinks: []SinkConfig{sc}, stateDir: dir, tags: nil, intake: 0, scripts: nil, clock: nil, + }) + second.start(t) + f := second.sinks["a"] + eventually(t, func() bool { return f.callCount() == 1 }) + got := f.snapshot()[0] + if got.ID != unit.ID || !bytes.Equal(got.Body, unit.Body) { + t.Fatalf("replay changed the batch: id %s vs %s", got.ID, unit.ID) + } + if !strings.HasPrefix(got.ID, prefixBatch) { + t.Fatalf("not a batch id: %s", got.ID) + } +} diff --git a/cmd/tail-watch/sink.go b/cmd/tail-watch/sink.go new file mode 100644 index 0000000..900d30b --- /dev/null +++ b/cmd/tail-watch/sink.go @@ -0,0 +1,158 @@ +package main + +import ( + "bytes" + "context" + "crypto/hmac" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "io" + "net" + "net/http" + "strconv" + "strings" + "time" +) + +const ( + requestTimeout = 10 * time.Second + maxResponseRead = 64 << 10 + signatureHeader = "Tail-Watch-Signature" +) + +// newSinkWithEnv builds the delivery code for one configured sink. env, when +// set, carries every connection so sinks follow the host routing table. +func newSinkWithEnv(sc SinkConfig, col CollectorInfo, env netEnv) (sink, error) { + client := newSinkClient(env) + switch sc.Type { + case sinkWebhook: + return newWebhookSink(sc, client), nil + case sinkSlack: + return newSlackSink(sc, col, client), nil + default: + return nil, fmt.Errorf("sink %s: unknown type %q", sc.Name, sc.Type) + } +} + +func newSinkClient(env netEnv) *http.Client { + tr := &http.Transport{ + Proxy: http.ProxyFromEnvironment, + DisableKeepAlives: true, + ForceAttemptHTTP2: true, + } + if env != nil { + tr.DialContext = env.Dial + } + return &http.Client{ + Transport: tr, + Timeout: requestTimeout, + CheckRedirect: func(*http.Request, []*http.Request) error { + return http.ErrUseLastResponse + }, + } +} + +// marshalJSON encodes v without HTML escaping, so payloads stay readable and +// the exact bytes are what gets signed. +func marshalJSON(v any) ([]byte, error) { + var buf bytes.Buffer + enc := json.NewEncoder(&buf) + enc.SetEscapeHTML(false) + if err := enc.Encode(v); err != nil { + return nil, err + } + return bytes.TrimRight(buf.Bytes(), "\n"), nil +} + +// signBody computes the Tail-Watch-Signature value for body sent at unix time +// t: lowercase hex HMAC-SHA256 over ".". +func signBody(secret string, t int64, body []byte) string { + mac := hmac.New(sha256.New, []byte(secret)) + mac.Write([]byte(strconv.FormatInt(t, 10) + ".")) + mac.Write(body) + return "t=" + strconv.FormatInt(t, 10) + ",v1=" + hex.EncodeToString(mac.Sum(nil)) +} + +// classifyStatus maps an HTTP response to a delivery result. +func classifyStatus(status int, h http.Header, now time.Time) deliveryResult { + res := deliveryResult{Class: resultOK, Status: status, Reason: "", RetryAfter: 0, Err: nil} + switch { + case status >= 200 && status < 300: + return res + case status == http.StatusRequestTimeout: + res.Class, res.Reason = resultRetry, failTimeout + case status == http.StatusTooManyRequests: + res.Class, res.Reason = resultRetry, failHTTP429 + res.RetryAfter = parseRetryAfter(h.Get("Retry-After"), now) + case status >= http.StatusInternalServerError: + res.Class, res.Reason = resultRetry, failHTTP5xx + default: + res.Class, res.Reason = resultPermanent, failHTTP4xx + } + return res +} + +// parseRetryAfter accepts delay seconds or an HTTP date. +func parseRetryAfter(v string, now time.Time) time.Duration { + v = strings.TrimSpace(v) + if v == "" { + return 0 + } + if secs, err := strconv.Atoi(v); err == nil { + return time.Duration(max(secs, 0)) * time.Second + } + if t, err := http.ParseTime(v); err == nil { + return max(t.Sub(now), 0) + } + return 0 +} + +// classifyError maps a transport error to a delivery result, with the URL +// reduced to scheme and host. +func classifyError(err error, rawURL string) deliveryResult { + res := deliveryResult{ + Class: resultRetry, Status: 0, Reason: failNetwork, RetryAfter: 0, + Err: errors.New(strings.ReplaceAll(err.Error(), rawURL, redactURL(rawURL))), + } + var ne net.Error + if errors.Is(err, context.DeadlineExceeded) || (errors.As(err, &ne) && ne.Timeout()) { + res.Reason = failTimeout + } + return res +} + +// httpPost is one POST shared by the webhook and Slack sinks. The URL is read +// at send time so a rotated secret needs no restart. +func httpPost( + ctx context.Context, + client *http.Client, + url string, + body []byte, + headers map[string]string, + now time.Time, +) deliveryResult { + req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(body)) + if err != nil { + return deliveryResult{ + Class: resultPermanent, Status: 0, Reason: failHTTP4xx, RetryAfter: 0, + Err: errors.New("invalid sink URL " + redactURL(url)), + } + } + for k, v := range headers { + req.Header.Set(k, v) + } + resp, err := client.Do(req) + if err != nil { + return classifyError(err, url) + } + defer resp.Body.Close() + _, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, maxResponseRead)) + return classifyStatus(resp.StatusCode, resp.Header, now) +} + +func secretUnavailable(err error) deliveryResult { + return deliveryResult{Class: resultRetry, Status: 0, Reason: failNetwork, RetryAfter: 0, Err: err} +} diff --git a/cmd/tail-watch/slack.go b/cmd/tail-watch/slack.go new file mode 100644 index 0000000..8b9c0f2 --- /dev/null +++ b/cmd/tail-watch/slack.go @@ -0,0 +1,268 @@ +package main + +import ( + "context" + "fmt" + "net/http" + "strings" + "time" +) + +const ( + slackMaxLines = 10 + slackMaxDeps = 10 + slackMaxErrorLen = 300 + slackMinInterval = time.Second + slackTimeLayout = "2006-01-02 15:04 UTC" + slackClockLayout = "15:04 UTC" + slackRedCircle = ":red_circle:" + slackGreenCircle = ":large_green_circle:" + slackWarnMarker = ":warning:" + slackInfoMarker = ":information_source:" + slackTestMarker = ":test_tube:" + slackTargetsBehind = "and %d targets behind it" +) + +// slackSink posts Block Kit messages to an incoming webhook. +type slackSink struct { + cfg SinkConfig + col CollectorInfo + client *http.Client +} + +func newSlackSink(cfg SinkConfig, col CollectorInfo, client *http.Client) *slackSink { + return &slackSink{cfg: cfg, col: col, client: client} +} + +func (s *slackSink) Name() string { return s.cfg.Name } + +func (s *slackSink) MinInterval() time.Duration { return slackMinInterval } + +type slackText struct { + Type string `json:"type"` + Text string `json:"text"` +} + +type slackBlock struct { + Type string `json:"type"` + Text *slackText `json:"text,omitempty"` + Elements []slackText `json:"elements,omitempty"` +} + +type slackMessage struct { + Text string `json:"text"` + Blocks []slackBlock `json:"blocks"` +} + +// Render is a pure function of the events and the sink's options. +func (s *slackSink) Render(evs []Event, _ string) ([]byte, error) { + return marshalJSON(renderSlack(evs, s.cfg, s.col)) +} + +func renderSlack(evs []Event, cfg SinkConfig, col CollectorInfo) slackMessage { + var blocks []slackBlock + var plain []string + if cfg.MentionOnPage != "" && hasPageDown(evs) { + blocks = append(blocks, mrkdwnBlock(cfg.MentionOnPage)) + plain = append(plain, cfg.MentionOnPage) + } + shown := evs + if len(shown) > slackMaxLines { + shown = shown[:slackMaxLines] + } + for _, ev := range shown { + rich, flat := slackLine(ev, cfg.StatusURL) + blocks = append(blocks, mrkdwnBlock(rich)) + plain = append(plain, flat) + } + if more := len(evs) - len(shown); more > 0 { + line := fmt.Sprintf("and %d more", more) + blocks = append(blocks, mrkdwnBlock(line)) + plain = append(plain, line) + } + name := col.Name + if name == "" { + name = "tail-watch" + } + when := "" + if len(evs) > 0 { + when = " · " + evs[0].Time.Time().UTC().Format(slackTimeLayout) + } + blocks = append(blocks, slackBlock{ + Type: "context", Text: nil, + Elements: []slackText{{Type: "mrkdwn", Text: slackEscape(name) + when}}, + }) + return slackMessage{Text: strings.Join(plain, "\n"), Blocks: blocks} +} + +func hasPageDown(evs []Event) bool { + for _, ev := range evs { + if ev.Type == EventDown && ev.Severity == SeverityPage { + return true + } + } + return false +} + +func mrkdwnBlock(text string) slackBlock { + return slackBlock{Type: "section", Text: &slackText{Type: "mrkdwn", Text: text}, Elements: nil} +} + +// slackEscape escapes the three characters Slack treats as control syntax. +func slackEscape(s string) string { + return strings.NewReplacer("&", "&", "<", "<", ">", ">").Replace(s) +} + +// slackLine renders one event as mrkdwn with links and as plain fallback text. +func slackLine(ev Event, statusURL string) (string, string) { + name := "" + if ev.Target != nil { + name = slackEscape(ev.Target.Name) + if ev.Target.Implicit { + name += " (learned)" + } + } + link := name + if statusURL != "" && ev.Target != nil { + link = fmt.Sprintf("<%s/#%s|%s>", strings.TrimRight(statusURL, "/"), ev.Target.Name, name) + } + switch ev.Type { + case EventDown: + return slackDown(ev, link, name) + case EventRecovered: + tail := "%s RECOVERED %%s %s after %s%s" + after := humanDuration(secondsOf(ev.DurationSeconds)) + return fmt.Sprintf(fmt.Sprintf(tail, slackGreenCircle, slackCheckRich(ev.Check), after, removedNote(ev)), link), + fmt.Sprintf(fmt.Sprintf(tail, slackGreenCircle, slackEscape(ev.Check), after, removedNote(ev)), name) + case EventFlapping: + head := "%s FLAPPING %%s %s%s" + return fmt.Sprintf(fmt.Sprintf(head, slackWarnMarker, slackCheckRich(ev.Check), layerNote(ev)), link), + fmt.Sprintf(fmt.Sprintf(head, slackWarnMarker, slackEscape(ev.Check), layerNote(ev)), name) + case EventStarted: + line := fmt.Sprintf("%s tail-watch started on %s", slackInfoMarker, slackEscape(ev.Collector.Name)) + return line, line + case EventTest: + line := fmt.Sprintf("%s tail-watch test notification from %s", slackTestMarker, slackEscape(ev.Collector.Name)) + return line, line + case EventDegraded: + fallthrough + default: + line := fmt.Sprintf("%s %s", slackInfoMarker, slackEscape(string(ev.Type))) + return line, line + } +} + +// slackCheckRich escapes a check for mrkdwn. A check holding a URL, such as +// "http:http://host/|text", is a code span: Slack would otherwise auto-link it +// and encode the "|" as "%7C". +func slackCheckRich(check string) string { + if strings.Contains(check, "://") { + return "`" + strings.ReplaceAll(check, "`", "'") + "`" + } + return slackEscape(check) +} + +func layerNote(ev Event) string { + if ev.Layer == "" { + return "" + } + return " (" + ev.Layer + ")" +} + +func removedNote(ev Event) string { + if ev.Reason == reasonRemoved { + return " (target removed)" + } + return "" +} + +func secondsOf(p *int64) int64 { + if p == nil { + return 0 + } + return *p +} + +func slackDown(ev Event, link, name string) (string, string) { + const headFmt = "%s DOWN %%s %s%s" + richHead := fmt.Sprintf(headFmt, slackRedCircle, slackCheckRich(ev.Check), layerNote(ev)) + flatHead := fmt.Sprintf(headFmt, slackRedCircle, slackEscape(ev.Check), layerNote(ev)) + var details []string + if t := ev.Target; t != nil { + details = append(details, "addr "+slackEscape(t.Addr)) + if t.Site != "" { + details = append(details, "site "+slackEscape(t.Site)) + } + if t.Parent != "" { + details = append(details, "parent "+slackEscape(t.Parent)) + } + } + if p := ev.Path; p != nil && p.Kind != "" { + path := "path " + p.Kind + if p.LatencyMS != nil { + path += fmt.Sprintf(" %d ms", *p.LatencyMS) + } + details = append(details, path) + } + lines := []string{""} + if len(details) > 0 { + lines = append(lines, strings.Join(details, " · ")) + } + if ev.Error != "" { + lines = append(lines, "error: "+slackEscape(truncate(ev.Error, slackMaxErrorLen))) + } + var tail []string + if ev.Since != nil { + tail = append(tail, "down since "+ev.Since.Time().UTC().Format(slackClockLayout)) + } + if ev.Failures > 0 { + tail = append(tail, fmt.Sprintf("%d failures", ev.Failures)) + } + tail = append(tail, "collector "+slackEscape(ev.Collector.Name)) + lines = append(lines, strings.Join(tail, " · ")) + if ev.Dependents != nil && len(*ev.Dependents) > 0 { + deps := *ev.Dependents + shown := deps + if len(shown) > slackMaxDeps { + shown = shown[:slackMaxDeps] + } + line := fmt.Sprintf(slackTargetsBehind, len(deps)) + ": " + slackEscape(strings.Join(shown, ", ")) + if len(deps) > len(shown) { + line += ", …" + } + lines = append(lines, line) + } + rest := strings.Join(lines, "\n") + return fmt.Sprintf(richHead, link) + rest, fmt.Sprintf(flatHead, name) + rest +} + +func truncate(s string, n int) string { + if len(s) <= n { + return s + } + return s[:n] + "…" +} + +// humanDuration renders whole seconds compactly: 45s, 12m, 1h5m. +func humanDuration(secs int64) string { + s := (time.Duration(secs) * time.Second).String() + if strings.HasSuffix(s, "m0s") { + s = strings.TrimSuffix(s, "0s") + } + if strings.HasSuffix(s, "h0m") { + s = strings.TrimSuffix(s, "0m") + } + return s +} + +func (s *slackSink) Send(ctx context.Context, body []byte, m sendMeta) deliveryResult { + url, err := s.cfg.URL.resolve() + if err != nil { + return secretUnavailable(fmt.Errorf("sink %s: url: %w", s.cfg.Name, err)) + } + headers := map[string]string{ + "Content-Type": "application/json", + "User-Agent": "tail-watch/" + versionString(), + } + return httpPost(ctx, s.client, url, body, headers, m.At) +} diff --git a/cmd/tail-watch/slack_test.go b/cmd/tail-watch/slack_test.go new file mode 100644 index 0000000..ea19242 --- /dev/null +++ b/cmd/tail-watch/slack_test.go @@ -0,0 +1,415 @@ +package main + +import ( + "encoding/json" + "fmt" + "net/http" + "path/filepath" + "strings" + "testing" + "time" +) + +func slackSinkCfg(mut func(*SinkConfig)) SinkConfig { + sc := testSink("slack") + sc.Type = sinkSlack + if mut != nil { + mut(&sc) + } + return sc +} + +func renderJSON(t *testing.T, evs []Event, cfg SinkConfig) []byte { + t.Helper() + b, err := marshalJSON(renderSlack(evs, cfg, goldenCollector())) + if err != nil { + t.Fatal(err) + } + return b +} + +func TestSlackGolden(t *testing.T) { + t.Parallel() + tr := goldenTransition() + down := newDownEvent("evt_D1", goldenCollector(), tr) + rec := newRecoveredEvent( + "evt_R1", + goldenCollector(), + &incident{ + Down: down, + ID: "", + Status: incidentOpen, + Target: targetDef{ + Name: "", + Addr: "", + Site: "", + Parent: "", + Severity: SeverityPage, + Implicit: false, + StableID: "", + NodeID: 0, + }, + DownSince: time.Time{}, + Recovery: nil, + Sinks: nil, + }, + tr.At.Add(12*time.Minute), + nil, + "", + ) + + router := goldenTransition() + router.Target = Target{ + Name: "nyc-router", + Addr: "100.64.0.9", + Site: "nyc", + Severity: SeverityPage, + Implicit: true, + Parent: "", + InsecureTLS: false, + Interval: 0, + Timeout: 0, + FailThreshold: 0, + RecoverThreshold: 0, + FlapLimit: 0, + FlapWindow: 0, + Network: NetworkTailnet, + Checks: nil, + StableID: "", + NodeID: 0, + } + router.Layer = layerUnrouted + for i := range 14 { + router.Dependents = append(router.Dependents, fmt.Sprintf("nyc-host-%02d", i+1)) + } + routerDown := newDownEvent("evt_D2", goldenCollector(), router) + + warn := goldenTransition() + warn.Target.Severity = SeverityWarn + warnDown := newDownEvent("evt_D3", goldenCollector(), warn) + + mention := slackSinkCfg(func(sc *SinkConfig) { sc.MentionOnPage = "" }) + linked := slackSinkCfg(func(sc *SinkConfig) { sc.StatusURL = "https://monitor-nyc.example.ts.net:9109" }) + + cases := map[string][]byte{ + "down": renderJSON(t, []Event{down}, slackSinkCfg(nil)), + "down-mention": renderJSON(t, []Event{down}, mention), + "recovered": renderJSON(t, []Event{rec}, mention), + "router-14-deps": renderJSON(t, []Event{routerDown}, slackSinkCfg(nil)), + "warn-with-mention": renderJSON(t, []Event{warnDown}, mention), + "down-status-link": renderJSON(t, []Event{down}, linked), + "recovered-removed": renderJSON( + t, + []Event{ + newRecoveredEvent( + "evt_R2", + goldenCollector(), + &incident{ + Down: down, + ID: "", + Status: incidentOpen, + Target: targetDef{ + Name: "", + Addr: "", + Site: "", + Parent: "", + Severity: SeverityPage, + Implicit: false, + StableID: "", + NodeID: 0, + }, + DownSince: time.Time{}, + Recovery: nil, + Sinks: nil, + }, + tr.At.Add(time.Hour+5*time.Minute), + nil, + reasonRemoved, + ), + }, + slackSinkCfg(nil), + ), + } + for name, raw := range cases { + assertGolden(t, filepath.Join("testdata", "slack", name+".json"), raw) + } +} + +func TestSlackMessageContent(t *testing.T) { + t.Parallel() + tr := goldenTransition() + down := newDownEvent("evt_D1", goldenCollector(), tr) + msg := renderSlack([]Event{down}, slackSinkCfg(nil), goldenCollector()) + if !strings.HasPrefix(msg.Text, ":red_circle: DOWN nyc-nas tcp:445 (service)") { + t.Fatalf("text starts %q", msg.Text) + } + if !strings.Contains(msg.Text, "connection refused") { + t.Fatalf("error missing from text %q", msg.Text) + } + if len(msg.Blocks) == 0 { + t.Fatal("no blocks") + } + + rec := newRecoveredEvent( + "evt_R1", + goldenCollector(), + &incident{ + Down: down, + ID: "", + Status: incidentOpen, + Target: targetDef{ + Name: "", + Addr: "", + Site: "", + Parent: "", + Severity: SeverityPage, + Implicit: false, + StableID: "", + NodeID: 0, + }, + DownSince: time.Time{}, + Recovery: nil, + Sinks: nil, + }, + tr.At.Add(12*time.Minute+30*time.Second), + nil, + "", + ) + got := renderSlack([]Event{rec}, slackSinkCfg(nil), goldenCollector()).Text + want := ":large_green_circle: RECOVERED nyc-nas tcp:445 after 14m" + if got != want { + t.Fatalf("recovered %q want %q", got, want) + } + + routerTr := goldenTransition() + for i := range 14 { + routerTr.Dependents = append(routerTr.Dependents, fmt.Sprintf("dep-%02d", i)) + } + text := renderSlack( + []Event{newDownEvent("evt_D2", goldenCollector(), routerTr)}, + slackSinkCfg(nil), + goldenCollector(), + ).Text + if !strings.Contains(text, "and 14 targets behind it") || !strings.Contains(text, "dep-09") || + strings.Contains(text, "dep-10") { + t.Fatalf("dependents line wrong: %q", text) + } + + implicit := goldenTransition() + implicit.Target.Implicit = true + text = renderSlack( + []Event{newDownEvent("evt_D3", goldenCollector(), implicit)}, + slackSinkCfg(nil), + goldenCollector(), + ).Text + if !strings.Contains(text, "nyc-nas (learned)") { + t.Fatalf("implicit target not marked: %q", text) + } +} + +func TestSlackMentionsOnlyPageDowns(t *testing.T) { + t.Parallel() + const mention = "" + cfg := slackSinkCfg(func(sc *SinkConfig) { sc.MentionOnPage = mention }) + tr := goldenTransition() + pageDown := newDownEvent("evt_D1", goldenCollector(), tr) + warnTr := goldenTransition() + warnTr.Target.Severity = SeverityWarn + warnDown := newDownEvent("evt_D2", goldenCollector(), warnTr) + rec := newRecoveredEvent( + "evt_R1", + goldenCollector(), + &incident{ + Down: pageDown, + ID: "", + Status: incidentOpen, + Target: targetDef{ + Name: "", + Addr: "", + Site: "", + Parent: "", + Severity: SeverityPage, + Implicit: false, + StableID: "", + NodeID: 0, + }, + DownSince: time.Time{}, + Recovery: nil, + Sinks: nil, + }, + tr.At.Add(time.Minute), + nil, + "", + ) + flap := newFlappingEvent("evt_F1", goldenCollector(), tr) + started := newStartedEvent("evt_S1", goldenCollector(), tr.At) + for name, ev := range map[string]Event{"warn": warnDown, "recovery": rec, "flapping": flap, "info": started} { + if strings.Contains(renderSlack([]Event{ev}, cfg, goldenCollector()).Text, mention) { + t.Fatalf("%s must not carry the mention", name) + } + } + msg := renderSlack([]Event{pageDown}, cfg, goldenCollector()) + if !strings.HasPrefix(msg.Text, mention+"\n") { + t.Fatalf("page down must start with the mention: %q", msg.Text) + } + mixed := renderSlack([]Event{warnDown, pageDown}, cfg, goldenCollector()) + if !strings.HasPrefix(mixed.Text, mention) { + t.Fatal("a group holding any page down starts with the mention") + } +} + +func TestSlackStatusLinks(t *testing.T) { + t.Parallel() + cfg := slackSinkCfg(func(sc *SinkConfig) { sc.StatusURL = "https://monitor-nyc.example.ts.net:9109/" }) + blocks := renderSlack( + []Event{newDownEvent("evt_D1", goldenCollector(), goldenTransition())}, + cfg, + goldenCollector(), + ).Blocks + if !strings.Contains(blocks[0].Text.Text, "") { + t.Fatalf("no status link: %q", blocks[0].Text.Text) + } +} + +func TestSlackEscapesUserText(t *testing.T) { + t.Parallel() + tr := goldenTransition() + tr.Err = fmt.Errorf("bad <@U123> & ") + text := renderSlack( + []Event{newDownEvent("evt_D1", goldenCollector(), tr)}, + slackSinkCfg(nil), + goldenCollector(), + ).Text + if strings.Contains(text, "<@U123>") || strings.Contains(text, "") { + t.Fatalf("unescaped control syntax: %q", text) + } +} + +func TestSlackGroupsBurstIntoOneMessage(t *testing.T) { + rc := newReceiver(t) + t.Setenv("TW_TEST_URL", rc.srv.URL) + sc := slackSinkCfg(func(sc *SinkConfig) { sc.URL = secretRef{Env: "TW_TEST_URL", File: ""} }) + h := realSinkHarness( + t, + harnessOpts{sinks: []SinkConfig{sc}, stateDir: "", tags: nil, intake: 0, scripts: nil, clock: nil}, + ) + h.n.mu.Lock() + for i := range 25 { + tr := goldenTransition() + tr.Target.Name = fmt.Sprintf("host-%02d", i) + h.n.enqueueLocked(newFlappingEvent(fmt.Sprintf("evt_%02d", i), goldenCollector(), tr)) + } + h.n.mu.Unlock() + h.start(t) + h.clock.waitTimers(t, 0) + if rc.count() != 0 { + t.Fatal("nothing should be sent before group_wait elapses") + } + h.clock.Advance(sc.GroupWait) + eventually(t, func() bool { return rc.count() == 1 }) + var msg slackMessage + if err := json.Unmarshal(rc.request(0).Body, &msg); err != nil { + t.Fatal(err) + } + lines := 0 + for _, b := range msg.Blocks { + if b.Type == "section" && strings.Contains(b.Text.Text, "FLAPPING") { + lines++ + } + } + if lines != 10 || !strings.Contains(msg.Text, "and 15 more") { + t.Fatalf("lines=%d text tail=%q", lines, msg.Text[max(0, len(msg.Text)-40):]) + } + time.Sleep(20 * time.Millisecond) + if rc.count() != 1 { + t.Fatalf("burst should be a single message, got %d", rc.count()) + } +} + +func TestSlackSendsAtMostOnePerSecond(t *testing.T) { + t.Parallel() + sc := slackSinkCfg(nil) + sc.Type = sinkWebhook + h := newHarness( + t, + harnessOpts{sinks: []SinkConfig{sc}, stateDir: "", tags: nil, intake: 0, scripts: nil, clock: nil}, + ) + h.sinks["slack"].interval = time.Second + h.start(t) + h.n.ObserveTransition(downTransition(1, h.clock.Now())) + f := h.sinks["slack"] + eventually(t, func() bool { return f.callCount() == 1 }) + h.n.ObserveTransition(upTransition(2, h.clock.Now())) + h.clock.waitTimers(t, 0) + h.clock.Advance(999 * time.Millisecond) + time.Sleep(20 * time.Millisecond) + if f.callCount() != 1 { + t.Fatal("second message sent inside one second") + } + h.clock.Advance(time.Millisecond) + eventually(t, func() bool { return f.callCount() == 2 }) +} + +func TestSlackRetryAfterThroughRealSink(t *testing.T) { + rc := newReceiver(t, 429, 200) + rc.extra[1] = func(w http.ResponseWriter) { w.Header().Set("Retry-After", "2") } + t.Setenv("TW_TEST_URL", rc.srv.URL) + sc := slackSinkCfg(func(sc *SinkConfig) { sc.URL = secretRef{Env: "TW_TEST_URL", File: ""} }) + h := realSinkHarness( + t, + harnessOpts{sinks: []SinkConfig{sc}, stateDir: "", tags: nil, intake: 0, scripts: nil, clock: nil}, + ) + h.start(t) + h.n.ObserveTransition(downTransition(1, h.clock.Now())) + h.clock.waitTimers(t, 0) + h.clock.Advance(sc.GroupWait) + eventually(t, func() bool { return rc.count() == 1 }) + h.clock.waitTimers(t, 1) + h.clock.Advance(time.Second + 500*time.Millisecond) + time.Sleep(20 * time.Millisecond) + if rc.count() != 1 { + t.Fatal("resent before Retry-After: 2") + } + h.clock.Advance(500 * time.Millisecond) + eventually(t, func() bool { return rc.count() == 2 }) + if string(rc.request(0).Body) != string(rc.request(1).Body) { + t.Fatal("Slack retry must resend the same message") + } +} + +func TestSlackURLCheckIsNotAutoLinked(t *testing.T) { + t.Parallel() + const check = "http:http://100.76.167.117:8080/|tail-watch-ok" + tr := goldenTransition() + tr.Check = check + down := newDownEvent("evt_D1", goldenCollector(), tr) + msg := renderSlack([]Event{down}, slackSinkCfg(nil), goldenCollector()) + rich := msg.Blocks[0].Text.Text + if !strings.Contains(rich, "`"+check+"`") { + t.Fatalf("a URL check must be a code span so Slack does not encode the pipe: %q", rich) + } + if !strings.HasPrefix(msg.Text, ":red_circle: DOWN nyc-nas "+check+" (service)") || + strings.Contains(msg.Text, "`") { + t.Fatalf("the plain text fallback must keep the exact check: %q", msg.Text) + } + rec := newRecoveredEvent("evt_R1", goldenCollector(), &incident{ + ID: "", Status: incidentOpen, Target: targetDef{ + Name: "", Addr: "", Site: "", Parent: "", Severity: SeverityPage, Implicit: false, StableID: "", NodeID: 0, + }, + Down: down, DownSince: time.Time{}, Recovery: nil, Sinks: nil, + }, tr.At.Add(time.Minute), nil, "") + if got := renderSlack([]Event{rec}, slackSinkCfg(nil), goldenCollector()).Blocks[0].Text.Text; !strings.Contains( + got, + "`"+check+"`", + ) { + t.Fatalf("recovery lost the code span: %q", got) + } + plain := renderSlack( + []Event{newDownEvent("evt_D2", goldenCollector(), goldenTransition())}, + slackSinkCfg(nil), + goldenCollector(), + ) + if strings.Contains(plain.Blocks[0].Text.Text, "`") { + t.Fatalf("a non-URL check must stay plain: %q", plain.Blocks[0].Text.Text) + } +} diff --git a/cmd/tail-watch/testdata/compose.live-notify.yaml b/cmd/tail-watch/testdata/compose.live-notify.yaml new file mode 100644 index 0000000..04577b1 --- /dev/null +++ b/cmd/tail-watch/testdata/compose.live-notify.yaml @@ -0,0 +1,107 @@ +# Merged over examples/compose.yaml by TestLiveNotifications. One tailnet node per +# check the collector runs, so each check can be failed and healed on its own: +# +# fx-icmp icmp ICMP is dropped with iptables to fail, and the rule removed to heal +# fx-tsping tsping the container is paused to fail +# fx-tcp tcp a listener on 9000, stopped with /tmp/down +# fx-http http an nc responder on 8080 serving "tail-watch-ok", stopped with /tmp/down +# lan-router icmp subnet router for 10.10.53.0/24, answering 10.10.53.1 +# +# Required: TS_FX_PREFIX, TS_KEY_ICMP, TS_KEY_TSPING, TS_KEY_TCP, TS_KEY_HTTP, +# TS_KEY_LAN. Sink URLs and secrets are passed through from the environment. +x-fixture: &fixture + image: tailscale/tailscale:v1.102.5 + cap_add: [NET_ADMIN, NET_RAW] + devices: [/dev/net/tun] + +x-fixture-env: &fixture-env + TS_USERSPACE: "false" + TS_ACCEPT_DNS: "false" + TS_STATE_DIR: /var/lib/tailscale + TS_SOCKET: /var/run/tailscale/tailscaled.sock + +services: + tailscale: + extra_hosts: + - host.docker.internal:host-gateway + + tail-watch: + environment: + - SLACK_WEBHOOK_URL + - OPS_WEBHOOK_URL + - OPS_WEBHOOK_SECRET + - HEARTBEAT_URL + volumes: + - notify-state:/var/lib/tail-watch + + fx-icmp: + <<: *fixture + hostname: ${TS_FX_PREFIX:?}-icmp + environment: + <<: *fixture-env + TS_AUTHKEY: ${TS_KEY_ICMP:?} + TS_HOSTNAME: ${TS_FX_PREFIX:?}-icmp + + fx-tsping: + <<: *fixture + hostname: ${TS_FX_PREFIX:?}-tsping + environment: + <<: *fixture-env + TS_AUTHKEY: ${TS_KEY_TSPING:?} + TS_HOSTNAME: ${TS_FX_PREFIX:?}-tsping + + fx-tcp: + <<: *fixture + hostname: ${TS_FX_PREFIX:?}-tcp + entrypoint: + - sh + - -c + - | + (while true; do + if [ -e /tmp/down ]; then sleep 1; else nc -l -p 9000 -e true 2>/dev/null; fi + done) & + exec /usr/local/bin/containerboot + environment: + <<: *fixture-env + TS_AUTHKEY: ${TS_KEY_TCP:?} + TS_HOSTNAME: ${TS_FX_PREFIX:?}-tcp + + fx-http: + <<: *fixture + hostname: ${TS_FX_PREFIX:?}-http + entrypoint: + - sh + - -c + - | + cat > /tmp/resp.sh <<'EOS' + while read -r l; do [ -z "$${l%?}" ] && break; done + printf 'HTTP/1.1 200 OK\r\nConnection: close\r\nContent-Length: 13\r\n\r\ntail-watch-ok' + EOS + (while true; do + if [ -e /tmp/down ]; then sleep 1; else nc -l -p 8080 -e sh /tmp/resp.sh 2>/dev/null; fi + done) & + exec /usr/local/bin/containerboot + environment: + <<: *fixture-env + TS_AUTHKEY: ${TS_KEY_HTTP:?} + TS_HOSTNAME: ${TS_FX_PREFIX:?}-http + + lan-router: + <<: *fixture + hostname: ${TS_FX_PREFIX:?}-lan + entrypoint: + - sh + - -c + - ip addr replace 10.10.53.1/32 dev lo && exec /usr/local/bin/containerboot + environment: + <<: *fixture-env + TS_AUTHKEY: ${TS_KEY_LAN:?} + TS_HOSTNAME: ${TS_FX_PREFIX:?}-lan + TS_ROUTES: 10.10.53.0/24 + sysctls: + net.ipv4.ip_forward: "1" + net.ipv4.conf.all.rp_filter: "2" + net.ipv4.conf.lo.rp_filter: "0" + +volumes: + notify-state: diff --git a/cmd/tail-watch/testdata/payload/collector.started.json b/cmd/tail-watch/testdata/payload/collector.started.json new file mode 100644 index 0000000..e8d2877 --- /dev/null +++ b/cmd/tail-watch/testdata/payload/collector.started.json @@ -0,0 +1,11 @@ +{ + "version": 1, + "id": "evt_01JAB3K9T6M4X2Q8R7N5W0YZCK", + "type": "collector.started", + "severity": "info", + "time": "2026-04-02T14:02:11Z", + "collector": { + "name": "monitor-nyc", + "tailscale_ip": "100.101.102.103" + } +} diff --git a/cmd/tail-watch/testdata/payload/target.down.implicit.json b/cmd/tail-watch/testdata/payload/target.down.implicit.json new file mode 100644 index 0000000..7c90d99 --- /dev/null +++ b/cmd/tail-watch/testdata/payload/target.down.implicit.json @@ -0,0 +1,31 @@ +{ + "version": 1, + "id": "evt_01JAB3K9T6M4X2Q8R7N5W0YZCE", + "type": "target.down", + "severity": "page", + "time": "2026-04-02T14:02:11Z", + "collector": { + "name": "monitor-nyc", + "tailscale_ip": "100.101.102.103" + }, + "target": { + "name": "nyc-router", + "addr": "100.64.0.9", + "site": "nyc", + "implicit": true + }, + "check": "tcp:445", + "layer": "unrouted", + "error": "connection refused", + "since": "2026-04-02T14:00:41Z", + "failures": 3, + "path": { + "kind": "direct", + "endpoint": "203.0.113.5:41641", + "latency_ms": 12 + }, + "dependents": [ + "nyc-nas", + "nyc-cam" + ] +} diff --git a/cmd/tail-watch/testdata/payload/target.down.json b/cmd/tail-watch/testdata/payload/target.down.json new file mode 100644 index 0000000..f88d5ed --- /dev/null +++ b/cmd/tail-watch/testdata/payload/target.down.json @@ -0,0 +1,28 @@ +{ + "version": 1, + "id": "evt_01JAB3K9T6M4X2Q8R7N5W0YZCD", + "type": "target.down", + "severity": "page", + "time": "2026-04-02T14:02:11Z", + "collector": { + "name": "monitor-nyc", + "tailscale_ip": "100.101.102.103" + }, + "target": { + "name": "nyc-nas", + "addr": "10.20.0.50", + "site": "nyc", + "parent": "nyc-router" + }, + "check": "tcp:445", + "layer": "service", + "error": "connection refused", + "since": "2026-04-02T14:00:41Z", + "failures": 3, + "path": { + "kind": "direct", + "endpoint": "203.0.113.5:41641", + "latency_ms": 12 + }, + "dependents": [] +} diff --git a/cmd/tail-watch/testdata/payload/target.down.warn.json b/cmd/tail-watch/testdata/payload/target.down.warn.json new file mode 100644 index 0000000..409dac6 --- /dev/null +++ b/cmd/tail-watch/testdata/payload/target.down.warn.json @@ -0,0 +1,28 @@ +{ + "version": 1, + "id": "evt_01JAB3K9T6M4X2Q8R7N5W0YZCF", + "type": "target.down", + "severity": "warn", + "time": "2026-04-02T14:02:11Z", + "collector": { + "name": "monitor-nyc", + "tailscale_ip": "100.101.102.103" + }, + "target": { + "name": "nyc-nas", + "addr": "10.20.0.50", + "site": "nyc", + "parent": "nyc-router" + }, + "check": "tcp:445", + "layer": "lan", + "error": "connection refused", + "since": "2026-04-02T14:00:41Z", + "failures": 3, + "path": { + "kind": "direct", + "endpoint": "203.0.113.5:41641", + "latency_ms": 12 + }, + "dependents": [] +} diff --git a/cmd/tail-watch/testdata/payload/target.flapping.json b/cmd/tail-watch/testdata/payload/target.flapping.json new file mode 100644 index 0000000..b97c661 --- /dev/null +++ b/cmd/tail-watch/testdata/payload/target.flapping.json @@ -0,0 +1,27 @@ +{ + "version": 1, + "id": "evt_01JAB3K9T6M4X2Q8R7N5W0YZCJ", + "type": "target.flapping", + "severity": "warn", + "time": "2026-04-02T14:02:11Z", + "collector": { + "name": "monitor-nyc", + "tailscale_ip": "100.101.102.103" + }, + "target": { + "name": "nyc-nas", + "addr": "10.20.0.50", + "site": "nyc", + "parent": "nyc-router" + }, + "check": "tcp:445", + "layer": "service", + "error": "connection refused", + "since": "2026-04-02T14:00:41Z", + "failures": 3, + "path": { + "kind": "direct", + "endpoint": "203.0.113.5:41641", + "latency_ms": 12 + } +} diff --git a/cmd/tail-watch/testdata/payload/target.recovered.json b/cmd/tail-watch/testdata/payload/target.recovered.json new file mode 100644 index 0000000..2ac2594 --- /dev/null +++ b/cmd/tail-watch/testdata/payload/target.recovered.json @@ -0,0 +1,31 @@ +{ + "version": 1, + "id": "evt_01JAB3K9T6M4X2Q8R7N5W0YZCG", + "type": "target.recovered", + "severity": "page", + "time": "2026-04-02T14:14:11Z", + "collector": { + "name": "monitor-nyc", + "tailscale_ip": "100.101.102.103" + }, + "target": { + "name": "nyc-nas", + "addr": "10.20.0.50", + "site": "nyc", + "parent": "nyc-router" + }, + "check": "tcp:445", + "error": "connection refused", + "since": "2026-04-02T14:00:41Z", + "failures": 3, + "path": { + "kind": "direct", + "endpoint": "203.0.113.5:41641", + "latency_ms": 12 + }, + "dependents": [ + "nyc-cam" + ], + "duration_seconds": 810, + "down_event_id": "evt_01JAB3K9T6M4X2Q8R7N5W0YZCD" +} diff --git a/cmd/tail-watch/testdata/payload/target.recovered.removed.json b/cmd/tail-watch/testdata/payload/target.recovered.removed.json new file mode 100644 index 0000000..c6f067f --- /dev/null +++ b/cmd/tail-watch/testdata/payload/target.recovered.removed.json @@ -0,0 +1,30 @@ +{ + "version": 1, + "id": "evt_01JAB3K9T6M4X2Q8R7N5W0YZCH", + "type": "target.recovered", + "severity": "page", + "time": "2026-04-02T14:14:11Z", + "collector": { + "name": "monitor-nyc", + "tailscale_ip": "100.101.102.103" + }, + "target": { + "name": "nyc-nas", + "addr": "10.20.0.50", + "site": "nyc", + "parent": "nyc-router" + }, + "check": "tcp:445", + "error": "connection refused", + "since": "2026-04-02T14:00:41Z", + "failures": 3, + "path": { + "kind": "direct", + "endpoint": "203.0.113.5:41641", + "latency_ms": 12 + }, + "dependents": [], + "duration_seconds": 810, + "down_event_id": "evt_01JAB3K9T6M4X2Q8R7N5W0YZCD", + "reason": "removed" +} diff --git a/cmd/tail-watch/testdata/payload/test.json b/cmd/tail-watch/testdata/payload/test.json new file mode 100644 index 0000000..4bb53b8 --- /dev/null +++ b/cmd/tail-watch/testdata/payload/test.json @@ -0,0 +1,10 @@ +{ + "version": 1, + "id": "evt_01JAB3K9T6M4X2Q8R7N5W0YZCM", + "type": "test", + "severity": "info", + "time": "2026-04-02T14:02:11Z", + "collector": { + "name": "monitor-nyc" + } +} diff --git a/cmd/tail-watch/testdata/slack/down-mention.json b/cmd/tail-watch/testdata/slack/down-mention.json new file mode 100644 index 0000000..e0e23d9 --- /dev/null +++ b/cmd/tail-watch/testdata/slack/down-mention.json @@ -0,0 +1,28 @@ +{ + "text": "\n:red_circle: DOWN nyc-nas tcp:445 (service)\naddr 10.20.0.50 · site nyc · parent nyc-router · path direct 12 ms\nerror: connection refused\ndown since 14:00 UTC · 3 failures · collector monitor-nyc", + "blocks": [ + { + "type": "section", + "text": { + "type": "mrkdwn", + "text": "" + } + }, + { + "type": "section", + "text": { + "type": "mrkdwn", + "text": ":red_circle: DOWN nyc-nas tcp:445 (service)\naddr 10.20.0.50 · site nyc · parent nyc-router · path direct 12 ms\nerror: connection refused\ndown since 14:00 UTC · 3 failures · collector monitor-nyc" + } + }, + { + "type": "context", + "elements": [ + { + "type": "mrkdwn", + "text": "monitor-nyc · 2026-04-02 14:02 UTC" + } + ] + } + ] +} diff --git a/cmd/tail-watch/testdata/slack/down-status-link.json b/cmd/tail-watch/testdata/slack/down-status-link.json new file mode 100644 index 0000000..16a3c8e --- /dev/null +++ b/cmd/tail-watch/testdata/slack/down-status-link.json @@ -0,0 +1,21 @@ +{ + "text": ":red_circle: DOWN nyc-nas tcp:445 (service)\naddr 10.20.0.50 · site nyc · parent nyc-router · path direct 12 ms\nerror: connection refused\ndown since 14:00 UTC · 3 failures · collector monitor-nyc", + "blocks": [ + { + "type": "section", + "text": { + "type": "mrkdwn", + "text": ":red_circle: DOWN tcp:445 (service)\naddr 10.20.0.50 · site nyc · parent nyc-router · path direct 12 ms\nerror: connection refused\ndown since 14:00 UTC · 3 failures · collector monitor-nyc" + } + }, + { + "type": "context", + "elements": [ + { + "type": "mrkdwn", + "text": "monitor-nyc · 2026-04-02 14:02 UTC" + } + ] + } + ] +} diff --git a/cmd/tail-watch/testdata/slack/down.json b/cmd/tail-watch/testdata/slack/down.json new file mode 100644 index 0000000..ced6555 --- /dev/null +++ b/cmd/tail-watch/testdata/slack/down.json @@ -0,0 +1,21 @@ +{ + "text": ":red_circle: DOWN nyc-nas tcp:445 (service)\naddr 10.20.0.50 · site nyc · parent nyc-router · path direct 12 ms\nerror: connection refused\ndown since 14:00 UTC · 3 failures · collector monitor-nyc", + "blocks": [ + { + "type": "section", + "text": { + "type": "mrkdwn", + "text": ":red_circle: DOWN nyc-nas tcp:445 (service)\naddr 10.20.0.50 · site nyc · parent nyc-router · path direct 12 ms\nerror: connection refused\ndown since 14:00 UTC · 3 failures · collector monitor-nyc" + } + }, + { + "type": "context", + "elements": [ + { + "type": "mrkdwn", + "text": "monitor-nyc · 2026-04-02 14:02 UTC" + } + ] + } + ] +} diff --git a/cmd/tail-watch/testdata/slack/recovered-removed.json b/cmd/tail-watch/testdata/slack/recovered-removed.json new file mode 100644 index 0000000..9937d04 --- /dev/null +++ b/cmd/tail-watch/testdata/slack/recovered-removed.json @@ -0,0 +1,21 @@ +{ + "text": ":large_green_circle: RECOVERED nyc-nas tcp:445 after 1h6m30s (target removed)", + "blocks": [ + { + "type": "section", + "text": { + "type": "mrkdwn", + "text": ":large_green_circle: RECOVERED nyc-nas tcp:445 after 1h6m30s (target removed)" + } + }, + { + "type": "context", + "elements": [ + { + "type": "mrkdwn", + "text": "monitor-nyc · 2026-04-02 15:07 UTC" + } + ] + } + ] +} diff --git a/cmd/tail-watch/testdata/slack/recovered.json b/cmd/tail-watch/testdata/slack/recovered.json new file mode 100644 index 0000000..11d9b1f --- /dev/null +++ b/cmd/tail-watch/testdata/slack/recovered.json @@ -0,0 +1,21 @@ +{ + "text": ":large_green_circle: RECOVERED nyc-nas tcp:445 after 13m30s", + "blocks": [ + { + "type": "section", + "text": { + "type": "mrkdwn", + "text": ":large_green_circle: RECOVERED nyc-nas tcp:445 after 13m30s" + } + }, + { + "type": "context", + "elements": [ + { + "type": "mrkdwn", + "text": "monitor-nyc · 2026-04-02 14:14 UTC" + } + ] + } + ] +} diff --git a/cmd/tail-watch/testdata/slack/router-14-deps.json b/cmd/tail-watch/testdata/slack/router-14-deps.json new file mode 100644 index 0000000..94c8e47 --- /dev/null +++ b/cmd/tail-watch/testdata/slack/router-14-deps.json @@ -0,0 +1,21 @@ +{ + "text": ":red_circle: DOWN nyc-router (learned) tcp:445 (unrouted)\naddr 100.64.0.9 · site nyc · path direct 12 ms\nerror: connection refused\ndown since 14:00 UTC · 3 failures · collector monitor-nyc\nand 14 targets behind it: nyc-host-01, nyc-host-02, nyc-host-03, nyc-host-04, nyc-host-05, nyc-host-06, nyc-host-07, nyc-host-08, nyc-host-09, nyc-host-10, …", + "blocks": [ + { + "type": "section", + "text": { + "type": "mrkdwn", + "text": ":red_circle: DOWN nyc-router (learned) tcp:445 (unrouted)\naddr 100.64.0.9 · site nyc · path direct 12 ms\nerror: connection refused\ndown since 14:00 UTC · 3 failures · collector monitor-nyc\nand 14 targets behind it: nyc-host-01, nyc-host-02, nyc-host-03, nyc-host-04, nyc-host-05, nyc-host-06, nyc-host-07, nyc-host-08, nyc-host-09, nyc-host-10, …" + } + }, + { + "type": "context", + "elements": [ + { + "type": "mrkdwn", + "text": "monitor-nyc · 2026-04-02 14:02 UTC" + } + ] + } + ] +} diff --git a/cmd/tail-watch/testdata/slack/warn-with-mention.json b/cmd/tail-watch/testdata/slack/warn-with-mention.json new file mode 100644 index 0000000..ced6555 --- /dev/null +++ b/cmd/tail-watch/testdata/slack/warn-with-mention.json @@ -0,0 +1,21 @@ +{ + "text": ":red_circle: DOWN nyc-nas tcp:445 (service)\naddr 10.20.0.50 · site nyc · parent nyc-router · path direct 12 ms\nerror: connection refused\ndown since 14:00 UTC · 3 failures · collector monitor-nyc", + "blocks": [ + { + "type": "section", + "text": { + "type": "mrkdwn", + "text": ":red_circle: DOWN nyc-nas tcp:445 (service)\naddr 10.20.0.50 · site nyc · parent nyc-router · path direct 12 ms\nerror: connection refused\ndown since 14:00 UTC · 3 failures · collector monitor-nyc" + } + }, + { + "type": "context", + "elements": [ + { + "type": "mrkdwn", + "text": "monitor-nyc · 2026-04-02 14:02 UTC" + } + ] + } + ] +} diff --git a/cmd/tail-watch/testdata/snippet.hujson b/cmd/tail-watch/testdata/snippet.hujson index 7f25408..a7b19f6 100644 --- a/cmd/tail-watch/testdata/snippet.hujson +++ b/cmd/tail-watch/testdata/snippet.hujson @@ -9,10 +9,12 @@ "grants": [ // The peer the test discovers is a member device. The grant also puts // it in the collector's netmap, which is all tsping (disco) needs. + // TestLiveNotifications also probes its fixture nodes with icmp, tcp:9000 + // (the tcp check) and tcp:8080 (the http check). { "src": ["tag:monitor"], "dst": ["autogroup:member"], - "ip": ["tcp:22"], + "ip": ["icmp:*", "tcp:22", "tcp:8080", "tcp:9000"], }, // Subnet targets, including the test router's 10.10.50.1. { diff --git a/cmd/tail-watch/webhook.go b/cmd/tail-watch/webhook.go new file mode 100644 index 0000000..05751bc --- /dev/null +++ b/cmd/tail-watch/webhook.go @@ -0,0 +1,71 @@ +package main + +import ( + "context" + "fmt" + "net/http" + "time" +) + +// webhookSink posts JSON events to a generic receiver, signed when it has a +// secret. +type webhookSink struct { + cfg SinkConfig + client *http.Client +} + +func newWebhookSink(cfg SinkConfig, client *http.Client) *webhookSink { + return &webhookSink{cfg: cfg, client: client} +} + +func (w *webhookSink) Name() string { return w.cfg.Name } + +func (w *webhookSink) MinInterval() time.Duration { return 0 } + +type batchBody struct { + Version int `json:"version"` + BatchID string `json:"batch_id"` + Events []Event `json:"events"` +} + +// Render is one event payload, or a batch envelope when batching. +func (w *webhookSink) Render(evs []Event, unitID string) ([]byte, error) { + if w.cfg.Batch { + return marshalJSON(batchBody{Version: eventVersion, BatchID: unitID, Events: evs}) + } + if len(evs) != 1 { + return nil, fmt.Errorf("webhook sink %s: %d events in a single-event request", w.cfg.Name, len(evs)) + } + return marshalJSON(evs[0]) +} + +func (w *webhookSink) Send(ctx context.Context, body []byte, m sendMeta) deliveryResult { + url, err := w.cfg.URL.resolve() + if err != nil { + return secretUnavailable(fmt.Errorf("sink %s: url: %w", w.cfg.Name, err)) + } + headers := map[string]string{ + "Content-Type": "application/json", + "User-Agent": "tail-watch/" + versionString(), + } + if w.cfg.Batch { + headers["Tail-Watch-Batch-Id"] = m.ID + } else { + headers["Tail-Watch-Event-Id"] = m.ID + } + for name, env := range w.cfg.HeadersEnv { + v, headerErr := (secretRef{Env: env, File: ""}).resolve() + if headerErr != nil { + return secretUnavailable(fmt.Errorf("sink %s: header %s: %w", w.cfg.Name, name, headerErr)) + } + headers[name] = v + } + if w.cfg.Secret.isSet() { + secret, secretErr := w.cfg.Secret.resolve() + if secretErr != nil { + return secretUnavailable(fmt.Errorf("sink %s: secret: %w", w.cfg.Name, secretErr)) + } + headers[signatureHeader] = signBody(secret, m.At.Unix(), body) + } + return httpPost(ctx, w.client, url, body, headers, m.At) +} diff --git a/cmd/tail-watch/webhook_test.go b/cmd/tail-watch/webhook_test.go new file mode 100644 index 0000000..0e32dcd --- /dev/null +++ b/cmd/tail-watch/webhook_test.go @@ -0,0 +1,490 @@ +package main + +import ( + "context" + "crypto/hmac" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "fmt" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strconv" + "strings" + "sync" + "testing" + "time" + + "github.com/prometheus/client_golang/prometheus/testutil" +) + +const ( + vectorSecret = "whsec_test_key" + vectorTime = int64(1700000000) + vectorBody = `{"id":"evt_X","type":"test"}` + vectorSignature = "t=1700000000,v1=1419cf10ea0dc484e3a06e7df47e222718af9df5f1bb5180b9c3bb9cde19722f" +) + +// verifySignature is the receiver side: it recomputes the HMAC independently of +// signBody and rejects timestamps outside tolerance. +func verifySignature(secret, header string, body []byte, now time.Time, tolerance time.Duration) bool { + var ts, sig string + for part := range strings.SplitSeq(header, ",") { + k, v, _ := strings.Cut(part, "=") + switch k { + case "t": + ts = v + case "v1": + sig = v + } + } + secs, err := strconv.ParseInt(ts, 10, 64) + if err != nil || now.Sub(time.Unix(secs, 0)).Abs() > tolerance { + return false + } + mac := hmac.New(sha256.New, []byte(secret)) + fmt.Fprintf(mac, "%s.%s", ts, body) + return hmac.Equal([]byte(hex.EncodeToString(mac.Sum(nil))), []byte(sig)) +} + +func TestSignBodyVector(t *testing.T) { + t.Parallel() + if got := signBody(vectorSecret, vectorTime, []byte(vectorBody)); got != vectorSignature { + t.Fatalf("signature %s, want independently computed %s", got, vectorSignature) + } + if signBody(vectorSecret, vectorTime+1, []byte(vectorBody)) == vectorSignature { + t.Fatal("changing the timestamp must change the signature") + } + if signBody(vectorSecret, vectorTime, []byte(vectorBody+" ")) == vectorSignature { + t.Fatal("changing the body must change the signature") + } + if signBody("other", vectorTime, []byte(vectorBody)) == vectorSignature { + t.Fatal("changing the secret must change the signature") + } +} + +func TestVerifySignatureRules(t *testing.T) { + t.Parallel() + now := time.Unix(vectorTime, 0) + if !verifySignature(vectorSecret, vectorSignature, []byte(vectorBody), now, 5*time.Minute) { + t.Fatal("valid signature rejected") + } + if verifySignature("wrong", vectorSignature, []byte(vectorBody), now, 5*time.Minute) { + t.Fatal("wrong secret accepted") + } + if verifySignature(vectorSecret, vectorSignature, []byte(vectorBody), now.Add(time.Hour), 5*time.Minute) { + t.Fatal("stale timestamp accepted") + } + if verifySignature(vectorSecret, vectorSignature, []byte(vectorBody+"x"), now, 5*time.Minute) { + t.Fatal("tampered body accepted") + } +} + +type received struct { + Header http.Header + Body []byte + At time.Time +} + +type receiver struct { + srv *httptest.Server + mu sync.Mutex + got []received + status []int + extra map[int]func(http.ResponseWriter) +} + +func newReceiver(t *testing.T, statuses ...int) *receiver { + t.Helper() + r := &receiver{status: statuses, extra: map[int]func(http.ResponseWriter){}, srv: nil, mu: sync.Mutex{}, got: nil} + r.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { + body := make([]byte, 0, 1024) + buf := make([]byte, 4096) + for { + n, err := req.Body.Read(buf) + body = append(body, buf[:n]...) + if err != nil { + break + } + } + r.mu.Lock() + r.got = append(r.got, received{Header: req.Header.Clone(), Body: body, At: time.Now()}) + status := http.StatusOK + if len(r.status) > 0 { + status = r.status[0] + r.status = r.status[1:] + } + hook := r.extra[len(r.got)] + r.mu.Unlock() + if hook != nil { + hook(w) + } + w.WriteHeader(status) + })) + t.Cleanup(r.srv.Close) + return r +} + +func (r *receiver) count() int { + r.mu.Lock() + defer r.mu.Unlock() + return len(r.got) +} + +func (r *receiver) request(i int) received { + r.mu.Lock() + defer r.mu.Unlock() + return r.got[i] +} + +func webhookFor(t *testing.T, url string, mut func(*SinkConfig)) *webhookSink { + t.Helper() + sc := testSink("hook") + sc.URL = urlFile(t, url) + if mut != nil { + mut(&sc) + } + return newWebhookSink(sc, newSinkClient(nil)) +} + +func urlFile(t *testing.T, url string) secretRef { + t.Helper() + path := filepath.Join(t.TempDir(), "url") + if err := os.WriteFile(path, []byte(url), 0o600); err != nil { + t.Fatal(err) + } + return secretRef{Env: "", File: path} +} + +func TestWebhookRequestFormatAndSignature(t *testing.T) { + t.Setenv("TW_TEST_SECRET", vectorSecret) + t.Setenv("TW_TEST_AUTH", "Bearer abc") + rc := newReceiver(t) + w := webhookFor(t, rc.srv.URL+"/hook/private", func(sc *SinkConfig) { + sc.Secret = secretRef{Env: "TW_TEST_SECRET", File: ""} + sc.HeadersEnv = map[string]string{"Authorization": "TW_TEST_AUTH"} + }) + ev := newDownEvent("evt_01JAB3K9T6M4X2Q8R7N5W0YZCD", goldenCollector(), goldenTransition()) + body, err := w.Render([]Event{ev}, ev.ID) + if err != nil { + t.Fatal(err) + } + now := time.Now() + res := w.Send(context.Background(), body, sendMeta{ID: ev.ID, At: now}) + if res.Class != resultOK || res.Status != 200 { + t.Fatalf("result %+v", res) + } + got := rc.request(0) + if got.Header.Get("Content-Type") != "application/json" || + !strings.HasPrefix(got.Header.Get("User-Agent"), "tail-watch/") || + got.Header.Get("Authorization") != "Bearer abc" || + got.Header.Get("Tail-Watch-Event-Id") != ev.ID { + t.Fatalf("headers %v", got.Header) + } + var decoded Event + if decodeErr := json.Unmarshal(got.Body, &decoded); decodeErr != nil || decoded.ID != ev.ID { + t.Fatalf("body %s (%v)", got.Body, decodeErr) + } + if !verifySignature(vectorSecret, got.Header.Get(signatureHeader), got.Body, now, time.Minute) { + t.Fatal("receiver could not verify the signature") + } + if verifySignature("different", got.Header.Get(signatureHeader), got.Body, now, time.Minute) { + t.Fatal("a different secret verified") + } +} + +func TestWebhookUnsignedWithoutSecret(t *testing.T) { + t.Parallel() + rc := newReceiver(t) + w := webhookFor(t, rc.srv.URL, nil) + ev := newTestEvent("evt_A", goldenCollector(), testNow()) + body, _ := w.Render([]Event{ev}, ev.ID) + if res := w.Send(context.Background(), body, sendMeta{ID: ev.ID, At: time.Now()}); res.Class != resultOK { + t.Fatalf("%+v", res) + } + if rc.request(0).Header.Get(signatureHeader) != "" { + t.Fatal("unsigned sink must not send a signature") + } +} + +func TestWebhookResultClassification(t *testing.T) { + t.Parallel() + now := time.Now() + tests := []struct { + status int + hdr string + class resultClass + reason string + after time.Duration + }{ + {200, "", resultOK, "", 0}, + {204, "", resultOK, "", 0}, + {500, "", resultRetry, failHTTP5xx, 0}, + {503, "", resultRetry, failHTTP5xx, 0}, + {408, "", resultRetry, failTimeout, 0}, + {429, "30", resultRetry, failHTTP429, 30 * time.Second}, + {429, "", resultRetry, failHTTP429, 0}, + {400, "", resultPermanent, failHTTP4xx, 0}, + {401, "", resultPermanent, failHTTP4xx, 0}, + {302, "", resultPermanent, failHTTP4xx, 0}, + } + for _, tt := range tests { + h := http.Header{} + if tt.hdr != "" { + h.Set("Retry-After", tt.hdr) + } + got := classifyStatus(tt.status, h, now) + if got.Class != tt.class || got.Reason != tt.reason || got.RetryAfter != tt.after { + t.Errorf("status %d: %+v", tt.status, got) + } + } + date := now.Add(45 * time.Second).UTC().Format(http.TimeFormat) + h := http.Header{} + h.Set("Retry-After", date) + if got := classifyStatus(429, h, now); got.RetryAfter < 40*time.Second || got.RetryAfter > 46*time.Second { + t.Fatalf("HTTP-date Retry-After = %s", got.RetryAfter) + } +} + +func TestWebhookOverHTTPMatrix(t *testing.T) { + t.Parallel() + tests := []struct { + name string + status int + class resultClass + reason string + }{ + {"server error", 500, resultRetry, failHTTP5xx}, + {"request timeout", 408, resultRetry, failTimeout}, + {"bad request", 400, resultPermanent, failHTTP4xx}, + {"redirect not followed", 302, resultPermanent, failHTTP4xx}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + rc := newReceiver(t, tt.status, http.StatusOK) + w := webhookFor(t, rc.srv.URL, nil) + ev := newTestEvent("evt_A", goldenCollector(), testNow()) + body, _ := w.Render([]Event{ev}, ev.ID) + res := w.Send(context.Background(), body, sendMeta{ID: ev.ID, At: time.Now()}) + if res.Class != tt.class || res.Reason != tt.reason { + t.Fatalf("%+v", res) + } + if rc.count() != 1 { + t.Fatalf("redirects must not be followed, saw %d requests", rc.count()) + } + }) + } +} + +func TestWebhookTimeoutAndNetworkErrors(t *testing.T) { + t.Parallel() + slow := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) { + select { + case <-r.Context().Done(): + case <-time.After(2 * time.Second): + } + })) + defer slow.Close() + w := webhookFor(t, slow.URL+"/secret/path", nil) + w.client.Timeout = 50 * time.Millisecond + ev := newTestEvent("evt_A", goldenCollector(), testNow()) + body, _ := w.Render([]Event{ev}, ev.ID) + res := w.Send(context.Background(), body, sendMeta{ID: ev.ID, At: time.Now()}) + if res.Class != resultRetry || res.Reason != failTimeout { + t.Fatalf("timeout classified as %+v", res) + } + if res.Err != nil && strings.Contains(res.Err.Error(), "/secret/path") { + t.Fatalf("error leaks the URL path: %v", res.Err) + } + closed := httptest.NewServer(http.NotFoundHandler()) + url := closed.URL + closed.Close() + w2 := webhookFor(t, url+"/secret/path", nil) + res = w2.Send(context.Background(), body, sendMeta{ID: ev.ID, At: time.Now()}) + if res.Class != resultRetry || res.Reason != failNetwork { + t.Fatalf("dial error classified as %+v", res) + } + if strings.Contains(res.Err.Error(), "/secret/path") { + t.Fatalf("error leaks the URL path: %v", res.Err) + } +} + +func TestWebhookThroughNotifierRetriesServerError(t *testing.T) { + rc := newReceiver(t, 500, 200) + t.Setenv("TW_TEST_URL", rc.srv.URL) + sc := testSink("hook") + sc.URL = secretRef{Env: "TW_TEST_URL", File: ""} + h := realSinkHarness( + t, + harnessOpts{sinks: []SinkConfig{sc}, stateDir: "", tags: nil, intake: 0, scripts: nil, clock: nil}, + ) + h.start(t) + before := h.clock.timers.Load() + h.n.ObserveTransition(downTransition(1, h.clock.Now())) + eventually(t, func() bool { return rc.count() == 1 }) + h.clock.waitTimers(t, before) + h.clock.Advance(time.Second) + eventually(t, func() bool { return rc.count() == 2 }) + if string(rc.request(0).Body) != string(rc.request(1).Body) { + t.Fatal("retry changed the body") + } + if rc.request(0).Header.Get("Tail-Watch-Event-Id") != rc.request(1).Header.Get("Tail-Watch-Event-Id") { + t.Fatal("retry changed the event id") + } +} + +func TestWebhookBatchedBurstSplits(t *testing.T) { + rc := newReceiver(t) + t.Setenv("TW_TEST_URL", rc.srv.URL) + sc := testSink("hook") + sc.URL = secretRef{Env: "TW_TEST_URL", File: ""} + sc.Batch = true + h := realSinkHarness( + t, + harnessOpts{sinks: []SinkConfig{sc}, stateDir: "", tags: nil, intake: 0, scripts: nil, clock: nil}, + ) + h.n.mu.Lock() + for i := range 250 { + id := fmt.Sprintf("evt_%03d", i) + h.n.enqueueLocked(newFlappingEvent(id, goldenCollector(), goldenTransition())) + } + h.n.commitLocked() + h.n.mu.Unlock() + h.start(t) + eventually(t, func() bool { return rc.count() == 2 }) + h.clock.waitTimers(t, 0) + h.clock.Advance(sc.GroupWait) + eventually(t, func() bool { return rc.count() == 3 }) + sizes := []int{} + var firstID string + next := 0 + for i := range 3 { + req := rc.request(i) + var b batchBody + if err := json.Unmarshal(req.Body, &b); err != nil { + t.Fatal(err) + } + sizes = append(sizes, len(b.Events)) + if req.Header.Get("Tail-Watch-Batch-Id") != b.BatchID || req.Header.Get("Tail-Watch-Event-Id") != "" || + !strings.HasPrefix(b.BatchID, "bat_") || b.Version != 1 { + t.Fatalf("batch envelope %+v headers %v", b, req.Header) + } + for _, e := range b.Events { + if e.ID != fmt.Sprintf("evt_%03d", next) { + t.Fatalf("batch order broken at %d: %s", next, e.ID) + } + next++ + } + if i == 0 { + firstID = b.BatchID + } + } + if fmt.Sprint(sizes) != "[100 100 50]" { + t.Fatalf("batch sizes %v", sizes) + } + if firstID == "" { + t.Fatal("no batch id") + } + if n := testutil.CollectAndCount(h.n.m.batchSizeH); n != 1 { + t.Fatalf("batch_size histogram series = %d", n) + } +} + +func TestWebhookBatchRetryKeepsBytesAndFreshSignature(t *testing.T) { + t.Setenv("TW_TEST_SECRET", vectorSecret) + rc := newReceiver(t, 503, 200) + t.Setenv("TW_TEST_URL", rc.srv.URL) + sc := testSink("hook") + sc.URL = secretRef{Env: "TW_TEST_URL", File: ""} + sc.Secret = secretRef{Env: "TW_TEST_SECRET", File: ""} + sc.Batch = true + h := realSinkHarness( + t, + harnessOpts{sinks: []SinkConfig{sc}, stateDir: "", tags: nil, intake: 0, scripts: nil, clock: nil}, + ) + h.start(t) + h.n.ObserveTransition(downTransition(1, h.clock.Now())) + h.clock.waitTimers(t, 0) + h.clock.Advance(sc.GroupWait) + eventually(t, func() bool { return rc.count() == 1 }) + h.clock.waitTimers(t, 1) + h.n.ObserveTransition(upTransition(2, h.clock.Now())) + h.clock.Advance(time.Second) + eventually(t, func() bool { return rc.count() >= 2 }) + a, b := rc.request(0), rc.request(1) + if string(a.Body) != string(b.Body) || a.Header.Get("Tail-Watch-Batch-Id") != b.Header.Get("Tail-Watch-Batch-Id") { + t.Fatal("batch retry must keep body bytes and batch id") + } + ta, _, _ := strings.Cut(strings.TrimPrefix(a.Header.Get(signatureHeader), "t="), ",") + tb, _, _ := strings.Cut(strings.TrimPrefix(b.Header.Get(signatureHeader), "t="), ",") + if ta == "" || tb == "" { + t.Fatal("signature headers missing") + } + var body batchBody + if err := json.Unmarshal(b.Body, &body); err != nil || len(body.Events) != 1 { + t.Fatalf("arrivals during retry must not join the active batch: %v %d", err, len(body.Events)) + } +} + +func TestWebhookBatchDropsWholeBatchOnPermanent(t *testing.T) { + rc := newReceiver(t, 400) + t.Setenv("TW_TEST_URL", rc.srv.URL) + sc := testSink("hook") + sc.URL = secretRef{Env: "TW_TEST_URL", File: ""} + sc.Batch = true + h := realSinkHarness( + t, + harnessOpts{sinks: []SinkConfig{sc}, stateDir: "", tags: nil, intake: 0, scripts: nil, clock: nil}, + ) + h.n.mu.Lock() + for i := range 5 { + h.n.enqueueLocked(newFlappingEvent(fmt.Sprintf("evt_%d", i), goldenCollector(), goldenTransition())) + } + h.n.mu.Unlock() + h.start(t) + h.clock.waitTimers(t, 0) + h.clock.Advance(sc.GroupWait) + eventually(t, func() bool { + return testutil.ToFloat64(h.n.m.droppedC.WithLabelValues("hook", reasonPermanent)) == 5 + }) +} + +func TestNoSecretsReachDisk(t *testing.T) { + rc := newReceiver(t, 503, 503, 503) + dir := filepath.Join(t.TempDir(), "state") + t.Setenv("TW_TEST_URL", rc.srv.URL+"/TOPSECRETPATH") + t.Setenv("TW_TEST_SECRET", "hmac-secret-value-123") + t.Setenv("TW_TEST_AUTH", "Bearer header-secret-456") + sc := testSink("hook") + sc.Queue = queueDisk + sc.URL = secretRef{Env: "TW_TEST_URL", File: ""} + sc.Secret = secretRef{Env: "TW_TEST_SECRET", File: ""} + sc.HeadersEnv = map[string]string{"Authorization": "TW_TEST_AUTH"} + h := realSinkHarness( + t, + harnessOpts{sinks: []SinkConfig{sc}, stateDir: dir, tags: nil, intake: 0, scripts: nil, clock: nil}, + ) + h.start(t) + before := h.clock.timers.Load() + h.n.ObserveTransition(downTransition(1, h.clock.Now())) + eventually(t, func() bool { return rc.count() >= 1 }) + h.clock.waitTimers(t, before) + entries, err := os.ReadDir(dir) + if err != nil || len(entries) == 0 { + t.Fatalf("state dir empty: %v", err) + } + for _, e := range entries { + raw, readErr := os.ReadFile(filepath.Join(dir, e.Name())) + if readErr != nil { + t.Fatal(readErr) + } + for _, secret := range []string{"TOPSECRETPATH", "hmac-secret-value-123", "header-secret-456", rc.srv.URL} { + if strings.Contains(string(raw), secret) { + t.Fatalf("%s contains %q", e.Name(), secret) + } + } + } +} diff --git a/examples/tail-watch.yaml b/examples/tail-watch.yaml index 7617dbe..172eb16 100644 --- a/examples/tail-watch.yaml +++ b/examples/tail-watch.yaml @@ -58,3 +58,29 @@ targets: - icmp - tcp:9100 - "http:http://192.168.1.50/|ready" + +# Notifications. Secrets come only from *_env or *_file keys. +# Try them with: tail-watch notify-test --config examples/tail-watch.yaml +sinks: + - name: ops-webhook + type: webhook + url_env: OPS_WEBHOOK_URL + secret_env: OPS_WEBHOOK_SECRET # HMAC key for Tail-Watch-Signature + min_severity: page + events: [target.down, target.recovered] + queue: disk # survives restarts; needs collector.state_dir + + - name: ops-slack + type: slack + url_env: SLACK_WEBHOOK_URL + min_severity: warn + mention_on_page: "" + group_wait: 10s + status_url: https://monitor-nyc.example.ts.net:9109 + match: + sites: [nyc, lab] + +# Pings a dead-man's-switch service so a dead collector still alerts. +heartbeat: + url_env: HEARTBEAT_URL + interval: 1m diff --git a/openspec/changes/add-notifications/.openspec.yaml b/openspec/changes/archive/2026-09-29-add-notifications/.openspec.yaml similarity index 100% rename from openspec/changes/add-notifications/.openspec.yaml rename to openspec/changes/archive/2026-09-29-add-notifications/.openspec.yaml diff --git a/openspec/changes/add-notifications/design.md b/openspec/changes/archive/2026-09-29-add-notifications/design.md similarity index 96% rename from openspec/changes/add-notifications/design.md rename to openspec/changes/archive/2026-09-29-add-notifications/design.md index c9ebb73..402ad93 100644 --- a/openspec/changes/add-notifications/design.md +++ b/openspec/changes/archive/2026-09-29-add-notifications/design.md @@ -193,6 +193,12 @@ Before starting any checks, load and replay the snapshot, register recorded implicit reachability targets, then restore open incidents as DOWN. Closing incidents only replay pending recovery intents and never restore DOWN. +The alert state machine refuses to remove a configured `DOWN` target but allows +removing a `DOWN` implicit one (see the `alert-state-machine` delta). The +learner asks for that only after authoritative reconciliation; the removal +leaves a removed, dirty snapshot and the notifier's reconcile pass closes the +incident with `reason: removed`. + Close explicitly removed configured targets with `reason: removed`. Keep unresolved implicit incidents through partial/failed discovery. Only a complete, authoritative topology reconciliation can establish that a restored implicit @@ -208,9 +214,10 @@ acknowledgment or an explicit measured drop. ### D8. Webhook sink (`webhook.go`) `http.Client` with `Timeout: 10s` and -`CheckRedirect: http.ErrUseLastResponse`; transport dials through the default -dialer (sinks follow the host routing table, including tailnet receivers -reached over MagicDNS). Signature: +`CheckRedirect: http.ErrUseLastResponse`; the transport dials through +`netEnv.Dial`, which uses the host's default dialer, so sinks follow the host +routing table, including tailnet receivers reached over MagicDNS, and the +"every network access goes through `netEnv`" rule holds. Signature: write `[]byte(t + ".")` and the exact body bytes to `mac := hmac.New(sha256.New, secret)`, then use `hex.EncodeToString(mac.Sum(nil))`. `Sum` does not consume message input. diff --git a/openspec/changes/add-notifications/proposal.md b/openspec/changes/archive/2026-09-29-add-notifications/proposal.md similarity index 91% rename from openspec/changes/add-notifications/proposal.md rename to openspec/changes/archive/2026-09-29-add-notifications/proposal.md index 70861e4..e458db0 100644 --- a/openspec/changes/add-notifications/proposal.md +++ b/openspec/changes/archive/2026-09-29-add-notifications/proposal.md @@ -68,9 +68,13 @@ Non-goals (later phases): ### Modified Capabilities -None. `collector-config` still lives only in the unarchived -`add-tsmon-collector` change, so sink and heartbeat config requirements are -specified in this change's own capabilities. +- `alert-state-machine`: removing a `DOWN` implicit target is allowed, so + authoritative topology reconciliation can close a restored learned router's + incident as `removed`. Configured targets still cannot be removed while `DOWN`. + +`collector-config` still lives only in the unarchived `add-tsmon-collector` +change, so sink and heartbeat config requirements are specified in this +change's own capabilities. ## Impact diff --git a/openspec/changes/archive/2026-09-29-add-notifications/specs/alert-state-machine/spec.md b/openspec/changes/archive/2026-09-29-add-notifications/specs/alert-state-machine/spec.md new file mode 100644 index 0000000..0d8ebe6 --- /dev/null +++ b/openspec/changes/archive/2026-09-29-add-notifications/specs/alert-state-machine/spec.md @@ -0,0 +1,56 @@ +# Spec Delta + +## MODIFIED Requirements + +### Requirement: Runtime target registration + +The state machine SHALL accept targets registered after startup, such as +implicit targets for learned parents that are not in the config. A registered +target SHALL start in `DOWN` when a restore is pending for its name (see +"Restore of open incidents"). Otherwise it SHALL start in `SUPPRESSED` when +its parent is already down, and in `UNKNOWN` in every other case. It SHALL follow every rule that applies to configured targets: +thresholds, flap damping, suppression, grace hold and transitions. Registering +a name that is already registered SHALL return an error. + +The state machine SHALL allow removing a target whose last emitted state is +not `DOWN`, and SHALL refuse removal with an error when a configured target is +`DOWN`, so an open incident always gets its recovery. An implicit target MAY be +removed while `DOWN`: the learner asks only after authoritative topology +reconciliation, and the removal leaves a terminal dirty snapshot from which the +notifier closes the incident with `reason: removed`. A refused caller retries +later. Removal +SHALL cancel the target's timers and drop its metric series. Any dependents +that were `SUPPRESSED` because of it SHALL be released, as when a parent +recovers. + +#### Scenario: Implicit parent is registered + +- **WHEN** an implicit target `nyc-router` is registered and its `tsping` + fails `fail_threshold` times +- **THEN** it emits `UNKNOWN -> DOWN` and suppresses the targets that have it + as parent + +#### Scenario: Remove a down target + +- **WHEN** removal is requested for a configured target whose last emitted + state is `DOWN` +- **THEN** the removal returns an error and the target stays registered + +#### Scenario: Remove a down implicit target + +- **WHEN** removal is requested for an implicit target whose last emitted state + is `DOWN` +- **THEN** it is removed, emits no transition, and its snapshot is a removed, + dirty view that the notifier reconciles into a `reason: removed` recovery + +#### Scenario: Remove a suppressed target + +- **WHEN** removal is requested for a `SUPPRESSED` target whose last emitted + state is `UP` +- **THEN** it is removed and emits no transition + +#### Scenario: Remove an up target + +- **WHEN** removal is requested for an `UP` target +- **THEN** it is removed, emits no transition, and `tailwatch_target_state` no + longer has a series for it diff --git a/openspec/changes/add-notifications/specs/collector-heartbeat/spec.md b/openspec/changes/archive/2026-09-29-add-notifications/specs/collector-heartbeat/spec.md similarity index 100% rename from openspec/changes/add-notifications/specs/collector-heartbeat/spec.md rename to openspec/changes/archive/2026-09-29-add-notifications/specs/collector-heartbeat/spec.md diff --git a/openspec/changes/add-notifications/specs/notification-delivery/spec.md b/openspec/changes/archive/2026-09-29-add-notifications/specs/notification-delivery/spec.md similarity index 100% rename from openspec/changes/add-notifications/specs/notification-delivery/spec.md rename to openspec/changes/archive/2026-09-29-add-notifications/specs/notification-delivery/spec.md diff --git a/openspec/changes/add-notifications/specs/notification-events/spec.md b/openspec/changes/archive/2026-09-29-add-notifications/specs/notification-events/spec.md similarity index 97% rename from openspec/changes/add-notifications/specs/notification-events/spec.md rename to openspec/changes/archive/2026-09-29-add-notifications/specs/notification-events/spec.md index 2bc9428..39f4020 100644 --- a/openspec/changes/add-notifications/specs/notification-events/spec.md +++ b/openspec/changes/archive/2026-09-29-add-notifications/specs/notification-events/spec.md @@ -79,8 +79,9 @@ Every event SHALL be serialized as one JSON object with `version` (integer, `layer`, `error`, `since`, `failures`, `path` (`kind`, `endpoint`, `derp_region`, `latency_ms`, each present only when known), and `dependents` (always an array on `target.down` and -`target.recovered`). `target.recovered` SHALL add `duration_seconds` and -`down_event_id`. Fields that do not apply SHALL be omitted, not sent as null. +`target.recovered`). `target.recovered` SHALL add `duration_seconds` (from the +outage's `since` to recovery) and `down_event_id`, and MAY repeat the down's +`check`, `error`, `since`, `failures` and `path`. Fields that do not apply SHALL be omitted, not sent as null. `layer` SHALL be one of `unrouted`, `control`, `tunnel`, `lan` or `service` on `target.down` and `target.flapping`, and SHALL be omitted on all other events. Payload changes within `version: 1` SHALL only add fields. diff --git a/openspec/changes/add-notifications/specs/notify-test/spec.md b/openspec/changes/archive/2026-09-29-add-notifications/specs/notify-test/spec.md similarity index 100% rename from openspec/changes/add-notifications/specs/notify-test/spec.md rename to openspec/changes/archive/2026-09-29-add-notifications/specs/notify-test/spec.md diff --git a/openspec/changes/add-notifications/specs/slack-sink/spec.md b/openspec/changes/archive/2026-09-29-add-notifications/specs/slack-sink/spec.md similarity index 100% rename from openspec/changes/add-notifications/specs/slack-sink/spec.md rename to openspec/changes/archive/2026-09-29-add-notifications/specs/slack-sink/spec.md diff --git a/openspec/changes/add-notifications/specs/webhook-sink/spec.md b/openspec/changes/archive/2026-09-29-add-notifications/specs/webhook-sink/spec.md similarity index 100% rename from openspec/changes/add-notifications/specs/webhook-sink/spec.md rename to openspec/changes/archive/2026-09-29-add-notifications/specs/webhook-sink/spec.md diff --git a/openspec/changes/add-notifications/tasks.md b/openspec/changes/archive/2026-09-29-add-notifications/tasks.md similarity index 75% rename from openspec/changes/add-notifications/tasks.md rename to openspec/changes/archive/2026-09-29-add-notifications/tasks.md index d8816eb..ebabd9a 100644 --- a/openspec/changes/add-notifications/tasks.md +++ b/openspec/changes/archive/2026-09-29-add-notifications/tasks.md @@ -2,70 +2,70 @@ ## 1. Config schema (`cmd/tail-watch/config.go`) -- [ ] 1.1 Add `SinkConfig`, `MatchConfig`, `HeartbeatConfig`, `secretRef` and the `sinks`/`heartbeat` fields to `Config`; verify decoding the design doc's full `sinks`/`heartbeat` example succeeds with `KnownFields(true)` -- [ ] 1.2 Add sink and heartbeat defaults (`min_severity: info`, all events, `max_age: 1h`, `queue: memory`, `queue_size: 1000`, `group_wait: 10s`, `max_batch: 100`, `heartbeat.interval: 1m`); verify a table test of resolved values -- [ ] 1.3 Add validation rules from design D12 to the error-collecting pass; verify one table case per rule plus a multi-error case, including `queue: disk` without `state_dir` and the reserved sink names `incidents`, `intake` and `state` -- [ ] 1.4 Implement `secretRef` resolution (env, file with whitespace trim, unset/empty/unreadable errors) and `redactURL`; verify tests with `t.Setenv` and temp files, and that `redactURL("https://hooks.slack.com/services/T0/B0/x")` returns `https://hooks.slack.com/…` -- [ ] 1.5 Extend `check-config` to print sinks (name, type, filters, queue mode) without secret values; verify a test asserting a set secret value never appears in the output +- [x] 1.1 Add `SinkConfig`, `MatchConfig`, `HeartbeatConfig`, `secretRef` and the `sinks`/`heartbeat` fields to `Config`; verify decoding the design doc's full `sinks`/`heartbeat` example succeeds with `KnownFields(true)` +- [x] 1.2 Add sink and heartbeat defaults (`min_severity: info`, all events, `max_age: 1h`, `queue: memory`, `queue_size: 1000`, `group_wait: 10s`, `max_batch: 100`, `heartbeat.interval: 1m`); verify a table test of resolved values +- [x] 1.3 Add validation rules from design D12 to the error-collecting pass; verify one table case per rule plus a multi-error case, including `queue: disk` without `state_dir` and the reserved sink names `incidents`, `intake` and `state` +- [x] 1.4 Implement `secretRef` resolution (env, file with whitespace trim, unset/empty/unreadable errors) and `redactURL`; verify tests with `t.Setenv` and temp files, and that `redactURL("https://hooks.slack.com/services/T0/B0/x")` returns `https://hooks.slack.com/…` +- [x] 1.5 Extend `check-config` to print sinks (name, type, filters, queue mode) without secret values; verify a test asserting a set secret value never appears in the output ## 2. Events and routing (`cmd/tail-watch/notify.go`) -- [ ] 2.1 Implement the `evt_`/`bat_` id generator (D2); verify tests for length 26 after prefix, Crockford alphabet, and lexical order across milliseconds using `tstest.Clock` -- [ ] 2.2 Define `Event` with JSON tags per the payload contract; verify golden-file tests (`testdata/payload/*.json`) for `target.down`, `target.recovered` (with and without `"reason": "removed"`), `target.flapping`, `collector.started` and `test` -- [ ] 2.3 Implement the Transition→Event mapping table from D1, severity rules and collector identity; verify table tests over synthetic `Transition`s including `UNKNOWN -> DOWN` (pages), `SUPPRESSED -> DOWN`, `SUPPRESSED -> UP` with an open incident, `UNKNOWN -> UP` (no event), `* -> SUPPRESSED` (no event), each layer value including `unrouted`, `layer` omitted on recovered, and `"implicit": true` for an implicit target (golden file) -- [ ] 2.4 Implement routing filters (`events`, `min_severity`, `match.sites`, `match.targets` globs, `match.tags` via `presence.TagsFor`) and the `test` bypass; verify table tests for each filter and a collector.started event ignoring `match` -- [ ] 2.5 Emit `collector.started` on `FirstRoundDone()`; verify with a fake signal that exactly one event is emitted +- [x] 2.1 Implement the `evt_`/`bat_` id generator (D2); verify tests for length 26 after prefix, Crockford alphabet, and lexical order across milliseconds using `tstest.Clock` +- [x] 2.2 Define `Event` with JSON tags per the payload contract; verify golden-file tests (`testdata/payload/*.json`) for `target.down`, `target.recovered` (with and without `"reason": "removed"`), `target.flapping`, `collector.started` and `test` +- [x] 2.3 Implement the Transition→Event mapping table from D1, severity rules and collector identity; verify table tests over synthetic `Transition`s including `UNKNOWN -> DOWN` (pages), `SUPPRESSED -> DOWN`, `SUPPRESSED -> UP` with an open incident, `UNKNOWN -> UP` (no event), `* -> SUPPRESSED` (no event), each layer value including `unrouted`, `layer` omitted on recovered, and `"implicit": true` for an implicit target (golden file) +- [x] 2.4 Implement routing filters (`events`, `min_severity`, `match.sites`, `match.targets` globs, `match.tags` via `presence.TagsFor`) and the `test` bypass; verify table tests for each filter and a collector.started event ignoring `match` +- [x] 2.5 Emit `collector.started` on `FirstRoundDone()`; verify with a fake signal that exactly one event is emitted ## 3. Queues (`cmd/tail-watch/queue.go`) -- [ ] 3.1 Implement `memQueue` (bounded ring, drop-oldest, `Peek`/`Ack`/`Len`); verify tests for ordering and overflow counting `queue_full` -- [ ] 3.2 Implement the versioned `notification-state.json` coordinator in `queue.go` and `incidents.go`: atomic incident+delivery-intent snapshot, temp/fsync/rename/directory-fsync, permissions 0700/0600; verify workers cannot send disk intents before commit and all selected disk sinks commit together. -- [ ] 3.3 Implement replay of the atomic snapshot in `queue.go`: ordered pending intents, committed attempt/batch identity, expiry/overflow plus eligibility updates; verify crashes before rename, after commit before worker distribution, and after `2xx` before acknowledgment commit, for both down and recovery; corrupt committed state must fail startup. -- [ ] 3.4 Implement failed-commit fallback in `queue.go` and `incidents.go`: retain event and incident mutations together in memory, count `sink="state"`, retry full-state persistence and log degradation; verify delivery during failed writes and restoration of durability after writes recover. -- [ ] 3.5 Warn when `state_dir` is group- or world-readable; verify a test with a 0755 directory captures the warning -- [ ] 3.6 Verify no secret reaches disk: test that after queueing to a signed webhook sink with an auth header, no file under `state_dir` contains the URL, secret or header value +- [x] 3.1 Implement `memQueue` (bounded ring, drop-oldest, `Peek`/`Ack`/`Len`); verify tests for ordering and overflow counting `queue_full` +- [x] 3.2 Implement the versioned `notification-state.json` coordinator in `queue.go` and `incidents.go`: atomic incident+delivery-intent snapshot, temp/fsync/rename/directory-fsync, permissions 0700/0600; verify workers cannot send disk intents before commit and all selected disk sinks commit together. +- [x] 3.3 Implement replay of the atomic snapshot in `queue.go`: ordered pending intents, committed attempt/batch identity, expiry/overflow plus eligibility updates; verify crashes before rename, after commit before worker distribution, and after `2xx` before acknowledgment commit, for both down and recovery; corrupt committed state must fail startup. +- [x] 3.4 Implement failed-commit fallback in `queue.go` and `incidents.go`: retain event and incident mutations together in memory, count `sink="state"`, retry full-state persistence and log degradation; verify delivery during failed writes and restoration of durability after writes recover. +- [x] 3.5 Warn when `state_dir` is group- or world-readable; verify a test with a 0755 directory captures the warning +- [x] 3.6 Verify no secret reaches disk: test that after queueing to a signed webhook sink with an auth header, no file under `state_dir` contains the URL, secret or header value ## 4. Dispatcher and delivery loop (`cmd/tail-watch/notify.go`) -- [ ] 4.0 Implement non-blocking 4096-slot intake in `notify.go`, marking dirty sequences on overflow; reconcile snapshots after capacity returns even without new transitions and ignore stale queued sequences; verify dropped recovery plus permanently UP target eventually gets its resolve, and concurrent newer dirty state remains pending. -- [ ] 4.1 Implement per-sink workers and the shared retry loop (D5 schedule, `Retry-After`, `max_age` expiry, late flag fixed at first attempt); verify under `tstest.Clock` that delays follow 1s, 2s, 4s… capped at 5m and that `Retry-After: 30` delays at least 30s -- [ ] 4.2 Verify worker network isolation in `notify_test.go` with a blocked receiver and another delivering sink; separately test/document serialized snapshot commits delay all durable enqueues while checks and non-blocking intake continue. -- [ ] 4.3 Verify per-sink ordering: a test where `down` is retried and `recovered` is queued delivers `down` first -- [ ] 4.4 Register `tailwatch_notify_sent_total`, `tailwatch_notify_failed_total`, `tailwatch_notify_dropped_total`, `tailwatch_notify_queue_len`, `tailwatch_notify_latency_seconds`, `tailwatch_notify_batch_size`, `tailwatch_notify_persist_errors_total`; verify with `prometheus/testutil` counts after a scripted delivery run +- [x] 4.0 Implement non-blocking 4096-slot intake in `notify.go`, marking dirty sequences on overflow; reconcile snapshots after capacity returns even without new transitions and ignore stale queued sequences; verify dropped recovery plus permanently UP target eventually gets its resolve, and concurrent newer dirty state remains pending. +- [x] 4.1 Implement per-sink workers and the shared retry loop (D5 schedule, `Retry-After`, `max_age` expiry, late flag fixed at first attempt); verify under `tstest.Clock` that delays follow 1s, 2s, 4s… capped at 5m and that `Retry-After: 30` delays at least 30s +- [x] 4.2 Verify worker network isolation in `notify_test.go` with a blocked receiver and another delivering sink; separately test/document serialized snapshot commits delay all durable enqueues while checks and non-blocking intake continue. +- [x] 4.3 Verify per-sink ordering: a test where `down` is retried and `recovered` is queued delivers `down` first +- [x] 4.4 Register `tailwatch_notify_sent_total`, `tailwatch_notify_failed_total`, `tailwatch_notify_dropped_total`, `tailwatch_notify_queue_len`, `tailwatch_notify_latency_seconds`, `tailwatch_notify_batch_size`, `tailwatch_notify_persist_errors_total`; verify with `prometheus/testutil` counts after a scripted delivery run ## 5. Incidents (`cmd/tail-watch/incidents.go`) -- [ ] 5.1 Implement open/closing incidents and delivery records keyed by event ID in `incidents.go`, with at most one open incident per target; commit closing state and recovery intents together; verify a new outage can open while old recovery retries, and closing incidents are never restored as DOWN. -- [ ] 5.2 Implement per-sink pending/attempted/acknowledged/dropped state and recovery eligibility in `notify.go`/`incidents.go`; verify recovery during down retries, filtered sink exclusion, never-attempted down drops, permanent failure after attempt, and acceptance with lost HTTP response. -- [ ] 5.3 Implement startup restoration in `incidents.go`/`main.go` using persisted implicit target definition and stable node identity before scheduling; retain implicit incidents during failed discovery, close them only on authoritative removal, and close explicitly removed configured targets; verify restart with unavailable presence, later rediscovery without duplicate down, actual removal and restore-refusal deduplication. +- [x] 5.1 Implement open/closing incidents and delivery records keyed by event ID in `incidents.go`, with at most one open incident per target; commit closing state and recovery intents together; verify a new outage can open while old recovery retries, and closing incidents are never restored as DOWN. +- [x] 5.2 Implement per-sink pending/attempted/acknowledged/dropped state and recovery eligibility in `notify.go`/`incidents.go`; verify recovery during down retries, filtered sink exclusion, never-attempted down drops, permanent failure after attempt, and acceptance with lost HTTP response. +- [x] 5.3 Implement startup restoration in `incidents.go`/`main.go` using persisted implicit target definition and stable node identity before scheduling; retain implicit incidents during failed discovery, close them only on authoritative removal, and close explicitly removed configured targets; verify restart with unavailable presence, later rediscovery without duplicate down, actual removal and restore-refusal deduplication. ## 6. Webhook sink (`cmd/tail-watch/webhook.go`) -- [ ] 6.1 Implement signed POST in `webhook.go` by writing timestamp, dot and exact body to HMAC then hex-encoding `Sum(nil)`; verify an independently computed fixed signature vector, changed timestamp/body, wrong secret, stale timestamp and real receiver validation. -- [ ] 6.2 Implement result classification; verify the retry matrix against `httptest.Server`: `500` then `200` retries, `429` with `Retry-After` retries after the delay, timeout retries, `408` retries as `timeout`, `400` and `302` do not retry and count `permanent` -- [ ] 6.3 Implement batched mode in `webhook.go` with stable retry bytes/ID persisted before first disk attempt; verify 250 events split 100/100/50, restart during retry preserves body and ID, arrivals form later batches, and expiry of any attempted member drops/counts the whole batch. -- [ ] 6.4 Document the webhook sink, payload, signature verification (with a Go snippet) and batched mode in `README.md`, and add a webhook sink to `examples/tail-watch.yaml`; verify markdownlint passes and `tail-watch check-config --config examples/tail-watch.yaml` exits 0 +- [x] 6.1 Implement signed POST in `webhook.go` by writing timestamp, dot and exact body to HMAC then hex-encoding `Sum(nil)`; verify an independently computed fixed signature vector, changed timestamp/body, wrong secret, stale timestamp and real receiver validation. +- [x] 6.2 Implement result classification; verify the retry matrix against `httptest.Server`: `500` then `200` retries, `429` with `Retry-After` retries after the delay, timeout retries, `408` retries as `timeout`, `400` and `302` do not retry and count `permanent` +- [x] 6.3 Implement batched mode in `webhook.go` with stable retry bytes/ID persisted before first disk attempt; verify 250 events split 100/100/50, restart during retry preserves body and ID, arrivals form later batches, and expiry of any attempted member drops/counts the whole batch. +- [x] 6.4 Document the webhook sink, payload, signature verification (with a Go snippet) and batched mode in `README.md`, and add a webhook sink to `examples/tail-watch.yaml`; verify markdownlint passes and `tail-watch check-config --config examples/tail-watch.yaml` exits 0 ## 7. Slack sink (`cmd/tail-watch/slack.go`) -- [ ] 7.1 Implement Block Kit rendering with text fallback, layer label, `(learned)` suffix for implicit targets, dependents line (up to 10 names), `status_url` links and `mention_on_page` only on page downs; verify golden-file tests (`testdata/slack/*.json`) for down, recovered, router with 14 dependents, warn target with mention configured -- [ ] 7.2 Implement grouping (10 lines + "and N more") and the 1 msg/s limiter on the shared retry loop; verify under `tstest.Clock` that 25 events inside `group_wait` become one message with 10 lines and "and 15 more" -- [ ] 7.3 Document the Slack sink in `README.md` and add one to `examples/tail-watch.yaml`; verify markdownlint and `check-config` pass +- [x] 7.1 Implement Block Kit rendering with text fallback, layer label, `(learned)` suffix for implicit targets, dependents line (up to 10 names), `status_url` links and `mention_on_page` only on page downs; verify golden-file tests (`testdata/slack/*.json`) for down, recovered, router with 14 dependents, warn target with mention configured +- [x] 7.2 Implement grouping (10 lines + "and N more") and the 1 msg/s limiter on the shared retry loop; verify under `tstest.Clock` that 25 events inside `group_wait` become one message with 10 lines and "and 15 more" +- [x] 7.3 Document the Slack sink in `README.md` and add one to `examples/tail-watch.yaml`; verify markdownlint and `check-config` pass ## 8. Heartbeat and notify-test (`cmd/tail-watch/heartbeat.go`, `cmd/tail-watch/notifytest.go`) -- [ ] 8.1 Implement the heartbeat ticker (starts on `FirstRoundDone()`, `GET`, 10s timeout, `tailwatch_heartbeat_failures_total`, stops on shutdown); verify under `tstest.Clock` with an `httptest.Server` that 5 intervals yield 5 requests and a failing receiver increments the counter -- [ ] 8.2 Implement `tail-watch notify-test [--sink] [--config] [--timeout]` using real sinks and the shared retry loop, no queues or `state_dir`, OS-hostname fallback when tailscaled is unreachable; verify tests: two `200` sinks exit 0; one `401` sink exits 1; unknown `--sink` lists sink names -- [ ] 8.3 Document heartbeat, notify-test, queue modes, `state_dir`, open incidents, alerting on `tailwatch_notify_dropped_total`/`tailwatch_notify_queue_len`, and the egress guidance (no exit node; tailnet receivers need a `tag:monitor` grant) in `README.md`; verify markdownlint passes +- [x] 8.1 Implement the heartbeat ticker (starts on `FirstRoundDone()`, `GET`, 10s timeout, `tailwatch_heartbeat_failures_total`, stops on shutdown); verify under `tstest.Clock` with an `httptest.Server` that 5 intervals yield 5 requests and a failing receiver increments the counter +- [x] 8.2 Implement `tail-watch notify-test [--sink] [--config] [--timeout]` using real sinks and the shared retry loop, no queues or `state_dir`, OS-hostname fallback when tailscaled is unreachable; verify tests: two `200` sinks exit 0; one `401` sink exits 1; unknown `--sink` lists sink names +- [x] 8.3 Document heartbeat, notify-test, queue modes, `state_dir`, open incidents, alerting on `tailwatch_notify_dropped_total`/`tailwatch_notify_queue_len`, and the egress guidance (no exit node; tailnet receivers need a `tag:monitor` grant) in `README.md`; verify markdownlint passes ## 9. Wiring (`cmd/tail-watch/main.go`) -- [ ] 9.1 Construct the incident store and notifier before the prober, register the notifier as the alert machine's `transitionConsumer`, start the heartbeat, and stop workers on shutdown; verify `tail-watch --once` still works with sinks configured and emits no notifications -- [ ] 9.2 Add `notify-test` to subcommand dispatch and `--help`; verify `go run ./cmd/tail-watch notify-test --help` prints its flags +- [x] 9.1 Construct the incident store and notifier before the prober, register the notifier as the alert machine's `transitionConsumer`, start the heartbeat, and stop workers on shutdown; verify `tail-watch --once` still works with sinks configured and emits no notifications +- [x] 9.2 Add `notify-test` to subcommand dispatch and `--help`; verify `go run ./cmd/tail-watch notify-test --help` prints its flags ## 10. Validation -- [ ] 10.1 Run `go build ./...`, `go vet ./...`, `go test -race ./...`; all pass -- [ ] 10.2 Run `golangci-lint run ./...` and `go run golang.org/x/vuln/cmd/govulncheck@latest ./...`; zero issues and no reachable vulnerabilities -- [ ] 10.3 Run markdownlint on changed Markdown and `openspec validate add-notifications --strict`; both pass -- [ ] 10.4 On a real tailnet, run `tail-watch notify-test` against a real Slack channel and a webhook receiver that verifies signatures, then stop a monitored service and confirm one down and one recovery in both, and one recovery after restarting tail-watch mid-outage with `queue: disk`; record results in the PR description +- [x] 10.1 Run `go build ./...`, `go vet ./...`, `go test -race ./...`; all pass +- [x] 10.2 Run `golangci-lint run ./...` and `go run golang.org/x/vuln/cmd/govulncheck@latest ./...`; zero issues and no reachable vulnerabilities +- [x] 10.3 Run markdownlint on changed Markdown and `openspec validate add-notifications --strict`; both pass +- [x] 10.4 On a real tailnet, run `tail-watch notify-test` against a real Slack channel and a webhook receiver that verifies signatures, then stop a monitored service and confirm one down and one recovery in both, and one recovery after restarting tail-watch mid-outage with `queue: disk`; record results in the PR description diff --git a/openspec/specs/alert-state-machine/spec.md b/openspec/specs/alert-state-machine/spec.md index eba8d0d..06db3ba 100644 --- a/openspec/specs/alert-state-machine/spec.md +++ b/openspec/specs/alert-state-machine/spec.md @@ -219,8 +219,12 @@ thresholds, flap damping, suppression, grace hold and transitions. Registering a name that is already registered SHALL return an error. The state machine SHALL allow removing a target whose last emitted state is -not `DOWN`, and SHALL refuse removal with an error when it is `DOWN`, so an -open incident always gets its recovery. The caller retries later. Removal +not `DOWN`, and SHALL refuse removal with an error when a configured target is +`DOWN`, so an open incident always gets its recovery. An implicit target MAY be +removed while `DOWN`: the learner asks only after authoritative topology +reconciliation, and the removal leaves a terminal dirty snapshot from which the +notifier closes the incident with `reason: removed`. A refused caller retries +later. Removal SHALL cancel the target's timers and drop its metric series. Any dependents that were `SUPPRESSED` because of it SHALL be released, as when a parent recovers. @@ -234,10 +238,17 @@ recovers. #### Scenario: Remove a down target -- **WHEN** removal is requested for a target whose last emitted state is - `DOWN` +- **WHEN** removal is requested for a configured target whose last emitted + state is `DOWN` - **THEN** the removal returns an error and the target stays registered +#### Scenario: Remove a down implicit target + +- **WHEN** removal is requested for an implicit target whose last emitted state + is `DOWN` +- **THEN** it is removed, emits no transition, and its snapshot is a removed, + dirty view that the notifier reconciles into a `reason: removed` recovery + #### Scenario: Remove a suppressed target - **WHEN** removal is requested for a `SUPPRESSED` target whose last emitted diff --git a/openspec/specs/collector-heartbeat/spec.md b/openspec/specs/collector-heartbeat/spec.md new file mode 100644 index 0000000..f6f34cd --- /dev/null +++ b/openspec/specs/collector-heartbeat/spec.md @@ -0,0 +1,40 @@ +# collector-heartbeat Specification + +## Purpose + +Lets an external dead-man's-switch service page when the collector, its host +or its uplink disappears, since a dead collector cannot report itself. + +## Requirements + +### Requirement: Heartbeat configuration + +The optional top-level `heartbeat` section SHALL require exactly one of +`url_env` or `url_file` and SHALL accept `interval` (default `1m`). The +heartbeat URL SHALL follow the same secret-handling rules as sink URLs. + +#### Scenario: No heartbeat + +- **WHEN** the config has no `heartbeat` section +- **THEN** tail-watch sends no heartbeat requests + +### Requirement: Heartbeat pings + +When configured, tail-watch SHALL send an HTTP `GET` to the heartbeat URL every +`interval`, starting once every configured target/check has produced at least +one result, with a 10s request +timeout. A failed ping SHALL be logged and counted in +`tailwatch_heartbeat_failures_total` and SHALL NOT be retried before the next +interval. Heartbeats SHALL stop when tail-watch shuts down. + +#### Scenario: Regular pings + +- **WHEN** `interval` is `1m` and the collector runs for 5 minutes after every + check has produced a result +- **THEN** the heartbeat URL receives 5 requests, give or take one + +#### Scenario: Collector stops + +- **WHEN** the tail-watch process exits +- **THEN** no further heartbeat requests are sent, so the external service + alerts diff --git a/openspec/specs/notification-delivery/spec.md b/openspec/specs/notification-delivery/spec.md new file mode 100644 index 0000000..63b7564 --- /dev/null +++ b/openspec/specs/notification-delivery/spec.md @@ -0,0 +1,326 @@ +# notification-delivery Specification + +## Purpose + +Defines how events move from the alert state machine to each sink without +blocking checks: per-sink queues and ordering, memory and crash-safe disk +queues, open-incident tracking across restarts, delayed-delivery marking, +secret handling, and the metrics that reveal lost alerts. + +## Requirements + +### Requirement: Non-blocking dispatch + +Accepting a state transition from the alert state machine SHALL be a bounded, +non-blocking enqueue: it SHALL NOT perform network I/O, disk I/O, or wait on +any sink. Mapping, incident updates, routing and persistence SHALL happen off +that call path. If the intake buffer is full, the transition SHALL be dropped, +its target marked dirty for snapshot reconciliation, logged, and counted in +`tailwatch_notify_dropped_total{sink="intake",reason="queue_full"}`. Each sink SHALL have its own worker and its own queue. Network I/O of one sink +SHALL NOT block another sink or any check. Snapshot commits are serialized and +can delay all durable enqueues and workers awaiting attempt-state commit; this +limitation SHALL be +measured and documented. The notifier SHALL reconcile dirty targets within +one second after intake/commit capacity becomes available to begin reconciliation, +including when no +new transitions arrive. An open incident whose notification-visible state is +UP SHALL get a recovery intent. A current DOWN without an open incident SHALL +get a down intent. Intermediate unreported blips may be coalesced. Dirty marks +SHALL be acknowledged only through the reconciled sequence, preserving newer +updates. Queued transitions at or below a target's reconciled sequence SHALL +be ignored. Target removal SHALL retain a terminal snapshot until reconciled. + +#### Scenario: Slow disk + +- **WHEN** `state_dir` is on a disk where each fsync takes 2 seconds +- **THEN** accepting a transition still returns without waiting for any + fsync + +#### Scenario: Lost recovery reconciled + +- **WHEN** intake overflows on recovery and the target stays UP permanently +- **THEN** after capacity returns, the notifier reconciles its dirty snapshot + and commits a recovery intent without requiring another state change + +#### Scenario: Slow webhook + +- **WHEN** a webhook receiver takes 10s per request and a Slack sink is also + configured +- **THEN** Slack messages are delivered without waiting for the webhook + +### Requirement: In-order delivery per sink + +Within one sink, delivery attempts SHALL preserve event order. A recovery SHALL +wait until its down is acknowledged or terminally dropped, +then follow the per-sink incident delivery-state rules. An unattempted down and +its recovery MAY share one batch/group, with down first and their attempt and +acknowledgment states committed together. Network uncertainty +means receiver observation cannot be guaranteed; receivers must handle unknown +resolve IDs. + +#### Scenario: Down retried, recovery queued + +- **WHEN** a `target.down` is being retried and its `target.recovered` is + emitted +- **THEN** recovery waits for the down to become terminal and is sent only if + its per-sink delivery state makes recovery eligible + +### Requirement: Bounded queues + +Each sink's queue SHALL hold at most `queue_size` events (default 1000). When +full, the oldest queued event SHALL be dropped and counted with reason +`queue_full`. + +#### Scenario: Overflow + +- **WHEN** a sink's queue holds 1000 events and another arrives +- **THEN** the oldest event is dropped and + `tailwatch_notify_dropped_total{reason="queue_full"}` increases by 1 + +### Requirement: Late delivery flag + +An event first attempted more than 1 minute after its `time` SHALL carry +`"late": true`, keep its original `time`, and keep that body for all later +attempts. + +#### Scenario: Uplink outage + +- **WHEN** the collector's uplink is down for 5 minutes and events queue up +- **THEN** they are delivered after the uplink returns with their original + `time` and `"late": true` + +### Requirement: Queue modes + +Each sink SHALL accept `queue: memory` (default) or `queue: disk`. `queue: disk` +without `collector.state_dir` SHALL be a config error. A memory queue SHALL +lose pending events on restart. + +#### Scenario: Disk without state_dir + +- **WHEN** a sink sets `queue: disk` and `collector.state_dir` is unset +- **THEN** config loading fails with an error naming the sink + +### Requirement: Atomic notification durability + +With `state_dir`, one coordinator SHALL own a versioned +`/notification-state.json` snapshot containing incident records, +per-sink delivery states, sequence allocation, and pending disk-queue events. +An incident mutation and the corresponding disk-sink enqueue intents SHALL be +committed in the same snapshot. The coordinator SHALL write a temporary file, +fsync it, rename it over the snapshot, then fsync the directory before exposing +new disk-backed deliveries to workers. No independent `incidents.json` or +per-event file SHALL be authoritative. + +A disk-backed incident SHALL retain its closing event and original down ID +until all associated durable delivery intents have been acknowledged or +explicitly dropped. Worker results SHALL update the snapshot through the same +coordinator. Startup SHALL replay pending intents in per-sink sequence order, +applying expiry and queue limits with their associated delivery-state updates +in the same commit. Memory-sink pending payloads SHALL NOT be restored, but +recorded acknowledgment/uncertainty and incident identity SHALL survive when +`state_dir` is configured. Abandoned temporary files SHALL be removed. + +Delivery SHALL be at least once after a successful durable commit, subject to +explicit expiry, overflow and permanent-failure policies. A crash before commit +SHALL leave the old snapshot intact; a crash after commit SHALL replay the new +snapshot. Snapshot parse failure SHALL stop startup with an actionable error, +not silently discard open incidents. + +#### Scenario: Crash between incident change and queue distribution + +- **WHEN** a recovery is committed and tail-watch crashes before any worker sees it +- **THEN** replay finds the closing incident and its recovery intents with the + original down ID and sends them without requiring a new outage + +#### Scenario: Corrupt committed snapshot + +- **WHEN** the committed notification snapshot cannot be decoded +- **THEN** startup fails with an actionable error and preserves the file for repair + +#### Scenario: Crash before commit + +- **WHEN** tail-watch crashes after writing a new snapshot's temporary file but before + rename +- **THEN** startup uses the old snapshot, retains any old open incident, and + removes the temporary file + +#### Scenario: Crash after delivery + +- **WHEN** a worker gets `2xx` but tail-watch crashes before its acknowledgment commits +- **THEN** replay may resend the event with the same ID + +#### Scenario: Stale event at startup + +- **WHEN** replay expires a pending down that was never attempted +- **THEN** that sink's down state and dependent recovery intent are updated in + the same commit, and no orphan recovery is sent to it + +### Requirement: Disk write fallback + +If committing the atomic snapshot fails, its event and incident mutations +SHALL stay together in authoritative memory and pending events may be delivered +from memory instead. The coordinator SHALL retry persistence of the full current +state. The failure SHALL be logged once until a write succeeds again, and +`tailwatch_notify_persist_errors_total{sink="state"}` SHALL increase. The sink +name `state` SHALL be reserved. Durability SHALL be explicitly degraded until +a subsequent successful snapshot commit. + +#### Scenario: Read-only state_dir + +- **WHEN** `state_dir` is read-only and a `target.down` is emitted +- **THEN** the event is still delivered and the persist-error counter + increases + +### Requirement: State directory safety + +tail-watch SHALL create `state_dir` and its subdirectories with mode 0700 and write +files with mode 0600. Files SHALL hold event payloads and incident records +only, never URLs, secrets or headers. tail-watch SHALL log a warning when +`state_dir` is readable by group or others. + +#### Scenario: Secrets never persisted + +- **WHEN** a disk-queued webhook sink has a secret and an auth header +- **THEN** no file under `state_dir` contains the URL, the secret or the + header value + +### Requirement: Open incident tracking + +Open incidents SHALL always be indexed in memory by target name, with open and +closing delivery records keyed by incident ID. With +`state_dir`, they SHALL be persisted in the atomic notification snapshot. Each +record SHALL retain target definition and identity (including `implicit`, stable +node ID, address and reachability check for learned routers), tripping check, +event ID, `since`, severity, and per-sink delivery state. An incident SHALL have +an open or closing lifecycle; recovery SHALL atomically move it to closing and +create delivery intents before it may be garbage-collected. A closing incident +SHALL NOT be restored as DOWN or prevent a later outage from opening a new +incident with a new ID. Delivery records SHALL remain keyed by incident ID until +all intents are terminal; only one incident per target may be open. + +Before scheduling checks, tail-watch SHALL restore open configured incidents as DOWN +without a second page and recreate recorded implicit reachability targets by +stable node identity before restoring them. Recovery SHALL use the original +`down_event_id`, severity and `since`. A restore refusal SHALL log an error and +retain the incident; further downs SHALL be deduplicated while it is open. + +An explicitly removed configured target SHALL close with `reason: removed`. +An unresolved implicit target SHALL be retained until authoritative topology +reconciliation confirms removal; completion of failed discovery attempts SHALL +NOT count as removal. Removed sinks SHALL be skipped and logged. Persistence +errors SHALL retain authoritative in-memory state, be counted and logged, and +be retried; crash durability is not guaranteed while writes fail. + +#### Scenario: Restart during an outage + +- **WHEN** tail-watch restarts while `nyc-nas` has an open incident and later recovers +- **THEN** no second down is created and its recovery has the original down ID + +#### Scenario: Implicit router cannot be rediscovered + +- **WHEN** an implicit router has an open incident and first-pass discovery fails +- **THEN** its recorded reachability target and incident remain, without a + `reason: removed` recovery + +#### Scenario: Removed configured target + +- **WHEN** a configured target with an open incident is deleted from config +- **THEN** a `reason: removed` recovery is committed for eligible recorded sinks + +#### Scenario: New outage while recovery is pending + +- **WHEN** a closing incident's recovery is retrying and the same target fails again +- **THEN** a new incident opens with a new down ID, queued after the old recovery + +### Requirement: Per-sink incident delivery state + +For each incident and selected sink, the coordinator SHALL distinguish +`pending` (queued, never attempted), `attempted` (receiver may have accepted), +`acknowledged` (`2xx`), and `dropped`, retaining whether any attempt occurred. +It SHALL mark a disk event attempted durably before making its request. A +recovery generated while the down is pending or retrying SHALL be retained as +an intent and wait behind it, except when both share an ordered batch/group +whose attempt states are committed together. An acknowledged down SHALL make its recovery +eligible even when filters changed. A never-attempted down that is dropped +SHALL discard its recovery intent. If any attempt occurred but delivery was +not acknowledged, the recovery SHALL remain eligible after the down becomes +terminal: the receiver may have accepted a request whose response was lost. +Receivers SHALL therefore tolerate duplicate events and resolves for unknown +IDs. A sink filtered out of the down SHALL never receive its recovery. + +Expiry, overflow and permanent failures SHALL update these states and pending +intents atomically. Guarantees SHALL be qualified by these explicit drop +policies; losses SHALL be logged and counted. On restart, lost memory-sink +pending payloads SHALL be marked dropped and counted with reason `restart`; +never-attempted downs become ineligible while attempted downs retain recovery +eligibility. Routing and delivery acknowledgments +SHALL NOT rely solely on a list of selected sink names. + +#### Scenario: Recovery while down is retrying + +- **WHEN** a down gets `503`, its recovery is generated, and the down later gets `200` +- **THEN** recovery waits and is subsequently sent to the same sink + +#### Scenario: Down dropped before first attempt + +- **WHEN** overflow or expiry drops a never-attempted down +- **THEN** its recovery is discarded for that sink and the drop is counted + +#### Scenario: Receiver accepted but response was lost + +- **WHEN** the down request times out after acceptance and eventually expires +- **THEN** recovery is still attempted with the original down ID + +### Requirement: Secret handling + +Sink URLs, HMAC secrets, `headers_env` values and the heartbeat URL SHALL come +only from `*_env` or `*_file` keys; `*_file` values SHALL have surrounding +whitespace trimmed. tail-watch SHALL fail at startup when a referenced variable is +unset or empty or a referenced file is unreadable, and SHALL re-read the value +at send time so rotation needs no restart. Logs and errors SHALL show URLs only +as scheme and host, with everything after the host replaced by `/…`. + +#### Scenario: Unset variable + +- **WHEN** a sink sets `url_env: SLACK_WEBHOOK_URL` and the variable is unset +- **THEN** tail-watch exits non-zero naming the sink and the variable + +#### Scenario: Redacted log + +- **WHEN** delivery to `https://hooks.slack.com/services/T0/B0/xyz` fails +- **THEN** the log line shows `https://hooks.slack.com/…` and not the path + +### Requirement: Notification metrics + +tail-watch SHALL export, in the `tailwatch` namespace: + +- `tailwatch_notify_sent_total{sink,type}` +- `tailwatch_notify_failed_total{sink,reason}` with reason `timeout`, `http_5xx`, + `http_429`, `http_4xx_permanent` or `network` +- `tailwatch_notify_dropped_total{sink,reason}` with reason `queue_full`, + `expired`, `permanent` or `restart` +- `tailwatch_notify_persist_errors_total{sink}` +- `tailwatch_notify_batch_size{sink}` histogram, for batched webhook sinks only +- `tailwatch_notify_queue_len{sink}` gauge +- `tailwatch_notify_latency_seconds{sink}` histogram of time from event creation + to confirmed delivery +- `tailwatch_open_incidents` gauge, when `state_dir` is set + +#### Scenario: Failed attempt counted + +- **WHEN** a webhook attempt gets `503` +- **THEN** `tailwatch_notify_failed_total{reason="http_5xx"}` increases by 1 for + that sink + +### Requirement: Durable attempt identity + +Disk sinks SHALL persist the first-attempt body and any batch ID/membership before +sending, and replay identical bodies and IDs after restart; only signature time +may change. If any member of an attempted batch expires, the whole batch SHALL +be dropped and every event counted, retaining per-sink attempted state. Newly +arriving events SHALL NOT change an active batch. + +#### Scenario: Restart during a batch retry + +- **WHEN** a batch fails and the collector restarts before its next attempt +- **THEN** replay sends the same batch ID and body with a fresh signature diff --git a/openspec/specs/notification-events/spec.md b/openspec/specs/notification-events/spec.md new file mode 100644 index 0000000..f07caea --- /dev/null +++ b/openspec/specs/notification-events/spec.md @@ -0,0 +1,199 @@ +# notification-events Specification + +## Purpose + +Defines the events tail-watch emits when alert state changes, their severities, the +versioned JSON payload that receivers depend on, and how each sink chooses +which events it receives. + +## Requirements + +### Requirement: Event types + +tail-watch SHALL emit exactly these event types in phase 1: + +- `target.down` on `UP -> DOWN`, on `UNKNOWN -> DOWN` (a target already + failing at startup with no open incident), or on `SUPPRESSED -> DOWN` when a + dependent is still failing after its parent recovers. +- `target.recovered` when a target with an open incident reaches `UP`, + whatever state it came from. +- `target.flapping` when a target crosses `flap_limit`. +- `collector.started` once, when every configured target/check has produced + at least one result. +- `test`, sent only by `tail-watch notify-test`. + +The type `target.degraded` SHALL be reserved in the payload contract and SHALL +NOT be emitted in phase 1. Transitions into `SUPPRESSED` and `UNKNOWN -> UP` +SHALL NOT emit events. + +#### Scenario: Down then recovered + +- **WHEN** target `nyc-nas` goes `UP -> DOWN` and later `DOWN -> UP` +- **THEN** tail-watch emits one `target.down` and then one `target.recovered` for + `nyc-nas` + +#### Scenario: Suppressed dependent + +- **WHEN** `nyc-router` goes DOWN and its dependent `nyc-nas` moves to + `SUPPRESSED` +- **THEN** no event is emitted for `nyc-nas`, and the `target.down` for + `nyc-router` lists `nyc-nas` in `dependents` + +#### Scenario: Paged target's parent later fails + +- **WHEN** `nyc-nas` is DOWN with an open incident and its parent goes DOWN +- **THEN** `nyc-nas` remains DOWN and emits exactly one recovery when its own + checks meet the recovery threshold + +#### Scenario: Down at startup + +- **WHEN** the collector starts, `nyc-nas` has no open incident, and its checks + fail `fail_threshold` times +- **THEN** tail-watch emits one `target.down` for `nyc-nas` + +#### Scenario: Silent startup + +- **WHEN** the collector starts and every target goes `UNKNOWN -> UP` +- **THEN** no `target.recovered` events are emitted and one + `collector.started` event is emitted after the first full round + +### Requirement: Event severity + +Each event SHALL carry a `severity` of `info`, `warn` or `page`. `target.down` +SHALL use the target's configured `severity` (default `page`). +`target.recovered` SHALL use the severity of the `target.down` it resolves. +`target.flapping` SHALL be `warn`. `collector.started` and `test` SHALL be +`info`. + +#### Scenario: Warn-only target + +- **WHEN** target `nyc-printer` has `severity: warn` and goes DOWN +- **THEN** its `target.down` event has `"severity": "warn"` + +### Requirement: Payload contract + +Every event SHALL be serialized as one JSON object with `version` (integer, +`1`), `id`, `type`, `severity`, `time` (RFC 3339 UTC) and `collector` +(`name`, `tailscale_ip`). Target events SHALL add `target` (`name`, `addr`, +`site`, `parent`, and `implicit: true` only for implicit targets), `check`, +`layer`, `error`, `since`, `failures`, `path` +(`kind`, `endpoint`, `derp_region`, `latency_ms`, each present only when +known), and `dependents` (always an array on `target.down` and +`target.recovered`). `target.recovered` SHALL add `duration_seconds` (from the +outage's `since` to recovery) and `down_event_id`, and MAY repeat the down's +`check`, `error`, `since`, `failures` and `path`. Fields that do not apply SHALL be omitted, not sent as null. +`layer` SHALL be one of `unrouted`, `control`, `tunnel`, `lan` or `service` +on `target.down` and `target.flapping`, and SHALL be omitted on all other +events. Payload changes within `version: 1` SHALL only add fields. + +#### Scenario: Down payload + +- **WHEN** `nyc-nas` goes DOWN on `tcp:445` at layer `service` over a direct + path +- **THEN** the payload has `"type": "target.down"`, `"check": "tcp:445"`, + `"layer": "service"`, `"path": {"kind": "direct", ...}`, `"failures": 3` and + `"dependents": []` + +#### Scenario: Recovered references its down + +- **WHEN** `nyc-nas` recovers 12 minutes after a down event with id + `evt_01JAB3K9T6M4X2Q8R7N5W0YZCD` +- **THEN** the recovered payload has + `"down_event_id": "evt_01JAB3K9T6M4X2Q8R7N5W0YZCD"` and + `"duration_seconds": 720` + +#### Scenario: Unrouted address + +- **WHEN** a target goes DOWN because no peer handles its address +- **THEN** the payload has `"layer": "unrouted"` + +#### Scenario: Implicit router + +- **WHEN** the learned router `nyc-router`, which is not in the config, goes + DOWN +- **THEN** the payload's `target` object has `"name": "nyc-router"` and + `"implicit": true` + +#### Scenario: No layer on recovery + +- **WHEN** a `target.recovered` is emitted +- **THEN** the payload has no `layer` field + +### Requirement: Event identifiers + +Each event SHALL have an `id` of the form `evt_` followed by 26 Crockford +base32 characters whose leading characters encode the creation time in +milliseconds, so ids sort by creation time. The `id` SHALL stay the same across +every delivery attempt and every sink. + +#### Scenario: Retries keep the id + +- **WHEN** a webhook delivery of event `evt_X` is retried twice +- **THEN** all three requests carry `id` `evt_X` + +#### Scenario: Ids sort by time + +- **WHEN** event A is created 1 ms before event B +- **THEN** A's id sorts lexically before B's + +### Requirement: Collector identity + +The `collector.name` field SHALL be the collector node's Tailscale hostname, +and `collector.tailscale_ip` its first Tailscale IPv4 address (or IPv6 when it +has no IPv4), as reported by tailscaled at startup. + +#### Scenario: Collector fields + +- **WHEN** the collector node is `monitor-nyc` with IP `100.101.102.103` +- **THEN** every payload has + `"collector": {"name": "monitor-nyc", "tailscale_ip": "100.101.102.103"}` + +### Requirement: Sink routing filters + +Each sink SHALL accept `events` (list of event types; default all), +`min_severity` (`info`, `warn` or `page`, ordered `info < warn < page`; default +`info`), and `match` with optional `sites`, `targets` (shell-style globs) and +`tags`. An event SHALL reach a sink only when its type is listed, its severity +is at least `min_severity`, and it satisfies every `match` key given (any +listed value may match). `tags` SHALL match the tags of the Tailscale node the +target resolves to; a target with no node tags never matches a `tags` filter. +Events without a target (`collector.started`) SHALL ignore `match`. `test` +events SHALL bypass all filters. + +#### Scenario: Severity floor + +- **WHEN** a sink has `min_severity: page` and a `target.flapping` (`warn`) + event is emitted +- **THEN** that sink does not receive it + +#### Scenario: Site match + +- **WHEN** a sink has `match: {sites: [nyc, sfo]}` and a target in site `lon` + goes DOWN +- **THEN** that sink does not receive the event + +#### Scenario: Target glob + +- **WHEN** a sink has `match: {targets: ["nyc-*"]}` and `nyc-nas` goes DOWN +- **THEN** that sink receives the event + +### Requirement: Recoveries follow their pages + +A recovery SHALL bypass current routing filters for sinks selected for its +matching down, subject to the per-sink delivery-state rules in +`notification-delivery`. It SHALL NOT go to sinks filtered out of the down or +whose down was dropped before any attempt. If an attempted down's acknowledgment +is uncertain, recovery SHALL still be eligible; receivers MUST tolerate resolves +for unknown down IDs. Pending downs SHALL retain recovery intents until delivery +or dropping determines eligibility, except that an unattempted down and its +recovery may share one ordered batch/group with atomic attempt state. + +#### Scenario: Filter would drop the recovery + +- **WHEN** a sink with `events: [target.down]` acknowledged a down for `nyc-nas` +- **THEN** it also receives attempts for the matching recovery + +#### Scenario: Sink skipped the down + +- **WHEN** a sink filtered out the down because of `match.sites` +- **THEN** it receives no matching recovery diff --git a/openspec/specs/notify-test/spec.md b/openspec/specs/notify-test/spec.md new file mode 100644 index 0000000..ba7c73f --- /dev/null +++ b/openspec/specs/notify-test/spec.md @@ -0,0 +1,45 @@ +# notify-test Specification + +## Purpose + +Gives operators a one-command way to prove each configured sink works end to +end, through the real signing, rendering and retry code, after every config +change. + +## Requirements + +### Requirement: Notify-test command + +`tail-watch notify-test [--sink NAME] [--config PATH] [--timeout DUR]` SHALL load +the config, build a `test` event (severity `info`), and send it through each +selected sink's real delivery code, including signing, Slack rendering and the +retry policy, bypassing routing filters and queues. Without `--sink` it SHALL +select every sink. It SHALL print one line per sink with the final HTTP status +or error, and exit 0 only when every selected sink returned `2xx` within +`--timeout` (default 30s). + +#### Scenario: All sinks succeed + +- **WHEN** two sinks both answer `200` +- **THEN** notify-test prints `200` for each and exits 0 + +#### Scenario: One sink fails + +- **WHEN** the webhook sink answers `401` +- **THEN** notify-test prints `401` for that sink and exits 1 + +#### Scenario: Unknown sink + +- **WHEN** `--sink nope` names no configured sink +- **THEN** notify-test exits non-zero listing the configured sink names + +### Requirement: Notify-test without tailscaled + +notify-test SHALL work when tailscaled is unreachable, filling +`collector.name` from the OS hostname and omitting `collector.tailscale_ip`. +It SHALL NOT write to `state_dir`. + +#### Scenario: Daemon down + +- **WHEN** notify-test runs on a host where the LocalAPI socket is missing +- **THEN** it still sends the test event and reports each sink's status diff --git a/openspec/specs/slack-sink/spec.md b/openspec/specs/slack-sink/spec.md new file mode 100644 index 0000000..d2480a2 --- /dev/null +++ b/openspec/specs/slack-sink/spec.md @@ -0,0 +1,97 @@ +# slack-sink Specification + +## Purpose + +Defines how tail-watch posts events to a Slack channel through an incoming webhook: +message content, mentions for pages, grouping to stay within Slack's rate +limit, and links to the collector's status page. + +## Requirements + +### Requirement: Slack sink configuration + +A sink with `type: slack` SHALL require exactly one of `url_env` or +`url_file` holding a Slack incoming-webhook URL, and SHALL accept +`mention_on_page`, `group_wait` (default `10s`) and `status_url`, plus the +shared routing and queue keys. It SHALL reject `batch`, `max_batch`, +`secret_env`, `secret_file` and `headers_env`. + +#### Scenario: Batch on Slack + +- **WHEN** a Slack sink sets `batch: true` +- **THEN** config loading fails with an error naming the sink and key + +### Requirement: Message content + +Each Slack message SHALL contain Block Kit blocks and a plain-text `text` +fallback with the same information. A `target.down` line SHALL show a red +marker, `DOWN`, target name, check and layer, then the address, site, parent, +path kind and latency, the error, and the down-since time, failure count and +collector name. The layer SHALL be the payload `layer` value (`unrouted`, +`control`, `tunnel`, `lan` or `service`); an implicit target's name SHALL be +followed by `(learned)`. A `target.recovered` line SHALL read +`:large_green_circle: RECOVERED after `. A +`target.down` with dependents SHALL add "and N targets behind it" and list up +to 10 dependent names. + +#### Scenario: Down message + +- **WHEN** `nyc-nas` goes DOWN on `tcp:445` at layer `service` +- **THEN** the message text starts with + `:red_circle: DOWN nyc-nas tcp:445 (service)` and includes the error + `connection refused` + +#### Scenario: Router with dependents + +- **WHEN** `nyc-router` goes DOWN with 14 dependents +- **THEN** the message says "and 14 targets behind it" and names 10 of them + +### Requirement: Page mentions + +`mention_on_page` SHALL be prepended only to `target.down` events with +severity `page`. It SHALL NOT appear on recoveries, flapping notices, warnings +or info events. + +#### Scenario: Warn target + +- **WHEN** a `severity: warn` target goes DOWN on a sink with + `mention_on_page: ""` +- **THEN** the message does not contain `` + +#### Scenario: Recovery + +- **WHEN** a paged target recovers +- **THEN** the recovery message does not contain the mention + +### Requirement: Message grouping + +The Slack sink SHALL group events that arrive within `group_wait` of the first +pending event into one message, keeping queue order. A grouped message SHALL +show at most 10 event lines followed by "and N more". + +#### Scenario: Burst of 25 + +- **WHEN** 25 events arrive within one `group_wait` +- **THEN** Slack receives one message with 10 event lines and "and 15 more" + +### Requirement: Status links + +When `status_url` is set, each event line for a target SHALL link to the +collector status page for that target. + +#### Scenario: Link present + +- **WHEN** `status_url` is `https://monitor-nyc.example.ts.net:9109` +- **THEN** the `nyc-nas` line links to a URL under that address + +### Requirement: Slack delivery policy + +The Slack sink SHALL apply the same retry, `Retry-After`, permanent-failure and +`max_age` rules as the webhook sink, and SHALL send at most one message per +second. + +#### Scenario: Slack rate limit + +- **WHEN** Slack answers `429` with `Retry-After: 2` +- **THEN** the sink waits at least 2 seconds before resending the same + message diff --git a/openspec/specs/webhook-sink/spec.md b/openspec/specs/webhook-sink/spec.md new file mode 100644 index 0000000..21809d8 --- /dev/null +++ b/openspec/specs/webhook-sink/spec.md @@ -0,0 +1,137 @@ +# webhook-sink Specification + +## Purpose + +Defines how tail-watch delivers events to a generic HTTP receiver: request format, +HMAC signing compatible with Tailscale's own webhooks, retry and expiry policy, +and the optional batched mode for rate-limited receivers. + +## Requirements + +### Requirement: Webhook sink configuration + +A sink with `type: webhook` SHALL require exactly one of `url_env` or +`url_file`, and SHALL accept `secret_env` or `secret_file` (HMAC key), +`headers_env` (map of header name to environment variable name), `max_age` +(default `1h`), `batch` (default `false`), `group_wait` (default `10s`) and +`max_batch` (default `100`), plus the shared routing and queue keys. Sink names +SHALL be unique, match `[A-Za-z0-9._-]+`, and not be `incidents`, `intake` or `state`. + +#### Scenario: Both URL sources + +- **WHEN** a webhook sink sets both `url_env` and `url_file` +- **THEN** config loading fails with an error naming the sink + +#### Scenario: No signing secret + +- **WHEN** a webhook sink has no `secret_env` or `secret_file` +- **THEN** tail-watch starts, logs a warning that the sink is unsigned, and sends + requests without a `Tail-Watch-Signature` header + +### Requirement: Webhook request format + +Each delivery SHALL be an HTTP `POST` to the sink URL with +`Content-Type: application/json`, `User-Agent: tail-watch/`, the headers +from `headers_env`, and a body holding exactly one event payload. It SHALL set +`Tail-Watch-Event-Id: `. It SHALL NOT follow redirects. Each request SHALL time +out after 10s. + +#### Scenario: Single event + +- **WHEN** a `target.down` is delivered to a non-batched webhook sink +- **THEN** the receiver gets one POST whose body is that event's JSON payload + and whose `Tail-Watch-Event-Id` equals the payload `id` + +#### Scenario: Extra auth header + +- **WHEN** `headers_env: {Authorization: OPS_WEBHOOK_AUTH}` is set and + `OPS_WEBHOOK_AUTH=Bearer abc` +- **THEN** each request carries `Authorization: Bearer abc` + +### Requirement: Request signing + +When the sink has a secret, each request SHALL carry +`Tail-Watch-Signature: t=,v1=`, where `v1` is the lowercase hex +HMAC-SHA256, keyed by the secret, of the string `.`. `t` SHALL be the +time of that attempt, so a retry carries a fresh `t` over identical body bytes. + +#### Scenario: Receiver verifies + +- **WHEN** a receiver computes HMAC-SHA256 with the shared secret over + `.` from the request +- **THEN** the result equals the `v1` value + +#### Scenario: Wrong secret + +- **WHEN** a receiver verifies with a different secret +- **THEN** the computed value differs from `v1` + +#### Scenario: Independent signing vector + +- **WHEN** signing a fixed timestamp and exact body with a fixed key +- **THEN** the signature equals a fixture computed independently of the sender's + signing helper, and changing either timestamp or body changes the signature + +### Requirement: Delivery result handling + +A `2xx` response SHALL mark the event delivered. `408`, `429`, `5xx`, network +errors and timeouts SHALL be retried with increasing delay, capped at 5 +minutes between attempts. On `429` with a `Retry-After` header (seconds or +HTTP date), the next attempt SHALL wait at least that long, capped at 5 +minutes. Any other status, including `3xx` and other `4xx`, SHALL be a +permanent failure: logged once, counted, and not retried. An event older than +the sink's `max_age` at the time of an attempt SHALL be dropped and counted as +expired instead of sent. + +#### Scenario: Server error then success + +- **WHEN** the receiver answers `500` then `200` +- **THEN** tail-watch sends the event twice and marks it delivered + +#### Scenario: Rate limited + +- **WHEN** the receiver answers `429` with `Retry-After: 30` +- **THEN** the next attempt is sent no sooner than 30 seconds later + +#### Scenario: Bad request + +- **WHEN** the receiver answers `400` +- **THEN** tail-watch does not retry, logs the failure once, and increments + `tailwatch_notify_dropped_total{reason="permanent"}` + +#### Scenario: Expired while retrying + +- **WHEN** a receiver keeps answering `503` past the event's `max_age` +- **THEN** the event is dropped and counted with reason `expired` + +### Requirement: Batched mode + +With `batch: true`, the sink SHALL start a window on the first queued event +and send when `group_wait` elapses or `max_batch` events are waiting, +whichever comes first. The body SHALL be +`{"version": 1, "batch_id": "bat_", "events": [ ... ]}` where each element +is exactly a single-event payload, in queue order. The request SHALL carry +`Tail-Watch-Batch-Id` instead of `Tail-Watch-Event-Id`, and `Tail-Watch-Signature` SHALL cover +the whole body. A retry SHALL resend identical body bytes and `batch_id` with a +fresh signature. Events arriving during retries SHALL go into the next batch. A +`target.down` and its `target.recovered` in the same window SHALL both be sent, +in order. A permanent failure SHALL drop the whole batch and count every event +in it. + +#### Scenario: Large burst + +- **WHEN** 250 events arrive within one `group_wait` window and `max_batch` is + 100 +- **THEN** the receiver gets three batches of 100, 100 and 50 events, in + order + +#### Scenario: Batch retry + +- **WHEN** a batch POST receives `503` and is retried +- **THEN** the retry has the same body bytes and `Tail-Watch-Batch-Id`, and a + `Tail-Watch-Signature` with a newer `t` + +#### Scenario: Blip inside a window + +- **WHEN** `nyc-nas` goes down and recovers inside one window +- **THEN** the batch holds both events, down before recovered diff --git a/scripts/live-stack.sh b/scripts/live-stack.sh new file mode 100755 index 0000000..e0b4098 --- /dev/null +++ b/scripts/live-stack.sh @@ -0,0 +1,109 @@ +#!/usr/bin/env bash +# Bring up (or tear down) only the tailnet side of the live notification test: +# the collector's tailscale node plus one fixture node per check. It does not +# start the tail-watch process or run any scenario. See README "Live notification +# test". Credentials come from the pulumi env in TAIL_WATCH_PULUMI_ENV +# (default tailscale-phase-2/tailscale). +# +# scripts/live-stack.sh up create ephemeral nodes and print their addresses +# scripts/live-stack.sh down remove the containers and this stack's devices +set -euo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +PROJECT="${TAIL_WATCH_LIVE_PROJECT:-tail-watch-live-stack}" +OUT="${TAIL_WATCH_LIVE_DIR:-${TMPDIR:-/tmp}/$PROJECT}" +ENVNAME="${TAIL_WATCH_PULUMI_ENV:-tailscale-phase-2/tailscale}" +FILES=(-f "$ROOT/examples/compose.yaml" -f "$ROOT/cmd/tail-watch/testdata/compose.live-notify.yaml") +SERVICES=(fx-icmp fx-tsping fx-tcp fx-http lan-router) + +creds() { + pulumi env open "$ENVNAME" --format json | python3 -c ' +import json,sys +t=json.load(sys.stdin)["tailscale"] +print(t["apiKey"]); print(t["tailnet"])' +} + +api() { curl -fsS -u "$APIKEY:" -H 'Content-Type: application/json' "$@"; } + +mint_key() { + api -X POST "https://api.tailscale.com/api/v2/tailnet/$TAILNET/keys" \ + -d '{"capabilities":{"devices":{"create":{"reusable":false,"ephemeral":true,"preauthorized":true'"${1:+,\"tags\":[\"$1\"]}"'}}},"expirySeconds":900}' | + python3 -c 'import json,sys; print(json.load(sys.stdin)["key"])' +} + +compose() { + docker compose --project-name "$PROJECT" "${FILES[@]}" "$@" +} + +devices() { + api "https://api.tailscale.com/api/v2/tailnet/$TAILNET/devices?fields=all" +} + +load_creds() { + local out + out="$(creds)" + APIKEY="$(sed -n 1p <<<"$out")" + TAILNET="$(sed -n 2p <<<"$out")" +} + +up() { + load_creds + mkdir -p "$OUT" + export TS_FX_PREFIX="$PROJECT" + export TS_HOSTNAME="$PROJECT" + export TS_AUTHKEY TS_KEY_ICMP TS_KEY_TSPING TS_KEY_TCP TS_KEY_HTTP TS_KEY_LAN + # TAIL_WATCH_LIVE_TAG defaults to untagged; set it to tag:monitor once the policy + # in cmd/tail-watch/testdata/snippet.hujson is merged. + TS_AUTHKEY="$(mint_key "${TAIL_WATCH_LIVE_TAG:-}")" + TS_KEY_ICMP="$(mint_key)" + TS_KEY_TSPING="$(mint_key)" + TS_KEY_TCP="$(mint_key)" + TS_KEY_HTTP="$(mint_key)" + TS_KEY_LAN="$(mint_key)" + export TAIL_WATCH_CONFIG_FILE="$OUT/tail-watch.yaml" + : >"$TAIL_WATCH_CONFIG_FILE" + compose up --detach --wait --wait-timeout 120 tailscale "${SERVICES[@]}" + + local id + for _ in $(seq 1 30); do + id="$(devices | python3 -c ' +import json,sys +want=sys.argv[1].lower() +for d in json.load(sys.stdin)["devices"]: + if d["hostname"].lower()==want: print(d["id"])' "$PROJECT-lan")" + [ -n "$id" ] && break + sleep 2 + done + [ -n "$id" ] || { echo "lan router never registered" >&2; exit 1; } + api -X POST "https://api.tailscale.com/api/v2/device/$id/routes" -d '{"routes":["10.10.53.0/24"]}' >/dev/null + + echo "tailnet nodes (project $PROJECT):" + for s in tailscale "${SERVICES[@]}"; do + printf ' %-10s %s\n' "$s" "$(compose exec --no-TTY "$s" tailscale ip -4 2>/dev/null | head -1)" + done + echo "LAN target behind lan-router: 10.10.53.1" + echo + echo "Write a config at $TAIL_WATCH_CONFIG_FILE, then start the collector with:" + echo " TAIL_WATCH_CONFIG_FILE=$TAIL_WATCH_CONFIG_FILE docker compose --project-name $PROJECT ${FILES[*]} up -d --no-deps tail-watch" + echo "Tear down with: scripts/live-stack.sh down" +} + +down() { + load_creds + export TS_FX_PREFIX="$PROJECT" TS_KEY_ICMP=x TS_KEY_TSPING=x TS_KEY_TCP=x TS_KEY_HTTP=x TS_KEY_LAN=x TS_AUTHKEY=x + export TAIL_WATCH_CONFIG_FILE="$OUT/tail-watch.yaml" + compose down --volumes --remove-orphans || true + devices | python3 -c ' +import json,sys +p=sys.argv[1].lower() +for d in json.load(sys.stdin)["devices"]: + h=d["hostname"].lower() + if h==p or h.startswith(p+"-"): print(d["id"])' "$PROJECT" | + while read -r id; do api -X DELETE "https://api.tailscale.com/api/v2/device/$id" >/dev/null && echo "deleted device $id"; done +} + +case "${1:-}" in +up) up ;; +down) down ;; +*) echo "usage: $0 up|down" >&2; exit 2 ;; +esac From f225bf665fd071aae039ea9e4630cd2626bfc248 Mon Sep 17 00:00:00 2001 From: Richard Shade Date: Tue, 29 Sep 2026 12:24:05 -0500 Subject: [PATCH 2/2] openspec: propose user documentation restructure Add the add-user-documentation change. It proposes moving the README into a docs/ tree organized by Diataxis quadrant (tutorial, how-to, reference, explanation), leaving a short README as the landing page. New pages cover a first-monitor tutorial, the config, CLI and metrics references, and troubleshooting. Contributor material moves to docs/contributing/. The change sets skip_specs because it alters no system behavior. --- .../add-user-documentation/.openspec.yaml | 3 + .../changes/add-user-documentation/design.md | 98 +++++++++++++++++++ .../add-user-documentation/proposal.md | 72 ++++++++++++++ .../changes/add-user-documentation/tasks.md | 53 ++++++++++ 4 files changed, 226 insertions(+) create mode 100644 openspec/changes/add-user-documentation/.openspec.yaml create mode 100644 openspec/changes/add-user-documentation/design.md create mode 100644 openspec/changes/add-user-documentation/proposal.md create mode 100644 openspec/changes/add-user-documentation/tasks.md diff --git a/openspec/changes/add-user-documentation/.openspec.yaml b/openspec/changes/add-user-documentation/.openspec.yaml new file mode 100644 index 0000000..c7c6694 --- /dev/null +++ b/openspec/changes/add-user-documentation/.openspec.yaml @@ -0,0 +1,3 @@ +schema: spec-driven +created: 2026-09-29 +skip_specs: true diff --git a/openspec/changes/add-user-documentation/design.md b/openspec/changes/add-user-documentation/design.md new file mode 100644 index 0000000..ab5fcd8 --- /dev/null +++ b/openspec/changes/add-user-documentation/design.md @@ -0,0 +1,98 @@ +# Design + +## Context + +The only user documentation is `README.md` (about 625 lines, with 336 lines of +uncommitted additions from `add-notifications`). Reference facts live in three other +places: the Go source (`config.go`, `config_sinks.go`, `notify_metrics.go`, the flag +sets in `main.go`), the archived specs under `openspec/specs/`, and +`tailnet-reachability-monitor-design.md`. `examples/tail-watch.yaml` and +`examples/compose.yaml` are the only runnable samples. markdownlint runs with MD013 at +90 columns (tables and code blocks exempt) and ignores `.claude/**` and `node_modules/**`. + +## Goals / Non-Goals + +**Goals:** + +- Every operator question maps to exactly one page: learn, do, look up, or understand. +- Every command and YAML snippet in the docs is checked against the built binary, not + copied from memory. +- A fact lives in one page; other pages link to it. + +**Non-Goals:** + +- Documenting internal architecture (`netEnv`, clocks, mutex layout). That stays in + `CLAUDE.md` and `openspec/`. +- Auto-generating reference pages from code. The config surface is small enough to write + by hand and check. + +## Decisions + +### Diataxis layout under `docs/` + +```text +docs/ + README.md index, one line per page + tutorial/first-monitor.md + how-to/ deploy.md acls.md remote-subnets.md + slack-sink.md webhook-sink.md troubleshoot.md + reference/ config.md checks.md cli.md metrics.md payload.md + explanation/ failure-layers.md alert-state.md incidents-and-queues.md + contributing/ live-notification-test.md releases.md +``` + +Alternative: a single long `docs/guide.md`. Rejected because it recreates the README +problem and gives readers no stable links. + +### README becomes a landing page + +Keep the purpose paragraph, requirements summary, a five-line quickstart, and a link +list. Alternative: keep the README complete and add `docs/` as extras. Rejected: two +copies of the sink options and payload drift as soon as one changes. + +### Accuracy is checked mechanically where possible + +- Each YAML snippet that is a full config is saved and run through + `tail-watch check-config`. +- Each shell command is run, or, when it needs a tailnet, is marked as such and matches a + command exercised by `scripts/live-stack.sh` or the live tests. +- Metric names in `reference/metrics.md` are compared against the `Name:` entries in + `notify_metrics.go` and the collector metrics, so none is documented that does not exist. + +Alternative: extract snippets into Go tests. Rejected as over-built for a docs change. + +### Source of truth per page + +| Page | Source | +| ---- | ------ | +| `reference/config.md` | `config.go`, `config_sinks.go`, design doc "Config" | +| `reference/cli.md` | flag sets in `main.go`, `notifytest.go` | +| `reference/metrics.md` | metric registrations, design doc "Metrics" | +| `reference/payload.md` | `event.go`, `testdata/payload` golden files | +| `explanation/*` | archived specs, design doc "What a failed check means", "Alert state machine" | + +Where code and design doc disagree, the code and archived spec win and the page follows +them; note the mismatch in the task that found it. + +### Tutorial uses the shipped example + +The tutorial starts from `examples/compose.yaml`, adds one failing target, and ends with +the alert arriving (Slack webhook or a local webhook receiver). This keeps it runnable +without inventing a new sample. Alternative: a self-contained tutorial with its own +compose file. Deferred: revisit only if the example proves too far from a first run. + +## Risks / Trade-offs + +- [Docs describe behavior that `add-notifications` still changes] → Apply after that + change is archived; the README diff is the input. +- [Restructure loses README content] → Move first, then trim; a task diffs old README + headings against the new pages and confirms each is accounted for. +- [Existing README anchors are linked from elsewhere] → Search the repo and release notes + for `README.md#` links and keep or redirect them. +- [Reference pages drift from code over time] → The metrics/config checks above are + repeatable, and `CLAUDE.md` gets a note to update `docs/` with config or metric + changes. + +## Open Questions + +- Should `docs/` link check run in CI, or only locally in the validation group? diff --git a/openspec/changes/add-user-documentation/proposal.md b/openspec/changes/add-user-documentation/proposal.md new file mode 100644 index 0000000..dcdb5df --- /dev/null +++ b/openspec/changes/add-user-documentation/proposal.md @@ -0,0 +1,72 @@ +# Proposal + +## Why + +`README.md` has grown to about 625 lines and carries four kinds of content at once: +explanation (failure layers, alert state), how-to (Docker, ACLs, Slack webhook), reference +(checks, sink options, payload) and contributor notes (live notification test, +releases). A new operator has no path from "what is this" to "first alert in Slack", and +nothing documents the full `tail-watch.yaml` schema, the `tailwatch_*` metrics, or the +flags in one place. Phase 1 is feature-complete once `add-notifications` lands, so the +behavior is stable enough to document now. + +## What Changes + +- Restructure user documentation into `docs/` following the Diataxis quadrants + (tutorial, how-to, reference, explanation), written with the `documentation-writer` + skill. +- Reduce `README.md` to a short overview: what tail-watch is, a five-line quickstart, + and links into `docs/`. Move the existing README material into the matching pages + rather than keeping two copies. +- Add pages that do not exist today: a first-monitor tutorial, a configuration reference, + a metrics reference, a CLI reference (flags, `check-config`, `notify-test`), and a + troubleshooting guide. +- Split contributor material (live notification test, development and releases) into + `docs/contributing/` so it is out of the operator path. +- Lint the new Markdown with markdownlint and keep it in the existing lint task. + +This change adds documentation only. It does not change code, config keys, metric names or +payload fields; every documented fact is checked against the source, the archived specs +and `tailnet-reachability-monitor-design.md`. + +Implements no design-doc section. The design doc has no user-documentation section, and +documentation is not a phase 1 deliverable there. It describes behavior from the design +doc sections "Check types", "What a failed check means", "Alert state machine", +"Notifications", "Config", "ACLs" and "Metrics". + +Phase: MVP (phase 1). Non-goals: + +- Phase 2 and 3 features (PagerDuty, Slack bot-token mode, `target.degraded`, quorum + across collectors, tsnet mode). Pages mention them only as "not yet supported". +- A generated documentation site or hosting (mkdocs, Hugo, GitHub Pages). +- Man pages, shell completion, or in-binary help text changes. +- Translating or versioning the docs per release. + +## Capabilities + +### New Capabilities + +None. Documentation does not change system behavior, so no spec changes. The change sets +`skip_specs: true` in `.openspec.yaml`. + +### Modified Capabilities + +None. + +## Impact + +- New: `docs/` tree (tutorial, how-to, reference, explanation, contributing). +- Changed: `README.md` (slimmed), `CLAUDE.md` (layout note for `docs/`), markdownlint + configuration if `docs/` needs exclusions or the line-length rule needs adjusting. +- No Go code, dependency, or runtime impact. +- Sequencing: this change should be applied after `add-notifications` is archived, so the + notification pages describe final behavior. The uncommitted README edits from that + change are the source for those pages. + +## Assumptions + +- The primary reader is a tailnet admin or SRE who knows Tailscale ACLs but not tail-watch. +- The README is restructured, not left alongside new pages. +- The tutorial builds on `examples/compose.yaml` and `examples/tail-watch.yaml`. + +These were proposed during exploration and not contradicted; revise them here if wrong. diff --git a/openspec/changes/add-user-documentation/tasks.md b/openspec/changes/add-user-documentation/tasks.md new file mode 100644 index 0000000..5b1ee4b --- /dev/null +++ b/openspec/changes/add-user-documentation/tasks.md @@ -0,0 +1,53 @@ +# Tasks + +## 1. Outline and scaffolding + +- [ ] 1.1 Use the `documentation-writer` skill to confirm audience, goal and scope per page, and get the outline approved; verify the approved outline matches the tree in `design.md` +- [ ] 1.2 Create `docs/README.md` index and the empty quadrant directories; verify every planned page is listed and links resolve +- [ ] 1.3 Add a heading-by-heading map from current `README.md` sections to target pages (kept in the change directory, not committed to `docs/`); verify no README section is unassigned +- [ ] 1.4 Search the repo for `README.md#` anchors and inbound links (`CLAUDE.md`, `CHANGELOG.md`, `examples/`); verify each is listed with its new target + +## 2. Reference pages + +- [ ] 2.1 Write `docs/reference/config.md` from `cmd/tail-watch/config.go` and `config_sinks.go`; verify every key in `examples/tail-watch.yaml` and every `sinks` key appears with type, default and required/optional +- [ ] 2.2 Save each full-config snippet from `docs/reference/config.md` and run `tail-watch check-config`; verify all pass +- [ ] 2.3 Write `docs/reference/cli.md` (flags, `TAIL_WATCH_CONFIG`, `TAIL_WATCH_LOG_LEVEL`, `check-config`, `notify-test`, exit codes) from `cmd/tail-watch/main.go` and `notifytest.go`; verify against `tail-watch --help` and `notify-test --help` +- [ ] 2.4 Write `docs/reference/metrics.md` from the metric registrations under `cmd/tail-watch/`; verify each documented `tailwatch_*` name appears in the source and each registered name is documented +- [ ] 2.5 Write `docs/reference/checks.md` (ICMP, TCP, HTTP, tsping options and semantics) from `checks.go`; verify against the `reachability-checks` spec +- [ ] 2.6 Write `docs/reference/payload.md` (event types, fields, severities, signature header) from `event.go` and `testdata/payload`; verify the field list matches the golden files + +## 3. Explanation pages + +- [ ] 3.1 Write `docs/explanation/failure-layers.md` from the README "Failure layers" section and the `failure-classification` and `dependency-learning` specs; verify each layer name matches the `failure_layer` metric label values +- [ ] 3.2 Write `docs/explanation/alert-state.md` covering grace, flap suppression and dependency suppression, from the `alert-state-machine` and `dependency-suppression` specs; verify each state and timer name matches the config keys in `reference/config.md` +- [ ] 3.3 Write `docs/explanation/incidents-and-queues.md` covering `state_dir`, at-least-once delivery, replay and open incidents; verify against the `add-notifications` spec set once archived + +## 4. How-to guides + +- [ ] 4.1 Write `docs/how-to/deploy.md` (binary, Docker, Compose without a dedicated host) from README "Build and run" and "Docker"; verify the Docker build command runs and `--version` prints +- [ ] 4.2 Write `docs/how-to/acls.md` from README "ACLs"; verify the policy snippet is valid HuJSON +- [ ] 4.3 Write `docs/how-to/remote-subnets.md` from README "Remote subnets and local LANs"; verify each `network` value and startup failure named in it exists in `netenv.go` and `config.go` +- [ ] 4.4 Write `docs/how-to/slack-sink.md` including getting a webhook URL and using `*_env`; verify the config snippet passes `check-config` and `notify-test --sink` steps match `notifytest.go` +- [ ] 4.5 Write `docs/how-to/webhook-sink.md` (signing, headers, batching, retries); verify the signature verification example against `webhook_test.go` +- [ ] 4.6 Write `docs/how-to/troubleshoot.md` (userspace tailscaled refusal, missing `tag:monitor`, ICMP permissions, socket path, missing secrets, corrupt state); verify each symptom's message text against the source strings + +## 5. Tutorial + +- [ ] 5.1 Write `docs/tutorial/first-monitor.md` starting from `examples/compose.yaml`, adding a failing target, and ending with a delivered alert; verify by following it on a clean checkout with `scripts/live-stack.sh` or the Compose file +- [ ] 5.2 Record any step that needed a fix while following it, and update the page; verify a second read-through needs no unwritten step + +## 6. README and contributor split + +- [ ] 6.1 Move "Live notification test" and "Development and releases" from `README.md` to `docs/contributing/`; verify commands still match `CLAUDE.md` and `scripts/live-stack.sh` +- [ ] 6.2 Trim `README.md` to overview, quickstart and links; verify against the 1.3 map that every removed section exists in a page +- [ ] 6.3 Fix or redirect the inbound links found in 1.4; verify a grep for the old anchors finds none +- [ ] 6.4 Add a `docs/` layout and "update docs with config or metric changes" note to `CLAUDE.md`; verify the note matches the final tree +- [ ] 6.5 Add a `CHANGELOG.md` entry if the release process lists documentation; verify against `release-please-config.json` + +## 7. Validation + +- [ ] 7.1 Run `mise exec -- go build ./...`, `go vet ./...` and `go test ./...`; verify all pass, confirming no code changed +- [ ] 7.2 Run `mise run lint` (golangci-lint) and `govulncheck ./...`; verify no findings +- [ ] 7.3 Run `markdownlint` over `README.md`, `docs/` and `openspec/changes/add-user-documentation/`; verify zero findings +- [ ] 7.4 Run a link check over `docs/` and `README.md` (local, relative links); verify none are broken +- [ ] 7.5 Run `mise exec -- openspec validate add-user-documentation --strict`; verify it passes