diff --git a/.goreleaser.yaml b/.goreleaser.yaml index e5d28c7..b12a8b2 100644 --- a/.goreleaser.yaml +++ b/.goreleaser.yaml @@ -28,6 +28,7 @@ archives: files: - LICENSE - README.md + - docs/**/* - examples/tail-watch.yaml checksum: diff --git a/CLAUDE.md b/CLAUDE.md index 334f204..01c222a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -63,6 +63,17 @@ authoritative reconciliation, and the removed dirty snapshot makes the notifier close the incident with `reason: removed`. +## Documentation + +- User docs live in `docs/`, split by Diataxis quadrant: `tutorial/`, `how-to/`, + `reference/`, `explanation/`, and `contributing/`. `docs/README.md` is the index and + `README.md` is only an overview with links. A fact lives in one page; link to it. +- When a config key, flag, metric, event field, or log message changes, update + `reference/config.md`, `reference/cli.md`, `reference/metrics.md`, + `reference/payload.md`, or `how-to/troubleshoot.md` in the same change. +- Check a config snippet with `tail-watch check-config`. `check-config` and `--once` + do not resolve `*_env` secrets, so it passes with them unset. + ## Commits - Follow the Tailscale commit style in `AGENTS.md`: a `dir: verb ...` diff --git a/README.md b/README.md index 56d91cb..c3738d6 100644 --- a/README.md +++ b/README.md @@ -6,694 +6,46 @@ a status page, Prometheus metrics, and a health check. It answers a different question from the online/offline webhook: not whether control thinks a node is connected, but whether traffic to that -node or to a subnet behind it actually gets through. +node or to a subnet behind it actually gets through. When a check fails, it +names the layer that broke and sends one alert per outage to Slack or a webhook. ## Requirements - A host running `tailscaled` in kernel mode. Userspace networking cannot carry ICMP to LAN addresses, so this build refuses it at startup. -- Tag the host `tag:monitor`. tail-watch logs a warning when that tag is - missing, because ACL grants are usually written for the tag. -- The process must be able to open the LocalAPI socket. On Linux that is - `/var/run/tailscale/tailscaled.sock` unless you pass `--socket`. +- Tag the host `tag:monitor`, and grant that tag access to what you monitor. +- A process that can open the LocalAPI socket, by default + `/var/run/tailscale/tailscaled.sock`. -ICMP needs either unprivileged ping (`net.ipv4.ping_group_range`) or -`CAP_NET_RAW`. If neither is available, ICMP checks fail with that -guidance and TCP, HTTP, and tsping checks still run. - -## Remote subnets and local LANs - -`network` selects the path a target is supposed to take: - -- `tailnet` (the default) means IP checks should use Tailscale. On Linux, - a target whose address is a literal non-Tailscale IP is treated as a - remote subnet. Startup fails until the host accepts subnet routes: - - ```bash - tailscale set --accept-routes - ``` - - An advertised route is not an installed route. After accepting routes, - check the route the host will actually use: - - ```bash - ip route get 10.20.0.1 - ``` - - The answer should go out the Tailscale interface (`tailscale0`), not a - local gateway. Repeat that for each remote subnet you monitor. - -- `local` means the host's ordinary route, including a LAN the collector - is plugged into at a site. These checks keep working when that site's - Tailscale router is down. They cannot use `tsping`. Put a `network: - local` device in its own target; do not point an HTTP check at a - different host than the target's `addr`. - -`examples/tail-watch.yaml` has both kinds of target. `nyc-core-switch` and -`nyc-nas` are remote subnet devices behind `nyc-subnet-router`. -`lab-printer` is on the collector's own LAN. - -## Build and run +## Quickstart ```bash mise install mise run build -./bin/tail-watch --config examples/tail-watch.yaml -``` - -Config path order is `--config`, then `TAIL_WATCH_CONFIG`, then -`/etc/tail-watch/tail-watch.yaml`. - -Validate a file without contacting tailscaled: - -```bash ./bin/tail-watch check-config --config examples/tail-watch.yaml +./bin/tail-watch --config examples/tail-watch.yaml ``` -Run every check once, print a line each, and exit non-zero on failure: - -```bash -./bin/tail-watch --once --config examples/tail-watch.yaml -``` - -`--version` prints the build version. `--log-level` (or `TAIL_WATCH_LOG_LEVEL`) -is `debug`, `info`, `warn`, or `error`. Logs are JSON on stderr. Debug -logs include every check result. - -| Flag | Env | Default | -| --- | --- | --- | -| `--config` | `TAIL_WATCH_CONFIG` | `/etc/tail-watch/tail-watch.yaml` | -| `--listen` | | `127.0.0.1:9109` or `collector.listen` | -| `--socket` | | platform LocalAPI socket | -| `--log-level` | `TAIL_WATCH_LOG_LEVEL` | `info` | -| `--once` | | run one pass and exit | -| `--version` | | print the version and exit | - -HTTP endpoints on the listen address: - -- `/` prober status page -- `/metrics` Prometheus text, `tailwatch_` series labeled `target`, `check`, - `site`, and `class`. The prefix has no hyphen because Prometheus metric - names cannot contain one. -- `/healthz` returns `200` and `ok` - -SIGINT and SIGTERM stop new work, cancel running checks, and exit 0. - -## Docker - -The image listens as user `tail-watch` (uid 65532) and does not embed a -Tailscale node. Use host networking so checks share the host routes, and -mount the LocalAPI socket. That socket has to be readable by uid 65532, -or pass a `--user` / `--group-add` that can read it. - -```bash -docker build -t tail-watch . -docker run --rm --network host \ - -v /var/run/tailscale/tailscaled.sock:/var/run/tailscale/tailscaled.sock \ - -v /etc/tail-watch/tail-watch.yaml:/etc/tail-watch/tail-watch.yaml:ro \ - tail-watch -``` - -### Docker Compose without a dedicated host - -`examples/compose.yaml` runs a kernel-mode `tailscaled` container and puts -tail-watch in its network namespace, so checks follow that node's `tailscale0` -routes, including accepted subnet routes. Mint an auth key tagged -`tag:monitor`: - -```bash -TS_AUTHKEY=tskey-auth-... docker compose -f examples/compose.yaml up -d -``` - -tail-watch starts once the node reports healthy. It reads -`examples/tail-watch.yaml` unless `TAIL_WATCH_CONFIG_FILE` names another file. The -`tailscale` service sets `net.ipv4.ping_group_range`, so the non-root tail-watch -image can send ICMP without `CAP_NET_RAW`. `network: local` targets reach -only what the container's Docker network can reach. - -`collector.state_dir` holds notification state and, with the UI on, the monitors -created there. Do not put secrets in the config file. - -## ACLs - -The collector sees only what policy allows. A missing grant shows up as -`no peer handles ; check grants to tag:monitor`. - -```jsonc -{ - "tagOwners": { "tag:monitor": ["autogroup:admin"] }, - "grants": [ - { - "src": ["tag:monitor"], - "dst": ["tag:router", "tag:server"], - "ip": ["*"] - }, - { - "src": ["tag:monitor"], - "dst": ["10.20.0.0/16"], - "ip": ["icmp:*", "tcp:22", "tcp:445", "tcp:443", "tcp:5001"] - } - ] -} -``` - -## Checks - -| Check | Passes when | -| --- | --- | -| `icmp` | one echo reply arrives before the timeout | -| `tcp:` | a TCP handshake completes; the socket is closed without sending | -| `http:` | status is 2xx or 3xx; optional `\|text` must appear in the first 1 MiB | -| `tsping` | tailscaled reports a disco pong for the target address | - -`http` does not follow redirects. `insecure_tls: true` on a target skips -certificate verification for devices that present a private certificate. -`tls` checks are rejected until phase 2. - -An explicit `parent` must name another target whose only check is `icmp` -or `tsping`. Parent cycles are rejected. That parent is for suppression -only. It does not change which tailnet node handles the address. - -## Failure layers - -A passing check has no layer. A failing check is labeled with one layer: - -| Layer | Meaning | -| --- | --- | -| `unrouted` | no peer owns or routes the address | -| `control` | the handling node is known offline | -| `tunnel` | the handling node is up, but disco fails | -| `service` | the device answered, or the target is a Tailscale node | -| `lan` | disco works and the device does not answer | - -`network: local` targets use only `service` and `lan`. They do not consult -tailnet presence or disco. - -## Alert state - -Each target is `UNKNOWN`, `UP`, `DOWN`, or `SUPPRESSED`. It leaves `UNKNOWN` -for `UP` after one success on every check, and that move is silent. Any check -that fails `fail_threshold` times in a row moves the target to `DOWN`. It -returns to `UP` after `recover_threshold` consecutive successes on every check -that failed, while its other checks are still passing. The defaults are 3 -failures and 2 successes. - -More than `flap_limit` state changes inside `flap_window` (defaults 4 and 15m) mark the -target flapping. tail-watch emits one flapping notice, then stays quiet until -the state has been stable for another `flap_window`. While that mark is set, -`tailwatch_target_state` is 4. - -A reachability parent (`icmp` or `tsping` only) that is `DOWN` or -`SUPPRESSED` moves its `UNKNOWN` and `UP` descendants to `SUPPRESSED` and -names them on its transition. A descendant that is already `DOWN` keeps its -own incident. A `control` or `tunnel` failure waits for the parent's detection -window (`interval × fail_threshold + timeout`) before it can page, so a dead -router produces one alert instead of one per device. `unrouted`, `lan`, and -`service` failures page immediately. When the parent recovers, or a topology -change leaves no down ancestor, tail-watch rechecks each released target. The -target then goes `UP` silently, or `DOWN` if the fresh checks keep failing. -If flap damping left an earlier `DOWN` unresolved, suppression preserves that -incident. The child sends its recovery after release when flap damping permits, -keeping the original outage start time; continued failures do not open a second -incident. - -With more than one collector, quorum stays in Alertmanager until collectors -share state: - -```yaml -- alert: TargetDownQuorum - expr: count by (target) (tailwatch_target_state == 2) >= 2 - for: 1m -``` - -That rule pages only when two collectors agree the target is down. - -For a remote LAN target, tail-watch learns the handling node with one disco -ping at startup, every 10 minutes, and when primary routes change. A failed -ping keeps the previous node unless the primary route changed, in which case -the route is used. An explicit `parent` still wins for suppression. - -The learned parent is a configured target only when that target's address is -the handling node and its only check is `tsping`. A service check on the same -node does not suppress the LAN devices behind it. Otherwise tail-watch adds -an implicit `tsping` target named for the node's short MagicDNS name, adding -the stable node ID and then a number if that name is taken. An idle implicit -target that is still down stays until it recovers. While a target has no -handling node, tail-watch logs this once, and again only if the grant returns -and is then lost: - -```text -target lab-pdu: no peer handles 10.99.0.5; check grants to tag:monitor -``` - -Prometheus series use the `tailwatch_` prefix: - -| Metric | Kind | -| --- | --- | -| `tailwatch_failure_layer_total{target,layer}` | counter, one increment per failure | -| `tailwatch_path{target,kind}` | gauge, `direct`, `derp`, and `peer_relay` | -| `tailwatch_target_parent{target,parent}` | gauge, 1 for the current parent | -| `tailwatch_target_state{target}` | gauge, 0 unknown through 4 flapping | - -## Monitor UI - -Off by default, tail-watch reads its monitors from the YAML file at startup and -never changes them. With the UI on, it also serves a page at `/ui/` and a JSON API -under `/api/v1/` for adding, editing, and deleting monitors while it runs, with no -restart and no lost grace timers. Monitors from the YAML file appear in the list as -read-only; only monitors created in the UI can be changed there. - -Turn it on with `--ui` or `collector.ui: true`. The flag wins, so `--ui=false` turns -off a config that enables it. The UI needs `collector.state_dir`, because monitors -created in the UI are kept in `state_dir/targets.json`. tail-watch exits at startup -without it, and ignores the UI under `--once`. - -```bash -tail-watch --config /etc/tail-watch/tail-watch.yaml --ui -``` - -`targets.json` uses the YAML field names and is written atomically with mode 0600. -The `defaults` block of the YAML file applies to UI monitors. A UI monitor is -accepted only if the YAML loader would accept the same target, and a monitor whose -name matches a YAML target is skipped at startup with a warning; the YAML target -wins. With the UI off, `targets.json` is left alone and its monitors do not run; a -startup warning says how many are inactive. - -An edit is a delete followed by an add. Editing or deleting a `DOWN` monitor closes -its open incident with `reason: removed`, and a new incident opens only if the new -definition fails `fail_threshold` checks. A monitor that other monitors name as -`parent` cannot be deleted until they are changed. Editing a parent briefly releases -its suppressed dependents, which can page while the parent is down. - -![The monitor list: YAML monitors are read-only, UI monitors have Edit and Delete, and monitors behind a down parent show SUPPRESSED.](docs/images/monitor-ui-list.png) - -The form shows validation errors from the same rules the YAML loader uses, next to -the field they belong to, and creates nothing until they are fixed. - -![The add monitor form with an invalid TCP port reported under Checks.](docs/images/monitor-ui-form-error.png) - -Deleting a `DOWN` monitor asks first, and says that its open incident closes. - -![The delete confirmation for a down monitor.](docs/images/monitor-ui-delete-down.png) - -### Reaching the UI - -The default `listen` address is `127.0.0.1:9109`, which only the host itself can -reach. To open the UI from another device on the tailnet, bind an address that -device can reach, such as the node's tailnet IP, and grant that device access to the -port. Grants cover the whole port, so they also expose `/metrics`. - -```jsonc -{ - "grants": [ - { - "src": ["autogroup:admin"], - "dst": ["tag:monitor"], - "ip": ["tcp:9109"] - } - ] -} -``` - -In the Compose layout, tail-watch shares the `tailscale` container's network -namespace, so every process in that namespace can reach the loopback listener. - -### Risks - -There is no login. Access control is the tailnet ACL and the listen address, and -tail-watch warns at startup when the address is not loopback. Whoever can reach the -port can make the collector probe any address it can route to. The API refuses -requests that are not `application/json` and requests whose `Origin` names another -host, which blocks cross-site form posts and cross-origin fetches. It does not stop -DNS rebinding, where the attacker's page and the `Host` header agree. Every write is -logged with the caller's address. - -## Notifications - -Every alert transition becomes a versioned JSON event that tail-watch delivers to -the sinks in the `sinks` list. A probe never waits on a sink: each sink has its own -worker and bounded queue, and a slow receiver cannot delay a check or another sink. -Sinks are `webhook` and `slack`. Secrets come only from `*_env` or `*_file` keys, -never from the YAML itself, and are read again on every send, so a rotated secret -needs no restart. Logs show a URL only as scheme and host (`https://hooks.slack.com/…`). - -| Event | When | -| --- | --- | -| `target.down` | a target goes `DOWN` and has no open incident | -| `target.recovered` | a target with an open incident reaches `UP` | -| `target.flapping` | a target crosses `flap_limit` | -| `collector.started` | once, after every target and check has a first result | -| `test` | only from `tail-watch notify-test` | - -A start with everything `UP` is silent apart from `collector.started`. A target -that is already failing at startup pages once. - -Severity is `info`, `warn`, or `page`. `target.down` uses the target's `severity` -(default `page`), a recovery uses the severity of the down it resolves, flapping is -`warn`, and `collector.started` and `test` are `info`. - -### Sink options - -```yaml -sinks: - - name: ops-webhook - type: webhook - url_env: OPS_WEBHOOK_URL # or url_file - secret_env: OPS_WEBHOOK_SECRET # HMAC key; without it requests are unsigned - headers_env: - Authorization: OPS_WEBHOOK_AUTH # header name: env var holding the value - min_severity: page # info < warn < page - events: [target.down, target.recovered] - max_age: 1h - queue: disk # needs collector.state_dir - queue_size: 1000 - batch: true - group_wait: 10s - max_batch: 100 - - name: ops-slack - type: slack - url_env: SLACK_WEBHOOK_URL - mention_on_page: "" - status_url: https://monitor-nyc.example.ts.net:9109 - group_wait: 10s - match: - sites: [nyc, sfo] # also targets (globs) and tags -``` - -Every sink accepts `events` (default all), `min_severity` (default `info`), `match` -with `sites`, `targets` and `tags`, `max_age` (default `1h`), `queue` (`memory` or -`disk`, default `memory`), and `queue_size` (default `1000`). An event reaches a sink -only when its type is listed, its severity is high enough, and every `match` key -matches. `tags` compares the Tailscale tags of the node the target resolves to. The -names `incidents`, `intake` and `state` are reserved because they appear as metric -labels. `tail-watch check-config` prints each sink's name, type, filters and queue mode, -never a secret. - -A recovery ignores the current filters: it goes to every sink that was selected for its -down, so a sink with `events: [target.down]` still hears when the target recovers. A -sink that never saw the down never sees the recovery. - -### Payload - -Every event is one JSON object. Within `version: 1` fields are only ever added. - -```json -{ - "version": 1, - "id": "evt_01JAB3K9T6M4X2Q8R7N5W0YZCD", - "type": "target.down", - "severity": "page", - "time": "2026-04-02T14:02:11Z", - "collector": { "name": "monitor-nyc", "tailscale_ip": "100.101.102.103" }, - "target": { "name": "nyc-nas", "addr": "10.20.0.50", "site": "nyc", "parent": "nyc-router" }, - "check": "tcp:445", - "layer": "service", - "error": "connection refused", - "since": "2026-04-02T14:00:41Z", - "failures": 3, - "path": { "kind": "direct", "endpoint": "203.0.113.5:41641", "latency_ms": 12 }, - "dependents": [] -} -``` - -`layer` is `unrouted`, `control`, `tunnel`, `lan`, or `service` and appears only on -`target.down` and `target.flapping`. `target.implicit: true` marks a router tail-watch -learned rather than one from your config. `target.recovered` adds `duration_seconds` -and `down_event_id`, and `reason: "removed"` when the target disappeared instead of -recovering. Fields that do not apply are omitted, not sent as `null`. `late: true` -marks an event first attempted more than a minute after its `time`; it keeps its -original `time`. - -`id` is `evt_` plus a 26 character Crockford base32 string that sorts by creation time. -It is the same across every retry and every sink. Delivery is at least once, so -receivers must deduplicate on `id` and must accept a `target.recovered` whose -`down_event_id` they never saw. - -### Webhook sink - -Each event is one `POST` with `Content-Type: application/json`, -`User-Agent: tail-watch/`, `Tail-Watch-Event-Id`, and any `headers_env` -headers. Redirects are not followed and each request times out after 10s. - -| Response | What tail-watch does | -| --- | --- | -| `2xx` | delivered | -| `408`, `5xx`, timeout, network error | retry with growing delay | -| `429` | retry, waiting at least `Retry-After` | -| anything else, including `3xx` and other `4xx` | drop, log once, count `permanent` | - -The delay is 1s, 2s, 4s and so on with 0.5–1.5× jitter, never more than 5 minutes. An -event older than `max_age` when an attempt is due is dropped and counted as `expired`. - -With a secret, each request carries `Tail-Watch-Signature: t=,v1=`, -where `v1` is the lowercase hex HMAC-SHA256 of `.` under the secret. A retry -sends identical body bytes with a fresh `t`. Verify it, and reject stale timestamps: - -```go -func verify(secret, header string, body []byte, tolerance time.Duration) bool { - var ts, sig string - for _, part := range strings.Split(header, ",") { - k, v, _ := strings.Cut(part, "=") - switch k { - case "t": - ts = v - case "v1": - sig = v - } - } - secs, err := strconv.ParseInt(ts, 10, 64) - if err != nil || time.Since(time.Unix(secs, 0)).Abs() > tolerance { - return false - } - mac := hmac.New(sha256.New, []byte(secret)) - fmt.Fprintf(mac, "%s.%s", ts, body) - return hmac.Equal([]byte(hex.EncodeToString(mac.Sum(nil))), []byte(sig)) -} -``` - -With `batch: true` the sink waits `group_wait` after the first queued event, or until -`max_batch` events are waiting, and sends -`{"version": 1, "batch_id": "bat_…", "events": [...]}` with `Tail-Watch-Batch-Id` in -place of `Tail-Watch-Event-Id`. The signature covers the whole body. A retry resends the -same bytes and `batch_id`; events that arrive meanwhile go into the next batch, and a -permanent failure or expiry drops the whole batch and counts every event in it. - -### Slack sink - -A Slack sink posts to an incoming-webhook URL. Each message carries Block Kit blocks -and a plain-text fallback: - -```text -:red_circle: DOWN nyc-nas tcp:445 (service) -addr 10.20.0.50 · site nyc · parent nyc-router · path direct 12 ms -error: connection refused -down since 14:00 UTC · 3 failures · collector monitor-nyc -``` - -A router learned at runtime shows `(learned)` after its name. A down with dependents adds -`and N targets behind it` and names up to 10. `mention_on_page` is prepended only to a -`target.down` of severity `page`, never to recoveries, flapping notices or warnings. With -`status_url` each target line links to `/#`. Events that arrive -within `group_wait` of the first become one message of at most 10 lines followed by -`and N more`, and the sink sends at most one message per second. Retry, `Retry-After`, -permanent-failure and `max_age` rules match the webhook sink. Slack sinks reject `batch`, -`max_batch`, `secret_*` and `headers_env`. - -### Queues, `state_dir` and open incidents - -`queue: memory` (the default) loses pending events when tail-watch restarts. `queue: disk` -needs `collector.state_dir` and keeps them. With `state_dir`, tail-watch keeps one file, -`notification-state.json`, holding open incidents, per-sink delivery state and the -pending events of disk sinks. Each change is written to a temporary file, fsynced, -renamed over the file, and the directory is fsynced, and only then can a worker send -the new events. The directory is created `0700` and the file `0600`; tail-watch warns if -the directory is readable by group or others. The file holds hostnames, addresses and -event payloads, never URLs, secrets or header values. If the file cannot be decoded, -startup fails with the path so you can repair it; tail-watch never silently forgets -open incidents. - -Open incidents are tracked even without `state_dir`. With it, a restart during an outage -does not page again, and the eventual recovery names the original `down_event_id`. A -configured target deleted from the config while an incident was open closes with -`reason: removed`. If the snapshot cannot be written, tail-watch keeps delivering from -memory, logs the loss of durability once, counts `tailwatch_notify_persist_errors_total`, -and retries the whole state every second. - -Delivery is at least once after a successful commit, and the limits are explicit and -counted: an event dropped because the queue is full, expired, permanently rejected, or -lost on a memory queue at restart shows up in `tailwatch_notify_dropped_total`. A `down` -dropped before any request was made takes its unsent `recovered` with it; if a request -was ever made, the `recovered` is still sent, because the receiver may have accepted a -request whose response was lost. Each sink keeps strict order. - -Disk commits are serialized. A slow disk can delay every durable enqueue and any worker -waiting to record an attempt, but it never delays a check, and the intake that receives -transitions from the alert state machine never blocks: if its 4096 slots fill, the -transition is dropped, counted as `dropped{sink="intake"}`, and the target's state is -reconciled within a second once there is room, so a lost recovery is still sent. - -| Metric | Kind | -| --- | --- | -| `tailwatch_notify_sent_total{sink,type}` | counter | -| `tailwatch_notify_failed_total{sink,reason}` | counter, reason `timeout`, `http_5xx`, `http_429`, `http_4xx_permanent`, `network` | -| `tailwatch_notify_dropped_total{sink,reason}` | counter, reason `queue_full`, `expired`, `permanent`, `restart` | -| `tailwatch_notify_persist_errors_total{sink}` | counter, sink is always `state` | -| `tailwatch_notify_queue_len{sink}` | gauge | -| `tailwatch_notify_latency_seconds{sink}` | histogram, event creation to confirmed delivery | -| `tailwatch_notify_batch_size{sink}` | histogram, batched webhook sinks only | -| `tailwatch_open_incidents` | gauge, when `state_dir` is set | -| `tailwatch_heartbeat_failures_total` | counter | - -Alert on `increase(tailwatch_notify_dropped_total[15m]) > 0` and on a -`tailwatch_notify_queue_len` that stays above zero: both mean an alert did not get out. - -### Watching the watcher - -A dead collector cannot report itself, so point `heartbeat` at a dead-man's-switch -service: - -```yaml -heartbeat: - url_env: HEARTBEAT_URL # or url_file - interval: 1m -``` - -tail-watch sends a `GET` every `interval`, starting once every target and check has a -first result, with a 10s timeout. A failed ping is logged, counted in -`tailwatch_heartbeat_failures_total`, and not retried before the next interval. When the -process exits the pings stop and the service alerts. - -### Trying it out - -```bash -./bin/tail-watch notify-test --config examples/tail-watch.yaml -./bin/tail-watch notify-test --sink ops-slack --timeout 10s -``` - -`notify-test` sends one `test` event through each sink's real signing, rendering and -retry code, bypassing filters, queues and `state_dir`. It prints one line per sink with -the HTTP status or error and exits 0 only when every selected sink answered `2xx` within -`--timeout` (default 30s). It works when tailscaled is down, using the OS hostname as -the collector name. Run it after every config change. - -### Egress path - -Sinks use the host's ordinary routing table, so the collector's node must reach them. -tail-watch warns at startup when an exit node is active, because notifications would -then leave through the exit node. A receiver on your own tailnet needs a grant from -`tag:monitor`: - -```jsonc -{ "src": ["tag:monitor"], "dst": ["tag:ops-receiver"], "ip": ["tcp:443"] } -``` - -### Live notification test - -`TestLiveNotifications` runs the whole notification path on a real tailnet. It is off -unless `TAIL_WATCH_LIVE_TAILNET=1`. It starts the collector and one fixture node per -check (`icmp`, `tsping`, `tcp`, `http`, and a LAN target behind a subnet router), fails and -heals each one, restarts the collector mid-outage with `queue: disk`, and runs -`notify-test`. A signature-verifying webhook receiver in the test process checks every -event and heartbeat. The compose fixtures are in -`cmd/tail-watch/testdata/compose.live-notify.yaml`. Devices are ephemeral, named -`tail-watch-live-notify-*`, and deleted afterwards. - -```bash -TAIL_WATCH_LIVE_TAILNET=1 TAIL_WATCH_LIVE_TAG= \ - TAIL_WATCH_SLACK_PULUMI_ENV=/tail-watch-live/slack \ - go test -count=1 -timeout 18m -run TestLiveNotifications -v ./cmd/tail-watch -``` - -With `TAIL_WATCH_LIVE_TAG` unset the collector joins as `tag:monitor`, so the tailnet -policy must own that tag and grant it the fixtures: merge -`cmd/tail-watch/testdata/snippet.hujson`. The grants that matter are `tcp:9000` and -`tcp:8080` to members (the `tcp` and `http` fixtures) and `icmp` to `10.10.0.0/16` (the LAN -target). Without the two TCP grants those fixtures are down from the start, with -`i/o timeout` errors because the packets are dropped. Setting `TAIL_WATCH_LIVE_TAG=` to -empty runs the collector as an untagged member node, which needs no policy change. - -Tailnet credentials come from the `tailscale-phase-2/tailscale` pulumi env -(`TAIL_WATCH_PULUMI_ENV` overrides it). Slack is optional: without a webhook the test -runs the webhook sink only. Slack is one-way, so the test checks tail-watch's own sent and -failed counters for that sink; look in the channel for the messages. -`TAIL_WATCH_LIVE_MENTION` sets `mention_on_page` for the Slack sink. - -The Slack sink uses a memory queue, so an event still inside its `group_wait` window when -the test restarts the collector is lost and counted as `dropped{reason="restart"}`. The -test accepts that reason and fails on any other drop or failure. The webhook sink uses -`queue: disk` and must survive the restart without paging again. - -To bring up only the tailnet side, without the collector process or any scenario, use -`scripts/live-stack.sh up`. It mints ephemeral keys from the pulumi env, starts the -collector's tailscale node and the five fixture nodes, approves the LAN route, and prints -each node's address. `scripts/live-stack.sh down` removes the containers and deletes -that stack's devices. The webhook can live in a pulumi env such as -`richard-shade-gmail-com/tail-watch-live/slack`; pass its name as -`TAIL_WATCH_SLACK_PULUMI_ENV`. To rotate it, run -`pulumi env set environmentVariables.SLACK_WEBHOOK_URL '' --secret`. - -Set `queue: disk` on the Slack sink if an alert must survive a collector restart. The -default `queue: memory` loses events still waiting in the group window. - -#### Getting a Slack webhook URL - -tail-watch posts to an incoming webhook URL. It does not use bot or app-level tokens -(`xoxb-`, `xapp-`), and a bot token cannot produce one. Slack's own guide, -[Sending messages using incoming webhooks](https://docs.slack.dev/messaging/sending-messages-using-incoming-webhooks/), -is the reference; its steps are: - -1. Create an app at , or open an existing one. -2. In the app's settings sidebar choose **Incoming Webhooks** and switch - **Activate Incoming Webhooks** on. -3. Click **Add New Webhook to Workspace**, pick the channel, and click **Allow**. -4. Copy the URL from **Webhook URLs for Your Workspace**. It starts with - `https://hooks.slack.com/services/`. - -Slack has also documented the `incoming-webhook` scope, which produces a webhook during -an OAuth install and shows a channel picker. If your app's sidebar has no -**Incoming Webhooks** entry, that is the alternative to try. The settings pages change -over time, so if these steps do not match what you see, treat this section as out of date -and check Slack's guide. - -Check the URL with -`curl -X POST -H 'Content-type: application/json' --data '{"text":"hello"}' "$SLACK_WEBHOOK_URL"`, -which answers `ok`. Store it as a secret, not in a file: - -```bash -pulumi env init /tail-watch-live/slack -pulumi env set /tail-watch-live/slack \ - environmentVariables.SLACK_WEBHOOK_URL 'https://hooks.slack.com/services/...' --secret -``` - -## Development and releases - -`mise run check` builds, tests, and lints with the pinned tools. The lint -configuration is based on -[maratori/golangci-lint-config](https://github.com/maratori/golangci-lint-config), -with this module's import prefix. CI also runs race tests and builds release -archives for Linux, macOS, and Windows on amd64 and arm64. +The status page is at `http://127.0.0.1:9109/`, and metrics are at `/metrics`. To run +without a dedicated host, or to receive your first alert, follow the +[tutorial](docs/tutorial/first-monitor.md). -Use Conventional Commits in squash-merge titles (`fix: ...`, `feat: ...`, -or `feat!: ...` for breaking changes). After checks pass on `main`, -release-please opens or updates a release PR containing the version and -changelog. Merge that PR to create a GitHub release; GoReleaser then attaches -the archives and `checksums.txt`. Each archive includes `tail-watch`, the license, -this README, and `examples/tail-watch.yaml`. The binary's `--version` reports the -release version. +## Documentation -Repository setup: in **Settings → Actions → General → Workflow permissions**, -enable **Allow GitHub Actions to create and approve pull requests**. -The workflow uses the built-in `GITHUB_TOKEN`. PRs created with that token -do not automatically start CI; close and reopen the release PR to trigger -its checks. Publishing runs in the release workflow itself because tags -created with that token also do not trigger a separate workflow. +The [documentation index](docs/README.md) lists every page. The main ones: -To validate packaging locally without publishing: +- Learn: [Set up your first monitor](docs/tutorial/first-monitor.md) +- Do: [Deploy](docs/how-to/deploy.md), [write the ACLs](docs/how-to/acls.md), + [send alerts to Slack](docs/how-to/slack-sink.md), and + [troubleshoot](docs/how-to/troubleshoot.md) +- Look up: [configuration](docs/reference/config.md), + [command line](docs/reference/cli.md), [metrics](docs/reference/metrics.md), and + [event payload](docs/reference/payload.md) +- Understand: [failure layers](docs/explanation/failure-layers.md) and + [alert state](docs/explanation/alert-state.md) -```bash -mise run release-check -``` +## Contributing -Snapshot archives appear in `dist/`. If publishing fails after the release -has been created, rerun the failed job or manually run the **Release** workflow -with its existing tag (for example, `v0.1.0`). +`mise run check` builds, tests, and lints with the pinned tools. See +[Development and releases](docs/contributing/releases.md) and the +[live notification test](docs/contributing/live-notification-test.md). diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..492977e --- /dev/null +++ b/docs/README.md @@ -0,0 +1,42 @@ +# tail-watch documentation + +tail-watch checks whether traffic to your tailnet nodes and the subnets behind them +gets through, and tells you which layer failed. These pages are grouped by what you +need to do. + +## Tutorial + +- [Set up your first monitor](tutorial/first-monitor.md): from the example Compose + file to an alert in Slack. + +## How-to guides + +- [Deploy tail-watch](how-to/deploy.md): binary, Docker, and Compose without a + dedicated host. +- [Write the ACLs](how-to/acls.md): grants for `tag:monitor`. +- [Monitor remote subnets and local LANs](how-to/remote-subnets.md) +- [Manage monitors in the UI](how-to/monitor-ui.md) +- [Send alerts to Slack](how-to/slack-sink.md) +- [Send alerts to a webhook](how-to/webhook-sink.md): signing, batching, retries. +- [Watch the watcher with a heartbeat](how-to/heartbeat.md) +- [Troubleshoot](how-to/troubleshoot.md) + +## Reference + +- [Configuration](reference/config.md) +- [Command line](reference/cli.md) +- [Checks](reference/checks.md) +- [Metrics](reference/metrics.md) +- [Event payload](reference/payload.md) + +## Explanation + +- [Failure layers](explanation/failure-layers.md) +- [Alert state](explanation/alert-state.md) +- [Incidents and queues](explanation/incidents-and-queues.md) +- [Monitor UI risks](explanation/monitor-ui-risks.md) + +## Contributing + +- [Live notification test](contributing/live-notification-test.md) +- [Development and releases](contributing/releases.md) diff --git a/docs/contributing/live-notification-test.md b/docs/contributing/live-notification-test.md new file mode 100644 index 0000000..8174357 --- /dev/null +++ b/docs/contributing/live-notification-test.md @@ -0,0 +1,66 @@ +# Live notification test + +This page is for contributors who need to run the notification path against a +real tailnet. + +`TestLiveNotifications` runs the whole notification path on a real tailnet. It is off +unless `TAIL_WATCH_LIVE_TAILNET=1`. It starts the collector and one fixture node per +check (`icmp`, `tsping`, `tcp`, `http`, and a LAN target behind a subnet router), fails and +heals each one, restarts the collector mid-outage with `queue: disk`, and runs +`notify-test`. A signature-verifying webhook receiver in the test process checks every +event and heartbeat. The compose fixtures are in +`cmd/tail-watch/testdata/compose.live-notify.yaml`. Devices are ephemeral, named +`tail-watch-live-notify-*`, and deleted afterwards. + +```bash +TAIL_WATCH_LIVE_TAILNET=1 TAIL_WATCH_LIVE_TAG= \ + TAIL_WATCH_SLACK_PULUMI_ENV=/tail-watch-live/slack \ + go test -count=1 -timeout 18m -run TestLiveNotifications -v ./cmd/tail-watch +``` + +## Tailnet policy and tags + +With `TAIL_WATCH_LIVE_TAG` unset the collector joins as `tag:monitor`, so the tailnet +policy must own that tag and grant it the fixtures: merge +`cmd/tail-watch/testdata/snippet.hujson`. The grants that matter are `tcp:9000` and +`tcp:8080` to members (the `tcp` and `http` fixtures) and `icmp` to `10.10.0.0/16` (the LAN +target). Without the two TCP grants those fixtures are down from the start, with +`i/o timeout` errors because the packets are dropped. Setting `TAIL_WATCH_LIVE_TAG=` to +empty runs the collector as an untagged member node, which needs no policy change. + +## Credentials + +Tailnet credentials come from the `tailscale-phase-2/tailscale` pulumi env +(`TAIL_WATCH_PULUMI_ENV` overrides it). Slack is optional: without a webhook the test +runs the webhook sink only. Slack is one-way, so the test checks tail-watch's own sent and +failed counters for that sink; look in the channel for the messages. +`TAIL_WATCH_LIVE_MENTION` sets `mention_on_page` for the Slack sink. + +The test reads the webhook from `SLACK_WEBHOOK_URL`, or from the pulumi env named by +`TAIL_WATCH_SLACK_PULUMI_ENV`. Never write it to a file or a log. To get a webhook URL, +see the [Slack sink how-to](../how-to/slack-sink.md). + +The webhook can live in a pulumi env such as `/tail-watch-live/slack`; pass its name +as `TAIL_WATCH_SLACK_PULUMI_ENV`. To rotate it, run: + +```bash +pulumi env set environmentVariables.SLACK_WEBHOOK_URL '' --secret +``` + +## Restart behavior + +The Slack sink uses a memory queue, so an event still inside its `group_wait` window when +the test restarts the collector is lost and counted as `dropped{reason="restart"}`. The +test accepts that reason and fails on any other drop or failure. The webhook sink uses +`queue: disk` and must survive the restart without paging again. + +Set `queue: disk` on the Slack sink if an alert must survive a collector restart. The +default `queue: memory` loses events still waiting in the group window. + +## Bringing up only the tailnet side + +To bring up only the tailnet side, without the collector process or any scenario, use +`scripts/live-stack.sh up`. It mints ephemeral keys from the pulumi env, starts the +collector's tailscale node, the four fixture nodes, and the LAN router, approves the LAN +route, and prints each node's address. `scripts/live-stack.sh down` removes the containers +and deletes that stack's devices. diff --git a/docs/contributing/releases.md b/docs/contributing/releases.md new file mode 100644 index 0000000..3e309ed --- /dev/null +++ b/docs/contributing/releases.md @@ -0,0 +1,44 @@ +# Development and releases + +This page is for contributors who build, check, and release tail-watch. + +`mise run check` builds, tests, and lints with the pinned tools. The lint +configuration is based on +[maratori/golangci-lint-config](https://github.com/maratori/golangci-lint-config), +with this module's import prefix. CI also runs race tests and builds release +archives for Linux, macOS, and Windows on amd64 and arm64. + +## Releasing + +Use Conventional Commits in squash-merge titles (`fix: ...`, `feat: ...`, +or `feat!: ...` for breaking changes). After checks pass on `main`, +release-please opens or updates a release PR containing the version and +changelog. Merge that PR to create a GitHub release; GoReleaser then attaches +the archives and `checksums.txt`. Each archive includes `tail-watch`, the license, +the README, the `docs/` tree, and `examples/tail-watch.yaml`. The binary's +`--version` reports the release version. + +release-please owns `CHANGELOG.md`; do not edit it by hand. It lists `feat`, `fix`, +`perf`, `revert` and `docs` commits, so a documentation change appears under +**Documentation** when its squash-merge title starts with `docs:`. + +## Repository setup + +In **Settings → Actions → General → Workflow permissions**, +enable **Allow GitHub Actions to create and approve pull requests**. +The workflow uses the built-in `GITHUB_TOKEN`. PRs created with that token +do not automatically start CI; close and reopen the release PR to trigger +its checks. Publishing runs in the release workflow itself because tags +created with that token also do not trigger a separate workflow. + +## Validating packaging locally + +To validate packaging locally without publishing: + +```bash +mise run release-check +``` + +Snapshot archives appear in `dist/`. If publishing fails after the release +has been created, rerun the failed job or manually run the **Release** workflow +with its existing tag (for example, `v0.1.0`). diff --git a/docs/explanation/alert-state.md b/docs/explanation/alert-state.md new file mode 100644 index 0000000..51f379b --- /dev/null +++ b/docs/explanation/alert-state.md @@ -0,0 +1,208 @@ +# Alert state + +A check result is a fact about one moment. An alert is a decision that a +human should look. Between them sits a state machine whose job is to turn a +noisy stream of results into as few notifications as possible without hiding +a real outage. This page explains what that machine does and why it is +shaped the way it is, so you can predict its behavior and tune it. + +The keys named here are set under `defaults:` or on each target. Their +defaults and syntax are in the [configuration reference](../reference/config.md). +The state values as exported are in the [metrics reference](../reference/metrics.md). + +## One state per target + +Every target is in exactly one of four states: + +| State | Meaning | +| --- | --- | +| `UNKNOWN` | No verdict yet since the process started. | +| `UP` | Every check has been passing. | +| `DOWN` | A check failed `fail_threshold` times in a row. | +| `SUPPRESSED` | A parent that the target depends on is down. | + +The states are held in memory only. A restart begins again at `UNKNOWN`, and +the durable part of an outage is the incident record described in +[Incidents and queues](incidents-and-queues.md). That split is deliberate: +the state machine can stay simple and cheap because a restart costs a few +check cycles of "we do not know yet", not a lost incident. + +The gauge `tailwatch_target_state` adds a fifth value, 4, for a target that is +flapping. Flapping is a mark on top of the state and not a state of its own, +and it takes precedence in the gauge while it is set. + +## Why thresholds, and why they are per check + +One lost packet must not page anyone. A target therefore goes `DOWN` only +after `fail_threshold` consecutive failures of a single check (default 3), and +a success resets that check's count. With the default 30 second interval that +is roughly a minute and a half from first failure to page, and the time of the +first failure in the run is what the alert reports as `since`, so the outage +start is not pushed back by the waiting. + +Counters are kept per target and check pair, not per target. Consider a target +with `icmp` and `tcp:445`. If the counter were shared, a failing port +interleaved with a passing ping would never accumulate three failures, and a +real fault would be hidden by an unrelated healthy check. Per check counters +mean that any one check reaching the threshold is enough. + +Going down is quick and coming back is slow on purpose. A target returns to +`UP` only after `recover_threshold` consecutive successes (default 2) on every +check that failed while it was down, and only when none of its other checks is +currently failing. If `tcp:445` recovers while `icmp` has started failing, the +target stays `DOWN`. The asymmetry favors a recovery message that stays true +over one that is retracted a minute later. + +### Starting up + +A fresh process has no history, so a target leaves `UNKNOWN` in one of two +ways. If every check succeeds once, the target becomes `UP` and says nothing. +Startup is not news, and a restart during a healthy period would otherwise +produce a burst of "recovered" messages. If a check reaches `fail_threshold` +first, the target goes straight from `UNKNOWN` to `DOWN` and pages once, which +is why a target that is already broken when you start the collector is +reported. + +A separate `collector.started` event announces that every target has produced a +first result, so a silent start is still distinguishable from a dead collector. + +## Flap damping + +A target that alternates between failing and passing can defeat thresholds: +each cycle is long enough to page and short enough to be gone by the time +someone looks. `flap_limit` and `flap_window` (defaults 4 and 15 minutes) +bound the damage. + +The machine counts `UP` and `DOWN` changes inside a sliding window. When a +change would push the count past `flap_limit`, it emits one flapping notice in +place of that change's normal notification, and then goes quiet. While the mark +is set the target's state is still tracked, but nothing is emitted. The mark +clears when the target has been stable for a full `flap_window`. If the +settled state differs from the last state that was reported, one ordinary +notification catches the receiver up. If it matches, nothing is sent. + +Two consequences follow from that design. + +- The receiver's picture stays coherent. A target that flapped and settled + `UP` after being reported `DOWN` still gets its recovery, so no incident is + left open on the receiving side. +- Damping trades latency for quiet. A real outage that begins during a flap + will not page until the target has been stable, or until the flap window + runs out. If you cannot accept that, raise `flap_limit` or shrink + `flap_window` for that target rather than turning damping off elsewhere. + +## Parent suppression + +The most common cause of a wall of alerts on a tailnet is not forty broken +devices. It is one broken subnet router with forty devices behind it. Without +suppression, each device would page separately and the one useful alert, the +router, would be buried among its symptoms. + +Each target can have a parent: a target that must work for it to be +reachable. The parent comes from an explicit `parent` key, or from the +dependency learning described in [Failure layers](failure-layers.md). Two +rules keep the mechanism honest. + +- **A parent is a reachability target.** It may have only an `icmp` or a + `tsping` check. A service check on the same node is not a parent, so a + failed web interface cannot silence the devices behind that router. +- **A down parent means `DOWN` or `SUPPRESSED`.** Suppression is transitive. If + a switch behind a router is itself suppressed, its own dependents are too. + +When a parent goes down, every dependent in `UNKNOWN` or `UP` moves to +`SUPPRESSED` at the same instant, silently. The parent's own notification names +them in `dependents`, so one message carries the whole blast radius. A dependent +that is already `DOWN` keeps its incident, because it was reported on its own +merits and dropping it would leave an alert with no resolution. Checks on +suppressed targets keep running, which is what lets the machine notice the +moment the situation changes. + +### The detection window + +Suppression has a race. The device's checks and the router's checks run on +independent schedules, so a device can reach its threshold a few seconds before +the router does. Paging on the device then produces exactly the duplicate that +suppression was meant to prevent. + +The answer is a short wait. When a target with a parent reaches its threshold +on a `control` or `tunnel` failure and the parent is not yet down, the target +does not go `DOWN` immediately. It waits up to the parent's detection window, +which is the parent's `interval` times its `fail_threshold`, plus its +`timeout`. That is the longest the parent could take to reach its own verdict +if it started failing at the same moment. If the parent goes down in that +time, the target is suppressed. If not, the target goes `DOWN` at the end of +the wait, provided its check is still failing, and the alert keeps the +original layer. + +Only `control` and `tunnel` failures wait, because only those layers say "the +path to the parent's side is broken". A `lan`, `service` or `unrouted` failure +pages immediately. By definition the tunnel to the router works for a `lan` +or `service` failure, so the router is not the cause. For `unrouted`, no router +exists to blame. See [Failure layers](failure-layers.md) for how those labels are +chosen. + +### Release + +When the parent recovers, is removed, or a topology change leaves a target +with no down ancestor, tail-watch does not wait for the next scheduled check. +It re-runs every check of each released target promptly, at most eight at a +time, so a recovered router does not leave forty devices in limbo for a full +interval. A target whose fresh checks pass returns to `UP` silently. One whose +checks keep failing counts `fail_threshold` failures from the release, then +goes `DOWN` with its own alert. + +The counters are reset on release, and results from checks that began before +it are ignored. Without that, a timeout that started while the router was +down would be blamed on the device as a fresh failure. A failover in a +high-availability pair works the same way: if learning moves a suppressed +device from a dead router to a live one, the device is released at once, even +though the dead router never recovers. + +One subtle case exists for incidents that were already reported. If flap +damping left a `DOWN` unresolved before the target was suppressed, that +incident is preserved through suppression and release. The target sends its +recovery once damping permits, with the original outage start time, and +continued failures do not open a second incident. + +## Quorum across collectors + +One collector sees the tailnet from one place. If its uplink fails it will +report every target down, and one unhealthy collector cannot tell you whether +the problem is the target or itself. State is not shared between collectors, so +tail-watch does not vote. + +Quorum is left to Alertmanager, which already knows how to combine sources. +Each collector exports `tailwatch_target_state`, and a rule can require +agreement: + +```yaml +- alert: TargetDownQuorum + expr: count by (target) (tailwatch_target_state == 2) >= 2 + for: 1m +``` + +That pages only when two collectors agree the target is `DOWN`. The rejected +alternative was a gossip protocol among collectors. It would add a network +dependency between the very machines you are trying to make trustworthy, to +solve a problem Alertmanager users have already solved. + +## Choosing values + +The defaults suit a 30 second interval and mostly stable devices. A few +relationships help when you tune them: + +- Time to page is about `interval` times `fail_threshold`, plus `timeout` for + the last attempt. Shortening the interval shortens the detection window of + every parent that uses it. +- A larger `recover_threshold` makes recovery slower and flapping less likely. +- Raising `flap_limit` makes a target flap less readily. A larger + `flap_window` keeps it quiet for longer after it settles. +- `severity` on a target controls how loud the resulting events are, not + whether the state machine reaches `DOWN`. See + [Incidents and queues](incidents-and-queues.md) for routing. + +## Related pages + +- [Configuration reference](../reference/config.md) for every key named here. +- [Metrics reference](../reference/metrics.md) for `tailwatch_target_state`. +- [Failure layers](failure-layers.md) for the labels that drive the wait. diff --git a/docs/explanation/failure-layers.md b/docs/explanation/failure-layers.md new file mode 100644 index 0000000..1404cad --- /dev/null +++ b/docs/explanation/failure-layers.md @@ -0,0 +1,186 @@ +# Failure layers + +A monitoring alert that says only "nyc-nas is down" leaves the hard part to +you. On a tailnet with subnet routers, a device can be unreachable because +the control plane thinks its router is offline, because the WireGuard tunnel +to the router is broken, because the LAN behind the router lost the device, +or because the device is fine and one service on it is not. Each cause has a +different owner and a different fix. tail-watch labels every failed check +with one of five layers so the alert starts the diagnosis instead of +postponing it. + +This page explains how the label is chosen, why it is chosen in that order, +and where it can mislead you. The label values and metric are listed in the +[metrics reference](../reference/metrics.md) and the +[event payload reference](../reference/payload.md). + +## The five layers + +A passing check has no layer. A failing check has exactly one, and the value +appears as the `layer` label on `tailwatch_failure_layer_total` and as the +`layer` field on `target.down` and `target.flapping` events. + +| Layer | What it says | +| --- | --- | +| `unrouted` | No peer owns or routes the address. | +| `control` | Tailscale's control plane reports the handling node offline. | +| `tunnel` | The handling node looks online, but a disco ping to it fails. | +| `service` | The device demonstrably answered, or the target is a node. | +| `lan` | The tunnel works, but the device behind the node does not answer. | + +The "handling node" is the tailnet node that carries traffic for the target. +For a target that is itself a Tailscale node, that is the node. For a device +on a subnet, it is the subnet router that advertises the route. Nothing else +in the model makes sense without this idea: tail-watch never reasons about the +device directly until it has established that the path to the handling node is +sound. + +## Deciding from the bottom up + +The classifier walks the stack from the lowest layer to the highest and stops +at the first layer that explains the failure. The order matters, because a +failure low in the stack makes every higher check fail too. If the router is +offline, the device's TCP port also refuses to connect, and reporting that as +a service problem would send you to the wrong team. + +1. If no peer owns the address or advertises a route that covers it, nothing + could ever have carried the traffic. That is `unrouted`. It usually points + at a missing route approval or a missing grant rather than at an outage, + which is why tail-watch also logs a hint about grants to `tag:monitor` the + first time it notices (see the + [ACL how-to](../how-to/acls.md)). +2. If the handling node exists and the control plane says it is offline, that + is `control`. The claim is cheap to check and it is authoritative about + what Tailscale believes, so it comes before any active probe. +3. If the node is not known to be offline, tail-watch sends a disco ping to + it. A failed ping is `tunnel`: the coordination server thinks the node is + fine, but no WireGuard path works. Typical causes are a firewall that + blocks UDP, a node whose tailscaled has hung, or a path that flaps. +4. If the tunnel works, the failure is on the far side of the handling node. + When the device gave evidence that it is alive, the layer is `service`. A + refused TCP connection, an HTTP status or body mismatch, and a TLS error + are all answers, just not the answer the check wanted. A timed-out check + is ambiguous, so tail-watch also sends one ICMP echo to the device. If + the echo succeeds, the host is up and the layer is `service`. +5. If the tunnel works and the device gives no sign of life, the layer is + `lan`. The path into the site is fine and the device is not on it. + +A target that is itself a Tailscale node never gets `lan`, because there is no +LAN behind it. Where the walk would otherwise say `lan`, it says `service`. + +### Failed `tsping` checks + +A `tsping` check is a disco ping, which is the same probe the classifier would +send in step 3. Rather than probing twice, tail-watch treats the failed check +as that probe. The result is `control` when presence shows the node offline +and `tunnel` otherwise. + +### Local targets + +A target with `network: local` is not reached through the tailnet, so there is +no handling node to consult. Presence, routes and disco do not apply, and the +only layers available are `service` (the device answered in some way) and +`lan` (it did not). + +## What the classifier knows, and what it does not + +The classifier uses the tailnet's own view of itself. That view comes from +the local tailscaled over its IPN bus, which reports which peers exist, +whether the control plane counts them online, and which routes they serve. +tail-watch keeps a live copy of it. + +The copy has one property that shapes the layers: it distinguishes "known +offline" from "unknown". When the bus connection is down, or has not yet +delivered its first snapshot, every peer is unknown. The classifier does not +guess `control` from missing data. It carries on to the disco ping and +decides between `tunnel` and the layers above it. So a tailscaled restart +degrades precision (a genuinely offline router shows up as `tunnel` for a +short time) but never invents a control-plane claim. + +Presence never raises an alert by itself. A router flipping offline while +every check still passes produces no failure and no layer. The layer only +describes a check that has already failed. + +## Learning what a device depends on + +`unrouted`, `control` and `tunnel` all depend on knowing the handling node, +and you should not have to declare it by hand for every device. For each +remote LAN target, tail-watch sends one disco ping at startup, every 10 +minutes, and when primary routes change. The pong names the node that +actually carried it. That node becomes the target's parent for classification +and for suppression (see [Alert state](alert-state.md)). + +The design choices behind that: + +- **The pong is authoritative, routes are the fallback.** In a + high-availability pair, the route table names the primary, but the node that + answers is the one that matters. The learned parent follows whichever router + the pong reports, so a failover changes the parent without configuration. + When the ping fails, a changed primary route is used, and otherwise the + previously learned node is kept, so a router that is down at the moment of + learning is still the parent you would want to blame. +- **A parent must be a reachability target.** A learned parent maps only to a + configured target whose sole check is `tsping`. A router that also runs a + monitored web interface has a `tsping` target and a separate service target, + and only the first can suppress devices. Otherwise a broken admin page on a + router would silence every device behind it. When no such target exists, + tail-watch creates an implicit `tsping` target named after the node and + removes it when nothing depends on it, although an implicit target that is + still down stays until it recovers, so its incident is not orphaned. +- **A configured `parent` always wins.** It is for suppression only. It does + not change which node the classifier treats as the handler, because the + handler is a fact about routing and the parent is a statement about blame. + A device behind a second switch can name that switch as its parent without + pretending that the switch carries its tailnet traffic. +- **Cycles are rejected.** A learned edge that would close a loop with the + configured graph is refused and the previous graph stays. + +If no peer handles an address at all, tail-watch logs the missing grant once, +and logs again only if the address resolves and is later lost, so a permanent +misconfiguration does not fill the log. + +## Path kind is not a layer + +For every `tsping` result, tail-watch also records how the pong arrived: +`direct`, `derp`, or `peer_relay`, on `tailwatch_path{target,kind}`. A path +change never fails a check and never changes a layer. A node that falls back +to DERP is slower, not broken, and an alert would be noise. The metric exists +so that you can graph the slowdown and correlate it with a later `tunnel` +failure. + +## Cost of supplemental probes + +Step 3 and step 4 can need an extra disco ping or ICMP echo. When forty +devices behind one router fail in the same second, sending forty pings +to the same router would add load exactly when the network is in trouble and +would tell you nothing new. Supplemental results are cached per handling node +(for disco) and per address (for ICMP), reused for 10 seconds or the target's +interval, whichever is shorter, and at most one probe per key is in flight; +concurrent callers wait for it and share the result. A fresh passing result +from the target's own `icmp` or `tsping` check replaces a supplemental probe. +These probes are not checks, so they do not appear on the status page or in +the prober result metrics. + +## Limits of the labels + +- The layer is a best explanation from the vantage point of one collector. A + `tunnel` verdict means that this collector cannot reach the node, not that + nobody can. That is one reason to run more than one collector (see the + quorum rule in [Alert state](alert-state.md)). +- `service` means "something answered", not "the service is healthy". An ICMP + reply with a timed-out TCP check is `service`, on the reasoning that a + live host with a dead port is a service problem for the owner of the + service. +- `lan` is a conclusion by elimination. tail-watch cannot see the LAN. It + sees that the tunnel works and the device is silent, and that combination + is the definition. +- The layer on an alert is the layer of the check that tripped the threshold. + A later failure of a different check does not rewrite it. + +## Related pages + +- [Checks reference](../reference/checks.md) for what each check tests. +- [Alert state](alert-state.md) for how layers decide whether an alert is held + back behind a parent. +- [Troubleshooting](../how-to/troubleshoot.md) for what to do about each + layer. diff --git a/docs/explanation/incidents-and-queues.md b/docs/explanation/incidents-and-queues.md new file mode 100644 index 0000000..3d9e400 --- /dev/null +++ b/docs/explanation/incidents-and-queues.md @@ -0,0 +1,207 @@ +# Incidents and queues + +A notification system has two promises that pull against each other. An alert +must not be lost, and a slow or dead receiver must never stop the monitoring +that produces alerts. This page explains how tail-watch keeps both promises, +what it does when it cannot, and the limits it tells you about instead of +hiding. + +The sink keys mentioned here are in the +[configuration reference](../reference/config.md). The metrics are in the +[metrics reference](../reference/metrics.md), and the fields on the wire are +in the [payload reference](../reference/payload.md). Setup instructions are in +the [webhook](../how-to/webhook-sink.md) and [Slack](../how-to/slack-sink.md) +guides. + +## Events and incidents + +The alert state machine emits transitions. The notifier turns them into events +(`target.down`, `target.recovered`, `target.flapping`, and the rest) and +groups the down and recovery of one outage into an incident. + +An incident is the memory that pairs a recovery with the down it resolves. It +records the tripping check, the severity, the start time and the id of the +`target.down` event. A recovery carries that id as `down_event_id`, which lets +a receiver close the exact alert it opened, for example an incident in a paging tool, +instead of guessing by target name. + +There is at most one open incident per target. If a target fails again while +its recovery is still being delivered, a new incident opens with a new id +queued behind the old recovery. Incidents exist even without `state_dir`, +which keeps duplicate downs from being sent while one is open. + +## The path from a check to a receiver + +The design goal is that nothing on the way from a probe to a network socket +can make a probe wait. + +1. The alert state machine hands each transition to the notifier through an + intake with 4096 slots. That call only writes to the channel. It does no + disk or network work, so a check can never block on notification. +2. The notifier maps the transition to an event, updates the incident, decides + which sinks should receive it, and commits the result. +3. Each sink has its own worker and its own bounded queue. A receiver that + takes ten seconds per request delays only its own sink, not another sink + and not a check. + +Delivery within one sink is strictly ordered. A recovery waits until its down +has been acknowledged or has been dropped for good. The alternative, sending in +parallel for throughput, would allow a recovery to arrive first and a receiver +to resolve something it has not yet opened. Ordering costs some throughput +under a backlog, which is a good trade for an alerting channel. + +## At-least-once delivery + +Exactly-once delivery across a network is not achievable, so tail-watch picks +the failure mode that is safe for alerts: an event may be delivered more than +once, but a committed event is not silently forgotten. + +Every event has an id that stays the same across every retry and every sink. +That gives receivers what they need to deduplicate. The contract has a second +half that receivers must accept: a `target.recovered` can arrive for a down the +receiver never saw. That happens when a request reached the receiver but the +response was lost, when the down was dropped, or when the receiver was added +later. Treat an unknown `down_event_id` as a no-op, not as an error. + +The same reasoning explains an apparent oddity. When a down was never +attempted, for example it was dropped from a full queue, its recovery is +discarded too, because sending a resolve for something never sent is only +noise. When a down was attempted but not acknowledged, its recovery is still +sent. The receiver may have accepted a request whose response was lost, and +leaving that alert open forever would be worse than a stray resolve. + +Recoveries also ignore filters that changed since the down. A sink that +received a down always gets the matching recovery, even if its `events` list +would now exclude it, and a sink that never saw the down never sees a recovery. + +An event first attempted more than a minute after its creation time is marked +`late` and keeps its original timestamp, so a receiver can tell a delayed +delivery from a fresh one. + +## Queues: memory or disk + +Each sink chooses where pending events live. + +| Mode | Behavior | Suits | +| --- | --- | --- | +| `queue: memory` | Fast and default; pending events vanish on restart. | Chat sinks where a late message is worth little. | +| `queue: disk` | Pending events survive restart; needs `collector.state_dir`. | Sinks that feed paging or automation. | + +The trade-off is between complexity and safety. A memory queue has no failure +mode beyond process death. A disk queue requires a directory that the +collector can write, and a slow disk can add latency to every durable enqueue. +It never delays a check, but it can delay delivery. + +Queues are bounded by `queue_size` (default 1000). When one is full, the +oldest event is dropped, not the newest. A backlog is most often the sign of a +receiver that has been away for a while, and by then the newest events describe +the current state of the world while the oldest describe history that has +probably resolved. The events also expire after `max_age` (default 1 hour), for +the same reason: a page for a target that recovered hours ago is worse than no +page. + +## State on disk + +With `collector.state_dir`, tail-watch keeps one file, +`notification-state.json`. It holds open incidents, the delivery state of every +incident on every sink, and the pending events of disk sinks. It deliberately +does not hold URLs, secrets or header values, so a copied or backed up state +directory is not a credential leak. + +Two decisions explain how it is written. + +**One file, one commit.** An incident change and the queue intents that follow +from it are written in the same snapshot. Splitting them across files would +create a window where a crash leaves an incident closed with its recovery never +queued, or a recovery queued for an incident that does not exist. The snapshot +is written to a temporary file, synced, renamed over the old one, and the +directory is synced. Only then do workers see the new work. A crash before the +rename leaves the old snapshot intact and a crash after it replays the new one. +That is why delivery is at least once *after a successful commit*, and not +before it. + +**Attempts are recorded before they are made.** A disk sink marks an event +"attempted" durably before sending it. If the process dies between the send +and the acknowledgment, replay knows a request may have gone out. It resends +the identical body and id, and it keeps the recovery eligible. The cost is a +possible duplicate. + +A snapshot that cannot be decoded stops startup with an error that names the +file. The alternative, starting with empty state, would forget every open +incident and page again, or worse, never send the recoveries. Refusing to start +is loud, and the operator decides what to do with the file. + +If a write fails, for example the disk is read-only or full, the collector does +not stop notifying. It keeps the authoritative state in memory, delivers from +memory, logs the loss of durability once, increments +`tailwatch_notify_persist_errors_total`, and retries persisting the full state +every second. Durability is degraded until a write succeeds, and that is +something you should alert on, because a crash in that window loses queued +events. + +## Restarts and open incidents + +At startup, before any check is scheduled, the notifier restores open +incidents into the alert state machine as `DOWN` with their original start +time. Restoring an incident is not a new alert. No second `target.down` is +sent, and the eventual recovery carries the original `down_event_id` and +duration. Without this, every restart during an outage would either re-page or +strand the original alert without a resolution. + +Learned routers add a wrinkle. An implicit target exists only because +dependency learning created it, so at restart it might not be rediscoverable +yet. The notifier records enough to recreate it and does not treat a failed +discovery pass as evidence that the router is gone. It closes such an incident +with `reason: removed` only after a reconciliation that saw the full topology. +A target that was deleted from the configuration, or a monitor deleted in the +UI, closes its incident the same way, so a receiver is not left with an alert +for something that no longer exists. + +## Intake overflow and reconciliation + +The intake is bounded, and a full intake drops the transition instead of +blocking. That looks alarming until you consider the alternative: a blocked +intake would stall the state machine, which would stall probing. The design +keeps a second, independent source of truth instead. The alert state machine +holds a snapshot of the latest notification-visible state of every target and +marks a target dirty when its transition could not be delivered. + +Once per second, the notifier reconciles dirty targets against their +snapshots. An open incident whose target is now `UP` gets a recovery. A target +that is `DOWN` with no incident gets a down. A recovery lost to overflow is +therefore still sent, without waiting for another state change. The price is +coalescing: if a target blipped down and up while the notifier was behind, the +receiver may never hear about the blip. That is acceptable, because the blip +was, by definition, not what the receiver would have wanted a page for. + +Removal follows the same route. A target removed from the collector reaches the +notifier only through a terminal snapshot, so the incident close is not lost if +the transition was. + +## When alerts are lost + +tail-watch does not promise that no alert is ever lost. It promises that a loss +has a named reason and increments a counter. Every drop increments +`tailwatch_notify_dropped_total` with one reason: + +| Reason | What happened | +| --- | --- | +| `queue_full` | The sink queue, or the intake with `sink="intake"`, was full. | +| `expired` | The event was older than `max_age` when an attempt was due. | +| `permanent` | The receiver answered with a status that will never succeed. | +| `restart` | A memory queue lost the event when the process restarted. | + +Failed attempts that will be retried are counted separately in +`tailwatch_notify_failed_total`. The recommended alerts are on any increase of +the dropped counter and on a queue length that stays above zero, since both +mean an alert did not get out. + +Notification also has an obvious blind spot: a collector that is dead cannot +say so. The [heartbeat how-to](../how-to/heartbeat.md) covers pointing the +collector at a dead-man's-switch service that alarms on silence. + +## Related pages + +- [Alert state](alert-state.md) for how transitions are produced. +- [Event payload](../reference/payload.md) for the fields receivers depend on. +- [Troubleshooting](../how-to/troubleshoot.md) for missing notifications. diff --git a/docs/explanation/monitor-ui-risks.md b/docs/explanation/monitor-ui-risks.md new file mode 100644 index 0000000..9ca837f --- /dev/null +++ b/docs/explanation/monitor-ui-risks.md @@ -0,0 +1,174 @@ +# Monitor UI risks + +The monitor UI lets someone add, edit and delete monitors while tail-watch +runs. That is a write capability on a service that otherwise only reads +configuration, and it is off by default for that reason. This page explains +what the UI protects, what it deliberately does not, and how to decide whether +the trade-off is acceptable on your tailnet. + +To turn the UI on and use it, see the [monitor UI how-to](../how-to/monitor-ui.md). +The `listen` and `state_dir` keys are in the +[configuration reference](../reference/config.md), and the flag is in the +[command line reference](../reference/cli.md). + +## What a write is worth to an attacker + +A monitor is an instruction to the collector to send probes to an address of +the writer's choosing. The collector runs on a tailnet node, with whatever +routes and grants its tag has. Someone who can create monitors can therefore +make the collector probe any address it can route to, and read the outcome +back in the monitor's state. In a network with a permissive collector tag, +that is a port scanner planted inside your network, and the results appear on +the status page. The writer can also delete monitors, which silences alerts, +and close open incidents as a side effect (see +[Incidents and queues](incidents-and-queues.md)). + +The UI does not expose secrets. Sink URLs and keys never pass through it, and +it can only change monitors that were created in the UI. Monitors from the YAML +file are shown as read-only. So the exposure is the ability to probe and to +silence, not credential theft. + +## No login, on purpose + +There is no login. tail-watch has no user database, no session handling and no +password to leak. Access control is the network: the listen address decides who +can open a socket, and the tailnet ACL decides who can reach that address. + +The alternatives were weighed against a small tool that is meant to run +unattended: + +- **Built-in accounts.** They add credentials to store and rotate, a login + page to defend, and a second identity system next to the one the tailnet + already has. For a monitor with a handful of operators, that is more attack + surface than the feature is worth. +- **Tailscale identity from the request.** This is the natural fit for a + tailnet service, because the caller's identity is already authenticated. It + is not built in today. The write path passes every request through a single + authorization decision made before any change, so identity-based + authorization can be added at that one point later. The decision currently + permits everything that reaches the listener. +- **Network scoping (chosen).** It relies on controls you already run and audit + and it has no state of its own. + +The cost is that everything reachable is trusted equally. Reading the monitor +list needs no more than reaching the port, and a grant gives write access to +everyone it covers. + +## The listen address is the security boundary + +The default `listen` address is `127.0.0.1:9109`. Only processes on the +collector host can reach it. That is a safe default, and it is also a reason +the UI is awkward to use from a laptop. + +To reach the UI from another device, you bind an address that device can reach, +such as the node's tailnet IP, and grant that device access to the port. Two +things follow. + +- **A grant covers a port, not a path.** The same listener serves `/metrics` + and the status page. Whoever can open the UI can also read the metrics. If + you scrape metrics from a different set of hosts than you want to edit + monitors from, the shared port is a constraint, and worth planning your grants + around (see the [ACL how-to](../how-to/acls.md)). +- **Loopback is shared inside the container network.** In the Compose layout, + tail-watch shares the `tailscale` container's network namespace, so every + process in that namespace can reach a loopback listener. "Loopback only" + means "only things on this network stack", which is not always the same as + "only this program". + +When the UI is on, tail-watch logs at startup that the write API is enabled +and the address it is on, and logs a warning when that address is not +loopback. Binding `0.0.0.0` is the case the warning exists for: it exposes the +write API on every interface, including any that face a LAN. + +## What the request guard does cover + +Because there is no login, a browser is the main way an outside party could +trick an authorized person into writing. A web page can make your browser send +requests to an address only your machine can reach, carrying nothing but the +request itself. The write API is guarded against the two common forms of that +attack. Every write goes through the same guard, in this order: + +1. **Content type.** A request that does not declare `application/json` is + refused with 415. A browser can send a cross-site HTML form without any + check, but it can only send form encoded content types. Refusing everything + that is not JSON turns that whole class of attack into an error. +2. **Origin.** A request whose `Origin` header names a host different from its + `Host` header is refused with 403. A cross-origin `fetch` with a JSON body + would need a permission the page cannot obtain, since the server sends no + cross-origin allowances, and the check refuses it in any case. + +Both checks run before the authorization decision and before anything changes. +Two details about the second check are worth knowing. A request with no +`Origin` header is accepted, because command line clients and scripts do not +send one. That is intentional, since the guard targets browsers, and it +means the guard is not access control. The dry-run validation endpoint is +subject to the same two checks but is not audited or authorized, because it +changes nothing. + +The list endpoint that returns the monitors is a read, and it is not guarded by +either check. + +The pages under `/ui/` also send a Content Security Policy that allows only +same-origin resources, and the page loads nothing from other origins. This +limits what an injected script could do, but it is separate from the write +guard. + +## What it does not cover: DNS rebinding + +The Origin check compares the `Origin` header to the `Host` header. It assumes +that a page on a foreign site will announce a foreign host. DNS rebinding +breaks that assumption. An attacker serves a page from a name they control, then +changes the name's DNS answer to the collector's address. The browser now +treats the collector as the same origin as the attacker's page, so `Origin` and +`Host` both name the attacker's hostname, and they match. The request carries +the right content type, and the guard sees nothing wrong. + +tail-watch does not defend against this. The guard cannot tell a rebound name +from a legitimate one without a list of names it should answer to, and it has +no such list, because the correct name depends on how you reach it: a MagicDNS +name, a tailnet IP, a hostname in the Compose network. Requiring a +configured `Host` allowlist would close the gap, at the cost of one more thing +to configure and to break. + +The mitigation is in the network. A rebinding attack needs a browser on your +tailnet, or on the collector host, to visit a hostile page while the collector +is reachable from it. Keep the write API on loopback unless someone needs it, +keep the set of devices with access to the port small, and prefer an admin +browser profile that you do not use for arbitrary sites. If you cannot accept +this risk, leave the UI off, or use it briefly and turn it off again. Editing the +YAML file and restarting remains a complete alternative. + +## The audit log + +Every create, replace and delete is logged at info level, whether it was +accepted or refused. Each entry has the operation, the monitor name, the +outcome and the caller's remote address. Refused requests are logged too, +including those rejected for content type or origin, which makes probing +visible. The caller address is the remote address of the connection, so at best the log +answers "which device did this", never "which person", because there is no +login to attribute it to. A request that arrives through a proxy or a shared +network namespace may show the address of the intermediary instead. + +The audit log is your way to reconstruct what changed. Ship it somewhere that +someone who can edit monitors cannot rewrite. + +## Choosing whether to enable it + +The UI makes sense when few, trusted operators manage monitors frequently and +the tailnet ACL already restricts the port to them. It makes less sense when +monitors change rarely, because the YAML file under version control gives review +and history that the UI does not, and monitors created in the UI live only in +`state_dir/targets.json`. That file is written atomically with owner-only +permissions, but nothing reviews it. + +If you enable it, the useful controls are, in order: keep the listener on +loopback until needed, scope the grant to the smallest set of admins, watch the +startup warning and the audit log, and remember that the guard is a defense +against browsers, not against people who can reach the port. + +## Related pages + +- [Manage monitors in the UI](../how-to/monitor-ui.md) +- [Write the ACLs](../how-to/acls.md) +- [Incidents and queues](incidents-and-queues.md) for what closing an incident + on delete means. diff --git a/docs/how-to/acls.md b/docs/how-to/acls.md new file mode 100644 index 0000000..913783d --- /dev/null +++ b/docs/how-to/acls.md @@ -0,0 +1,69 @@ +# Write the ACLs for tag:monitor + +The collector sees only what your tailnet policy allows. Without a grant, a +target fails with `no peer handles ; check grants to tag:monitor`. + +Prerequisites: + +- Permission to edit the tailnet policy file. +- The collector's node joined with the tag `tag:monitor`. tail-watch logs + `node is missing tag:monitor` at startup otherwise. + +Steps: + +1. Make sure the tag has an owner: + + ```hujson + "tagOwners": { "tag:monitor": ["autogroup:admin"] }, + ``` + +2. Grant `tag:monitor` access to the tagged nodes you probe. Grant the + subnets behind a router separately, and only the ports your checks use: + + ```hujson + { + "tagOwners": { "tag:monitor": ["autogroup:admin"] }, + "grants": [ + { + "src": ["tag:monitor"], + "dst": ["tag:router", "tag:server"], + "ip": ["*"] + }, + { + "src": ["tag:monitor"], + "dst": ["10.20.0.0/16"], + "ip": ["icmp:*", "tcp:22", "tcp:445", "tcp:443", "tcp:5001"] + } + ] + } + ``` + + Replace `tag:router`, `tag:server` and `10.20.0.0/16` with your own tags + and subnets. Each `tcp:` in a target's checks needs a matching + entry, and `icmp:*` covers `icmp` checks. + +3. If a notification receiver is a node on your tailnet, grant the collector + access to it. Sinks use the host's ordinary routing table, so the + collector's node must reach them: + + ```hujson + { "src": ["tag:monitor"], "dst": ["tag:ops-receiver"], "ip": ["tcp:443"] } + ``` + + Add this object to the same `grants` list. + +4. Save the policy. Subnet routes also need to be approved and accepted; see + [Monitor remote subnets and local LANs](remote-subnets.md). + +Verify: + +```bash +./bin/tail-watch --once --config examples/tail-watch.yaml +``` + +Each line ends `ok` for a granted check. A blocked TCP check reports an +`i/o timeout`, because the packets are dropped. If a LAN target has no +handling node, the log has `no peer handles ; check grants to +tag:monitor`. See [Troubleshoot](troubleshoot.md). + +For the UI port and its grant, see [Manage monitors in the UI](monitor-ui.md). diff --git a/docs/how-to/deploy.md b/docs/how-to/deploy.md new file mode 100644 index 0000000..578577c --- /dev/null +++ b/docs/how-to/deploy.md @@ -0,0 +1,135 @@ +# Deploy tail-watch + +Run the collector as a binary, as a Docker container next to the host's +`tailscaled`, or with Docker Compose when you have no dedicated host. + +tail-watch needs a kernel-mode `tailscaled`. It refuses to start against +userspace networking. See [Troubleshoot](troubleshoot.md) if startup fails. + +## Run the binary + +Prerequisites: + +- A host running `tailscaled` in kernel mode, tagged `tag:monitor`. +- The [ACL grants](acls.md) for that tag. +- [mise](https://mise.jdx.dev/) to install the pinned Go toolchain. + +Steps: + +1. Build the binary: + + ```bash + mise install + mise run build + ``` + +2. Validate the config. This does not contact `tailscaled`: + + ```bash + ./bin/tail-watch check-config --config examples/tail-watch.yaml + ``` + +3. Start the collector: + + ```bash + ./bin/tail-watch --config examples/tail-watch.yaml + ``` + + The config path is `--config`, then `TAIL_WATCH_CONFIG`, then + `/etc/tail-watch/tail-watch.yaml`. + +Verify: + +```bash +curl -s http://127.0.0.1:9109/healthz +``` + +The answer is `ok`. Open `http://127.0.0.1:9109/` for the status page. +To run every check once and exit non-zero on failure, use +`./bin/tail-watch --once --config examples/tail-watch.yaml`. + +Flags, environment variables and defaults are in the +[command-line reference](../reference/cli.md). + +## Run in Docker + +The image runs as user `tail-watch` (uid 65532) and does not embed a +Tailscale node. Use host networking so checks share the host routes, and +mount the LocalAPI socket. + +Prerequisites: the same host as above, and a socket that uid 65532 can read. +If it cannot, pass a `--user` or `--group-add` that can. + +Steps: + +1. Build the image: + + ```bash + docker build -t tail-watch . + ``` + +2. Run it: + + ```bash + docker run --rm --network host \ + -v /var/run/tailscale/tailscaled.sock:/var/run/tailscale/tailscaled.sock \ + -v /etc/tail-watch/tail-watch.yaml:/etc/tail-watch/tail-watch.yaml:ro \ + tail-watch + ``` + +Verify: `curl -s http://127.0.0.1:9109/healthz` answers `ok`. If the +container exits, its JSON log on stderr names the reason. For ICMP checks the +container needs `net.ipv4.ping_group_range` to include uid 65532; see +[Troubleshoot](troubleshoot.md). + +If you use `collector.state_dir` (`/var/lib/tail-watch` in the example), mount +a volume there that uid 65532 can write. + +## Run with Docker Compose and no dedicated host + +`examples/compose.yaml` runs a kernel-mode `tailscaled` container and puts +tail-watch in its network namespace. Checks follow that node's `tailscale0` +routes, including accepted subnet routes. + +Prerequisites: + +- Docker with the Compose plugin, and access to `/dev/net/tun`. +- An auth key tagged `tag:monitor`. Create it in the Tailscale admin console. + +Steps: + +1. Put your targets in a config file. The example reads + `examples/tail-watch.yaml`. To use another file, set + `TAIL_WATCH_CONFIG_FILE`. +2. Start the stack: + + ```bash + TS_AUTHKEY=tskey-auth-... docker compose -f examples/compose.yaml up -d + ``` + + tail-watch starts once the `tailscale` service reports healthy. + +Verify: + +```bash +docker compose -f examples/compose.yaml ps +docker compose -f examples/compose.yaml logs tail-watch +``` + +The log shows a `collector identity` line with the tag `tag:monitor`. The +listener is on the loopback address of the shared namespace, so run +`docker compose -f examples/compose.yaml exec tailscale wget -qO- http://127.0.0.1:9109/healthz` +to reach it. + +Things to know: + +- The `tailscale` service sets `net.ipv4.ping_group_range`, so the non-root + tail-watch image can send ICMP without `CAP_NET_RAW`. +- `network: local` targets reach only what the container's Docker network + can reach. To monitor a LAN, use the host deployment instead. See + [Monitor remote subnets and local LANs](remote-subnets.md). +- Do not put secrets in the config file. Sinks read them from `*_env` or + `*_file` keys; see [Send alerts to Slack](slack-sink.md). +- To keep notification state and UI monitors across restarts, mount a volume + at `collector.state_dir`. See + [Manage monitors in the UI](monitor-ui.md). diff --git a/docs/how-to/heartbeat.md b/docs/how-to/heartbeat.md new file mode 100644 index 0000000..3cc9f5c --- /dev/null +++ b/docs/how-to/heartbeat.md @@ -0,0 +1,45 @@ +# Watch the watcher with a heartbeat + +A dead collector cannot report itself. Point `heartbeat` at a dead-man's-switch +service that alerts when the pings stop. + +Prerequisites: an account with a dead-man's-switch or cron-monitoring service +that gives you a ping URL and alerts when it stops receiving pings. + +Steps: + +1. Put the ping URL in an environment variable, or in a file for `url_file`. + Do not write it into the config. +2. Add the block to the config. Set exactly one of `url_env` or `url_file`: + + ```yaml + heartbeat: + url_env: HEARTBEAT_URL + interval: 1m + ``` + +3. Set the interval below the service's grace period. +4. Validate the file, then restart. `check-config` does not read the URL + variable, but startup fails if it is unset: + + ```bash + ./bin/tail-watch check-config --config /etc/tail-watch/tail-watch.yaml + ``` + +Verify: the service shows pings arriving each `interval`. tail-watch sends a +`GET` starting once every target and check has a first result, with a 10s +timeout. When the process exits the pings stop and the service alerts. + +A failed ping is logged, counted in `tailwatch_heartbeat_failures_total`, and +not retried before the next interval. Alert on that counter; see the +[metrics reference](../reference/metrics.md). + +Exit-node egress warning: the heartbeat leaves through the host's ordinary +routing table, like the notification sinks. tail-watch warns at startup with +`exit node is active; notifications will route through the exit node` when +an exit node is set, and the pings would then come from the exit node's +address. Clear the exit node on the collector or make sure the service does not +filter by source address. + +`tail-watch --once` never sends a heartbeat. If the URL variable is unset, +startup fails; see [Troubleshoot](troubleshoot.md). diff --git a/docs/how-to/monitor-ui.md b/docs/how-to/monitor-ui.md new file mode 100644 index 0000000..6666aab --- /dev/null +++ b/docs/how-to/monitor-ui.md @@ -0,0 +1,119 @@ +# Manage monitors in the UI + +The monitor UI adds, edits and deletes monitors while tail-watch runs, with +no restart and no lost grace timers. It is off by default. + +Read [Monitor UI risks](../explanation/monitor-ui-risks.md) first: there is +no login, and whoever can reach the port can make the collector probe any +address it can route to. + +## Turn the UI on + +Prerequisites: a writable `collector.state_dir`. Monitors created in the UI +are stored in `state_dir/targets.json`. + +Steps: + +1. Set `collector.state_dir` in the config: + + ```yaml + collector: + listen: 127.0.0.1:9109 + state_dir: /var/lib/tail-watch + ui: true + targets: + - name: nyc-subnet-router + addr: nyc-router + checks: [tsping] + ``` + +2. Enable the UI with `collector.ui: true` (above) or the flag: + + ```bash + tail-watch --config /etc/tail-watch/tail-watch.yaml --ui + ``` + + The flag wins over the file, so `--ui=false` turns off a config that + enables it. Under `--once` the UI is ignored. +3. Restart tail-watch. + +Verify: the log has `monitor write API enabled`, and +`http://127.0.0.1:9109/ui/` shows the monitor list. If tail-watch exits with +`the UI needs collector.state_dir to store monitors`, set `state_dir`. + +## Reach the UI from another device + +The default `listen` address, `127.0.0.1:9109`, is reachable only from the +host. To open the UI from another device on the tailnet, bind an address that +device can reach and grant it access. + +Steps: + +1. Bind the node's tailnet IP with `--listen` or `collector.listen`, for + example `--listen 100.101.102.103:9109`. tail-watch then warns + `monitor write API is reachable beyond loopback; access is controlled only + by the network`. +2. Grant the device access to the port: + + ```hujson + { + "grants": [ + { + "src": ["autogroup:admin"], + "dst": ["tag:monitor"], + "ip": ["tcp:9109"] + } + ] + } + ``` + + A grant covers the whole port, so it also exposes `/metrics`. +3. Open `http://:9109/ui/` from that device. + +In the Compose layout tail-watch shares the `tailscale` container's network +namespace, so every process in that namespace can reach a loopback listener. + +## Add, edit and delete a monitor + +Steps: + +1. Click **Add monitor**. Fill in name, address, checks and the other fields. + The `defaults` block of the YAML file applies to UI monitors. +2. Click **Check** to validate without saving, then **Save**. Errors appear + next to the field they belong to, and nothing is created until they are + fixed. They come from the same rules the YAML loader uses. + + ![The add monitor form with an invalid TCP port reported under Checks.](../images/monitor-ui-form-error.png) + +3. The list shows every monitor. YAML monitors are read-only; only monitors + created in the UI have **Edit** and **Delete**. Monitors behind a down + parent show `SUPPRESSED`. + + ![The monitor list: YAML monitors are read-only, UI monitors have Edit and Delete, and monitors behind a down parent show SUPPRESSED.](../images/monitor-ui-list.png) + +4. To delete, click **Delete** and confirm. Deleting a `DOWN` monitor asks + first, and says its open incident closes. + + ![The delete confirmation for a down monitor.](../images/monitor-ui-delete-down.png) + +Behavior to expect: + +- An edit is a delete followed by an add. Editing or deleting a `DOWN` + monitor closes its open incident with `reason: removed`. A new incident + opens only if the new definition fails `fail_threshold` checks. +- A monitor that other monitors name as `parent` cannot be deleted until + they are changed. The API answers with `"" is the parent of [...]; + delete or change those first`. +- Editing a parent briefly releases its suppressed dependents, which can page + while the parent is down. +- A monitor whose name matches a YAML target is skipped at startup with a + warning, and the YAML target wins. +- `targets.json` uses the YAML field names, is written atomically, and has + mode 0600. +- With the UI off, `targets.json` is left alone and its monitors do not run. + A startup warning, `dynamic monitors are inactive: the UI is off`, gives + the count. + +Every write is logged with the caller's address as a `monitor write` line +with `op`, `target`, `status`, `outcome` and `remote`. To find writes, filter +the JSON log on that message. diff --git a/docs/how-to/remote-subnets.md b/docs/how-to/remote-subnets.md new file mode 100644 index 0000000..ac60e5b --- /dev/null +++ b/docs/how-to/remote-subnets.md @@ -0,0 +1,92 @@ +# Monitor remote subnets and local LANs + +Each target has a `network` setting that says which path its checks should +take. Pick the value that matches where the device is. + +## Monitor a subnet behind a Tailscale router + +Use `network: tailnet` (the default) for devices reached through a subnet +router. + +Prerequisites: + +- A subnet router advertising the route, with the route approved in the + admin console. +- [Grants](acls.md) from `tag:monitor` to the subnet. + +Steps: + +1. On the collector host, accept subnet routes. On Linux, startup fails + until you do, with `remote subnet targets require accepted routes; run + tailscale set --accept-routes`: + + ```bash + tailscale set --accept-routes + ``` + +2. Check the route the host will actually use. An advertised route is not an + installed route: + + ```bash + ip route get 10.20.0.1 + ``` + + The answer must go out `tailscale0`, not a local gateway. Repeat this + for each remote subnet you monitor. + +3. Add the targets. Point a `tsping` target at the router, and name it as the + `parent` of the devices behind it: + + ```yaml + targets: + - name: nyc-subnet-router + addr: nyc-router + site: nyc + network: tailnet + checks: [tsping] + + - name: nyc-core-switch + addr: 10.20.0.1 + site: nyc + network: tailnet + parent: nyc-subnet-router + checks: [icmp] + ``` + +Verify: `./bin/tail-watch check-config --config ` accepts the file, and +`./bin/tail-watch --once --config ` prints `ok` for each check. + +## Monitor a LAN the collector is plugged into + +Use `network: local` for a device on the collector's own LAN. These checks +use the host's ordinary route and keep working when that site's Tailscale +router is down. + +Steps: + +1. Give the device its own target with `network: local`: + + ```yaml + targets: + - name: lab-printer + addr: 192.168.1.50 + site: lab + network: local + severity: warn + checks: + - icmp + - tcp:9100 + - "http:http://192.168.1.50/|ready" + ``` + +2. Keep the HTTP check on the same host as `addr`. +3. Do not add `tsping`. It is rejected for `network: local`. + +Verify: `./bin/tail-watch --once --config ` shows the checks passing. +A failure on a local target is labeled `service` or `lan`, never `control` +or `tunnel`. The layers are explained in +[Failure layers](../explanation/failure-layers.md). + +`examples/tail-watch.yaml` has both kinds of target. The check syntax is in +the [checks reference](../reference/checks.md), and the target fields are in +the [configuration reference](../reference/config.md). diff --git a/docs/how-to/slack-sink.md b/docs/how-to/slack-sink.md new file mode 100644 index 0000000..f345060 --- /dev/null +++ b/docs/how-to/slack-sink.md @@ -0,0 +1,89 @@ +# Send alerts to Slack + +Post alert transitions to a Slack channel through an incoming webhook. + +Prerequisites: + +- A Slack workspace where you can create an app. +- A collector that can reach `hooks.slack.com` (see the egress note in + [Send alerts to a webhook](webhook-sink.md)). + +## Get an incoming webhook URL + +tail-watch posts to an incoming webhook URL. It does not use bot or app-level +tokens (`xoxb-`, `xapp-`), and a bot token cannot produce a webhook URL. +Slack's guide, +[Sending messages using incoming webhooks](https://docs.slack.dev/messaging/sending-messages-using-incoming-webhooks/), +is the reference. + +1. Create an app at , or open an + existing one. +2. In the app's settings sidebar choose **Incoming Webhooks** and switch + **Activate Incoming Webhooks** on. +3. Click **Add New Webhook to Workspace**, pick the channel, and click + **Allow**. +4. Copy the URL from **Webhook URLs for Your Workspace**. It starts with + `https://hooks.slack.com/services/`. + +If the sidebar has no **Incoming Webhooks** entry, Slack's `incoming-webhook` +scope produces a webhook during an OAuth install. Slack changes these pages +over time; if the steps do not match, follow Slack's guide. + +Check the URL. Slack answers `ok`: + +```bash +curl -X POST -H 'Content-type: application/json' \ + --data '{"text":"hello"}' "$SLACK_WEBHOOK_URL" +``` + +Store the URL as a secret in your environment, secret manager or a file +outside the config. Never put it in the YAML. + +## Add the sink + +1. Add a sink to the config. `url_env` names the environment variable that + holds the URL; use `url_file` for a file instead: + + ```yaml + sinks: + - name: ops-slack + type: slack + url_env: SLACK_WEBHOOK_URL + min_severity: warn + mention_on_page: "" + status_url: https://monitor-nyc.example.ts.net:9109 + group_wait: 10s + queue: disk + match: + sites: [nyc, lab] + ``` + + - `mention_on_page` is prepended only to a `target.down` of severity + `page`, never to recoveries, flapping notices or warnings. + - `status_url` turns each target line into a link to + `/#`. + - `group_wait` batches events that arrive together into one message. + - `queue: disk` needs `collector.state_dir`. With the default + `queue: memory`, an event still waiting in the group window is lost when + tail-watch restarts. Use `disk` if an alert must survive a restart. + - A Slack sink rejects `batch`, `max_batch`, `secret_*` and `headers_env`. +2. Validate the file. `check-config` prints each sink's name, type, filters + and queue mode, never a secret. It does not read the secret variables: + + ```bash + ./bin/tail-watch check-config --config /etc/tail-watch/tail-watch.yaml + ``` + +3. Send a test message: + + ```bash + ./bin/tail-watch notify-test --sink ops-slack --timeout 10s + ``` + +Verify: `notify-test` prints `ops-slack slack 200` and exits 0, and the message +appears in the channel. A missing variable prints +`environment variable SLACK_WEBHOOK_URL is unset or empty`; see +[Troubleshoot](troubleshoot.md). Run `notify-test` after every config change. + +For the message fields see the [payload reference](../reference/payload.md). +For sink options see the [configuration reference](../reference/config.md). diff --git a/docs/how-to/troubleshoot.md b/docs/how-to/troubleshoot.md new file mode 100644 index 0000000..e6351ac --- /dev/null +++ b/docs/how-to/troubleshoot.md @@ -0,0 +1,40 @@ +# Troubleshoot + +Find the message tail-watch printed, then apply the fix. Startup failures +print to stderr and exit 1; log lines are JSON on stderr. + +## Startup failures and warnings + +| Symptom | Cause | Fix | +| --- | --- | --- | +| `tailscaled is using userspace networking; tail-watch requires kernel-mode tailscaled` | `tailscaled` runs with `--tun=userspace-networking`, which cannot carry ICMP to LAN addresses | Run `tailscaled` in kernel mode. In Compose set `TS_USERSPACE: "false"` as `examples/compose.yaml` does. See [Deploy](deploy.md) | +| `node is missing tag:monitor` (a warning) | The collector node does not carry the tag your grants are written for | Join with an auth key tagged `tag:monitor`, or re-tag the node. See [ACLs](acls.md) | +| `cannot open an ICMP socket: set net.ipv4.ping_group_range or grant CAP_NET_RAW` | The process cannot open an ICMP socket. TCP, HTTP and `tsping` checks still run | Allow unprivileged ping: `sysctl -w net.ipv4.ping_group_range="0 2147483647"` (or a range that includes the collector's group), or grant `CAP_NET_RAW`. In Docker, set the sysctl on the container as the example Compose file does | +| `tailscaled is not reachable at : ...` | The LocalAPI socket is missing, wrong, or unreadable by the collector's user. On Linux the default is `/var/run/tailscale/tailscaled.sock` | Pass `--socket `, check `tailscaled` is running, and make the socket readable by the collector's user. In Docker mount the socket and use a `--user` or `--group-add` that can read it (the image runs as uid 65532) | +| `tailscaled state is ; run tailscale up` | `tailscaled` is running but not logged in | Run `tailscale up` | +| `remote subnet targets require accepted routes; run tailscale set --accept-routes` | A `network: tailnet` target has a literal non-Tailscale IP and the host does not accept subnet routes | Run `tailscale set --accept-routes`, then check `ip route get `. See [Remote subnets](remote-subnets.md) | +| `exit node is active; notifications will route through the exit node` (a warning) | An exit node is set on the collector, so sinks and the heartbeat egress through it | Clear the exit node, or make sure receivers accept that source. See [Webhook](webhook-sink.md) | +| `sink : url: environment variable is unset or empty` (also `secret:`, `header :`, `heartbeat:`) | A `*_env` variable is unset or empty at startup. Secret files report `secret file is empty` | Export the variable in the service's environment. Compose and systemd do not inherit your shell. Run `notify-test` to confirm. See [Slack](slack-sink.md) and [Heartbeat](heartbeat.md) | +| `notification state is corrupt (...); repair or move the file aside and restart, but note open incidents are forgotten if it is removed` | `notification-state.json` in `collector.state_dir` cannot be decoded, so startup stops rather than forget open incidents | Restore or repair the file from a backup. If you move it aside, open incidents are forgotten and their recoveries are not sent. See [Incidents and queues](../explanation/incidents-and-queues.md) | +| `the UI needs collector.state_dir to store monitors` | `--ui` or `collector.ui: true` is set with no `state_dir` | Set `collector.state_dir`, or turn the UI off with `--ui=false`. See [Monitor UI](monitor-ui.md) | +| `queue: disk requires collector.state_dir` | A sink uses the disk queue with no `state_dir` | Set `collector.state_dir` or use `queue: memory` | +| `dynamic monitors are inactive: the UI is off` (a warning) | `targets.json` has monitors but the UI is off, so they do not run | Turn the UI on, or ignore it | +| `webhook sink is unsigned: no secret_env or secret_file` (a warning) | A webhook sink has no HMAC key | Add `secret_env`. See [Webhook](webhook-sink.md) | + +## Check failures + +| Symptom | Cause | Fix | +| --- | --- | --- | +| `no peer handles ; check grants to tag:monitor` | No peer owns or routes the address for this collector: a missing grant, an unapproved or unaccepted route, or the wrong address | Add the grant from `tag:monitor` and approve the route. See [ACLs](acls.md) and [Remote subnets](remote-subnets.md). tail-watch logs it once per target, again only if the grant returns and is lost | +| TCP check ends with `i/o timeout` | The grant does not allow that port, so packets are dropped | Add `tcp:` to the grant | + +## Validate before you restart + +- `tail-watch check-config --config ` validates the file without + contacting `tailscaled`. +- `tail-watch notify-test` sends one test event per sink and reports the + HTTP status or the error. +- `tail-watch --once --config ` runs every check once. + +Flags are in the [command-line reference](../reference/cli.md). To see every +check result in the log, use `--log-level debug`. diff --git a/docs/how-to/webhook-sink.md b/docs/how-to/webhook-sink.md new file mode 100644 index 0000000..59d8c67 --- /dev/null +++ b/docs/how-to/webhook-sink.md @@ -0,0 +1,137 @@ +# Send alerts to a webhook + +Deliver each alert event as a signed JSON `POST` to your own receiver. + +Prerequisites: + +- A receiver that accepts `POST` with `Content-Type: application/json` and + answers `2xx`. +- The receiver must deduplicate on the event `id`, because delivery is at + least once. See the [payload reference](../reference/payload.md). + +## Add the sink + +1. Add a sink. Secrets come only from environment variables or files: + + ```yaml + sinks: + - name: ops-webhook + type: webhook + url_env: OPS_WEBHOOK_URL + secret_env: OPS_WEBHOOK_SECRET + headers_env: + Authorization: OPS_WEBHOOK_AUTH + min_severity: page + events: [target.down, target.recovered] + max_age: 1h + queue: disk + queue_size: 1000 + batch: true + group_wait: 10s + max_batch: 100 + ``` + + - `secret_env` (or `secret_file`) is the HMAC key. Without it requests are + unsigned, and tail-watch logs `webhook sink is unsigned: no secret_env + or secret_file`. + - `headers_env` maps a header name to the environment variable holding its + value. + - `queue: disk` needs `collector.state_dir`. + - `batch: true` waits `group_wait` after the first queued event, or until + `max_batch` events are waiting, and sends one request. +2. Validate the file. `check-config` does not read the secret variables: + + ```bash + ./bin/tail-watch check-config --config /etc/tail-watch/tail-watch.yaml + ``` + + Export the variables in the collector's environment before you run + `notify-test` or start tail-watch. + +3. Send a test event through the real signing and retry code: + + ```bash + ./bin/tail-watch notify-test --sink ops-webhook + ``` + +Verify: `notify-test` prints `ops-webhook webhook 200` and exits 0 when the +receiver answers `2xx` within `--timeout` (default 30s). It bypasses filters, +queues and `state_dir`, and works when `tailscaled` is down. + +## What the receiver sees + +Each event is one `POST` with these headers: + +- `Content-Type: application/json` +- `User-Agent: tail-watch/` +- `Tail-Watch-Event-Id`, or `Tail-Watch-Batch-Id` for a batched sink +- `Tail-Watch-Signature: t=,v1=` when a secret is set +- any `headers_env` headers + +Redirects are not followed and each request times out after 10s. A batched +request has the body `{"version": 1, "batch_id": "bat_...", "events": [...]}`. + +| Response | What tail-watch does | +| --- | --- | +| `2xx` | delivered | +| `408`, `5xx`, timeout, network error | retry with growing delay | +| `429` | retry, waiting at least `Retry-After` | +| anything else, including `3xx` and other `4xx` | drop, log once, count `permanent` | + +The delay is 1s, 2s, 4s and so on with 0.5 to 1.5 times jitter, never more +than 5 minutes. An event older than `max_age` when an attempt is due is +dropped and counted as `expired`. A retry sends identical body bytes with a +fresh `t`. + +## Verify the signature in the receiver + +`v1` is the lowercase hex HMAC-SHA256 of `.` under the secret. The +signature covers the whole body, including a batch. Reject stale +timestamps: + +```go +func verify(secret, header string, body []byte, tolerance time.Duration) bool { + var ts, sig string + for _, part := range strings.Split(header, ",") { + k, v, _ := strings.Cut(part, "=") + switch k { + case "t": + ts = v + case "v1": + sig = v + } + } + secs, err := strconv.ParseInt(ts, 10, 64) + if err != nil || time.Since(time.Unix(secs, 0)).Abs() > tolerance { + return false + } + mac := hmac.New(sha256.New, []byte(secret)) + fmt.Fprintf(mac, "%s.%s", ts, body) + return hmac.Equal([]byte(hex.EncodeToString(mac.Sum(nil))), []byte(sig)) +} +``` + +To check your implementation, the repository's test vector uses the secret +`whsec_test_key`, the timestamp `1700000000` and the body +`{"id":"evt_X","type":"test"}`. The header must be: + +```text +t=1700000000,v1=1419cf10ea0dc484e3a06e7df47e222718af9df5f1bb5180b9c3bb9cde19722f +``` + +You can reproduce the digest with: + +```bash +printf '%s' '1700000000.{"id":"evt_X","type":"test"}' \ + | openssl dgst -sha256 -hmac whsec_test_key +``` + +## Make sure the collector can reach the receiver + +Sinks use the host's ordinary routing table. tail-watch warns at startup with +`exit node is active; notifications will route through the exit node` when an +exit node is set. A receiver on your own tailnet needs a grant from +`tag:monitor`; see [Write the ACLs](acls.md). + +For how queues, retries and drops behave, see +[Incidents and queues](../explanation/incidents-and-queues.md). diff --git a/docs/reference/checks.md b/docs/reference/checks.md new file mode 100644 index 0000000..367cf6f --- /dev/null +++ b/docs/reference/checks.md @@ -0,0 +1,125 @@ +# Checks reference + +A check is one string in a target's `checks` list. Each check runs on its target's +`interval` and is cancelled at its target's `timeout`. A check that times out counts +as a failure. First runs are spread across the interval so checks do not start in a +burst. + +| Check | Syntax | Passes when | +| --- | --- | --- | +| ICMP echo | `icmp` | One echo reply arrives before the timeout. | +| TCP connect | `tcp:` | A TCP connection to `addr:` is established. | +| HTTP request | `http:` or `http:\|` | The status is 2xx or 3xx and, if `` is given, the body contains it. | +| Tailscale ping | `tsping` | tailscaled reports a disco pong. | +| TLS | `tls:...` | Rejected. Not supported in this version. | + +The same check string cannot appear twice in one target. Every check that fails +gets a failure layer; see [Failure layers](../explanation/failure-layers.md). + +## Path and network + +Checks use host sockets and do not bind to an interface. The target's `network` +declares the intended path: + +- `tailnet` (default): IP checks should use Tailscale. On Linux, a target whose + `addr` is a literal non-Tailscale IP is a remote subnet and needs accepted + routes at startup. +- `local`: the host's ordinary route. `tsping` is rejected on a `local` target. + +See [Monitor remote subnets and local LANs](../how-to/remote-subnets.md). + +## `icmp` + +Sends one echo request to the target's address, IPv4 or IPv6. A DNS name is +resolved first and an IPv4 address is preferred. It uses an unprivileged ICMP +socket when the host allows one and otherwise a raw socket. + +| Result | Detail | +| --- | --- | +| Pass | Latency is the round-trip time. | +| Fail | `timeout: ...` when no reply arrives. | +| Fail | `cannot open an ICMP socket: set net.ipv4.ping_group_range or grant CAP_NET_RAW` when neither socket type can be opened. | + +## `tcp:` + +`` is an integer from 1 to 65535. The check opens a connection to +`addr:` and closes it immediately without sending data. Latency is the +connect time. A refused connection fails with an error containing +`connection refused`. + +## `http:` + +```yaml +checks: + - "http:https://10.20.0.50:5001/" + - "http:https://10.20.0.50:5001/|DSM" +``` + +The URL must use `http` or `https` and include a host. The host must equal the +target's `addr`, ignoring case, a trailing dot, and equivalent IP forms. To check +a different host, define a separate target. Quote the string in YAML because it +contains a colon. + +Behavior: + +- Sends `GET`. A response of 200 through 399 passes the status test. Any other + status fails with `HTTP status `. +- Redirects are not followed. A `302` counts as up. +- The text after the first `|` is a literal, case-sensitive substring that must + appear in the first 1 MiB of the body. If it is missing, the check fails with + `body missing ""; first 300 bytes: ""`. The URL part therefore + cannot contain `|`. +- Opens a new connection for every run, ignores `HTTP_PROXY` and related + environment variables, and verifies TLS certificates. +- With `insecure_tls: true` on the target, certificate verification is skipped for + every `http:` check on that target. + +## `tsping` + +Resolves `addr` to a Tailscale or subnet-routed IP and sends a disco ping through +the local tailscaled. It passes when tailscaled reports a pong before the timeout, +and records the path kind, the endpoint or DERP region, and the latency. + +Name resolution for a non-IP `addr` prefers, in order: the peer's full MagicDNS +name, its short MagicDNS name, its hostname. Matching ignores case. If several +peers tie, the first by name is used and a warning is logged. If no peer matches, +the name is resolved with DNS. + +| Path kind | Meaning | +| --- | --- | +| `direct` | Direct UDP path. The endpoint is recorded. | +| `derp` | Relayed through a DERP region. The region code is recorded. | +| `peer_relay` | Relayed through a peer relay. | + +Failures: + +| Error | Cause | +| --- | --- | +| `no peer handles ; check grants to tag:monitor` | No peer in the collector's netmap routes the address, usually a missing grant. See [Write the ACLs](../how-to/acls.md). | +| `tsping returned no result` | tailscaled answered without a result. | +| other text | The error reported by tailscaled. | + +## `tls:` (rejected) + +Any check starting with `tls:` is rejected at load time with +`tls checks are not supported in this version`. + +## Parent rules + +`parent` is for suppression only. It does not change which node handles the +address. A parent must: + +- name another target in the same config; +- have exactly one check, and that check must be `icmp` or `tsping`; +- not create a cycle. + +An explicit `parent` is allowed on a `local` target. See +[Alert state](../explanation/alert-state.md) for what a down parent does to its +dependents. + +## Result record + +Every run produces a result with the target name, the check string (for example +`icmp` or `tcp:445`), the start time, the latency, the error if any, and for +`tsping` the path details. Under `--once`, each result is printed as +` ok ` or ` fail `. diff --git a/docs/reference/cli.md b/docs/reference/cli.md new file mode 100644 index 0000000..98f82f4 --- /dev/null +++ b/docs/reference/cli.md @@ -0,0 +1,161 @@ +# Command line reference + +```text +tail-watch [flags] +tail-watch check-config [--config ] +tail-watch notify-test [flags] +``` + +Without a subcommand, tail-watch runs the collector. + +## Collector flags + +| Flag | Type | Default | Meaning | +| --- | --- | --- | --- | +| `-config` | string | see below | Config file path. Overrides `TAIL_WATCH_CONFIG`. | +| `-listen` | string | `collector.listen`, else `127.0.0.1:9109` | HTTP listen address. | +| `-socket` | string | platform default | tailscaled LocalAPI socket. | +| `-log-level` | string | `info` | `debug`, `info`, `warn`, or `error`. Overrides `TAIL_WATCH_LOG_LEVEL`. | +| `-once` | bool | `false` | Run every check once, print one line each, and exit. | +| `-ui` | bool | `collector.ui`, else `false` | Serve the monitor UI and API. `--ui=false` turns off a config that enables it. | +| `-version` | bool | `false` | Print the version and exit. | + +Go's flag syntax applies: `-flag` and `--flag` are equivalent, and a value is +given as `--flag value` or `--flag=value`. `-h` and `--help` print usage and +exit 0. + +The default socket is `/var/run/tailscale/tailscaled.sock` on Linux. + +### Precedence + +| Setting | Order, first match wins | +| --- | --- | +| Config path | `--config`, `TAIL_WATCH_CONFIG`, `/etc/tail-watch/tail-watch.yaml` | +| Listen address | `--listen`, `collector.listen`, `127.0.0.1:9109` | +| UI | `--ui` or `--ui=false`, `collector.ui`, off | +| Log level | `--log-level`, `TAIL_WATCH_LOG_LEVEL`, `info` | + +### Environment variables + +| Variable | Meaning | +| --- | --- | +| `TAIL_WATCH_CONFIG` | Config file path when `--config` is not given. | +| `TAIL_WATCH_LOG_LEVEL` | Log level when `--log-level` is not given. | + +Log levels are case-insensitive. `warning` is accepted as `warn`. Logs are JSON, +one object per line, on stderr. At `debug`, every check result is logged. + +### `--once` + +Runs the preflight, runs every check of every configured target once, prints one +line per result, and exits. It starts no HTTP server, builds no notifier, resolves +no secrets, sends no heartbeat, and ignores the UI. + +```text + ok + fail +``` + +### Startup checks + +Before serving, the collector fails with exit code 1 when: + +- the config file cannot be read or is invalid; +- the UI is on and `collector.state_dir` is unset; +- tailscaled is unreachable, or its state is not `Running`; +- tailscaled uses userspace networking (kernel mode is required); +- a target is a remote subnet address on Linux and the host has not accepted + routes (`tailscale set --accept-routes`); +- a sink or heartbeat secret cannot be read or is empty (server mode only); +- the notification state file cannot be decoded (server mode only). + +It logs a warning, and continues, when the node lacks `tag:monitor`, when an exit +node is active, when a webhook sink has no secret, and when the UI is reachable +beyond loopback. + +## `check-config` + +```bash +tail-watch check-config [--config ] +``` + +Loads and validates the config. It does not contact tailscaled and does not +resolve secrets, so it needs no environment variables. On success it prints one +line per target, one per sink, and one for the heartbeat if configured, then +exits 0. + +```text +nyc-router interval=15s timeout=5s network=tailnet checks=tsping +nyc-nas interval=30s timeout=5s network=tailnet checks=icmp,tcp:445,http:https://10.20.0.50:5001/|DSM +sink ops-webhook type=webhook min_severity=page events=target.down,target.recovered queue=disk match=sites:nyc targets:nyc-* tags:tag:router +sink ops-slack type=slack min_severity=warn events=all queue=memory +heartbeat interval=1m0s +``` + +On failure it prints every problem to stderr and exits 1. The only flag is +`-config`. + +## `notify-test` + +```bash +tail-watch notify-test [--config ] [--sink ] [--timeout ] [--socket ] +``` + +Sends one `test` event through each selected sink's real signing, rendering, and +retry code. It bypasses filters, queues, and `state_dir`. It needs the secrets +that the sinks reference, and it works when tailscaled is unreachable, in which +case the collector name in the payload is the OS hostname and `tailscale_ip` is +omitted. + +| Flag | Type | Default | Meaning | +| --- | --- | --- | --- | +| `-config` | string | see [Precedence](#precedence) | Config file path. | +| `-sink` | string | every sink | Send only to the sink with this name. | +| `-timeout` | duration | `30s` | Give up on a sink after this long, including retries. | +| `-socket` | string | platform default | tailscaled LocalAPI socket, used to read the collector identity. | + +Sinks are tried in parallel. One line is printed per sink, in config order: + +| Line | Meaning | +| --- | --- | +| ` ` | Delivered. `` is the 2xx response code. | +| ` ` | Failed. A permanent rejection such as `404` shows the status code. | +| ` error: timed out after retries (last: )` | Timeout reached while retrying. | +| ` error: ` | A secret could not be read, or the request could not be built. | + +Retryable failures (`408`, `429`, `5xx`, network errors) are retried with backoff +until `--timeout`. See [Event payload](payload.md) for the request format. + +## Exit codes + +| Code | Meaning | +| --- | --- | +| `0` | Success. For the collector: clean shutdown, or `--once` with every check passing. For `notify-test`: every selected sink answered 2xx. `--help` also exits 0. | +| `1` | Failure. Config or startup error, HTTP server error, a check failed under `--once`, or a sink failed under `notify-test`. | +| `2` | Usage error. Unknown flag, invalid `--log-level` or `TAIL_WATCH_LOG_LEVEL`, or `notify-test` with no sinks configured or an unknown `--sink`. | + +## HTTP endpoints + +Served on the listen address by the collector, not under `--once`. + +| Path | Response | +| --- | --- | +| `/` | Prober status page: each check's recent results, latency, and state. | +| `/metrics` | Prometheus text exposition. See [Metrics](metrics.md). | +| `/healthz` | `200` with body `ok`. Says the process is serving, not that targets are up. | +| `/ui/` | Monitor UI. `404` unless the UI is on. | +| `/api/v1/targets` | Monitor API: `GET` list, `POST` create. `404` unless the UI is on. | +| `/api/v1/targets/{name}` | Monitor API: `PUT` replace, `DELETE` remove. | +| `/api/v1/validate` | Monitor API: `POST` validate a monitor without saving it. | + +Write requests to `/api/v1/` must be `application/json` and must not carry an +`Origin` for another host. There is no authentication. See +[Manage monitors in the UI](../how-to/monitor-ui.md) and +[Monitor UI risks](../explanation/monitor-ui-risks.md). + +## Signals + +`SIGINT` and `SIGTERM` stop check scheduling and cancel running checks. The +collector then stops the notifier and shuts down the HTTP server, waiting up to the +longest target timeout plus 5 seconds. It exits 0. Under `--once`, a signal stops +the run and exits 0. diff --git a/docs/reference/config.md b/docs/reference/config.md new file mode 100644 index 0000000..8f75253 --- /dev/null +++ b/docs/reference/config.md @@ -0,0 +1,292 @@ +# Configuration reference + +tail-watch reads one YAML file. This page lists every key, its type, its default, +and the rules the loader enforces. Validate a file without contacting tailscaled: + +```bash +tail-watch check-config --config /etc/tail-watch/tail-watch.yaml +``` + +See [Command line](cli.md) for how the file path is chosen and [Checks](checks.md) +for the `checks` syntax. + +## File rules + +- The file must not be empty. +- Unknown keys are rejected at every level. +- Durations use Go syntax with a unit, such as `500ms`, `30s`, `15m`, `1h`. A + bare number is invalid. +- The loader reports every problem it finds in one pass, each prefixed with the + key path, such as `targets[2].checks[1]`. +- Secret values are never written in the file. Keys ending in `_env` name an + environment variable, and keys ending in `_file` name a file. + +## Top-level keys + +| Key | Type | Required | Default | +| --- | --- | --- | --- | +| `collector` | mapping | no | see below | +| `defaults` | mapping | no | see below | +| `targets` | list of targets | no | empty | +| `sinks` | list of sinks | no | empty | +| `heartbeat` | mapping | no | off | + +## Complete example + +Every key, set to its default where it has one. This file passes +`tail-watch check-config`. + +```yaml +collector: + listen: 127.0.0.1:9109 + state_dir: /var/lib/tail-watch + ui: false + +defaults: + interval: 30s + timeout: 5s + fail_threshold: 3 + recover_threshold: 2 + flap_limit: 4 + flap_window: 15m + +targets: + - name: nyc-router + addr: nyc-router + site: nyc + severity: page + network: tailnet + interval: 15s + timeout: 5s + fail_threshold: 3 + recover_threshold: 2 + flap_limit: 4 + flap_window: 15m + checks: [tsping] + + - name: nyc-nas + addr: 10.20.0.50 + site: nyc + severity: warn + parent: nyc-router + insecure_tls: true + checks: + - icmp + - tcp:445 + - "http:https://10.20.0.50:5001/|DSM" + + - name: lab-printer + addr: 192.168.1.50 + site: lab + network: local + checks: [icmp, tcp:9100] + +sinks: + - name: ops-webhook + type: webhook + url_env: OPS_WEBHOOK_URL + secret_env: OPS_WEBHOOK_SECRET + headers_env: + Authorization: OPS_WEBHOOK_AUTH + min_severity: page + events: [target.down, target.recovered] + match: + sites: [nyc] + targets: ["nyc-*"] + tags: [tag:router] + max_age: 1h + queue: disk + queue_size: 1000 + batch: true + group_wait: 10s + max_batch: 100 + + - name: ops-slack + type: slack + url_env: SLACK_WEBHOOK_URL + min_severity: warn + mention_on_page: "" + status_url: https://monitor-nyc.example.ts.net:9109 + group_wait: 10s + max_age: 1h + queue: memory + queue_size: 1000 + +heartbeat: + url_env: HEARTBEAT_URL + interval: 1m +``` + +## `collector` + +| Key | Type | Required | Default | Meaning | +| --- | --- | --- | --- | --- | +| `listen` | string | no | `127.0.0.1:9109` | HTTP listen address, `host:port`. `--listen` overrides it. | +| `state_dir` | string | no | unset | Directory for durable state. Created with mode `0700` when missing. | +| `ui` | bool | no | `false` | Serve the monitor UI and API. `--ui` overrides it, and `--ui=false` turns it off. | + +`state_dir` holds: + +- `notification-state.json`: open incidents, per-sink delivery state, and the + pending events of `queue: disk` sinks. Mode `0600`. Written whenever + `state_dir` is set and at least one sink exists. +- `targets.json`: monitors created in the UI. Mode `0600`. Read only when the UI + is on. + +`state_dir` is required when `ui` is on (startup exits with an error otherwise) +and when any sink sets `queue: disk`. `--once` ignores `ui`. + +## `defaults` + +Each key is the fallback for the same key on every target, including monitors +created in the UI. + +| Key | Type | Default | Constraint | +| --- | --- | --- | --- | +| `interval` | duration | `30s` | greater than 0 | +| `timeout` | duration | `5s` | greater than 0 and less than `interval` | +| `fail_threshold` | integer | `3` | at least 1 | +| `recover_threshold` | integer | `2` | at least 1 | +| `flap_limit` | integer | `4` | at least 1 | +| `flap_window` | duration | `15m` | greater than 0 | + +Constraints are checked on each target after defaults and overrides are merged. + +## `targets[]` + +| Key | Type | Required | Default | Meaning | +| --- | --- | --- | --- | --- | +| `name` | string | yes | | Unique identifier. Must match `[A-Za-z0-9._-]+`. Appears as the `target` metric label and in payloads. | +| `addr` | string | yes | | IP address or DNS name, including a MagicDNS short name. | +| `checks` | list of strings | yes | | Non-empty, no duplicates. See [Checks](checks.md). | +| `site` | string | no | empty | Free-form group label. Used for the `site` metric label and sink `match.sites`. | +| `severity` | string | no | `page` | `page` or `warn`. Severity of the `target.down` event. | +| `network` | string | no | `tailnet` | `tailnet` or `local`. | +| `parent` | string | no | empty | Name of another target whose only check is `icmp` or `tsping`. | +| `insecure_tls` | bool | no | `false` | Skip certificate verification for `http:` checks on this target. | +| `interval` | duration | no | `defaults.interval` | Time between runs of each check. | +| `timeout` | duration | no | `defaults.timeout` | Per-run cancellation time. | +| `fail_threshold` | integer | no | `defaults.fail_threshold` | Consecutive failures that mark the target `DOWN`. | +| `recover_threshold` | integer | no | `defaults.recover_threshold` | Consecutive successes that mark it `UP` again. | +| `flap_limit` | integer | no | `defaults.flap_limit` | State changes within `flap_window` that mark it flapping. | +| `flap_window` | duration | no | `defaults.flap_window` | Window for counting state changes. | + +Rules enforced across targets: + +- `tsping` is invalid when `network` is `local`. +- An `http:` check's URL host must equal `addr`. Comparison ignores case and a + trailing dot, and treats equivalent IP forms as equal. +- `parent` must name a target in the same list, and that target must have + exactly one check, either `icmp` or `tsping`. +- Parent cycles are rejected. +- `name` must be unique. + +For how the state, flap, and parent settings behave at run time, see +[Alert state](../explanation/alert-state.md) and +[Failure layers](../explanation/failure-layers.md). For `network`, see +[Monitor remote subnets and local LANs](../how-to/remote-subnets.md). + +## `sinks[]` + +A sink is a notification destination. For delivery behavior, see +[Incidents and queues](../explanation/incidents-and-queues.md), and for setup, see +[Send alerts to Slack](../how-to/slack-sink.md) and +[Send alerts to a webhook](../how-to/webhook-sink.md). + +### Common keys + +| Key | Type | Required | Default | Meaning | +| --- | --- | --- | --- | --- | +| `name` | string | yes | | Unique. Must match `[A-Za-z0-9._-]+`. `incidents`, `intake`, and `state` are reserved. | +| `type` | string | yes | | `webhook` or `slack`. | +| `url_env` | string | one of `url_env`, `url_file` | | Environment variable holding the destination URL. | +| `url_file` | string | one of `url_env`, `url_file` | | File holding the destination URL. | +| `min_severity` | string | no | `info` | `info`, `warn`, or `page`. Events below it are not sent. | +| `events` | list of strings | no | all types | Event types to send. See the list below. | +| `match` | mapping | no | no filter | Target filters. See below. | +| `max_age` | duration | no | `1h` | Age after which an undelivered event is dropped. Greater than 0. | +| `queue` | string | no | `memory` | `memory` or `disk`. `disk` requires `collector.state_dir`. | +| `queue_size` | integer | no | `1000` | Maximum events waiting. At least 1. | +| `group_wait` | duration | no | `10s` | How long to collect events into one request. Greater than 0. | + +Exactly one of `url_env` and `url_file` must be set. Values from a file have +surrounding whitespace trimmed. All secrets are read again on every send, so a +rotated secret needs no restart. + +`group_wait` groups events for a `slack` sink, and for a `webhook` sink only when +`batch` is `true`. + +Valid `events` values: + +| Value | Emitted | +| --- | --- | +| `target.down` | yes | +| `target.recovered` | yes | +| `target.flapping` | yes | +| `collector.started` | yes | +| `test` | yes, by `notify-test` only | +| `target.degraded` | never in this version; accepted and reserved | + +An event reaches a sink only when its type is listed (or `events` is unset), its +severity is at least `min_severity`, and every key in `match` matches. The `test` +event ignores all filters. An event with no target, such as `collector.started`, +ignores `match`. A recovery goes to the sinks that were selected for its down, +regardless of the current filters. + +### `match` + +All keys are optional. Within one key, any listed value may match. Across keys, +all must match. + +| Key | Type | Matches | +| --- | --- | --- | +| `sites` | list of strings | The target's `site`, exactly. | +| `targets` | list of strings | The target's `name`, as a glob (`*`, `?`, `[...]`). An invalid glob is rejected. | +| `tags` | list of strings | Any Tailscale tag on the node the target resolves to, such as `tag:router`. | + +### Webhook-only keys + +| Key | Type | Required | Default | Meaning | +| --- | --- | --- | --- | --- | +| `secret_env` | string | no | unset | Environment variable holding the HMAC key. | +| `secret_file` | string | no | unset | File holding the HMAC key. | +| `headers_env` | mapping | no | empty | Header name to the name of an environment variable holding its value. | +| `batch` | bool | no | `false` | Send several events per request. | +| `max_batch` | integer | no | `100` | Events per batch at most. At least 1. | + +Set at most one of `secret_env` and `secret_file`. Without a secret, requests are +unsigned and the collector logs a warning at startup. See +[Event payload](payload.md) for the signature and batch envelope. + +### Slack-only keys + +| Key | Type | Required | Default | Meaning | +| --- | --- | --- | --- | --- | +| `mention_on_page` | string | no | empty | Text prepended to a `target.down` of severity `page`, such as ``. | +| `status_url` | string | no | empty | Base URL. Each target line links to `/#`. | + +A `slack` sink rejects `batch`, `max_batch`, `secret_env`, `secret_file`, and +`headers_env`. A `webhook` sink rejects `mention_on_page` and `status_url`. + +## `heartbeat` + +Pings a dead-man's-switch URL with `GET`. The section is off when omitted. + +| Key | Type | Required | Default | Meaning | +| --- | --- | --- | --- | --- | +| `url_env` | string | one of `url_env`, `url_file` | | Environment variable holding the URL. | +| `url_file` | string | one of `url_env`, `url_file` | | File holding the URL. | +| `interval` | duration | no | `1m` | Time between pings. Greater than 0. | + +Pings start once every target and check has a first result. Each request times +out after 10s, and a non-2xx response or an error is counted in +`tailwatch_heartbeat_failures_total` and not retried early. See +[Watch the watcher with a heartbeat](../how-to/heartbeat.md). + +## Secret resolution + +At server startup, every `url_env`, `url_file`, `secret_env`, `secret_file`, +`headers_env` value, and the heartbeat URL must resolve to a non-empty value, or +startup exits with an error. `check-config` and `--once` do not resolve secrets. +`check-config` prints each sink's name, type, filters, and queue mode, and never a +secret. diff --git a/docs/reference/metrics.md b/docs/reference/metrics.md new file mode 100644 index 0000000..712d575 --- /dev/null +++ b/docs/reference/metrics.md @@ -0,0 +1,97 @@ +# Metrics reference + +The collector serves Prometheus text at `/metrics` on the listen address. Every +series starts with `tailwatch_`. The prefix has no hyphen because Prometheus metric +names cannot contain one. The default Go and process collectors are also exposed +and are not listed here. + +## Check series + +One set per target and check pair, registered by the scheduler when the check is +added and removed when the target is removed. Labels on every series: + +| Label | Value | +| --- | --- | +| `name` | `/`, such as `nyc-nas/tcp:445`. | +| `class` | Check kind: `icmp`, `tcp`, `http`, or `tsping`. | +| `target` | Target name. | +| `check` | Check string as configured, such as `tcp:445`. | +| `site` | Target `site`, empty when unset. | + +| Metric | Type | Extra labels | Meaning | +| --- | --- | --- | --- | +| `tailwatch_interval_secs` | gauge | | Probe interval in seconds. | +| `tailwatch_start_secs` | gauge | | Start time of the latest run, seconds since the epoch. | +| `tailwatch_end_secs` | gauge | | End time of the latest run, seconds since the epoch. | +| `tailwatch_latency_millis` | gauge | | Latency of the latest run in milliseconds. Absent when the latest run reported none. | +| `tailwatch_result` | gauge | | `1` if the latest run succeeded, `0` if it failed. | +| `tailwatch_in_flight` | gauge | | Runs in progress. | +| `tailwatch_attempts_total` | counter | `status` | Runs, by `status` `ok` or `fail`. | +| `tailwatch_seconds_total` | counter | `status` | Time spent running, in seconds, by `status`. | + +`tailwatch_result`, `tailwatch_end_secs`, and the two counters appear after the +first run finishes. These series come from the Tailscale `prober` package. + +## Classification and state series + +| Metric | Type | Labels | Meaning | +| --- | --- | --- | --- | +| `tailwatch_failure_layer_total` | counter | `target`, `layer` | Failed check results by assigned failure layer. Increments once per failed result. `layer` is `unrouted`, `control`, `tunnel`, `lan`, or `service`. | +| `tailwatch_path` | gauge | `target`, `kind` | Current `tsping` path kind. One series per `kind` (`direct`, `derp`, `peer_relay`); the active kind is `1` and the others are `0`. | +| `tailwatch_target_parent` | gauge | `target`, `parent` | `1` while `parent` is the target's current parent, configured or learned. Removed when the parent changes. | +| `tailwatch_target_state` | gauge | `target` | Alert state: `0` unknown, `1` up, `2` down, `3` suppressed, `4` flapping. | + +See [Failure layers](../explanation/failure-layers.md) and +[Alert state](../explanation/alert-state.md). + +## Notification series + +Registered only when at least one sink is configured, except +`tailwatch_heartbeat_failures_total`. The `sink` label is the sink `name`. + +| Metric | Type | Labels | Meaning | +| --- | --- | --- | --- | +| `tailwatch_notify_sent_total` | counter | `sink`, `type` | Notifications delivered, by event type. | +| `tailwatch_notify_failed_total` | counter | `sink`, `reason` | Failed delivery attempts. `reason` is `timeout`, `http_5xx`, `http_429`, `http_4xx_permanent`, or `network`. | +| `tailwatch_notify_dropped_total` | counter | `sink`, `reason` | Notifications dropped without delivery. `reason` is `queue_full`, `expired`, `permanent`, or `restart`. | +| `tailwatch_notify_persist_errors_total` | counter | `sink` | Failed writes of notification state. `sink` is always `state`. | +| `tailwatch_notify_queue_len` | gauge | `sink` | Events waiting in the sink's queue. | +| `tailwatch_notify_latency_seconds` | histogram | `sink` | Seconds from event creation to confirmed delivery. Buckets: 0.1, 0.5, 1, 5, 10, 30, 60, 300, 900, 3600. | +| `tailwatch_notify_batch_size` | histogram | `sink` | Events per request for batched webhook sinks. Buckets: 1, 2, 5, 10, 25, 50, 100, 250, 500. | +| `tailwatch_open_incidents` | gauge | | Incidents opened and not yet closed. Registered only when `collector.state_dir` is set. | + +`tailwatch_notify_dropped_total` also uses `sink="intake"` with `reason="queue_full"` +when the 4096-slot intake channel is full. The alerter's next reconcile, within a +second, repairs the lost transition. + +Drop reasons: + +| Reason | Cause | +| --- | --- | +| `queue_full` | The sink queue or the intake channel was full. | +| `expired` | The event was older than the sink's `max_age` when an attempt was due. | +| `permanent` | The receiver returned a non-retryable status, such as `404`. | +| `restart` | The event sat in a `memory` queue when the collector restarted. | + +See [Incidents and queues](../explanation/incidents-and-queues.md). + +## Heartbeat series + +| Metric | Type | Labels | Meaning | +| --- | --- | --- | --- | +| `tailwatch_heartbeat_failures_total` | counter | | Heartbeat pings that failed. Registered only when `heartbeat` is configured. | + +## Suggested alerts + +```yaml +- alert: TailWatchNotificationLoss + expr: increase(tailwatch_notify_dropped_total[15m]) > 0 +- alert: TailWatchQueueStuck + expr: min_over_time(tailwatch_notify_queue_len[15m]) > 0 +- alert: TargetDownQuorum + expr: count by (target) (tailwatch_target_state == 2) >= 2 + for: 1m +``` + +The first two mean an alert did not get out. The third pages only when two +collectors agree that a target is down. diff --git a/docs/reference/payload.md b/docs/reference/payload.md new file mode 100644 index 0000000..2c2019c --- /dev/null +++ b/docs/reference/payload.md @@ -0,0 +1,242 @@ +# Event payload reference + +Every notification is a JSON event. This page describes the version 1 payload, the +batch envelope, the webhook request, and the signature. For the Slack rendering of +the same events, see [Send alerts to Slack](../how-to/slack-sink.md). For how events +are produced and queued, see +[Incidents and queues](../explanation/incidents-and-queues.md). + +Within `version: 1`, changes only add fields. Receivers must ignore fields they do +not know. Fields that do not apply are omitted, never sent as `null`. + +## Event types + +| `type` | `severity` | Sent when | +| --- | --- | --- | +| `target.down` | The target's `severity`: `page` or `warn` | A target goes `DOWN` and has no open incident. | +| `target.recovered` | The severity of the down it resolves | A target with an open incident returns to `UP`, or the target is removed. | +| `target.flapping` | `warn` | A target crosses `flap_limit` state changes within `flap_window`. | +| `collector.started` | `info` | Once, after every target and check has a first result. | +| `test` | `info` | Only from `tail-watch notify-test`. | +| `target.degraded` | | Reserved. Never sent in this version. | + +Severity ranks `info` < `warn` < `page`. A sink's `min_severity` compares against it. +A start with every target `UP` sends only `collector.started`. A target already +failing at startup produces one `target.down`. + +## Fields + +| Field | Type | Present on | Meaning | +| --- | --- | --- | --- | +| `version` | integer | all | Always `1`. | +| `id` | string | all | Event id, see [Ids](#ids). | +| `type` | string | all | One of the types above. | +| `severity` | string | all | `info`, `warn`, or `page`. | +| `time` | string | all | Creation time, RFC 3339 in UTC with second precision. | +| `late` | boolean | any, when true | The first attempt happened more than one minute after `time`. The original `time` is kept. | +| `collector` | object | all | The sending collector. | +| `collector.name` | string | all | The node's hostname. | +| `collector.tailscale_ip` | string | when known | The node's first Tailscale IPv4 address, or IPv6 when it has none. | +| `target` | object | target events | The affected target. | +| `target.name` | string | target events | Target name. | +| `target.addr` | string | target events | Target address. | +| `target.site` | string | when set | Target site. | +| `target.parent` | string | when set | Parent target name. | +| `target.implicit` | boolean | when true | The target is a router tail-watch learned, not one from the config. | +| `check` | string | target events | The check that failed, such as `tcp:445`. | +| `layer` | string | down, flapping | Failure layer: `unrouted`, `control`, `tunnel`, `lan`, or `service`. | +| `error` | string | target events | The error from the failing check. | +| `since` | string | target events | When the target's outage began, RFC 3339 UTC. | +| `failures` | integer | target events | Consecutive failures that opened the incident. | +| `path` | object | when known | How the latest probe reached the target. | +| `path.kind` | string | when known | `direct`, `derp`, or `peer_relay`. | +| `path.endpoint` | string | direct paths | Remote UDP endpoint, `ip:port`. | +| `path.derp_region` | string | derp paths | DERP region code. | +| `path.latency_ms` | integer | when measured | Latency in milliseconds. | +| `dependents` | array of strings | down, recovered | Targets suppressed behind this one. Empty array when none. | +| `duration_seconds` | integer | recovered | Seconds from `since` to the recovery. | +| `down_event_id` | string | recovered | The `id` of the `target.down` it resolves. | +| `reason` | string | recovered, when set | `removed` when the target was deleted or edited while `DOWN`, rather than recovered. | + +`collector.started` and `test` carry only `version`, `id`, `type`, `severity`, +`time`, `collector`, and `late` when applicable. `target.recovered` has no `layer`. +`target.flapping` has no `dependents`. + +## Examples + +Full examples for each shape are in `cmd/tail-watch/testdata/payload/` in the +repository. + +### `target.down` + +```json +{ + "version": 1, + "id": "evt_01JAB3K9T6M4X2Q8R7N5W0YZCD", + "type": "target.down", + "severity": "page", + "time": "2026-04-02T14:02:11Z", + "collector": { + "name": "monitor-nyc", + "tailscale_ip": "100.101.102.103" + }, + "target": { + "name": "nyc-nas", + "addr": "10.20.0.50", + "site": "nyc", + "parent": "nyc-router" + }, + "check": "tcp:445", + "layer": "service", + "error": "connection refused", + "since": "2026-04-02T14:00:41Z", + "failures": 3, + "path": { + "kind": "direct", + "endpoint": "203.0.113.5:41641", + "latency_ms": 12 + }, + "dependents": [] +} +``` + +A learned router adds `"implicit": true` inside `target`. + +### `target.recovered` + +```json +{ + "version": 1, + "id": "evt_01JAB3K9T6M4X2Q8R7N5W0YZCG", + "type": "target.recovered", + "severity": "page", + "time": "2026-04-02T14:14:11Z", + "collector": { + "name": "monitor-nyc", + "tailscale_ip": "100.101.102.103" + }, + "target": { + "name": "nyc-nas", + "addr": "10.20.0.50", + "site": "nyc", + "parent": "nyc-router" + }, + "check": "tcp:445", + "error": "connection refused", + "since": "2026-04-02T14:00:41Z", + "failures": 3, + "path": { + "kind": "direct", + "endpoint": "203.0.113.5:41641", + "latency_ms": 12 + }, + "dependents": [ + "nyc-cam" + ], + "duration_seconds": 810, + "down_event_id": "evt_01JAB3K9T6M4X2Q8R7N5W0YZCD" +} +``` + +When the target was removed instead of recovering, the event also carries +`"reason": "removed"`. + +### `collector.started` and `test` + +```json +{ + "version": 1, + "id": "evt_01JAB3K9T6M4X2Q8R7N5W0YZCK", + "type": "collector.started", + "severity": "info", + "time": "2026-04-02T14:02:11Z", + "collector": { + "name": "monitor-nyc", + "tailscale_ip": "100.101.102.103" + } +} +``` + +## Ids + +An event `id` is `evt_` followed by 26 Crockford base32 characters +(`0123456789ABCDEFGHJKMNPQRSTVWXYZ`): 48 bits of Unix milliseconds followed by 80 +random bits. Ids sort by creation time across milliseconds. The id is the same +across every retry and every sink. + +A batch id is `bat_` followed by 26 characters of the same form. + +Delivery is at least once. Receivers must deduplicate on `id`, and must accept a +`target.recovered` whose `down_event_id` they never saw. + +## Webhook request + +Each request is a `POST` with these headers: + +| Header | Value | +| --- | --- | +| `Content-Type` | `application/json` | +| `User-Agent` | `tail-watch/` | +| `Tail-Watch-Event-Id` | The event `id`. Sent when `batch` is `false`. | +| `Tail-Watch-Batch-Id` | The `batch_id`. Sent when `batch` is `true`. | +| `Tail-Watch-Signature` | The signature. Sent when the sink has a secret. | +| any `headers_env` name | The value of the environment variable it names. | + +Redirects are not followed and each request times out after 10 seconds. + +| Response | Result | +| --- | --- | +| `2xx` | Delivered. | +| `408`, `5xx`, timeout, network error | Retried. | +| `429` | Retried, waiting at least `Retry-After` (seconds or an HTTP date). | +| Anything else, including `3xx` and other `4xx` | Dropped, logged once, counted as `permanent`. | + +The retry delay is `min(2^(attempt-1) seconds, 5 minutes)` scaled by 0.5 to 1.5 +times jitter, raised to `Retry-After` when longer, and never more than 5 minutes. +An event older than the sink's `max_age` when an attempt is due is dropped and +counted as `expired`. + +### Batch envelope + +With `batch: true`, the request body is: + +```json +{ + "version": 1, + "batch_id": "bat_01JAB3K9T6M4X2Q8R7N5W0YZCP", + "events": [] +} +``` + +`events` holds full event objects, in queue order. The sink waits `group_wait` after +the first queued event, or until `max_batch` events are waiting. A retry resends +identical body bytes and the same `batch_id`. Events that arrive meanwhile go into +the next batch. A permanent failure or expiry drops the whole batch, and every event +in it is counted. + +## Signature + +With `secret_env` or `secret_file`, each request carries: + +```text +Tail-Watch-Signature: t=,v1= +``` + +`v1` is the lowercase hex HMAC-SHA256, keyed with the secret, of the ASCII string +`.` followed by the exact request body bytes. For a batch the body is the whole +envelope. A retry sends the same body bytes with a fresh `t`. + +To verify, parse `t` and `v1`, reject a `t` outside your tolerance, recompute the +HMAC over `.`, and compare in constant time. Verify against the raw +bytes, not a re-serialized copy. + +```bash +{ printf '%s.' "$T"; cat body.json; } | openssl dgst -sha256 -hmac "$SECRET" -hex +``` + +## `late` + +The body of the first attempt is fixed when the request is first formed. If that +moment is more than one minute after the event's `time`, the event carries +`"late": true`, for example after a receiver outage or a collector restart. The +original `time` is never changed. diff --git a/docs/tutorial/first-monitor.md b/docs/tutorial/first-monitor.md new file mode 100644 index 0000000..57c03cc --- /dev/null +++ b/docs/tutorial/first-monitor.md @@ -0,0 +1,175 @@ +# Set up your first monitor + +In this tutorial you run tail-watch beside a Tailscale node, watch one device on your +tailnet, make that device fail, and see the alert arrive in Slack. It takes about 20 +minutes. + +You need: + +- Docker with Compose, on a machine that has `/dev/net/tun`. +- A tailnet where you can edit the policy file and create an auth key. +- A Slack workspace where you can add an incoming webhook. +- One device on your tailnet that you can reach by its MagicDNS name. This tutorial + calls it `my-device`. + +## 1. Allow the collector in your policy + +tail-watch runs as a node tagged `tag:monitor`. In your tailnet policy file, own the tag +and let it reach `my-device`. Replace `tag:server` with the tag your device has, or use +its user or IP instead: + +```jsonc +{ + "tagOwners": { "tag:monitor": ["autogroup:admin"] }, + "grants": [ + { + "src": ["tag:monitor"], + "dst": ["tag:server"], + "ip": ["*"] + } + ] +} +``` + +[Write the ACLs](../how-to/acls.md) explains each field. + +## 2. Create an auth key + +In the admin console, open **Settings → Keys** and generate an auth key. Tick +**Pre-approved** if your tailnet requires device approval, and choose `tag:monitor` under +**Tags**. Copy the key and export it: + +```bash +export TS_AUTHKEY=tskey-auth-... +``` + +## 3. Get a Slack webhook URL + +Follow [Get an incoming webhook URL](../how-to/slack-sink.md#get-an-incoming-webhook-url), +then export the URL: + +```bash +export SLACK_WEBHOOK_URL='https://hooks.slack.com/services/...' +``` + +## 4. Write a config + +Create `first-monitor.yaml` next to `examples/compose.yaml`. It has one target, the +tailnet ping of `my-device`, and one Slack sink: + +```yaml +collector: + listen: 127.0.0.1:9109 + state_dir: /var/lib/tail-watch + +defaults: + interval: 10s + timeout: 3s + fail_threshold: 2 + recover_threshold: 1 + +targets: + - name: my-device + addr: my-device + checks: [tsping] + +sinks: + - name: ops-slack + type: slack + url_env: SLACK_WEBHOOK_URL + group_wait: 5s +``` + +Check it. This reads the file and does not contact Tailscale or Slack: + +```bash +docker build -t tail-watch:local . +docker run --rm -v "$PWD/first-monitor.yaml:/first-monitor.yaml:ro" \ + -e SLACK_WEBHOOK_URL tail-watch:local \ + check-config --config /first-monitor.yaml +``` + +The output lists the target and the `ops-slack` sink. The shorter thresholds only make +the tutorial faster; the [defaults](../reference/config.md) are better for real use. + +## 5. Pass the webhook to the container + +`examples/compose.yaml` does not forward your environment to tail-watch. Add an override +file named `first-monitor.compose.yaml`: + +```yaml +services: + tail-watch: + environment: + SLACK_WEBHOOK_URL: ${SLACK_WEBHOOK_URL:?set SLACK_WEBHOOK_URL} +``` + +## 6. Start the stack + +```bash +TAIL_WATCH_CONFIG_FILE=./first-monitor.yaml \ + docker compose -f examples/compose.yaml -f first-monitor.compose.yaml up -d --build +``` + +The `tail-watch` node appears in your admin console with the `tag:monitor` tag. Check +the logs: + +```bash +docker compose -f examples/compose.yaml -f first-monitor.compose.yaml logs tail-watch +``` + +The last line is `listening` on `127.0.0.1:9109`. The `state_dir is readable by group +or others` warning is expected here, because the image creates that directory with mode +`0755`; it does not stop anything from working. Now send a test message through the real +Slack code: + +```bash +docker compose -f examples/compose.yaml -f first-monitor.compose.yaml \ + exec tail-watch /usr/local/bin/tail-watch notify-test \ + --config /etc/tail-watch/tail-watch.yaml +``` + +Slack shows a test message, and the command prints `ops-slack slack 200` and exits 0. +If it does not, see [Troubleshoot](../how-to/troubleshoot.md). + +You should also see `collector.started` arrive in the channel once every target has its +first result. + +## 7. Watch it work + +The status page runs on the node's loopback address, so open it from inside the +container namespace: + +```bash +docker compose -f examples/compose.yaml exec tailscale wget -qO- http://127.0.0.1:9109/ +``` + +`my-device` shows as up. + +## 8. Break it + +Stop Tailscale on `my-device`, or power the device off. Within about 30 seconds, +tail-watch marks the target `DOWN` and Slack shows a message that starts +`:red_circle: DOWN my-device`, with the failure layer in brackets. + +The layer says where the path broke. [Failure layers](../explanation/failure-layers.md) +explains why `control` and `tunnel` differ from `service` and `lan`. + +## 9. Fix it + +Start Tailscale on `my-device` again. After one successful check, tail-watch sends a +recovery message that names the original outage start and how long it lasted. + +## Clean up + +```bash +docker compose -f examples/compose.yaml -f first-monitor.compose.yaml down -v +``` + +Delete the `tail-watch` machine from the admin console if it does not expire on its own. + +## Next steps + +- [Deploy tail-watch](../how-to/deploy.md) on a host instead of Compose. +- [Monitor remote subnets](../how-to/remote-subnets.md) behind a router. +- [Configuration reference](../reference/config.md) for every key. diff --git a/openspec/changes/add-user-documentation/.openspec.yaml b/openspec/changes/archive/2026-09-29-add-user-documentation/.openspec.yaml similarity index 100% rename from openspec/changes/add-user-documentation/.openspec.yaml rename to openspec/changes/archive/2026-09-29-add-user-documentation/.openspec.yaml diff --git a/openspec/changes/add-user-documentation/design.md b/openspec/changes/archive/2026-09-29-add-user-documentation/design.md similarity index 88% rename from openspec/changes/add-user-documentation/design.md rename to openspec/changes/archive/2026-09-29-add-user-documentation/design.md index ab5fcd8..34320f0 100644 --- a/openspec/changes/add-user-documentation/design.md +++ b/openspec/changes/archive/2026-09-29-add-user-documentation/design.md @@ -34,16 +34,23 @@ sets in `main.go`), the archived specs under `openspec/specs/`, and docs/ README.md index, one line per page tutorial/first-monitor.md - how-to/ deploy.md acls.md remote-subnets.md - slack-sink.md webhook-sink.md troubleshoot.md + how-to/ deploy.md acls.md remote-subnets.md monitor-ui.md + slack-sink.md webhook-sink.md heartbeat.md + troubleshoot.md reference/ config.md checks.md cli.md metrics.md payload.md explanation/ failure-layers.md alert-state.md incidents-and-queues.md + monitor-ui-risks.md contributing/ live-notification-test.md releases.md ``` Alternative: a single long `docs/guide.md`. Rejected because it recreates the README problem and gives readers no stable links. +The monitor UI, heartbeat and egress path sections of the README have no page in the +first outline. They get `how-to/monitor-ui.md`, `explanation/monitor-ui-risks.md` and +`how-to/heartbeat.md`; the egress-path grant goes into the sink how-tos, and +`notify-test` goes into `reference/cli.md` with links from each sink page. + ### README becomes a landing page Keep the purpose paragraph, requirements summary, a five-line quickstart, and a link diff --git a/openspec/changes/add-user-documentation/proposal.md b/openspec/changes/archive/2026-09-29-add-user-documentation/proposal.md similarity index 100% rename from openspec/changes/add-user-documentation/proposal.md rename to openspec/changes/archive/2026-09-29-add-user-documentation/proposal.md diff --git a/openspec/changes/archive/2026-09-29-add-user-documentation/readme-map.md b/openspec/changes/archive/2026-09-29-add-user-documentation/readme-map.md new file mode 100644 index 0000000..aa0d37d --- /dev/null +++ b/openspec/changes/archive/2026-09-29-add-user-documentation/readme-map.md @@ -0,0 +1,43 @@ +# README migration map + +Working record for tasks 1.3 and 1.4. Not part of `docs/`. + +## Heading map + +| README section (before) | Target | +| ----------------------- | ------ | +| Intro (lines 1-9) | `README.md` overview | +| Requirements | `README.md` summary; `docs/how-to/deploy.md`; `docs/how-to/troubleshoot.md` | +| Remote subnets and local LANs | `docs/how-to/remote-subnets.md` | +| Build and run | `docs/how-to/deploy.md`; flags in `docs/reference/cli.md` | +| Docker | `docs/how-to/deploy.md` | +| Docker Compose without a dedicated host | `docs/how-to/deploy.md`; `docs/tutorial/first-monitor.md` | +| ACLs | `docs/how-to/acls.md` | +| Checks | `docs/reference/checks.md` | +| Failure layers | `docs/explanation/failure-layers.md` | +| Alert state | `docs/explanation/alert-state.md`; `docs/reference/metrics.md` | +| Monitor UI | `docs/how-to/monitor-ui.md` | +| Monitor UI / Reaching the UI | `docs/how-to/monitor-ui.md` | +| Monitor UI / Risks | `docs/explanation/monitor-ui-risks.md` | +| Notifications | `docs/reference/payload.md`; `docs/explanation/incidents-and-queues.md` | +| Notifications / Sink options | `docs/reference/config.md` | +| Notifications / Payload | `docs/reference/payload.md` | +| Notifications / Webhook sink | `docs/how-to/webhook-sink.md` | +| Notifications / Slack sink | `docs/how-to/slack-sink.md` | +| Notifications / Queues, `state_dir` and open incidents | `docs/explanation/incidents-and-queues.md`; metrics in `docs/reference/metrics.md` | +| Notifications / Watching the watcher | `docs/how-to/heartbeat.md` | +| Notifications / Trying it out | `docs/reference/cli.md`; linked from sink pages | +| Notifications / Egress path | `docs/how-to/heartbeat.md`; `docs/how-to/acls.md` | +| Notifications / Live notification test | `docs/contributing/live-notification-test.md` | +| Live notification test / Getting a Slack webhook URL | `docs/how-to/slack-sink.md` | +| Development and releases | `docs/contributing/releases.md` | + +## Inbound links + +A search for `README.md#` and `README#` found no anchor links outside +`openspec/changes/archive`. Other references to the README: + +| Location | Reference | Action | +| -------- | --------- | ------ | +| `scripts/live-stack.sh` line 4 | `README "Live notification test"` | Point at `docs/contributing/live-notification-test.md` | +| `.goreleaser.yaml` `archives.files` | ships `README.md` | Trimmed README links into `docs/`; see open question in the change summary | diff --git a/openspec/changes/add-user-documentation/tasks.md b/openspec/changes/archive/2026-09-29-add-user-documentation/tasks.md similarity index 53% rename from openspec/changes/add-user-documentation/tasks.md rename to openspec/changes/archive/2026-09-29-add-user-documentation/tasks.md index 5b1ee4b..fe4cba2 100644 --- a/openspec/changes/add-user-documentation/tasks.md +++ b/openspec/changes/archive/2026-09-29-add-user-documentation/tasks.md @@ -2,52 +2,55 @@ ## 1. Outline and scaffolding -- [ ] 1.1 Use the `documentation-writer` skill to confirm audience, goal and scope per page, and get the outline approved; verify the approved outline matches the tree in `design.md` -- [ ] 1.2 Create `docs/README.md` index and the empty quadrant directories; verify every planned page is listed and links resolve -- [ ] 1.3 Add a heading-by-heading map from current `README.md` sections to target pages (kept in the change directory, not committed to `docs/`); verify no README section is unassigned -- [ ] 1.4 Search the repo for `README.md#` anchors and inbound links (`CLAUDE.md`, `CHANGELOG.md`, `examples/`); verify each is listed with its new target +- [x] 1.1 Use the `documentation-writer` skill to confirm audience, goal and scope per page, and get the outline approved; verify the approved outline matches the tree in `design.md` +- [x] 1.2 Create `docs/README.md` index and the empty quadrant directories; verify every planned page is listed and links resolve +- [x] 1.3 Add a heading-by-heading map from current `README.md` sections to target pages (kept in the change directory, not committed to `docs/`); verify no README section is unassigned +- [x] 1.4 Search the repo for `README.md#` anchors and inbound links (`CLAUDE.md`, `CHANGELOG.md`, `examples/`); verify each is listed with its new target ## 2. Reference pages -- [ ] 2.1 Write `docs/reference/config.md` from `cmd/tail-watch/config.go` and `config_sinks.go`; verify every key in `examples/tail-watch.yaml` and every `sinks` key appears with type, default and required/optional -- [ ] 2.2 Save each full-config snippet from `docs/reference/config.md` and run `tail-watch check-config`; verify all pass -- [ ] 2.3 Write `docs/reference/cli.md` (flags, `TAIL_WATCH_CONFIG`, `TAIL_WATCH_LOG_LEVEL`, `check-config`, `notify-test`, exit codes) from `cmd/tail-watch/main.go` and `notifytest.go`; verify against `tail-watch --help` and `notify-test --help` -- [ ] 2.4 Write `docs/reference/metrics.md` from the metric registrations under `cmd/tail-watch/`; verify each documented `tailwatch_*` name appears in the source and each registered name is documented -- [ ] 2.5 Write `docs/reference/checks.md` (ICMP, TCP, HTTP, tsping options and semantics) from `checks.go`; verify against the `reachability-checks` spec -- [ ] 2.6 Write `docs/reference/payload.md` (event types, fields, severities, signature header) from `event.go` and `testdata/payload`; verify the field list matches the golden files +- [x] 2.1 Write `docs/reference/config.md` from `cmd/tail-watch/config.go` and `config_sinks.go`; verify every key in `examples/tail-watch.yaml` and every `sinks` key, `heartbeat` key and monitor UI key (`collector.ui`, `state_dir`) appears with type, default and required/optional +- [x] 2.2 Save each full-config snippet from `docs/reference/config.md` and run `tail-watch check-config`; verify all pass +- [x] 2.3 Write `docs/reference/cli.md` (flags, `TAIL_WATCH_CONFIG`, `TAIL_WATCH_LOG_LEVEL`, `check-config`, `notify-test`, exit codes) from `cmd/tail-watch/main.go` and `notifytest.go`; verify against `tail-watch --help` and `notify-test --help` +- [x] 2.4 Write `docs/reference/metrics.md` from the metric registrations under `cmd/tail-watch/`; verify each documented `tailwatch_*` name appears in the source and each registered name is documented +- [x] 2.5 Write `docs/reference/checks.md` (ICMP, TCP, HTTP, tsping options and semantics) from `checks.go`; verify against the `reachability-checks` spec +- [x] 2.6 Write `docs/reference/payload.md` (event types, fields, severities, signature header) from `event.go` and `testdata/payload`; verify the field list matches the golden files ## 3. Explanation pages -- [ ] 3.1 Write `docs/explanation/failure-layers.md` from the README "Failure layers" section and the `failure-classification` and `dependency-learning` specs; verify each layer name matches the `failure_layer` metric label values -- [ ] 3.2 Write `docs/explanation/alert-state.md` covering grace, flap suppression and dependency suppression, from the `alert-state-machine` and `dependency-suppression` specs; verify each state and timer name matches the config keys in `reference/config.md` -- [ ] 3.3 Write `docs/explanation/incidents-and-queues.md` covering `state_dir`, at-least-once delivery, replay and open incidents; verify against the `add-notifications` spec set once archived +- [x] 3.1 Write `docs/explanation/failure-layers.md` from the README "Failure layers" section and the `failure-classification` and `dependency-learning` specs; verify each layer name matches the `failure_layer` metric label values +- [x] 3.2 Write `docs/explanation/alert-state.md` covering grace, flap suppression and dependency suppression, from the `alert-state-machine` and `dependency-suppression` specs; verify each state and timer name matches the config keys in `reference/config.md` +- [x] 3.3 Write `docs/explanation/incidents-and-queues.md` covering `state_dir`, at-least-once delivery, replay and open incidents; verify against the `add-notifications` spec set once archived +- [x] 3.4 Write `docs/explanation/monitor-ui-risks.md` (no login, listen address, Origin and content-type guard, DNS rebinding, audit log) from README "Risks" and the `monitor-ui` and `monitor-management` specs; verify each claim against `monitor_guard.go` ## 4. How-to guides -- [ ] 4.1 Write `docs/how-to/deploy.md` (binary, Docker, Compose without a dedicated host) from README "Build and run" and "Docker"; verify the Docker build command runs and `--version` prints -- [ ] 4.2 Write `docs/how-to/acls.md` from README "ACLs"; verify the policy snippet is valid HuJSON -- [ ] 4.3 Write `docs/how-to/remote-subnets.md` from README "Remote subnets and local LANs"; verify each `network` value and startup failure named in it exists in `netenv.go` and `config.go` -- [ ] 4.4 Write `docs/how-to/slack-sink.md` including getting a webhook URL and using `*_env`; verify the config snippet passes `check-config` and `notify-test --sink` steps match `notifytest.go` -- [ ] 4.5 Write `docs/how-to/webhook-sink.md` (signing, headers, batching, retries); verify the signature verification example against `webhook_test.go` -- [ ] 4.6 Write `docs/how-to/troubleshoot.md` (userspace tailscaled refusal, missing `tag:monitor`, ICMP permissions, socket path, missing secrets, corrupt state); verify each symptom's message text against the source strings +- [x] 4.1 Write `docs/how-to/deploy.md` (binary, Docker, Compose without a dedicated host) from README "Build and run" and "Docker"; verify the Docker build command runs and `--version` prints +- [x] 4.2 Write `docs/how-to/acls.md` from README "ACLs"; verify the policy snippet is valid HuJSON +- [x] 4.3 Write `docs/how-to/remote-subnets.md` from README "Remote subnets and local LANs"; verify each `network` value and startup failure named in it exists in `netenv.go` and `config.go` +- [x] 4.4 Write `docs/how-to/slack-sink.md` including getting a webhook URL and using `*_env`; verify the config snippet passes `check-config` and `notify-test --sink` steps match `notifytest.go` +- [x] 4.5 Write `docs/how-to/webhook-sink.md` (signing, headers, batching, retries); verify the signature verification example against `webhook_test.go` +- [x] 4.6 Write `docs/how-to/troubleshoot.md` (userspace tailscaled refusal, missing `tag:monitor`, ICMP permissions, socket path, missing secrets, corrupt state); verify each symptom's message text against the source strings +- [x] 4.7 Write `docs/how-to/monitor-ui.md` (enable, `state_dir`, `targets.json`, reaching the UI over the tailnet, edit and delete semantics, screenshots from `docs/images/`); verify flags and behavior against `ui_serve.go`, `dynamic.go` and `monitor_api.go` +- [x] 4.8 Write `docs/how-to/heartbeat.md` (dead-man's-switch config, egress path and exit-node warning); verify against `heartbeat.go` ## 5. Tutorial -- [ ] 5.1 Write `docs/tutorial/first-monitor.md` starting from `examples/compose.yaml`, adding a failing target, and ending with a delivered alert; verify by following it on a clean checkout with `scripts/live-stack.sh` or the Compose file -- [ ] 5.2 Record any step that needed a fix while following it, and update the page; verify a second read-through needs no unwritten step +- [x] 5.1 Write `docs/tutorial/first-monitor.md` starting from `examples/compose.yaml`, adding a failing target, and ending with a delivered alert; verify by following it on a clean checkout with `scripts/live-stack.sh` or the Compose file +- [x] 5.2 Record any step that needed a fix while following it, and update the page; verify a second read-through needs no unwritten step ## 6. README and contributor split -- [ ] 6.1 Move "Live notification test" and "Development and releases" from `README.md` to `docs/contributing/`; verify commands still match `CLAUDE.md` and `scripts/live-stack.sh` -- [ ] 6.2 Trim `README.md` to overview, quickstart and links; verify against the 1.3 map that every removed section exists in a page -- [ ] 6.3 Fix or redirect the inbound links found in 1.4; verify a grep for the old anchors finds none -- [ ] 6.4 Add a `docs/` layout and "update docs with config or metric changes" note to `CLAUDE.md`; verify the note matches the final tree -- [ ] 6.5 Add a `CHANGELOG.md` entry if the release process lists documentation; verify against `release-please-config.json` +- [x] 6.1 Move "Live notification test" and "Development and releases" from `README.md` to `docs/contributing/`; verify commands still match `CLAUDE.md` and `scripts/live-stack.sh` +- [x] 6.2 Trim `README.md` to overview, quickstart and links; verify against the 1.3 map that every removed section exists in a page +- [x] 6.3 Fix or redirect the inbound links found in 1.4; verify a grep for the old anchors finds none +- [x] 6.4 Add a `docs/` layout and "update docs with config or metric changes" note to `CLAUDE.md`; verify the note matches the final tree +- [x] 6.5 Add a `CHANGELOG.md` entry if the release process lists documentation; verify against `release-please-config.json` ## 7. Validation -- [ ] 7.1 Run `mise exec -- go build ./...`, `go vet ./...` and `go test ./...`; verify all pass, confirming no code changed -- [ ] 7.2 Run `mise run lint` (golangci-lint) and `govulncheck ./...`; verify no findings -- [ ] 7.3 Run `markdownlint` over `README.md`, `docs/` and `openspec/changes/add-user-documentation/`; verify zero findings -- [ ] 7.4 Run a link check over `docs/` and `README.md` (local, relative links); verify none are broken -- [ ] 7.5 Run `mise exec -- openspec validate add-user-documentation --strict`; verify it passes +- [x] 7.1 Run `mise exec -- go build ./...`, `go vet ./...` and `go test ./...`; verify all pass, confirming no code changed +- [x] 7.2 Run `mise run lint` (golangci-lint) and `govulncheck ./...`; verify no findings +- [x] 7.3 Run `markdownlint` over `README.md`, `docs/` and `openspec/changes/add-user-documentation/`; verify zero findings +- [x] 7.4 Run a link check over `docs/` and `README.md` (local, relative links); verify none are broken +- [x] 7.5 Run `mise exec -- openspec validate add-user-documentation --strict`; verify it passes diff --git a/release-please-config.json b/release-please-config.json index 2f31bec..838ed51 100644 --- a/release-please-config.json +++ b/release-please-config.json @@ -4,7 +4,29 @@ ".": { "release-type": "go", "include-component-in-tag": false, - "bump-minor-pre-major": true + "bump-minor-pre-major": true, + "changelog-sections": [ + { + "type": "feat", + "section": "Features" + }, + { + "type": "fix", + "section": "Bug Fixes" + }, + { + "type": "perf", + "section": "Performance Improvements" + }, + { + "type": "revert", + "section": "Reverts" + }, + { + "type": "docs", + "section": "Documentation" + } + ] } } } diff --git a/scripts/live-stack.sh b/scripts/live-stack.sh index e0b4098..841ccf3 100755 --- a/scripts/live-stack.sh +++ b/scripts/live-stack.sh @@ -1,9 +1,9 @@ #!/usr/bin/env bash # Bring up (or tear down) only the tailnet side of the live notification test: # the collector's tailscale node plus one fixture node per check. It does not -# start the tail-watch process or run any scenario. See README "Live notification -# test". Credentials come from the pulumi env in TAIL_WATCH_PULUMI_ENV -# (default tailscale-phase-2/tailscale). +# start the tail-watch process or run any scenario. See +# docs/contributing/live-notification-test.md. Credentials come from the pulumi +# env in TAIL_WATCH_PULUMI_ENV (default tailscale-phase-2/tailscale). # # scripts/live-stack.sh up create ephemeral nodes and print their addresses # scripts/live-stack.sh down remove the containers and this stack's devices