From cb7f49561163b3b126432323a18137851871cfc9 Mon Sep 17 00:00:00 2001 From: Richard Shade Date: Tue, 29 Sep 2026 15:44:13 -0500 Subject: [PATCH] docs,openspec: restructure user documentation into docs/ README.md had grown to about 625 lines and mixed tutorial, how-to, reference, explanation and contributor material, with no path from "what is this" to a first alert and no single place for the config schema, metrics or flags. Split it into docs/ along the Diataxis quadrants: a first-monitor tutorial, how-to guides (deploy, ACLs, remote subnets, Slack and webhook sinks, heartbeat, monitor UI, troubleshooting), reference pages (config, CLI, checks, metrics, payload), explanation pages (failure layers, alert state, incidents and queues, monitor UI risks), and contributor pages for the live notification test and releases. Trim README.md to an overview, a quickstart and links. Check each documented metric name against the registrations, and the flags against --help. Ship docs/ in the release archives, add a Documentation section to the release-please changelog, and point scripts/live-stack.sh at the moved live-test page. Note the docs layout in CLAUDE.md. Archive the add-user-documentation change. It has no spec deltas. Updates tailscale/tailscale#5967 Signed-off-by: Richard Shade --- .goreleaser.yaml | 1 + CLAUDE.md | 11 + README.md | 698 +----------------- docs/README.md | 42 ++ docs/contributing/live-notification-test.md | 66 ++ docs/contributing/releases.md | 44 ++ docs/explanation/alert-state.md | 208 ++++++ docs/explanation/failure-layers.md | 186 +++++ docs/explanation/incidents-and-queues.md | 207 ++++++ docs/explanation/monitor-ui-risks.md | 174 +++++ docs/how-to/acls.md | 69 ++ docs/how-to/deploy.md | 135 ++++ docs/how-to/heartbeat.md | 45 ++ docs/how-to/monitor-ui.md | 119 +++ docs/how-to/remote-subnets.md | 92 +++ docs/how-to/slack-sink.md | 89 +++ docs/how-to/troubleshoot.md | 40 + docs/how-to/webhook-sink.md | 137 ++++ docs/reference/checks.md | 125 ++++ docs/reference/cli.md | 161 ++++ docs/reference/config.md | 292 ++++++++ docs/reference/metrics.md | 97 +++ docs/reference/payload.md | 242 ++++++ docs/tutorial/first-monitor.md | 175 +++++ .../.openspec.yaml | 0 .../design.md | 11 +- .../proposal.md | 0 .../readme-map.md | 43 ++ .../tasks.md | 65 +- release-please-config.json | 24 +- scripts/live-stack.sh | 6 +- 31 files changed, 2894 insertions(+), 710 deletions(-) create mode 100644 docs/README.md create mode 100644 docs/contributing/live-notification-test.md create mode 100644 docs/contributing/releases.md create mode 100644 docs/explanation/alert-state.md create mode 100644 docs/explanation/failure-layers.md create mode 100644 docs/explanation/incidents-and-queues.md create mode 100644 docs/explanation/monitor-ui-risks.md create mode 100644 docs/how-to/acls.md create mode 100644 docs/how-to/deploy.md create mode 100644 docs/how-to/heartbeat.md create mode 100644 docs/how-to/monitor-ui.md create mode 100644 docs/how-to/remote-subnets.md create mode 100644 docs/how-to/slack-sink.md create mode 100644 docs/how-to/troubleshoot.md create mode 100644 docs/how-to/webhook-sink.md create mode 100644 docs/reference/checks.md create mode 100644 docs/reference/cli.md create mode 100644 docs/reference/config.md create mode 100644 docs/reference/metrics.md create mode 100644 docs/reference/payload.md create mode 100644 docs/tutorial/first-monitor.md rename openspec/changes/{add-user-documentation => archive/2026-09-29-add-user-documentation}/.openspec.yaml (100%) rename openspec/changes/{add-user-documentation => archive/2026-09-29-add-user-documentation}/design.md (88%) rename openspec/changes/{add-user-documentation => archive/2026-09-29-add-user-documentation}/proposal.md (100%) create mode 100644 openspec/changes/archive/2026-09-29-add-user-documentation/readme-map.md rename openspec/changes/{add-user-documentation => archive/2026-09-29-add-user-documentation}/tasks.md (53%) diff --git a/.goreleaser.yaml b/.goreleaser.yaml index e5d28c7..b12a8b2 100644 --- a/.goreleaser.yaml +++ b/.goreleaser.yaml @@ -28,6 +28,7 @@ archives: files: - LICENSE - README.md + - docs/**/* - examples/tail-watch.yaml checksum: diff --git a/CLAUDE.md b/CLAUDE.md index 334f204..01c222a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -63,6 +63,17 @@ authoritative reconciliation, and the removed dirty snapshot makes the notifier close the incident with `reason: removed`. +## Documentation + +- User docs live in `docs/`, split by Diataxis quadrant: `tutorial/`, `how-to/`, + `reference/`, `explanation/`, and `contributing/`. `docs/README.md` is the index and + `README.md` is only an overview with links. A fact lives in one page; link to it. +- When a config key, flag, metric, event field, or log message changes, update + `reference/config.md`, `reference/cli.md`, `reference/metrics.md`, + `reference/payload.md`, or `how-to/troubleshoot.md` in the same change. +- Check a config snippet with `tail-watch check-config`. `check-config` and `--once` + do not resolve `*_env` secrets, so it passes with them unset. + ## Commits - Follow the Tailscale commit style in `AGENTS.md`: a `dir: verb ...` diff --git a/README.md b/README.md index 56d91cb..c3738d6 100644 --- a/README.md +++ b/README.md @@ -6,694 +6,46 @@ a status page, Prometheus metrics, and a health check. It answers a different question from the online/offline webhook: not whether control thinks a node is connected, but whether traffic to that -node or to a subnet behind it actually gets through. +node or to a subnet behind it actually gets through. When a check fails, it +names the layer that broke and sends one alert per outage to Slack or a webhook. ## Requirements - A host running `tailscaled` in kernel mode. Userspace networking cannot carry ICMP to LAN addresses, so this build refuses it at startup. -- Tag the host `tag:monitor`. tail-watch logs a warning when that tag is - missing, because ACL grants are usually written for the tag. -- The process must be able to open the LocalAPI socket. On Linux that is - `/var/run/tailscale/tailscaled.sock` unless you pass `--socket`. +- Tag the host `tag:monitor`, and grant that tag access to what you monitor. +- A process that can open the LocalAPI socket, by default + `/var/run/tailscale/tailscaled.sock`. -ICMP needs either unprivileged ping (`net.ipv4.ping_group_range`) or -`CAP_NET_RAW`. If neither is available, ICMP checks fail with that -guidance and TCP, HTTP, and tsping checks still run. - -## Remote subnets and local LANs - -`network` selects the path a target is supposed to take: - -- `tailnet` (the default) means IP checks should use Tailscale. On Linux, - a target whose address is a literal non-Tailscale IP is treated as a - remote subnet. Startup fails until the host accepts subnet routes: - - ```bash - tailscale set --accept-routes - ``` - - An advertised route is not an installed route. After accepting routes, - check the route the host will actually use: - - ```bash - ip route get 10.20.0.1 - ``` - - The answer should go out the Tailscale interface (`tailscale0`), not a - local gateway. Repeat that for each remote subnet you monitor. - -- `local` means the host's ordinary route, including a LAN the collector - is plugged into at a site. These checks keep working when that site's - Tailscale router is down. They cannot use `tsping`. Put a `network: - local` device in its own target; do not point an HTTP check at a - different host than the target's `addr`. - -`examples/tail-watch.yaml` has both kinds of target. `nyc-core-switch` and -`nyc-nas` are remote subnet devices behind `nyc-subnet-router`. -`lab-printer` is on the collector's own LAN. - -## Build and run +## Quickstart ```bash mise install mise run build -./bin/tail-watch --config examples/tail-watch.yaml -``` - -Config path order is `--config`, then `TAIL_WATCH_CONFIG`, then -`/etc/tail-watch/tail-watch.yaml`. - -Validate a file without contacting tailscaled: - -```bash ./bin/tail-watch check-config --config examples/tail-watch.yaml +./bin/tail-watch --config examples/tail-watch.yaml ``` -Run every check once, print a line each, and exit non-zero on failure: - -```bash -./bin/tail-watch --once --config examples/tail-watch.yaml -``` - -`--version` prints the build version. `--log-level` (or `TAIL_WATCH_LOG_LEVEL`) -is `debug`, `info`, `warn`, or `error`. Logs are JSON on stderr. Debug -logs include every check result. - -| Flag | Env | Default | -| --- | --- | --- | -| `--config` | `TAIL_WATCH_CONFIG` | `/etc/tail-watch/tail-watch.yaml` | -| `--listen` | | `127.0.0.1:9109` or `collector.listen` | -| `--socket` | | platform LocalAPI socket | -| `--log-level` | `TAIL_WATCH_LOG_LEVEL` | `info` | -| `--once` | | run one pass and exit | -| `--version` | | print the version and exit | - -HTTP endpoints on the listen address: - -- `/` prober status page -- `/metrics` Prometheus text, `tailwatch_` series labeled `target`, `check`, - `site`, and `class`. The prefix has no hyphen because Prometheus metric - names cannot contain one. -- `/healthz` returns `200` and `ok` - -SIGINT and SIGTERM stop new work, cancel running checks, and exit 0. - -## Docker - -The image listens as user `tail-watch` (uid 65532) and does not embed a -Tailscale node. Use host networking so checks share the host routes, and -mount the LocalAPI socket. That socket has to be readable by uid 65532, -or pass a `--user` / `--group-add` that can read it. - -```bash -docker build -t tail-watch . -docker run --rm --network host \ - -v /var/run/tailscale/tailscaled.sock:/var/run/tailscale/tailscaled.sock \ - -v /etc/tail-watch/tail-watch.yaml:/etc/tail-watch/tail-watch.yaml:ro \ - tail-watch -``` - -### Docker Compose without a dedicated host - -`examples/compose.yaml` runs a kernel-mode `tailscaled` container and puts -tail-watch in its network namespace, so checks follow that node's `tailscale0` -routes, including accepted subnet routes. Mint an auth key tagged -`tag:monitor`: - -```bash -TS_AUTHKEY=tskey-auth-... docker compose -f examples/compose.yaml up -d -``` - -tail-watch starts once the node reports healthy. It reads -`examples/tail-watch.yaml` unless `TAIL_WATCH_CONFIG_FILE` names another file. The -`tailscale` service sets `net.ipv4.ping_group_range`, so the non-root tail-watch -image can send ICMP without `CAP_NET_RAW`. `network: local` targets reach -only what the container's Docker network can reach. - -`collector.state_dir` holds notification state and, with the UI on, the monitors -created there. Do not put secrets in the config file. - -## ACLs - -The collector sees only what policy allows. A missing grant shows up as -`no peer handles ; check grants to tag:monitor`. - -```jsonc -{ - "tagOwners": { "tag:monitor": ["autogroup:admin"] }, - "grants": [ - { - "src": ["tag:monitor"], - "dst": ["tag:router", "tag:server"], - "ip": ["*"] - }, - { - "src": ["tag:monitor"], - "dst": ["10.20.0.0/16"], - "ip": ["icmp:*", "tcp:22", "tcp:445", "tcp:443", "tcp:5001"] - } - ] -} -``` - -## Checks - -| Check | Passes when | -| --- | --- | -| `icmp` | one echo reply arrives before the timeout | -| `tcp:` | a TCP handshake completes; the socket is closed without sending | -| `http:` | status is 2xx or 3xx; optional `\|text` must appear in the first 1 MiB | -| `tsping` | tailscaled reports a disco pong for the target address | - -`http` does not follow redirects. `insecure_tls: true` on a target skips -certificate verification for devices that present a private certificate. -`tls` checks are rejected until phase 2. - -An explicit `parent` must name another target whose only check is `icmp` -or `tsping`. Parent cycles are rejected. That parent is for suppression -only. It does not change which tailnet node handles the address. - -## Failure layers - -A passing check has no layer. A failing check is labeled with one layer: - -| Layer | Meaning | -| --- | --- | -| `unrouted` | no peer owns or routes the address | -| `control` | the handling node is known offline | -| `tunnel` | the handling node is up, but disco fails | -| `service` | the device answered, or the target is a Tailscale node | -| `lan` | disco works and the device does not answer | - -`network: local` targets use only `service` and `lan`. They do not consult -tailnet presence or disco. - -## Alert state - -Each target is `UNKNOWN`, `UP`, `DOWN`, or `SUPPRESSED`. It leaves `UNKNOWN` -for `UP` after one success on every check, and that move is silent. Any check -that fails `fail_threshold` times in a row moves the target to `DOWN`. It -returns to `UP` after `recover_threshold` consecutive successes on every check -that failed, while its other checks are still passing. The defaults are 3 -failures and 2 successes. - -More than `flap_limit` state changes inside `flap_window` (defaults 4 and 15m) mark the -target flapping. tail-watch emits one flapping notice, then stays quiet until -the state has been stable for another `flap_window`. While that mark is set, -`tailwatch_target_state` is 4. - -A reachability parent (`icmp` or `tsping` only) that is `DOWN` or -`SUPPRESSED` moves its `UNKNOWN` and `UP` descendants to `SUPPRESSED` and -names them on its transition. A descendant that is already `DOWN` keeps its -own incident. A `control` or `tunnel` failure waits for the parent's detection -window (`interval × fail_threshold + timeout`) before it can page, so a dead -router produces one alert instead of one per device. `unrouted`, `lan`, and -`service` failures page immediately. When the parent recovers, or a topology -change leaves no down ancestor, tail-watch rechecks each released target. The -target then goes `UP` silently, or `DOWN` if the fresh checks keep failing. -If flap damping left an earlier `DOWN` unresolved, suppression preserves that -incident. The child sends its recovery after release when flap damping permits, -keeping the original outage start time; continued failures do not open a second -incident. - -With more than one collector, quorum stays in Alertmanager until collectors -share state: - -```yaml -- alert: TargetDownQuorum - expr: count by (target) (tailwatch_target_state == 2) >= 2 - for: 1m -``` - -That rule pages only when two collectors agree the target is down. - -For a remote LAN target, tail-watch learns the handling node with one disco -ping at startup, every 10 minutes, and when primary routes change. A failed -ping keeps the previous node unless the primary route changed, in which case -the route is used. An explicit `parent` still wins for suppression. - -The learned parent is a configured target only when that target's address is -the handling node and its only check is `tsping`. A service check on the same -node does not suppress the LAN devices behind it. Otherwise tail-watch adds -an implicit `tsping` target named for the node's short MagicDNS name, adding -the stable node ID and then a number if that name is taken. An idle implicit -target that is still down stays until it recovers. While a target has no -handling node, tail-watch logs this once, and again only if the grant returns -and is then lost: - -```text -target lab-pdu: no peer handles 10.99.0.5; check grants to tag:monitor -``` - -Prometheus series use the `tailwatch_` prefix: - -| Metric | Kind | -| --- | --- | -| `tailwatch_failure_layer_total{target,layer}` | counter, one increment per failure | -| `tailwatch_path{target,kind}` | gauge, `direct`, `derp`, and `peer_relay` | -| `tailwatch_target_parent{target,parent}` | gauge, 1 for the current parent | -| `tailwatch_target_state{target}` | gauge, 0 unknown through 4 flapping | - -## Monitor UI - -Off by default, tail-watch reads its monitors from the YAML file at startup and -never changes them. With the UI on, it also serves a page at `/ui/` and a JSON API -under `/api/v1/` for adding, editing, and deleting monitors while it runs, with no -restart and no lost grace timers. Monitors from the YAML file appear in the list as -read-only; only monitors created in the UI can be changed there. - -Turn it on with `--ui` or `collector.ui: true`. The flag wins, so `--ui=false` turns -off a config that enables it. The UI needs `collector.state_dir`, because monitors -created in the UI are kept in `state_dir/targets.json`. tail-watch exits at startup -without it, and ignores the UI under `--once`. - -```bash -tail-watch --config /etc/tail-watch/tail-watch.yaml --ui -``` - -`targets.json` uses the YAML field names and is written atomically with mode 0600. -The `defaults` block of the YAML file applies to UI monitors. A UI monitor is -accepted only if the YAML loader would accept the same target, and a monitor whose -name matches a YAML target is skipped at startup with a warning; the YAML target -wins. With the UI off, `targets.json` is left alone and its monitors do not run; a -startup warning says how many are inactive. - -An edit is a delete followed by an add. Editing or deleting a `DOWN` monitor closes -its open incident with `reason: removed`, and a new incident opens only if the new -definition fails `fail_threshold` checks. A monitor that other monitors name as -`parent` cannot be deleted until they are changed. Editing a parent briefly releases -its suppressed dependents, which can page while the parent is down. - -![The monitor list: YAML monitors are read-only, UI monitors have Edit and Delete, and monitors behind a down parent show SUPPRESSED.](docs/images/monitor-ui-list.png) - -The form shows validation errors from the same rules the YAML loader uses, next to -the field they belong to, and creates nothing until they are fixed. - -![The add monitor form with an invalid TCP port reported under Checks.](docs/images/monitor-ui-form-error.png) - -Deleting a `DOWN` monitor asks first, and says that its open incident closes. - -![The delete confirmation for a down monitor.](docs/images/monitor-ui-delete-down.png) - -### Reaching the UI - -The default `listen` address is `127.0.0.1:9109`, which only the host itself can -reach. To open the UI from another device on the tailnet, bind an address that -device can reach, such as the node's tailnet IP, and grant that device access to the -port. Grants cover the whole port, so they also expose `/metrics`. - -```jsonc -{ - "grants": [ - { - "src": ["autogroup:admin"], - "dst": ["tag:monitor"], - "ip": ["tcp:9109"] - } - ] -} -``` - -In the Compose layout, tail-watch shares the `tailscale` container's network -namespace, so every process in that namespace can reach the loopback listener. - -### Risks - -There is no login. Access control is the tailnet ACL and the listen address, and -tail-watch warns at startup when the address is not loopback. Whoever can reach the -port can make the collector probe any address it can route to. The API refuses -requests that are not `application/json` and requests whose `Origin` names another -host, which blocks cross-site form posts and cross-origin fetches. It does not stop -DNS rebinding, where the attacker's page and the `Host` header agree. Every write is -logged with the caller's address. - -## Notifications - -Every alert transition becomes a versioned JSON event that tail-watch delivers to -the sinks in the `sinks` list. A probe never waits on a sink: each sink has its own -worker and bounded queue, and a slow receiver cannot delay a check or another sink. -Sinks are `webhook` and `slack`. Secrets come only from `*_env` or `*_file` keys, -never from the YAML itself, and are read again on every send, so a rotated secret -needs no restart. Logs show a URL only as scheme and host (`https://hooks.slack.com/…`). - -| Event | When | -| --- | --- | -| `target.down` | a target goes `DOWN` and has no open incident | -| `target.recovered` | a target with an open incident reaches `UP` | -| `target.flapping` | a target crosses `flap_limit` | -| `collector.started` | once, after every target and check has a first result | -| `test` | only from `tail-watch notify-test` | - -A start with everything `UP` is silent apart from `collector.started`. A target -that is already failing at startup pages once. - -Severity is `info`, `warn`, or `page`. `target.down` uses the target's `severity` -(default `page`), a recovery uses the severity of the down it resolves, flapping is -`warn`, and `collector.started` and `test` are `info`. - -### Sink options - -```yaml -sinks: - - name: ops-webhook - type: webhook - url_env: OPS_WEBHOOK_URL # or url_file - secret_env: OPS_WEBHOOK_SECRET # HMAC key; without it requests are unsigned - headers_env: - Authorization: OPS_WEBHOOK_AUTH # header name: env var holding the value - min_severity: page # info < warn < page - events: [target.down, target.recovered] - max_age: 1h - queue: disk # needs collector.state_dir - queue_size: 1000 - batch: true - group_wait: 10s - max_batch: 100 - - name: ops-slack - type: slack - url_env: SLACK_WEBHOOK_URL - mention_on_page: "" - status_url: https://monitor-nyc.example.ts.net:9109 - group_wait: 10s - match: - sites: [nyc, sfo] # also targets (globs) and tags -``` - -Every sink accepts `events` (default all), `min_severity` (default `info`), `match` -with `sites`, `targets` and `tags`, `max_age` (default `1h`), `queue` (`memory` or -`disk`, default `memory`), and `queue_size` (default `1000`). An event reaches a sink -only when its type is listed, its severity is high enough, and every `match` key -matches. `tags` compares the Tailscale tags of the node the target resolves to. The -names `incidents`, `intake` and `state` are reserved because they appear as metric -labels. `tail-watch check-config` prints each sink's name, type, filters and queue mode, -never a secret. - -A recovery ignores the current filters: it goes to every sink that was selected for its -down, so a sink with `events: [target.down]` still hears when the target recovers. A -sink that never saw the down never sees the recovery. - -### Payload - -Every event is one JSON object. Within `version: 1` fields are only ever added. - -```json -{ - "version": 1, - "id": "evt_01JAB3K9T6M4X2Q8R7N5W0YZCD", - "type": "target.down", - "severity": "page", - "time": "2026-04-02T14:02:11Z", - "collector": { "name": "monitor-nyc", "tailscale_ip": "100.101.102.103" }, - "target": { "name": "nyc-nas", "addr": "10.20.0.50", "site": "nyc", "parent": "nyc-router" }, - "check": "tcp:445", - "layer": "service", - "error": "connection refused", - "since": "2026-04-02T14:00:41Z", - "failures": 3, - "path": { "kind": "direct", "endpoint": "203.0.113.5:41641", "latency_ms": 12 }, - "dependents": [] -} -``` - -`layer` is `unrouted`, `control`, `tunnel`, `lan`, or `service` and appears only on -`target.down` and `target.flapping`. `target.implicit: true` marks a router tail-watch -learned rather than one from your config. `target.recovered` adds `duration_seconds` -and `down_event_id`, and `reason: "removed"` when the target disappeared instead of -recovering. Fields that do not apply are omitted, not sent as `null`. `late: true` -marks an event first attempted more than a minute after its `time`; it keeps its -original `time`. - -`id` is `evt_` plus a 26 character Crockford base32 string that sorts by creation time. -It is the same across every retry and every sink. Delivery is at least once, so -receivers must deduplicate on `id` and must accept a `target.recovered` whose -`down_event_id` they never saw. - -### Webhook sink - -Each event is one `POST` with `Content-Type: application/json`, -`User-Agent: tail-watch/`, `Tail-Watch-Event-Id`, and any `headers_env` -headers. Redirects are not followed and each request times out after 10s. - -| Response | What tail-watch does | -| --- | --- | -| `2xx` | delivered | -| `408`, `5xx`, timeout, network error | retry with growing delay | -| `429` | retry, waiting at least `Retry-After` | -| anything else, including `3xx` and other `4xx` | drop, log once, count `permanent` | - -The delay is 1s, 2s, 4s and so on with 0.5–1.5× jitter, never more than 5 minutes. An -event older than `max_age` when an attempt is due is dropped and counted as `expired`. - -With a secret, each request carries `Tail-Watch-Signature: t=,v1=`, -where `v1` is the lowercase hex HMAC-SHA256 of `.` under the secret. A retry -sends identical body bytes with a fresh `t`. Verify it, and reject stale timestamps: - -```go -func verify(secret, header string, body []byte, tolerance time.Duration) bool { - var ts, sig string - for _, part := range strings.Split(header, ",") { - k, v, _ := strings.Cut(part, "=") - switch k { - case "t": - ts = v - case "v1": - sig = v - } - } - secs, err := strconv.ParseInt(ts, 10, 64) - if err != nil || time.Since(time.Unix(secs, 0)).Abs() > tolerance { - return false - } - mac := hmac.New(sha256.New, []byte(secret)) - fmt.Fprintf(mac, "%s.%s", ts, body) - return hmac.Equal([]byte(hex.EncodeToString(mac.Sum(nil))), []byte(sig)) -} -``` - -With `batch: true` the sink waits `group_wait` after the first queued event, or until -`max_batch` events are waiting, and sends -`{"version": 1, "batch_id": "bat_…", "events": [...]}` with `Tail-Watch-Batch-Id` in -place of `Tail-Watch-Event-Id`. The signature covers the whole body. A retry resends the -same bytes and `batch_id`; events that arrive meanwhile go into the next batch, and a -permanent failure or expiry drops the whole batch and counts every event in it. - -### Slack sink - -A Slack sink posts to an incoming-webhook URL. Each message carries Block Kit blocks -and a plain-text fallback: - -```text -:red_circle: DOWN nyc-nas tcp:445 (service) -addr 10.20.0.50 · site nyc · parent nyc-router · path direct 12 ms -error: connection refused -down since 14:00 UTC · 3 failures · collector monitor-nyc -``` - -A router learned at runtime shows `(learned)` after its name. A down with dependents adds -`and N targets behind it` and names up to 10. `mention_on_page` is prepended only to a -`target.down` of severity `page`, never to recoveries, flapping notices or warnings. With -`status_url` each target line links to `/#`. Events that arrive -within `group_wait` of the first become one message of at most 10 lines followed by -`and N more`, and the sink sends at most one message per second. Retry, `Retry-After`, -permanent-failure and `max_age` rules match the webhook sink. Slack sinks reject `batch`, -`max_batch`, `secret_*` and `headers_env`. - -### Queues, `state_dir` and open incidents - -`queue: memory` (the default) loses pending events when tail-watch restarts. `queue: disk` -needs `collector.state_dir` and keeps them. With `state_dir`, tail-watch keeps one file, -`notification-state.json`, holding open incidents, per-sink delivery state and the -pending events of disk sinks. Each change is written to a temporary file, fsynced, -renamed over the file, and the directory is fsynced, and only then can a worker send -the new events. The directory is created `0700` and the file `0600`; tail-watch warns if -the directory is readable by group or others. The file holds hostnames, addresses and -event payloads, never URLs, secrets or header values. If the file cannot be decoded, -startup fails with the path so you can repair it; tail-watch never silently forgets -open incidents. - -Open incidents are tracked even without `state_dir`. With it, a restart during an outage -does not page again, and the eventual recovery names the original `down_event_id`. A -configured target deleted from the config while an incident was open closes with -`reason: removed`. If the snapshot cannot be written, tail-watch keeps delivering from -memory, logs the loss of durability once, counts `tailwatch_notify_persist_errors_total`, -and retries the whole state every second. - -Delivery is at least once after a successful commit, and the limits are explicit and -counted: an event dropped because the queue is full, expired, permanently rejected, or -lost on a memory queue at restart shows up in `tailwatch_notify_dropped_total`. A `down` -dropped before any request was made takes its unsent `recovered` with it; if a request -was ever made, the `recovered` is still sent, because the receiver may have accepted a -request whose response was lost. Each sink keeps strict order. - -Disk commits are serialized. A slow disk can delay every durable enqueue and any worker -waiting to record an attempt, but it never delays a check, and the intake that receives -transitions from the alert state machine never blocks: if its 4096 slots fill, the -transition is dropped, counted as `dropped{sink="intake"}`, and the target's state is -reconciled within a second once there is room, so a lost recovery is still sent. - -| Metric | Kind | -| --- | --- | -| `tailwatch_notify_sent_total{sink,type}` | counter | -| `tailwatch_notify_failed_total{sink,reason}` | counter, reason `timeout`, `http_5xx`, `http_429`, `http_4xx_permanent`, `network` | -| `tailwatch_notify_dropped_total{sink,reason}` | counter, reason `queue_full`, `expired`, `permanent`, `restart` | -| `tailwatch_notify_persist_errors_total{sink}` | counter, sink is always `state` | -| `tailwatch_notify_queue_len{sink}` | gauge | -| `tailwatch_notify_latency_seconds{sink}` | histogram, event creation to confirmed delivery | -| `tailwatch_notify_batch_size{sink}` | histogram, batched webhook sinks only | -| `tailwatch_open_incidents` | gauge, when `state_dir` is set | -| `tailwatch_heartbeat_failures_total` | counter | - -Alert on `increase(tailwatch_notify_dropped_total[15m]) > 0` and on a -`tailwatch_notify_queue_len` that stays above zero: both mean an alert did not get out. - -### Watching the watcher - -A dead collector cannot report itself, so point `heartbeat` at a dead-man's-switch -service: - -```yaml -heartbeat: - url_env: HEARTBEAT_URL # or url_file - interval: 1m -``` - -tail-watch sends a `GET` every `interval`, starting once every target and check has a -first result, with a 10s timeout. A failed ping is logged, counted in -`tailwatch_heartbeat_failures_total`, and not retried before the next interval. When the -process exits the pings stop and the service alerts. - -### Trying it out - -```bash -./bin/tail-watch notify-test --config examples/tail-watch.yaml -./bin/tail-watch notify-test --sink ops-slack --timeout 10s -``` - -`notify-test` sends one `test` event through each sink's real signing, rendering and -retry code, bypassing filters, queues and `state_dir`. It prints one line per sink with -the HTTP status or error and exits 0 only when every selected sink answered `2xx` within -`--timeout` (default 30s). It works when tailscaled is down, using the OS hostname as -the collector name. Run it after every config change. - -### Egress path - -Sinks use the host's ordinary routing table, so the collector's node must reach them. -tail-watch warns at startup when an exit node is active, because notifications would -then leave through the exit node. A receiver on your own tailnet needs a grant from -`tag:monitor`: - -```jsonc -{ "src": ["tag:monitor"], "dst": ["tag:ops-receiver"], "ip": ["tcp:443"] } -``` - -### Live notification test - -`TestLiveNotifications` runs the whole notification path on a real tailnet. It is off -unless `TAIL_WATCH_LIVE_TAILNET=1`. It starts the collector and one fixture node per -check (`icmp`, `tsping`, `tcp`, `http`, and a LAN target behind a subnet router), fails and -heals each one, restarts the collector mid-outage with `queue: disk`, and runs -`notify-test`. A signature-verifying webhook receiver in the test process checks every -event and heartbeat. The compose fixtures are in -`cmd/tail-watch/testdata/compose.live-notify.yaml`. Devices are ephemeral, named -`tail-watch-live-notify-*`, and deleted afterwards. - -```bash -TAIL_WATCH_LIVE_TAILNET=1 TAIL_WATCH_LIVE_TAG= \ - TAIL_WATCH_SLACK_PULUMI_ENV=/tail-watch-live/slack \ - go test -count=1 -timeout 18m -run TestLiveNotifications -v ./cmd/tail-watch -``` - -With `TAIL_WATCH_LIVE_TAG` unset the collector joins as `tag:monitor`, so the tailnet -policy must own that tag and grant it the fixtures: merge -`cmd/tail-watch/testdata/snippet.hujson`. The grants that matter are `tcp:9000` and -`tcp:8080` to members (the `tcp` and `http` fixtures) and `icmp` to `10.10.0.0/16` (the LAN -target). Without the two TCP grants those fixtures are down from the start, with -`i/o timeout` errors because the packets are dropped. Setting `TAIL_WATCH_LIVE_TAG=` to -empty runs the collector as an untagged member node, which needs no policy change. - -Tailnet credentials come from the `tailscale-phase-2/tailscale` pulumi env -(`TAIL_WATCH_PULUMI_ENV` overrides it). Slack is optional: without a webhook the test -runs the webhook sink only. Slack is one-way, so the test checks tail-watch's own sent and -failed counters for that sink; look in the channel for the messages. -`TAIL_WATCH_LIVE_MENTION` sets `mention_on_page` for the Slack sink. - -The Slack sink uses a memory queue, so an event still inside its `group_wait` window when -the test restarts the collector is lost and counted as `dropped{reason="restart"}`. The -test accepts that reason and fails on any other drop or failure. The webhook sink uses -`queue: disk` and must survive the restart without paging again. - -To bring up only the tailnet side, without the collector process or any scenario, use -`scripts/live-stack.sh up`. It mints ephemeral keys from the pulumi env, starts the -collector's tailscale node and the five fixture nodes, approves the LAN route, and prints -each node's address. `scripts/live-stack.sh down` removes the containers and deletes -that stack's devices. The webhook can live in a pulumi env such as -`richard-shade-gmail-com/tail-watch-live/slack`; pass its name as -`TAIL_WATCH_SLACK_PULUMI_ENV`. To rotate it, run -`pulumi env set environmentVariables.SLACK_WEBHOOK_URL '' --secret`. - -Set `queue: disk` on the Slack sink if an alert must survive a collector restart. The -default `queue: memory` loses events still waiting in the group window. - -#### Getting a Slack webhook URL - -tail-watch posts to an incoming webhook URL. It does not use bot or app-level tokens -(`xoxb-`, `xapp-`), and a bot token cannot produce one. Slack's own guide, -[Sending messages using incoming webhooks](https://docs.slack.dev/messaging/sending-messages-using-incoming-webhooks/), -is the reference; its steps are: - -1. Create an app at , or open an existing one. -2. In the app's settings sidebar choose **Incoming Webhooks** and switch - **Activate Incoming Webhooks** on. -3. Click **Add New Webhook to Workspace**, pick the channel, and click **Allow**. -4. Copy the URL from **Webhook URLs for Your Workspace**. It starts with - `https://hooks.slack.com/services/`. - -Slack has also documented the `incoming-webhook` scope, which produces a webhook during -an OAuth install and shows a channel picker. If your app's sidebar has no -**Incoming Webhooks** entry, that is the alternative to try. The settings pages change -over time, so if these steps do not match what you see, treat this section as out of date -and check Slack's guide. - -Check the URL with -`curl -X POST -H 'Content-type: application/json' --data '{"text":"hello"}' "$SLACK_WEBHOOK_URL"`, -which answers `ok`. Store it as a secret, not in a file: - -```bash -pulumi env init /tail-watch-live/slack -pulumi env set /tail-watch-live/slack \ - environmentVariables.SLACK_WEBHOOK_URL 'https://hooks.slack.com/services/...' --secret -``` - -## Development and releases - -`mise run check` builds, tests, and lints with the pinned tools. The lint -configuration is based on -[maratori/golangci-lint-config](https://github.com/maratori/golangci-lint-config), -with this module's import prefix. CI also runs race tests and builds release -archives for Linux, macOS, and Windows on amd64 and arm64. +The status page is at `http://127.0.0.1:9109/`, and metrics are at `/metrics`. To run +without a dedicated host, or to receive your first alert, follow the +[tutorial](docs/tutorial/first-monitor.md). -Use Conventional Commits in squash-merge titles (`fix: ...`, `feat: ...`, -or `feat!: ...` for breaking changes). After checks pass on `main`, -release-please opens or updates a release PR containing the version and -changelog. Merge that PR to create a GitHub release; GoReleaser then attaches -the archives and `checksums.txt`. Each archive includes `tail-watch`, the license, -this README, and `examples/tail-watch.yaml`. The binary's `--version` reports the -release version. +## Documentation -Repository setup: in **Settings → Actions → General → Workflow permissions**, -enable **Allow GitHub Actions to create and approve pull requests**. -The workflow uses the built-in `GITHUB_TOKEN`. PRs created with that token -do not automatically start CI; close and reopen the release PR to trigger -its checks. Publishing runs in the release workflow itself because tags -created with that token also do not trigger a separate workflow. +The [documentation index](docs/README.md) lists every page. The main ones: -To validate packaging locally without publishing: +- Learn: [Set up your first monitor](docs/tutorial/first-monitor.md) +- Do: [Deploy](docs/how-to/deploy.md), [write the ACLs](docs/how-to/acls.md), + [send alerts to Slack](docs/how-to/slack-sink.md), and + [troubleshoot](docs/how-to/troubleshoot.md) +- Look up: [configuration](docs/reference/config.md), + [command line](docs/reference/cli.md), [metrics](docs/reference/metrics.md), and + [event payload](docs/reference/payload.md) +- Understand: [failure layers](docs/explanation/failure-layers.md) and + [alert state](docs/explanation/alert-state.md) -```bash -mise run release-check -``` +## Contributing -Snapshot archives appear in `dist/`. If publishing fails after the release -has been created, rerun the failed job or manually run the **Release** workflow -with its existing tag (for example, `v0.1.0`). +`mise run check` builds, tests, and lints with the pinned tools. See +[Development and releases](docs/contributing/releases.md) and the +[live notification test](docs/contributing/live-notification-test.md). diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..492977e --- /dev/null +++ b/docs/README.md @@ -0,0 +1,42 @@ +# tail-watch documentation + +tail-watch checks whether traffic to your tailnet nodes and the subnets behind them +gets through, and tells you which layer failed. These pages are grouped by what you +need to do. + +## Tutorial + +- [Set up your first monitor](tutorial/first-monitor.md): from the example Compose + file to an alert in Slack. + +## How-to guides + +- [Deploy tail-watch](how-to/deploy.md): binary, Docker, and Compose without a + dedicated host. +- [Write the ACLs](how-to/acls.md): grants for `tag:monitor`. +- [Monitor remote subnets and local LANs](how-to/remote-subnets.md) +- [Manage monitors in the UI](how-to/monitor-ui.md) +- [Send alerts to Slack](how-to/slack-sink.md) +- [Send alerts to a webhook](how-to/webhook-sink.md): signing, batching, retries. +- [Watch the watcher with a heartbeat](how-to/heartbeat.md) +- [Troubleshoot](how-to/troubleshoot.md) + +## Reference + +- [Configuration](reference/config.md) +- [Command line](reference/cli.md) +- [Checks](reference/checks.md) +- [Metrics](reference/metrics.md) +- [Event payload](reference/payload.md) + +## Explanation + +- [Failure layers](explanation/failure-layers.md) +- [Alert state](explanation/alert-state.md) +- [Incidents and queues](explanation/incidents-and-queues.md) +- [Monitor UI risks](explanation/monitor-ui-risks.md) + +## Contributing + +- [Live notification test](contributing/live-notification-test.md) +- [Development and releases](contributing/releases.md) diff --git a/docs/contributing/live-notification-test.md b/docs/contributing/live-notification-test.md new file mode 100644 index 0000000..8174357 --- /dev/null +++ b/docs/contributing/live-notification-test.md @@ -0,0 +1,66 @@ +# Live notification test + +This page is for contributors who need to run the notification path against a +real tailnet. + +`TestLiveNotifications` runs the whole notification path on a real tailnet. It is off +unless `TAIL_WATCH_LIVE_TAILNET=1`. It starts the collector and one fixture node per +check (`icmp`, `tsping`, `tcp`, `http`, and a LAN target behind a subnet router), fails and +heals each one, restarts the collector mid-outage with `queue: disk`, and runs +`notify-test`. A signature-verifying webhook receiver in the test process checks every +event and heartbeat. The compose fixtures are in +`cmd/tail-watch/testdata/compose.live-notify.yaml`. Devices are ephemeral, named +`tail-watch-live-notify-*`, and deleted afterwards. + +```bash +TAIL_WATCH_LIVE_TAILNET=1 TAIL_WATCH_LIVE_TAG= \ + TAIL_WATCH_SLACK_PULUMI_ENV=/tail-watch-live/slack \ + go test -count=1 -timeout 18m -run TestLiveNotifications -v ./cmd/tail-watch +``` + +## Tailnet policy and tags + +With `TAIL_WATCH_LIVE_TAG` unset the collector joins as `tag:monitor`, so the tailnet +policy must own that tag and grant it the fixtures: merge +`cmd/tail-watch/testdata/snippet.hujson`. The grants that matter are `tcp:9000` and +`tcp:8080` to members (the `tcp` and `http` fixtures) and `icmp` to `10.10.0.0/16` (the LAN +target). Without the two TCP grants those fixtures are down from the start, with +`i/o timeout` errors because the packets are dropped. Setting `TAIL_WATCH_LIVE_TAG=` to +empty runs the collector as an untagged member node, which needs no policy change. + +## Credentials + +Tailnet credentials come from the `tailscale-phase-2/tailscale` pulumi env +(`TAIL_WATCH_PULUMI_ENV` overrides it). Slack is optional: without a webhook the test +runs the webhook sink only. Slack is one-way, so the test checks tail-watch's own sent and +failed counters for that sink; look in the channel for the messages. +`TAIL_WATCH_LIVE_MENTION` sets `mention_on_page` for the Slack sink. + +The test reads the webhook from `SLACK_WEBHOOK_URL`, or from the pulumi env named by +`TAIL_WATCH_SLACK_PULUMI_ENV`. Never write it to a file or a log. To get a webhook URL, +see the [Slack sink how-to](../how-to/slack-sink.md). + +The webhook can live in a pulumi env such as `/tail-watch-live/slack`; pass its name +as `TAIL_WATCH_SLACK_PULUMI_ENV`. To rotate it, run: + +```bash +pulumi env set environmentVariables.SLACK_WEBHOOK_URL '' --secret +``` + +## Restart behavior + +The Slack sink uses a memory queue, so an event still inside its `group_wait` window when +the test restarts the collector is lost and counted as `dropped{reason="restart"}`. The +test accepts that reason and fails on any other drop or failure. The webhook sink uses +`queue: disk` and must survive the restart without paging again. + +Set `queue: disk` on the Slack sink if an alert must survive a collector restart. The +default `queue: memory` loses events still waiting in the group window. + +## Bringing up only the tailnet side + +To bring up only the tailnet side, without the collector process or any scenario, use +`scripts/live-stack.sh up`. It mints ephemeral keys from the pulumi env, starts the +collector's tailscale node, the four fixture nodes, and the LAN router, approves the LAN +route, and prints each node's address. `scripts/live-stack.sh down` removes the containers +and deletes that stack's devices. diff --git a/docs/contributing/releases.md b/docs/contributing/releases.md new file mode 100644 index 0000000..3e309ed --- /dev/null +++ b/docs/contributing/releases.md @@ -0,0 +1,44 @@ +# Development and releases + +This page is for contributors who build, check, and release tail-watch. + +`mise run check` builds, tests, and lints with the pinned tools. The lint +configuration is based on +[maratori/golangci-lint-config](https://github.com/maratori/golangci-lint-config), +with this module's import prefix. CI also runs race tests and builds release +archives for Linux, macOS, and Windows on amd64 and arm64. + +## Releasing + +Use Conventional Commits in squash-merge titles (`fix: ...`, `feat: ...`, +or `feat!: ...` for breaking changes). After checks pass on `main`, +release-please opens or updates a release PR containing the version and +changelog. Merge that PR to create a GitHub release; GoReleaser then attaches +the archives and `checksums.txt`. Each archive includes `tail-watch`, the license, +the README, the `docs/` tree, and `examples/tail-watch.yaml`. The binary's +`--version` reports the release version. + +release-please owns `CHANGELOG.md`; do not edit it by hand. It lists `feat`, `fix`, +`perf`, `revert` and `docs` commits, so a documentation change appears under +**Documentation** when its squash-merge title starts with `docs:`. + +## Repository setup + +In **Settings → Actions → General → Workflow permissions**, +enable **Allow GitHub Actions to create and approve pull requests**. +The workflow uses the built-in `GITHUB_TOKEN`. PRs created with that token +do not automatically start CI; close and reopen the release PR to trigger +its checks. Publishing runs in the release workflow itself because tags +created with that token also do not trigger a separate workflow. + +## Validating packaging locally + +To validate packaging locally without publishing: + +```bash +mise run release-check +``` + +Snapshot archives appear in `dist/`. If publishing fails after the release +has been created, rerun the failed job or manually run the **Release** workflow +with its existing tag (for example, `v0.1.0`). diff --git a/docs/explanation/alert-state.md b/docs/explanation/alert-state.md new file mode 100644 index 0000000..51f379b --- /dev/null +++ b/docs/explanation/alert-state.md @@ -0,0 +1,208 @@ +# Alert state + +A check result is a fact about one moment. An alert is a decision that a +human should look. Between them sits a state machine whose job is to turn a +noisy stream of results into as few notifications as possible without hiding +a real outage. This page explains what that machine does and why it is +shaped the way it is, so you can predict its behavior and tune it. + +The keys named here are set under `defaults:` or on each target. Their +defaults and syntax are in the [configuration reference](../reference/config.md). +The state values as exported are in the [metrics reference](../reference/metrics.md). + +## One state per target + +Every target is in exactly one of four states: + +| State | Meaning | +| --- | --- | +| `UNKNOWN` | No verdict yet since the process started. | +| `UP` | Every check has been passing. | +| `DOWN` | A check failed `fail_threshold` times in a row. | +| `SUPPRESSED` | A parent that the target depends on is down. | + +The states are held in memory only. A restart begins again at `UNKNOWN`, and +the durable part of an outage is the incident record described in +[Incidents and queues](incidents-and-queues.md). That split is deliberate: +the state machine can stay simple and cheap because a restart costs a few +check cycles of "we do not know yet", not a lost incident. + +The gauge `tailwatch_target_state` adds a fifth value, 4, for a target that is +flapping. Flapping is a mark on top of the state and not a state of its own, +and it takes precedence in the gauge while it is set. + +## Why thresholds, and why they are per check + +One lost packet must not page anyone. A target therefore goes `DOWN` only +after `fail_threshold` consecutive failures of a single check (default 3), and +a success resets that check's count. With the default 30 second interval that +is roughly a minute and a half from first failure to page, and the time of the +first failure in the run is what the alert reports as `since`, so the outage +start is not pushed back by the waiting. + +Counters are kept per target and check pair, not per target. Consider a target +with `icmp` and `tcp:445`. If the counter were shared, a failing port +interleaved with a passing ping would never accumulate three failures, and a +real fault would be hidden by an unrelated healthy check. Per check counters +mean that any one check reaching the threshold is enough. + +Going down is quick and coming back is slow on purpose. A target returns to +`UP` only after `recover_threshold` consecutive successes (default 2) on every +check that failed while it was down, and only when none of its other checks is +currently failing. If `tcp:445` recovers while `icmp` has started failing, the +target stays `DOWN`. The asymmetry favors a recovery message that stays true +over one that is retracted a minute later. + +### Starting up + +A fresh process has no history, so a target leaves `UNKNOWN` in one of two +ways. If every check succeeds once, the target becomes `UP` and says nothing. +Startup is not news, and a restart during a healthy period would otherwise +produce a burst of "recovered" messages. If a check reaches `fail_threshold` +first, the target goes straight from `UNKNOWN` to `DOWN` and pages once, which +is why a target that is already broken when you start the collector is +reported. + +A separate `collector.started` event announces that every target has produced a +first result, so a silent start is still distinguishable from a dead collector. + +## Flap damping + +A target that alternates between failing and passing can defeat thresholds: +each cycle is long enough to page and short enough to be gone by the time +someone looks. `flap_limit` and `flap_window` (defaults 4 and 15 minutes) +bound the damage. + +The machine counts `UP` and `DOWN` changes inside a sliding window. When a +change would push the count past `flap_limit`, it emits one flapping notice in +place of that change's normal notification, and then goes quiet. While the mark +is set the target's state is still tracked, but nothing is emitted. The mark +clears when the target has been stable for a full `flap_window`. If the +settled state differs from the last state that was reported, one ordinary +notification catches the receiver up. If it matches, nothing is sent. + +Two consequences follow from that design. + +- The receiver's picture stays coherent. A target that flapped and settled + `UP` after being reported `DOWN` still gets its recovery, so no incident is + left open on the receiving side. +- Damping trades latency for quiet. A real outage that begins during a flap + will not page until the target has been stable, or until the flap window + runs out. If you cannot accept that, raise `flap_limit` or shrink + `flap_window` for that target rather than turning damping off elsewhere. + +## Parent suppression + +The most common cause of a wall of alerts on a tailnet is not forty broken +devices. It is one broken subnet router with forty devices behind it. Without +suppression, each device would page separately and the one useful alert, the +router, would be buried among its symptoms. + +Each target can have a parent: a target that must work for it to be +reachable. The parent comes from an explicit `parent` key, or from the +dependency learning described in [Failure layers](failure-layers.md). Two +rules keep the mechanism honest. + +- **A parent is a reachability target.** It may have only an `icmp` or a + `tsping` check. A service check on the same node is not a parent, so a + failed web interface cannot silence the devices behind that router. +- **A down parent means `DOWN` or `SUPPRESSED`.** Suppression is transitive. If + a switch behind a router is itself suppressed, its own dependents are too. + +When a parent goes down, every dependent in `UNKNOWN` or `UP` moves to +`SUPPRESSED` at the same instant, silently. The parent's own notification names +them in `dependents`, so one message carries the whole blast radius. A dependent +that is already `DOWN` keeps its incident, because it was reported on its own +merits and dropping it would leave an alert with no resolution. Checks on +suppressed targets keep running, which is what lets the machine notice the +moment the situation changes. + +### The detection window + +Suppression has a race. The device's checks and the router's checks run on +independent schedules, so a device can reach its threshold a few seconds before +the router does. Paging on the device then produces exactly the duplicate that +suppression was meant to prevent. + +The answer is a short wait. When a target with a parent reaches its threshold +on a `control` or `tunnel` failure and the parent is not yet down, the target +does not go `DOWN` immediately. It waits up to the parent's detection window, +which is the parent's `interval` times its `fail_threshold`, plus its +`timeout`. That is the longest the parent could take to reach its own verdict +if it started failing at the same moment. If the parent goes down in that +time, the target is suppressed. If not, the target goes `DOWN` at the end of +the wait, provided its check is still failing, and the alert keeps the +original layer. + +Only `control` and `tunnel` failures wait, because only those layers say "the +path to the parent's side is broken". A `lan`, `service` or `unrouted` failure +pages immediately. By definition the tunnel to the router works for a `lan` +or `service` failure, so the router is not the cause. For `unrouted`, no router +exists to blame. See [Failure layers](failure-layers.md) for how those labels are +chosen. + +### Release + +When the parent recovers, is removed, or a topology change leaves a target +with no down ancestor, tail-watch does not wait for the next scheduled check. +It re-runs every check of each released target promptly, at most eight at a +time, so a recovered router does not leave forty devices in limbo for a full +interval. A target whose fresh checks pass returns to `UP` silently. One whose +checks keep failing counts `fail_threshold` failures from the release, then +goes `DOWN` with its own alert. + +The counters are reset on release, and results from checks that began before +it are ignored. Without that, a timeout that started while the router was +down would be blamed on the device as a fresh failure. A failover in a +high-availability pair works the same way: if learning moves a suppressed +device from a dead router to a live one, the device is released at once, even +though the dead router never recovers. + +One subtle case exists for incidents that were already reported. If flap +damping left a `DOWN` unresolved before the target was suppressed, that +incident is preserved through suppression and release. The target sends its +recovery once damping permits, with the original outage start time, and +continued failures do not open a second incident. + +## Quorum across collectors + +One collector sees the tailnet from one place. If its uplink fails it will +report every target down, and one unhealthy collector cannot tell you whether +the problem is the target or itself. State is not shared between collectors, so +tail-watch does not vote. + +Quorum is left to Alertmanager, which already knows how to combine sources. +Each collector exports `tailwatch_target_state`, and a rule can require +agreement: + +```yaml +- alert: TargetDownQuorum + expr: count by (target) (tailwatch_target_state == 2) >= 2 + for: 1m +``` + +That pages only when two collectors agree the target is `DOWN`. The rejected +alternative was a gossip protocol among collectors. It would add a network +dependency between the very machines you are trying to make trustworthy, to +solve a problem Alertmanager users have already solved. + +## Choosing values + +The defaults suit a 30 second interval and mostly stable devices. A few +relationships help when you tune them: + +- Time to page is about `interval` times `fail_threshold`, plus `timeout` for + the last attempt. Shortening the interval shortens the detection window of + every parent that uses it. +- A larger `recover_threshold` makes recovery slower and flapping less likely. +- Raising `flap_limit` makes a target flap less readily. A larger + `flap_window` keeps it quiet for longer after it settles. +- `severity` on a target controls how loud the resulting events are, not + whether the state machine reaches `DOWN`. See + [Incidents and queues](incidents-and-queues.md) for routing. + +## Related pages + +- [Configuration reference](../reference/config.md) for every key named here. +- [Metrics reference](../reference/metrics.md) for `tailwatch_target_state`. +- [Failure layers](failure-layers.md) for the labels that drive the wait. diff --git a/docs/explanation/failure-layers.md b/docs/explanation/failure-layers.md new file mode 100644 index 0000000..1404cad --- /dev/null +++ b/docs/explanation/failure-layers.md @@ -0,0 +1,186 @@ +# Failure layers + +A monitoring alert that says only "nyc-nas is down" leaves the hard part to +you. On a tailnet with subnet routers, a device can be unreachable because +the control plane thinks its router is offline, because the WireGuard tunnel +to the router is broken, because the LAN behind the router lost the device, +or because the device is fine and one service on it is not. Each cause has a +different owner and a different fix. tail-watch labels every failed check +with one of five layers so the alert starts the diagnosis instead of +postponing it. + +This page explains how the label is chosen, why it is chosen in that order, +and where it can mislead you. The label values and metric are listed in the +[metrics reference](../reference/metrics.md) and the +[event payload reference](../reference/payload.md). + +## The five layers + +A passing check has no layer. A failing check has exactly one, and the value +appears as the `layer` label on `tailwatch_failure_layer_total` and as the +`layer` field on `target.down` and `target.flapping` events. + +| Layer | What it says | +| --- | --- | +| `unrouted` | No peer owns or routes the address. | +| `control` | Tailscale's control plane reports the handling node offline. | +| `tunnel` | The handling node looks online, but a disco ping to it fails. | +| `service` | The device demonstrably answered, or the target is a node. | +| `lan` | The tunnel works, but the device behind the node does not answer. | + +The "handling node" is the tailnet node that carries traffic for the target. +For a target that is itself a Tailscale node, that is the node. For a device +on a subnet, it is the subnet router that advertises the route. Nothing else +in the model makes sense without this idea: tail-watch never reasons about the +device directly until it has established that the path to the handling node is +sound. + +## Deciding from the bottom up + +The classifier walks the stack from the lowest layer to the highest and stops +at the first layer that explains the failure. The order matters, because a +failure low in the stack makes every higher check fail too. If the router is +offline, the device's TCP port also refuses to connect, and reporting that as +a service problem would send you to the wrong team. + +1. If no peer owns the address or advertises a route that covers it, nothing + could ever have carried the traffic. That is `unrouted`. It usually points + at a missing route approval or a missing grant rather than at an outage, + which is why tail-watch also logs a hint about grants to `tag:monitor` the + first time it notices (see the + [ACL how-to](../how-to/acls.md)). +2. If the handling node exists and the control plane says it is offline, that + is `control`. The claim is cheap to check and it is authoritative about + what Tailscale believes, so it comes before any active probe. +3. If the node is not known to be offline, tail-watch sends a disco ping to + it. A failed ping is `tunnel`: the coordination server thinks the node is + fine, but no WireGuard path works. Typical causes are a firewall that + blocks UDP, a node whose tailscaled has hung, or a path that flaps. +4. If the tunnel works, the failure is on the far side of the handling node. + When the device gave evidence that it is alive, the layer is `service`. A + refused TCP connection, an HTTP status or body mismatch, and a TLS error + are all answers, just not the answer the check wanted. A timed-out check + is ambiguous, so tail-watch also sends one ICMP echo to the device. If + the echo succeeds, the host is up and the layer is `service`. +5. If the tunnel works and the device gives no sign of life, the layer is + `lan`. The path into the site is fine and the device is not on it. + +A target that is itself a Tailscale node never gets `lan`, because there is no +LAN behind it. Where the walk would otherwise say `lan`, it says `service`. + +### Failed `tsping` checks + +A `tsping` check is a disco ping, which is the same probe the classifier would +send in step 3. Rather than probing twice, tail-watch treats the failed check +as that probe. The result is `control` when presence shows the node offline +and `tunnel` otherwise. + +### Local targets + +A target with `network: local` is not reached through the tailnet, so there is +no handling node to consult. Presence, routes and disco do not apply, and the +only layers available are `service` (the device answered in some way) and +`lan` (it did not). + +## What the classifier knows, and what it does not + +The classifier uses the tailnet's own view of itself. That view comes from +the local tailscaled over its IPN bus, which reports which peers exist, +whether the control plane counts them online, and which routes they serve. +tail-watch keeps a live copy of it. + +The copy has one property that shapes the layers: it distinguishes "known +offline" from "unknown". When the bus connection is down, or has not yet +delivered its first snapshot, every peer is unknown. The classifier does not +guess `control` from missing data. It carries on to the disco ping and +decides between `tunnel` and the layers above it. So a tailscaled restart +degrades precision (a genuinely offline router shows up as `tunnel` for a +short time) but never invents a control-plane claim. + +Presence never raises an alert by itself. A router flipping offline while +every check still passes produces no failure and no layer. The layer only +describes a check that has already failed. + +## Learning what a device depends on + +`unrouted`, `control` and `tunnel` all depend on knowing the handling node, +and you should not have to declare it by hand for every device. For each +remote LAN target, tail-watch sends one disco ping at startup, every 10 +minutes, and when primary routes change. The pong names the node that +actually carried it. That node becomes the target's parent for classification +and for suppression (see [Alert state](alert-state.md)). + +The design choices behind that: + +- **The pong is authoritative, routes are the fallback.** In a + high-availability pair, the route table names the primary, but the node that + answers is the one that matters. The learned parent follows whichever router + the pong reports, so a failover changes the parent without configuration. + When the ping fails, a changed primary route is used, and otherwise the + previously learned node is kept, so a router that is down at the moment of + learning is still the parent you would want to blame. +- **A parent must be a reachability target.** A learned parent maps only to a + configured target whose sole check is `tsping`. A router that also runs a + monitored web interface has a `tsping` target and a separate service target, + and only the first can suppress devices. Otherwise a broken admin page on a + router would silence every device behind it. When no such target exists, + tail-watch creates an implicit `tsping` target named after the node and + removes it when nothing depends on it, although an implicit target that is + still down stays until it recovers, so its incident is not orphaned. +- **A configured `parent` always wins.** It is for suppression only. It does + not change which node the classifier treats as the handler, because the + handler is a fact about routing and the parent is a statement about blame. + A device behind a second switch can name that switch as its parent without + pretending that the switch carries its tailnet traffic. +- **Cycles are rejected.** A learned edge that would close a loop with the + configured graph is refused and the previous graph stays. + +If no peer handles an address at all, tail-watch logs the missing grant once, +and logs again only if the address resolves and is later lost, so a permanent +misconfiguration does not fill the log. + +## Path kind is not a layer + +For every `tsping` result, tail-watch also records how the pong arrived: +`direct`, `derp`, or `peer_relay`, on `tailwatch_path{target,kind}`. A path +change never fails a check and never changes a layer. A node that falls back +to DERP is slower, not broken, and an alert would be noise. The metric exists +so that you can graph the slowdown and correlate it with a later `tunnel` +failure. + +## Cost of supplemental probes + +Step 3 and step 4 can need an extra disco ping or ICMP echo. When forty +devices behind one router fail in the same second, sending forty pings +to the same router would add load exactly when the network is in trouble and +would tell you nothing new. Supplemental results are cached per handling node +(for disco) and per address (for ICMP), reused for 10 seconds or the target's +interval, whichever is shorter, and at most one probe per key is in flight; +concurrent callers wait for it and share the result. A fresh passing result +from the target's own `icmp` or `tsping` check replaces a supplemental probe. +These probes are not checks, so they do not appear on the status page or in +the prober result metrics. + +## Limits of the labels + +- The layer is a best explanation from the vantage point of one collector. A + `tunnel` verdict means that this collector cannot reach the node, not that + nobody can. That is one reason to run more than one collector (see the + quorum rule in [Alert state](alert-state.md)). +- `service` means "something answered", not "the service is healthy". An ICMP + reply with a timed-out TCP check is `service`, on the reasoning that a + live host with a dead port is a service problem for the owner of the + service. +- `lan` is a conclusion by elimination. tail-watch cannot see the LAN. It + sees that the tunnel works and the device is silent, and that combination + is the definition. +- The layer on an alert is the layer of the check that tripped the threshold. + A later failure of a different check does not rewrite it. + +## Related pages + +- [Checks reference](../reference/checks.md) for what each check tests. +- [Alert state](alert-state.md) for how layers decide whether an alert is held + back behind a parent. +- [Troubleshooting](../how-to/troubleshoot.md) for what to do about each + layer. diff --git a/docs/explanation/incidents-and-queues.md b/docs/explanation/incidents-and-queues.md new file mode 100644 index 0000000..3d9e400 --- /dev/null +++ b/docs/explanation/incidents-and-queues.md @@ -0,0 +1,207 @@ +# Incidents and queues + +A notification system has two promises that pull against each other. An alert +must not be lost, and a slow or dead receiver must never stop the monitoring +that produces alerts. This page explains how tail-watch keeps both promises, +what it does when it cannot, and the limits it tells you about instead of +hiding. + +The sink keys mentioned here are in the +[configuration reference](../reference/config.md). The metrics are in the +[metrics reference](../reference/metrics.md), and the fields on the wire are +in the [payload reference](../reference/payload.md). Setup instructions are in +the [webhook](../how-to/webhook-sink.md) and [Slack](../how-to/slack-sink.md) +guides. + +## Events and incidents + +The alert state machine emits transitions. The notifier turns them into events +(`target.down`, `target.recovered`, `target.flapping`, and the rest) and +groups the down and recovery of one outage into an incident. + +An incident is the memory that pairs a recovery with the down it resolves. It +records the tripping check, the severity, the start time and the id of the +`target.down` event. A recovery carries that id as `down_event_id`, which lets +a receiver close the exact alert it opened, for example an incident in a paging tool, +instead of guessing by target name. + +There is at most one open incident per target. If a target fails again while +its recovery is still being delivered, a new incident opens with a new id +queued behind the old recovery. Incidents exist even without `state_dir`, +which keeps duplicate downs from being sent while one is open. + +## The path from a check to a receiver + +The design goal is that nothing on the way from a probe to a network socket +can make a probe wait. + +1. The alert state machine hands each transition to the notifier through an + intake with 4096 slots. That call only writes to the channel. It does no + disk or network work, so a check can never block on notification. +2. The notifier maps the transition to an event, updates the incident, decides + which sinks should receive it, and commits the result. +3. Each sink has its own worker and its own bounded queue. A receiver that + takes ten seconds per request delays only its own sink, not another sink + and not a check. + +Delivery within one sink is strictly ordered. A recovery waits until its down +has been acknowledged or has been dropped for good. The alternative, sending in +parallel for throughput, would allow a recovery to arrive first and a receiver +to resolve something it has not yet opened. Ordering costs some throughput +under a backlog, which is a good trade for an alerting channel. + +## At-least-once delivery + +Exactly-once delivery across a network is not achievable, so tail-watch picks +the failure mode that is safe for alerts: an event may be delivered more than +once, but a committed event is not silently forgotten. + +Every event has an id that stays the same across every retry and every sink. +That gives receivers what they need to deduplicate. The contract has a second +half that receivers must accept: a `target.recovered` can arrive for a down the +receiver never saw. That happens when a request reached the receiver but the +response was lost, when the down was dropped, or when the receiver was added +later. Treat an unknown `down_event_id` as a no-op, not as an error. + +The same reasoning explains an apparent oddity. When a down was never +attempted, for example it was dropped from a full queue, its recovery is +discarded too, because sending a resolve for something never sent is only +noise. When a down was attempted but not acknowledged, its recovery is still +sent. The receiver may have accepted a request whose response was lost, and +leaving that alert open forever would be worse than a stray resolve. + +Recoveries also ignore filters that changed since the down. A sink that +received a down always gets the matching recovery, even if its `events` list +would now exclude it, and a sink that never saw the down never sees a recovery. + +An event first attempted more than a minute after its creation time is marked +`late` and keeps its original timestamp, so a receiver can tell a delayed +delivery from a fresh one. + +## Queues: memory or disk + +Each sink chooses where pending events live. + +| Mode | Behavior | Suits | +| --- | --- | --- | +| `queue: memory` | Fast and default; pending events vanish on restart. | Chat sinks where a late message is worth little. | +| `queue: disk` | Pending events survive restart; needs `collector.state_dir`. | Sinks that feed paging or automation. | + +The trade-off is between complexity and safety. A memory queue has no failure +mode beyond process death. A disk queue requires a directory that the +collector can write, and a slow disk can add latency to every durable enqueue. +It never delays a check, but it can delay delivery. + +Queues are bounded by `queue_size` (default 1000). When one is full, the +oldest event is dropped, not the newest. A backlog is most often the sign of a +receiver that has been away for a while, and by then the newest events describe +the current state of the world while the oldest describe history that has +probably resolved. The events also expire after `max_age` (default 1 hour), for +the same reason: a page for a target that recovered hours ago is worse than no +page. + +## State on disk + +With `collector.state_dir`, tail-watch keeps one file, +`notification-state.json`. It holds open incidents, the delivery state of every +incident on every sink, and the pending events of disk sinks. It deliberately +does not hold URLs, secrets or header values, so a copied or backed up state +directory is not a credential leak. + +Two decisions explain how it is written. + +**One file, one commit.** An incident change and the queue intents that follow +from it are written in the same snapshot. Splitting them across files would +create a window where a crash leaves an incident closed with its recovery never +queued, or a recovery queued for an incident that does not exist. The snapshot +is written to a temporary file, synced, renamed over the old one, and the +directory is synced. Only then do workers see the new work. A crash before the +rename leaves the old snapshot intact and a crash after it replays the new one. +That is why delivery is at least once *after a successful commit*, and not +before it. + +**Attempts are recorded before they are made.** A disk sink marks an event +"attempted" durably before sending it. If the process dies between the send +and the acknowledgment, replay knows a request may have gone out. It resends +the identical body and id, and it keeps the recovery eligible. The cost is a +possible duplicate. + +A snapshot that cannot be decoded stops startup with an error that names the +file. The alternative, starting with empty state, would forget every open +incident and page again, or worse, never send the recoveries. Refusing to start +is loud, and the operator decides what to do with the file. + +If a write fails, for example the disk is read-only or full, the collector does +not stop notifying. It keeps the authoritative state in memory, delivers from +memory, logs the loss of durability once, increments +`tailwatch_notify_persist_errors_total`, and retries persisting the full state +every second. Durability is degraded until a write succeeds, and that is +something you should alert on, because a crash in that window loses queued +events. + +## Restarts and open incidents + +At startup, before any check is scheduled, the notifier restores open +incidents into the alert state machine as `DOWN` with their original start +time. Restoring an incident is not a new alert. No second `target.down` is +sent, and the eventual recovery carries the original `down_event_id` and +duration. Without this, every restart during an outage would either re-page or +strand the original alert without a resolution. + +Learned routers add a wrinkle. An implicit target exists only because +dependency learning created it, so at restart it might not be rediscoverable +yet. The notifier records enough to recreate it and does not treat a failed +discovery pass as evidence that the router is gone. It closes such an incident +with `reason: removed` only after a reconciliation that saw the full topology. +A target that was deleted from the configuration, or a monitor deleted in the +UI, closes its incident the same way, so a receiver is not left with an alert +for something that no longer exists. + +## Intake overflow and reconciliation + +The intake is bounded, and a full intake drops the transition instead of +blocking. That looks alarming until you consider the alternative: a blocked +intake would stall the state machine, which would stall probing. The design +keeps a second, independent source of truth instead. The alert state machine +holds a snapshot of the latest notification-visible state of every target and +marks a target dirty when its transition could not be delivered. + +Once per second, the notifier reconciles dirty targets against their +snapshots. An open incident whose target is now `UP` gets a recovery. A target +that is `DOWN` with no incident gets a down. A recovery lost to overflow is +therefore still sent, without waiting for another state change. The price is +coalescing: if a target blipped down and up while the notifier was behind, the +receiver may never hear about the blip. That is acceptable, because the blip +was, by definition, not what the receiver would have wanted a page for. + +Removal follows the same route. A target removed from the collector reaches the +notifier only through a terminal snapshot, so the incident close is not lost if +the transition was. + +## When alerts are lost + +tail-watch does not promise that no alert is ever lost. It promises that a loss +has a named reason and increments a counter. Every drop increments +`tailwatch_notify_dropped_total` with one reason: + +| Reason | What happened | +| --- | --- | +| `queue_full` | The sink queue, or the intake with `sink="intake"`, was full. | +| `expired` | The event was older than `max_age` when an attempt was due. | +| `permanent` | The receiver answered with a status that will never succeed. | +| `restart` | A memory queue lost the event when the process restarted. | + +Failed attempts that will be retried are counted separately in +`tailwatch_notify_failed_total`. The recommended alerts are on any increase of +the dropped counter and on a queue length that stays above zero, since both +mean an alert did not get out. + +Notification also has an obvious blind spot: a collector that is dead cannot +say so. The [heartbeat how-to](../how-to/heartbeat.md) covers pointing the +collector at a dead-man's-switch service that alarms on silence. + +## Related pages + +- [Alert state](alert-state.md) for how transitions are produced. +- [Event payload](../reference/payload.md) for the fields receivers depend on. +- [Troubleshooting](../how-to/troubleshoot.md) for missing notifications. diff --git a/docs/explanation/monitor-ui-risks.md b/docs/explanation/monitor-ui-risks.md new file mode 100644 index 0000000..9ca837f --- /dev/null +++ b/docs/explanation/monitor-ui-risks.md @@ -0,0 +1,174 @@ +# Monitor UI risks + +The monitor UI lets someone add, edit and delete monitors while tail-watch +runs. That is a write capability on a service that otherwise only reads +configuration, and it is off by default for that reason. This page explains +what the UI protects, what it deliberately does not, and how to decide whether +the trade-off is acceptable on your tailnet. + +To turn the UI on and use it, see the [monitor UI how-to](../how-to/monitor-ui.md). +The `listen` and `state_dir` keys are in the +[configuration reference](../reference/config.md), and the flag is in the +[command line reference](../reference/cli.md). + +## What a write is worth to an attacker + +A monitor is an instruction to the collector to send probes to an address of +the writer's choosing. The collector runs on a tailnet node, with whatever +routes and grants its tag has. Someone who can create monitors can therefore +make the collector probe any address it can route to, and read the outcome +back in the monitor's state. In a network with a permissive collector tag, +that is a port scanner planted inside your network, and the results appear on +the status page. The writer can also delete monitors, which silences alerts, +and close open incidents as a side effect (see +[Incidents and queues](incidents-and-queues.md)). + +The UI does not expose secrets. Sink URLs and keys never pass through it, and +it can only change monitors that were created in the UI. Monitors from the YAML +file are shown as read-only. So the exposure is the ability to probe and to +silence, not credential theft. + +## No login, on purpose + +There is no login. tail-watch has no user database, no session handling and no +password to leak. Access control is the network: the listen address decides who +can open a socket, and the tailnet ACL decides who can reach that address. + +The alternatives were weighed against a small tool that is meant to run +unattended: + +- **Built-in accounts.** They add credentials to store and rotate, a login + page to defend, and a second identity system next to the one the tailnet + already has. For a monitor with a handful of operators, that is more attack + surface than the feature is worth. +- **Tailscale identity from the request.** This is the natural fit for a + tailnet service, because the caller's identity is already authenticated. It + is not built in today. The write path passes every request through a single + authorization decision made before any change, so identity-based + authorization can be added at that one point later. The decision currently + permits everything that reaches the listener. +- **Network scoping (chosen).** It relies on controls you already run and audit + and it has no state of its own. + +The cost is that everything reachable is trusted equally. Reading the monitor +list needs no more than reaching the port, and a grant gives write access to +everyone it covers. + +## The listen address is the security boundary + +The default `listen` address is `127.0.0.1:9109`. Only processes on the +collector host can reach it. That is a safe default, and it is also a reason +the UI is awkward to use from a laptop. + +To reach the UI from another device, you bind an address that device can reach, +such as the node's tailnet IP, and grant that device access to the port. Two +things follow. + +- **A grant covers a port, not a path.** The same listener serves `/metrics` + and the status page. Whoever can open the UI can also read the metrics. If + you scrape metrics from a different set of hosts than you want to edit + monitors from, the shared port is a constraint, and worth planning your grants + around (see the [ACL how-to](../how-to/acls.md)). +- **Loopback is shared inside the container network.** In the Compose layout, + tail-watch shares the `tailscale` container's network namespace, so every + process in that namespace can reach a loopback listener. "Loopback only" + means "only things on this network stack", which is not always the same as + "only this program". + +When the UI is on, tail-watch logs at startup that the write API is enabled +and the address it is on, and logs a warning when that address is not +loopback. Binding `0.0.0.0` is the case the warning exists for: it exposes the +write API on every interface, including any that face a LAN. + +## What the request guard does cover + +Because there is no login, a browser is the main way an outside party could +trick an authorized person into writing. A web page can make your browser send +requests to an address only your machine can reach, carrying nothing but the +request itself. The write API is guarded against the two common forms of that +attack. Every write goes through the same guard, in this order: + +1. **Content type.** A request that does not declare `application/json` is + refused with 415. A browser can send a cross-site HTML form without any + check, but it can only send form encoded content types. Refusing everything + that is not JSON turns that whole class of attack into an error. +2. **Origin.** A request whose `Origin` header names a host different from its + `Host` header is refused with 403. A cross-origin `fetch` with a JSON body + would need a permission the page cannot obtain, since the server sends no + cross-origin allowances, and the check refuses it in any case. + +Both checks run before the authorization decision and before anything changes. +Two details about the second check are worth knowing. A request with no +`Origin` header is accepted, because command line clients and scripts do not +send one. That is intentional, since the guard targets browsers, and it +means the guard is not access control. The dry-run validation endpoint is +subject to the same two checks but is not audited or authorized, because it +changes nothing. + +The list endpoint that returns the monitors is a read, and it is not guarded by +either check. + +The pages under `/ui/` also send a Content Security Policy that allows only +same-origin resources, and the page loads nothing from other origins. This +limits what an injected script could do, but it is separate from the write +guard. + +## What it does not cover: DNS rebinding + +The Origin check compares the `Origin` header to the `Host` header. It assumes +that a page on a foreign site will announce a foreign host. DNS rebinding +breaks that assumption. An attacker serves a page from a name they control, then +changes the name's DNS answer to the collector's address. The browser now +treats the collector as the same origin as the attacker's page, so `Origin` and +`Host` both name the attacker's hostname, and they match. The request carries +the right content type, and the guard sees nothing wrong. + +tail-watch does not defend against this. The guard cannot tell a rebound name +from a legitimate one without a list of names it should answer to, and it has +no such list, because the correct name depends on how you reach it: a MagicDNS +name, a tailnet IP, a hostname in the Compose network. Requiring a +configured `Host` allowlist would close the gap, at the cost of one more thing +to configure and to break. + +The mitigation is in the network. A rebinding attack needs a browser on your +tailnet, or on the collector host, to visit a hostile page while the collector +is reachable from it. Keep the write API on loopback unless someone needs it, +keep the set of devices with access to the port small, and prefer an admin +browser profile that you do not use for arbitrary sites. If you cannot accept +this risk, leave the UI off, or use it briefly and turn it off again. Editing the +YAML file and restarting remains a complete alternative. + +## The audit log + +Every create, replace and delete is logged at info level, whether it was +accepted or refused. Each entry has the operation, the monitor name, the +outcome and the caller's remote address. Refused requests are logged too, +including those rejected for content type or origin, which makes probing +visible. The caller address is the remote address of the connection, so at best the log +answers "which device did this", never "which person", because there is no +login to attribute it to. A request that arrives through a proxy or a shared +network namespace may show the address of the intermediary instead. + +The audit log is your way to reconstruct what changed. Ship it somewhere that +someone who can edit monitors cannot rewrite. + +## Choosing whether to enable it + +The UI makes sense when few, trusted operators manage monitors frequently and +the tailnet ACL already restricts the port to them. It makes less sense when +monitors change rarely, because the YAML file under version control gives review +and history that the UI does not, and monitors created in the UI live only in +`state_dir/targets.json`. That file is written atomically with owner-only +permissions, but nothing reviews it. + +If you enable it, the useful controls are, in order: keep the listener on +loopback until needed, scope the grant to the smallest set of admins, watch the +startup warning and the audit log, and remember that the guard is a defense +against browsers, not against people who can reach the port. + +## Related pages + +- [Manage monitors in the UI](../how-to/monitor-ui.md) +- [Write the ACLs](../how-to/acls.md) +- [Incidents and queues](incidents-and-queues.md) for what closing an incident + on delete means. diff --git a/docs/how-to/acls.md b/docs/how-to/acls.md new file mode 100644 index 0000000..913783d --- /dev/null +++ b/docs/how-to/acls.md @@ -0,0 +1,69 @@ +# Write the ACLs for tag:monitor + +The collector sees only what your tailnet policy allows. Without a grant, a +target fails with `no peer handles ; check grants to tag:monitor`. + +Prerequisites: + +- Permission to edit the tailnet policy file. +- The collector's node joined with the tag `tag:monitor`. tail-watch logs + `node is missing tag:monitor` at startup otherwise. + +Steps: + +1. Make sure the tag has an owner: + + ```hujson + "tagOwners": { "tag:monitor": ["autogroup:admin"] }, + ``` + +2. Grant `tag:monitor` access to the tagged nodes you probe. Grant the + subnets behind a router separately, and only the ports your checks use: + + ```hujson + { + "tagOwners": { "tag:monitor": ["autogroup:admin"] }, + "grants": [ + { + "src": ["tag:monitor"], + "dst": ["tag:router", "tag:server"], + "ip": ["*"] + }, + { + "src": ["tag:monitor"], + "dst": ["10.20.0.0/16"], + "ip": ["icmp:*", "tcp:22", "tcp:445", "tcp:443", "tcp:5001"] + } + ] + } + ``` + + Replace `tag:router`, `tag:server` and `10.20.0.0/16` with your own tags + and subnets. Each `tcp:` in a target's checks needs a matching + entry, and `icmp:*` covers `icmp` checks. + +3. If a notification receiver is a node on your tailnet, grant the collector + access to it. Sinks use the host's ordinary routing table, so the + collector's node must reach them: + + ```hujson + { "src": ["tag:monitor"], "dst": ["tag:ops-receiver"], "ip": ["tcp:443"] } + ``` + + Add this object to the same `grants` list. + +4. Save the policy. Subnet routes also need to be approved and accepted; see + [Monitor remote subnets and local LANs](remote-subnets.md). + +Verify: + +```bash +./bin/tail-watch --once --config examples/tail-watch.yaml +``` + +Each line ends `ok` for a granted check. A blocked TCP check reports an +`i/o timeout`, because the packets are dropped. If a LAN target has no +handling node, the log has `no peer handles ; check grants to +tag:monitor`. See [Troubleshoot](troubleshoot.md). + +For the UI port and its grant, see [Manage monitors in the UI](monitor-ui.md). diff --git a/docs/how-to/deploy.md b/docs/how-to/deploy.md new file mode 100644 index 0000000..578577c --- /dev/null +++ b/docs/how-to/deploy.md @@ -0,0 +1,135 @@ +# Deploy tail-watch + +Run the collector as a binary, as a Docker container next to the host's +`tailscaled`, or with Docker Compose when you have no dedicated host. + +tail-watch needs a kernel-mode `tailscaled`. It refuses to start against +userspace networking. See [Troubleshoot](troubleshoot.md) if startup fails. + +## Run the binary + +Prerequisites: + +- A host running `tailscaled` in kernel mode, tagged `tag:monitor`. +- The [ACL grants](acls.md) for that tag. +- [mise](https://mise.jdx.dev/) to install the pinned Go toolchain. + +Steps: + +1. Build the binary: + + ```bash + mise install + mise run build + ``` + +2. Validate the config. This does not contact `tailscaled`: + + ```bash + ./bin/tail-watch check-config --config examples/tail-watch.yaml + ``` + +3. Start the collector: + + ```bash + ./bin/tail-watch --config examples/tail-watch.yaml + ``` + + The config path is `--config`, then `TAIL_WATCH_CONFIG`, then + `/etc/tail-watch/tail-watch.yaml`. + +Verify: + +```bash +curl -s http://127.0.0.1:9109/healthz +``` + +The answer is `ok`. Open `http://127.0.0.1:9109/` for the status page. +To run every check once and exit non-zero on failure, use +`./bin/tail-watch --once --config examples/tail-watch.yaml`. + +Flags, environment variables and defaults are in the +[command-line reference](../reference/cli.md). + +## Run in Docker + +The image runs as user `tail-watch` (uid 65532) and does not embed a +Tailscale node. Use host networking so checks share the host routes, and +mount the LocalAPI socket. + +Prerequisites: the same host as above, and a socket that uid 65532 can read. +If it cannot, pass a `--user` or `--group-add` that can. + +Steps: + +1. Build the image: + + ```bash + docker build -t tail-watch . + ``` + +2. Run it: + + ```bash + docker run --rm --network host \ + -v /var/run/tailscale/tailscaled.sock:/var/run/tailscale/tailscaled.sock \ + -v /etc/tail-watch/tail-watch.yaml:/etc/tail-watch/tail-watch.yaml:ro \ + tail-watch + ``` + +Verify: `curl -s http://127.0.0.1:9109/healthz` answers `ok`. If the +container exits, its JSON log on stderr names the reason. For ICMP checks the +container needs `net.ipv4.ping_group_range` to include uid 65532; see +[Troubleshoot](troubleshoot.md). + +If you use `collector.state_dir` (`/var/lib/tail-watch` in the example), mount +a volume there that uid 65532 can write. + +## Run with Docker Compose and no dedicated host + +`examples/compose.yaml` runs a kernel-mode `tailscaled` container and puts +tail-watch in its network namespace. Checks follow that node's `tailscale0` +routes, including accepted subnet routes. + +Prerequisites: + +- Docker with the Compose plugin, and access to `/dev/net/tun`. +- An auth key tagged `tag:monitor`. Create it in the Tailscale admin console. + +Steps: + +1. Put your targets in a config file. The example reads + `examples/tail-watch.yaml`. To use another file, set + `TAIL_WATCH_CONFIG_FILE`. +2. Start the stack: + + ```bash + TS_AUTHKEY=tskey-auth-... docker compose -f examples/compose.yaml up -d + ``` + + tail-watch starts once the `tailscale` service reports healthy. + +Verify: + +```bash +docker compose -f examples/compose.yaml ps +docker compose -f examples/compose.yaml logs tail-watch +``` + +The log shows a `collector identity` line with the tag `tag:monitor`. The +listener is on the loopback address of the shared namespace, so run +`docker compose -f examples/compose.yaml exec tailscale wget -qO- http://127.0.0.1:9109/healthz` +to reach it. + +Things to know: + +- The `tailscale` service sets `net.ipv4.ping_group_range`, so the non-root + tail-watch image can send ICMP without `CAP_NET_RAW`. +- `network: local` targets reach only what the container's Docker network + can reach. To monitor a LAN, use the host deployment instead. See + [Monitor remote subnets and local LANs](remote-subnets.md). +- Do not put secrets in the config file. Sinks read them from `*_env` or + `*_file` keys; see [Send alerts to Slack](slack-sink.md). +- To keep notification state and UI monitors across restarts, mount a volume + at `collector.state_dir`. See + [Manage monitors in the UI](monitor-ui.md). diff --git a/docs/how-to/heartbeat.md b/docs/how-to/heartbeat.md new file mode 100644 index 0000000..3cc9f5c --- /dev/null +++ b/docs/how-to/heartbeat.md @@ -0,0 +1,45 @@ +# Watch the watcher with a heartbeat + +A dead collector cannot report itself. Point `heartbeat` at a dead-man's-switch +service that alerts when the pings stop. + +Prerequisites: an account with a dead-man's-switch or cron-monitoring service +that gives you a ping URL and alerts when it stops receiving pings. + +Steps: + +1. Put the ping URL in an environment variable, or in a file for `url_file`. + Do not write it into the config. +2. Add the block to the config. Set exactly one of `url_env` or `url_file`: + + ```yaml + heartbeat: + url_env: HEARTBEAT_URL + interval: 1m + ``` + +3. Set the interval below the service's grace period. +4. Validate the file, then restart. `check-config` does not read the URL + variable, but startup fails if it is unset: + + ```bash + ./bin/tail-watch check-config --config /etc/tail-watch/tail-watch.yaml + ``` + +Verify: the service shows pings arriving each `interval`. tail-watch sends a +`GET` starting once every target and check has a first result, with a 10s +timeout. When the process exits the pings stop and the service alerts. + +A failed ping is logged, counted in `tailwatch_heartbeat_failures_total`, and +not retried before the next interval. Alert on that counter; see the +[metrics reference](../reference/metrics.md). + +Exit-node egress warning: the heartbeat leaves through the host's ordinary +routing table, like the notification sinks. tail-watch warns at startup with +`exit node is active; notifications will route through the exit node` when +an exit node is set, and the pings would then come from the exit node's +address. Clear the exit node on the collector or make sure the service does not +filter by source address. + +`tail-watch --once` never sends a heartbeat. If the URL variable is unset, +startup fails; see [Troubleshoot](troubleshoot.md). diff --git a/docs/how-to/monitor-ui.md b/docs/how-to/monitor-ui.md new file mode 100644 index 0000000..6666aab --- /dev/null +++ b/docs/how-to/monitor-ui.md @@ -0,0 +1,119 @@ +# Manage monitors in the UI + +The monitor UI adds, edits and deletes monitors while tail-watch runs, with +no restart and no lost grace timers. It is off by default. + +Read [Monitor UI risks](../explanation/monitor-ui-risks.md) first: there is +no login, and whoever can reach the port can make the collector probe any +address it can route to. + +## Turn the UI on + +Prerequisites: a writable `collector.state_dir`. Monitors created in the UI +are stored in `state_dir/targets.json`. + +Steps: + +1. Set `collector.state_dir` in the config: + + ```yaml + collector: + listen: 127.0.0.1:9109 + state_dir: /var/lib/tail-watch + ui: true + targets: + - name: nyc-subnet-router + addr: nyc-router + checks: [tsping] + ``` + +2. Enable the UI with `collector.ui: true` (above) or the flag: + + ```bash + tail-watch --config /etc/tail-watch/tail-watch.yaml --ui + ``` + + The flag wins over the file, so `--ui=false` turns off a config that + enables it. Under `--once` the UI is ignored. +3. Restart tail-watch. + +Verify: the log has `monitor write API enabled`, and +`http://127.0.0.1:9109/ui/` shows the monitor list. If tail-watch exits with +`the UI needs collector.state_dir to store monitors`, set `state_dir`. + +## Reach the UI from another device + +The default `listen` address, `127.0.0.1:9109`, is reachable only from the +host. To open the UI from another device on the tailnet, bind an address that +device can reach and grant it access. + +Steps: + +1. Bind the node's tailnet IP with `--listen` or `collector.listen`, for + example `--listen 100.101.102.103:9109`. tail-watch then warns + `monitor write API is reachable beyond loopback; access is controlled only + by the network`. +2. Grant the device access to the port: + + ```hujson + { + "grants": [ + { + "src": ["autogroup:admin"], + "dst": ["tag:monitor"], + "ip": ["tcp:9109"] + } + ] + } + ``` + + A grant covers the whole port, so it also exposes `/metrics`. +3. Open `http://:9109/ui/` from that device. + +In the Compose layout tail-watch shares the `tailscale` container's network +namespace, so every process in that namespace can reach a loopback listener. + +## Add, edit and delete a monitor + +Steps: + +1. Click **Add monitor**. Fill in name, address, checks and the other fields. + The `defaults` block of the YAML file applies to UI monitors. +2. Click **Check** to validate without saving, then **Save**. Errors appear + next to the field they belong to, and nothing is created until they are + fixed. They come from the same rules the YAML loader uses. + + ![The add monitor form with an invalid TCP port reported under Checks.](../images/monitor-ui-form-error.png) + +3. The list shows every monitor. YAML monitors are read-only; only monitors + created in the UI have **Edit** and **Delete**. Monitors behind a down + parent show `SUPPRESSED`. + + ![The monitor list: YAML monitors are read-only, UI monitors have Edit and Delete, and monitors behind a down parent show SUPPRESSED.](../images/monitor-ui-list.png) + +4. To delete, click **Delete** and confirm. Deleting a `DOWN` monitor asks + first, and says its open incident closes. + + ![The delete confirmation for a down monitor.](../images/monitor-ui-delete-down.png) + +Behavior to expect: + +- An edit is a delete followed by an add. Editing or deleting a `DOWN` + monitor closes its open incident with `reason: removed`. A new incident + opens only if the new definition fails `fail_threshold` checks. +- A monitor that other monitors name as `parent` cannot be deleted until + they are changed. The API answers with `"" is the parent of [...]; + delete or change those first`. +- Editing a parent briefly releases its suppressed dependents, which can page + while the parent is down. +- A monitor whose name matches a YAML target is skipped at startup with a + warning, and the YAML target wins. +- `targets.json` uses the YAML field names, is written atomically, and has + mode 0600. +- With the UI off, `targets.json` is left alone and its monitors do not run. + A startup warning, `dynamic monitors are inactive: the UI is off`, gives + the count. + +Every write is logged with the caller's address as a `monitor write` line +with `op`, `target`, `status`, `outcome` and `remote`. To find writes, filter +the JSON log on that message. diff --git a/docs/how-to/remote-subnets.md b/docs/how-to/remote-subnets.md new file mode 100644 index 0000000..ac60e5b --- /dev/null +++ b/docs/how-to/remote-subnets.md @@ -0,0 +1,92 @@ +# Monitor remote subnets and local LANs + +Each target has a `network` setting that says which path its checks should +take. Pick the value that matches where the device is. + +## Monitor a subnet behind a Tailscale router + +Use `network: tailnet` (the default) for devices reached through a subnet +router. + +Prerequisites: + +- A subnet router advertising the route, with the route approved in the + admin console. +- [Grants](acls.md) from `tag:monitor` to the subnet. + +Steps: + +1. On the collector host, accept subnet routes. On Linux, startup fails + until you do, with `remote subnet targets require accepted routes; run + tailscale set --accept-routes`: + + ```bash + tailscale set --accept-routes + ``` + +2. Check the route the host will actually use. An advertised route is not an + installed route: + + ```bash + ip route get 10.20.0.1 + ``` + + The answer must go out `tailscale0`, not a local gateway. Repeat this + for each remote subnet you monitor. + +3. Add the targets. Point a `tsping` target at the router, and name it as the + `parent` of the devices behind it: + + ```yaml + targets: + - name: nyc-subnet-router + addr: nyc-router + site: nyc + network: tailnet + checks: [tsping] + + - name: nyc-core-switch + addr: 10.20.0.1 + site: nyc + network: tailnet + parent: nyc-subnet-router + checks: [icmp] + ``` + +Verify: `./bin/tail-watch check-config --config ` accepts the file, and +`./bin/tail-watch --once --config ` prints `ok` for each check. + +## Monitor a LAN the collector is plugged into + +Use `network: local` for a device on the collector's own LAN. These checks +use the host's ordinary route and keep working when that site's Tailscale +router is down. + +Steps: + +1. Give the device its own target with `network: local`: + + ```yaml + targets: + - name: lab-printer + addr: 192.168.1.50 + site: lab + network: local + severity: warn + checks: + - icmp + - tcp:9100 + - "http:http://192.168.1.50/|ready" + ``` + +2. Keep the HTTP check on the same host as `addr`. +3. Do not add `tsping`. It is rejected for `network: local`. + +Verify: `./bin/tail-watch --once --config ` shows the checks passing. +A failure on a local target is labeled `service` or `lan`, never `control` +or `tunnel`. The layers are explained in +[Failure layers](../explanation/failure-layers.md). + +`examples/tail-watch.yaml` has both kinds of target. The check syntax is in +the [checks reference](../reference/checks.md), and the target fields are in +the [configuration reference](../reference/config.md). diff --git a/docs/how-to/slack-sink.md b/docs/how-to/slack-sink.md new file mode 100644 index 0000000..f345060 --- /dev/null +++ b/docs/how-to/slack-sink.md @@ -0,0 +1,89 @@ +# Send alerts to Slack + +Post alert transitions to a Slack channel through an incoming webhook. + +Prerequisites: + +- A Slack workspace where you can create an app. +- A collector that can reach `hooks.slack.com` (see the egress note in + [Send alerts to a webhook](webhook-sink.md)). + +## Get an incoming webhook URL + +tail-watch posts to an incoming webhook URL. It does not use bot or app-level +tokens (`xoxb-`, `xapp-`), and a bot token cannot produce a webhook URL. +Slack's guide, +[Sending messages using incoming webhooks](https://docs.slack.dev/messaging/sending-messages-using-incoming-webhooks/), +is the reference. + +1. Create an app at , or open an + existing one. +2. In the app's settings sidebar choose **Incoming Webhooks** and switch + **Activate Incoming Webhooks** on. +3. Click **Add New Webhook to Workspace**, pick the channel, and click + **Allow**. +4. Copy the URL from **Webhook URLs for Your Workspace**. It starts with + `https://hooks.slack.com/services/`. + +If the sidebar has no **Incoming Webhooks** entry, Slack's `incoming-webhook` +scope produces a webhook during an OAuth install. Slack changes these pages +over time; if the steps do not match, follow Slack's guide. + +Check the URL. Slack answers `ok`: + +```bash +curl -X POST -H 'Content-type: application/json' \ + --data '{"text":"hello"}' "$SLACK_WEBHOOK_URL" +``` + +Store the URL as a secret in your environment, secret manager or a file +outside the config. Never put it in the YAML. + +## Add the sink + +1. Add a sink to the config. `url_env` names the environment variable that + holds the URL; use `url_file` for a file instead: + + ```yaml + sinks: + - name: ops-slack + type: slack + url_env: SLACK_WEBHOOK_URL + min_severity: warn + mention_on_page: "" + status_url: https://monitor-nyc.example.ts.net:9109 + group_wait: 10s + queue: disk + match: + sites: [nyc, lab] + ``` + + - `mention_on_page` is prepended only to a `target.down` of severity + `page`, never to recoveries, flapping notices or warnings. + - `status_url` turns each target line into a link to + `/#`. + - `group_wait` batches events that arrive together into one message. + - `queue: disk` needs `collector.state_dir`. With the default + `queue: memory`, an event still waiting in the group window is lost when + tail-watch restarts. Use `disk` if an alert must survive a restart. + - A Slack sink rejects `batch`, `max_batch`, `secret_*` and `headers_env`. +2. Validate the file. `check-config` prints each sink's name, type, filters + and queue mode, never a secret. It does not read the secret variables: + + ```bash + ./bin/tail-watch check-config --config /etc/tail-watch/tail-watch.yaml + ``` + +3. Send a test message: + + ```bash + ./bin/tail-watch notify-test --sink ops-slack --timeout 10s + ``` + +Verify: `notify-test` prints `ops-slack slack 200` and exits 0, and the message +appears in the channel. A missing variable prints +`environment variable SLACK_WEBHOOK_URL is unset or empty`; see +[Troubleshoot](troubleshoot.md). Run `notify-test` after every config change. + +For the message fields see the [payload reference](../reference/payload.md). +For sink options see the [configuration reference](../reference/config.md). diff --git a/docs/how-to/troubleshoot.md b/docs/how-to/troubleshoot.md new file mode 100644 index 0000000..e6351ac --- /dev/null +++ b/docs/how-to/troubleshoot.md @@ -0,0 +1,40 @@ +# Troubleshoot + +Find the message tail-watch printed, then apply the fix. Startup failures +print to stderr and exit 1; log lines are JSON on stderr. + +## Startup failures and warnings + +| Symptom | Cause | Fix | +| --- | --- | --- | +| `tailscaled is using userspace networking; tail-watch requires kernel-mode tailscaled` | `tailscaled` runs with `--tun=userspace-networking`, which cannot carry ICMP to LAN addresses | Run `tailscaled` in kernel mode. In Compose set `TS_USERSPACE: "false"` as `examples/compose.yaml` does. See [Deploy](deploy.md) | +| `node is missing tag:monitor` (a warning) | The collector node does not carry the tag your grants are written for | Join with an auth key tagged `tag:monitor`, or re-tag the node. See [ACLs](acls.md) | +| `cannot open an ICMP socket: set net.ipv4.ping_group_range or grant CAP_NET_RAW` | The process cannot open an ICMP socket. TCP, HTTP and `tsping` checks still run | Allow unprivileged ping: `sysctl -w net.ipv4.ping_group_range="0 2147483647"` (or a range that includes the collector's group), or grant `CAP_NET_RAW`. In Docker, set the sysctl on the container as the example Compose file does | +| `tailscaled is not reachable at : ...` | The LocalAPI socket is missing, wrong, or unreadable by the collector's user. On Linux the default is `/var/run/tailscale/tailscaled.sock` | Pass `--socket `, check `tailscaled` is running, and make the socket readable by the collector's user. In Docker mount the socket and use a `--user` or `--group-add` that can read it (the image runs as uid 65532) | +| `tailscaled state is ; run tailscale up` | `tailscaled` is running but not logged in | Run `tailscale up` | +| `remote subnet targets require accepted routes; run tailscale set --accept-routes` | A `network: tailnet` target has a literal non-Tailscale IP and the host does not accept subnet routes | Run `tailscale set --accept-routes`, then check `ip route get `. See [Remote subnets](remote-subnets.md) | +| `exit node is active; notifications will route through the exit node` (a warning) | An exit node is set on the collector, so sinks and the heartbeat egress through it | Clear the exit node, or make sure receivers accept that source. See [Webhook](webhook-sink.md) | +| `sink : url: environment variable is unset or empty` (also `secret:`, `header :`, `heartbeat:`) | A `*_env` variable is unset or empty at startup. Secret files report `secret file is empty` | Export the variable in the service's environment. Compose and systemd do not inherit your shell. Run `notify-test` to confirm. See [Slack](slack-sink.md) and [Heartbeat](heartbeat.md) | +| `notification state is corrupt (...); repair or move the file aside and restart, but note open incidents are forgotten if it is removed` | `notification-state.json` in `collector.state_dir` cannot be decoded, so startup stops rather than forget open incidents | Restore or repair the file from a backup. If you move it aside, open incidents are forgotten and their recoveries are not sent. See [Incidents and queues](../explanation/incidents-and-queues.md) | +| `the UI needs collector.state_dir to store monitors` | `--ui` or `collector.ui: true` is set with no `state_dir` | Set `collector.state_dir`, or turn the UI off with `--ui=false`. See [Monitor UI](monitor-ui.md) | +| `queue: disk requires collector.state_dir` | A sink uses the disk queue with no `state_dir` | Set `collector.state_dir` or use `queue: memory` | +| `dynamic monitors are inactive: the UI is off` (a warning) | `targets.json` has monitors but the UI is off, so they do not run | Turn the UI on, or ignore it | +| `webhook sink is unsigned: no secret_env or secret_file` (a warning) | A webhook sink has no HMAC key | Add `secret_env`. See [Webhook](webhook-sink.md) | + +## Check failures + +| Symptom | Cause | Fix | +| --- | --- | --- | +| `no peer handles ; check grants to tag:monitor` | No peer owns or routes the address for this collector: a missing grant, an unapproved or unaccepted route, or the wrong address | Add the grant from `tag:monitor` and approve the route. See [ACLs](acls.md) and [Remote subnets](remote-subnets.md). tail-watch logs it once per target, again only if the grant returns and is lost | +| TCP check ends with `i/o timeout` | The grant does not allow that port, so packets are dropped | Add `tcp:` to the grant | + +## Validate before you restart + +- `tail-watch check-config --config ` validates the file without + contacting `tailscaled`. +- `tail-watch notify-test` sends one test event per sink and reports the + HTTP status or the error. +- `tail-watch --once --config ` runs every check once. + +Flags are in the [command-line reference](../reference/cli.md). To see every +check result in the log, use `--log-level debug`. diff --git a/docs/how-to/webhook-sink.md b/docs/how-to/webhook-sink.md new file mode 100644 index 0000000..59d8c67 --- /dev/null +++ b/docs/how-to/webhook-sink.md @@ -0,0 +1,137 @@ +# Send alerts to a webhook + +Deliver each alert event as a signed JSON `POST` to your own receiver. + +Prerequisites: + +- A receiver that accepts `POST` with `Content-Type: application/json` and + answers `2xx`. +- The receiver must deduplicate on the event `id`, because delivery is at + least once. See the [payload reference](../reference/payload.md). + +## Add the sink + +1. Add a sink. Secrets come only from environment variables or files: + + ```yaml + sinks: + - name: ops-webhook + type: webhook + url_env: OPS_WEBHOOK_URL + secret_env: OPS_WEBHOOK_SECRET + headers_env: + Authorization: OPS_WEBHOOK_AUTH + min_severity: page + events: [target.down, target.recovered] + max_age: 1h + queue: disk + queue_size: 1000 + batch: true + group_wait: 10s + max_batch: 100 + ``` + + - `secret_env` (or `secret_file`) is the HMAC key. Without it requests are + unsigned, and tail-watch logs `webhook sink is unsigned: no secret_env + or secret_file`. + - `headers_env` maps a header name to the environment variable holding its + value. + - `queue: disk` needs `collector.state_dir`. + - `batch: true` waits `group_wait` after the first queued event, or until + `max_batch` events are waiting, and sends one request. +2. Validate the file. `check-config` does not read the secret variables: + + ```bash + ./bin/tail-watch check-config --config /etc/tail-watch/tail-watch.yaml + ``` + + Export the variables in the collector's environment before you run + `notify-test` or start tail-watch. + +3. Send a test event through the real signing and retry code: + + ```bash + ./bin/tail-watch notify-test --sink ops-webhook + ``` + +Verify: `notify-test` prints `ops-webhook webhook 200` and exits 0 when the +receiver answers `2xx` within `--timeout` (default 30s). It bypasses filters, +queues and `state_dir`, and works when `tailscaled` is down. + +## What the receiver sees + +Each event is one `POST` with these headers: + +- `Content-Type: application/json` +- `User-Agent: tail-watch/` +- `Tail-Watch-Event-Id`, or `Tail-Watch-Batch-Id` for a batched sink +- `Tail-Watch-Signature: t=,v1=` when a secret is set +- any `headers_env` headers + +Redirects are not followed and each request times out after 10s. A batched +request has the body `{"version": 1, "batch_id": "bat_...", "events": [...]}`. + +| Response | What tail-watch does | +| --- | --- | +| `2xx` | delivered | +| `408`, `5xx`, timeout, network error | retry with growing delay | +| `429` | retry, waiting at least `Retry-After` | +| anything else, including `3xx` and other `4xx` | drop, log once, count `permanent` | + +The delay is 1s, 2s, 4s and so on with 0.5 to 1.5 times jitter, never more +than 5 minutes. An event older than `max_age` when an attempt is due is +dropped and counted as `expired`. A retry sends identical body bytes with a +fresh `t`. + +## Verify the signature in the receiver + +`v1` is the lowercase hex HMAC-SHA256 of `.` under the secret. The +signature covers the whole body, including a batch. Reject stale +timestamps: + +```go +func verify(secret, header string, body []byte, tolerance time.Duration) bool { + var ts, sig string + for _, part := range strings.Split(header, ",") { + k, v, _ := strings.Cut(part, "=") + switch k { + case "t": + ts = v + case "v1": + sig = v + } + } + secs, err := strconv.ParseInt(ts, 10, 64) + if err != nil || time.Since(time.Unix(secs, 0)).Abs() > tolerance { + return false + } + mac := hmac.New(sha256.New, []byte(secret)) + fmt.Fprintf(mac, "%s.%s", ts, body) + return hmac.Equal([]byte(hex.EncodeToString(mac.Sum(nil))), []byte(sig)) +} +``` + +To check your implementation, the repository's test vector uses the secret +`whsec_test_key`, the timestamp `1700000000` and the body +`{"id":"evt_X","type":"test"}`. The header must be: + +```text +t=1700000000,v1=1419cf10ea0dc484e3a06e7df47e222718af9df5f1bb5180b9c3bb9cde19722f +``` + +You can reproduce the digest with: + +```bash +printf '%s' '1700000000.{"id":"evt_X","type":"test"}' \ + | openssl dgst -sha256 -hmac whsec_test_key +``` + +## Make sure the collector can reach the receiver + +Sinks use the host's ordinary routing table. tail-watch warns at startup with +`exit node is active; notifications will route through the exit node` when an +exit node is set. A receiver on your own tailnet needs a grant from +`tag:monitor`; see [Write the ACLs](acls.md). + +For how queues, retries and drops behave, see +[Incidents and queues](../explanation/incidents-and-queues.md). diff --git a/docs/reference/checks.md b/docs/reference/checks.md new file mode 100644 index 0000000..367cf6f --- /dev/null +++ b/docs/reference/checks.md @@ -0,0 +1,125 @@ +# Checks reference + +A check is one string in a target's `checks` list. Each check runs on its target's +`interval` and is cancelled at its target's `timeout`. A check that times out counts +as a failure. First runs are spread across the interval so checks do not start in a +burst. + +| Check | Syntax | Passes when | +| --- | --- | --- | +| ICMP echo | `icmp` | One echo reply arrives before the timeout. | +| TCP connect | `tcp:` | A TCP connection to `addr:` is established. | +| HTTP request | `http:` or `http:\|` | The status is 2xx or 3xx and, if `` is given, the body contains it. | +| Tailscale ping | `tsping` | tailscaled reports a disco pong. | +| TLS | `tls:...` | Rejected. Not supported in this version. | + +The same check string cannot appear twice in one target. Every check that fails +gets a failure layer; see [Failure layers](../explanation/failure-layers.md). + +## Path and network + +Checks use host sockets and do not bind to an interface. The target's `network` +declares the intended path: + +- `tailnet` (default): IP checks should use Tailscale. On Linux, a target whose + `addr` is a literal non-Tailscale IP is a remote subnet and needs accepted + routes at startup. +- `local`: the host's ordinary route. `tsping` is rejected on a `local` target. + +See [Monitor remote subnets and local LANs](../how-to/remote-subnets.md). + +## `icmp` + +Sends one echo request to the target's address, IPv4 or IPv6. A DNS name is +resolved first and an IPv4 address is preferred. It uses an unprivileged ICMP +socket when the host allows one and otherwise a raw socket. + +| Result | Detail | +| --- | --- | +| Pass | Latency is the round-trip time. | +| Fail | `timeout: ...` when no reply arrives. | +| Fail | `cannot open an ICMP socket: set net.ipv4.ping_group_range or grant CAP_NET_RAW` when neither socket type can be opened. | + +## `tcp:` + +`` is an integer from 1 to 65535. The check opens a connection to +`addr:` and closes it immediately without sending data. Latency is the +connect time. A refused connection fails with an error containing +`connection refused`. + +## `http:` + +```yaml +checks: + - "http:https://10.20.0.50:5001/" + - "http:https://10.20.0.50:5001/|DSM" +``` + +The URL must use `http` or `https` and include a host. The host must equal the +target's `addr`, ignoring case, a trailing dot, and equivalent IP forms. To check +a different host, define a separate target. Quote the string in YAML because it +contains a colon. + +Behavior: + +- Sends `GET`. A response of 200 through 399 passes the status test. Any other + status fails with `HTTP status `. +- Redirects are not followed. A `302` counts as up. +- The text after the first `|` is a literal, case-sensitive substring that must + appear in the first 1 MiB of the body. If it is missing, the check fails with + `body missing ""; first 300 bytes: ""`. The URL part therefore + cannot contain `|`. +- Opens a new connection for every run, ignores `HTTP_PROXY` and related + environment variables, and verifies TLS certificates. +- With `insecure_tls: true` on the target, certificate verification is skipped for + every `http:` check on that target. + +## `tsping` + +Resolves `addr` to a Tailscale or subnet-routed IP and sends a disco ping through +the local tailscaled. It passes when tailscaled reports a pong before the timeout, +and records the path kind, the endpoint or DERP region, and the latency. + +Name resolution for a non-IP `addr` prefers, in order: the peer's full MagicDNS +name, its short MagicDNS name, its hostname. Matching ignores case. If several +peers tie, the first by name is used and a warning is logged. If no peer matches, +the name is resolved with DNS. + +| Path kind | Meaning | +| --- | --- | +| `direct` | Direct UDP path. The endpoint is recorded. | +| `derp` | Relayed through a DERP region. The region code is recorded. | +| `peer_relay` | Relayed through a peer relay. | + +Failures: + +| Error | Cause | +| --- | --- | +| `no peer handles ; check grants to tag:monitor` | No peer in the collector's netmap routes the address, usually a missing grant. See [Write the ACLs](../how-to/acls.md). | +| `tsping returned no result` | tailscaled answered without a result. | +| other text | The error reported by tailscaled. | + +## `tls:` (rejected) + +Any check starting with `tls:` is rejected at load time with +`tls checks are not supported in this version`. + +## Parent rules + +`parent` is for suppression only. It does not change which node handles the +address. A parent must: + +- name another target in the same config; +- have exactly one check, and that check must be `icmp` or `tsping`; +- not create a cycle. + +An explicit `parent` is allowed on a `local` target. See +[Alert state](../explanation/alert-state.md) for what a down parent does to its +dependents. + +## Result record + +Every run produces a result with the target name, the check string (for example +`icmp` or `tcp:445`), the start time, the latency, the error if any, and for +`tsping` the path details. Under `--once`, each result is printed as +` ok ` or ` fail `. diff --git a/docs/reference/cli.md b/docs/reference/cli.md new file mode 100644 index 0000000..98f82f4 --- /dev/null +++ b/docs/reference/cli.md @@ -0,0 +1,161 @@ +# Command line reference + +```text +tail-watch [flags] +tail-watch check-config [--config ] +tail-watch notify-test [flags] +``` + +Without a subcommand, tail-watch runs the collector. + +## Collector flags + +| Flag | Type | Default | Meaning | +| --- | --- | --- | --- | +| `-config` | string | see below | Config file path. Overrides `TAIL_WATCH_CONFIG`. | +| `-listen` | string | `collector.listen`, else `127.0.0.1:9109` | HTTP listen address. | +| `-socket` | string | platform default | tailscaled LocalAPI socket. | +| `-log-level` | string | `info` | `debug`, `info`, `warn`, or `error`. Overrides `TAIL_WATCH_LOG_LEVEL`. | +| `-once` | bool | `false` | Run every check once, print one line each, and exit. | +| `-ui` | bool | `collector.ui`, else `false` | Serve the monitor UI and API. `--ui=false` turns off a config that enables it. | +| `-version` | bool | `false` | Print the version and exit. | + +Go's flag syntax applies: `-flag` and `--flag` are equivalent, and a value is +given as `--flag value` or `--flag=value`. `-h` and `--help` print usage and +exit 0. + +The default socket is `/var/run/tailscale/tailscaled.sock` on Linux. + +### Precedence + +| Setting | Order, first match wins | +| --- | --- | +| Config path | `--config`, `TAIL_WATCH_CONFIG`, `/etc/tail-watch/tail-watch.yaml` | +| Listen address | `--listen`, `collector.listen`, `127.0.0.1:9109` | +| UI | `--ui` or `--ui=false`, `collector.ui`, off | +| Log level | `--log-level`, `TAIL_WATCH_LOG_LEVEL`, `info` | + +### Environment variables + +| Variable | Meaning | +| --- | --- | +| `TAIL_WATCH_CONFIG` | Config file path when `--config` is not given. | +| `TAIL_WATCH_LOG_LEVEL` | Log level when `--log-level` is not given. | + +Log levels are case-insensitive. `warning` is accepted as `warn`. Logs are JSON, +one object per line, on stderr. At `debug`, every check result is logged. + +### `--once` + +Runs the preflight, runs every check of every configured target once, prints one +line per result, and exits. It starts no HTTP server, builds no notifier, resolves +no secrets, sends no heartbeat, and ignores the UI. + +```text + ok + fail +``` + +### Startup checks + +Before serving, the collector fails with exit code 1 when: + +- the config file cannot be read or is invalid; +- the UI is on and `collector.state_dir` is unset; +- tailscaled is unreachable, or its state is not `Running`; +- tailscaled uses userspace networking (kernel mode is required); +- a target is a remote subnet address on Linux and the host has not accepted + routes (`tailscale set --accept-routes`); +- a sink or heartbeat secret cannot be read or is empty (server mode only); +- the notification state file cannot be decoded (server mode only). + +It logs a warning, and continues, when the node lacks `tag:monitor`, when an exit +node is active, when a webhook sink has no secret, and when the UI is reachable +beyond loopback. + +## `check-config` + +```bash +tail-watch check-config [--config ] +``` + +Loads and validates the config. It does not contact tailscaled and does not +resolve secrets, so it needs no environment variables. On success it prints one +line per target, one per sink, and one for the heartbeat if configured, then +exits 0. + +```text +nyc-router interval=15s timeout=5s network=tailnet checks=tsping +nyc-nas interval=30s timeout=5s network=tailnet checks=icmp,tcp:445,http:https://10.20.0.50:5001/|DSM +sink ops-webhook type=webhook min_severity=page events=target.down,target.recovered queue=disk match=sites:nyc targets:nyc-* tags:tag:router +sink ops-slack type=slack min_severity=warn events=all queue=memory +heartbeat interval=1m0s +``` + +On failure it prints every problem to stderr and exits 1. The only flag is +`-config`. + +## `notify-test` + +```bash +tail-watch notify-test [--config ] [--sink ] [--timeout ] [--socket ] +``` + +Sends one `test` event through each selected sink's real signing, rendering, and +retry code. It bypasses filters, queues, and `state_dir`. It needs the secrets +that the sinks reference, and it works when tailscaled is unreachable, in which +case the collector name in the payload is the OS hostname and `tailscale_ip` is +omitted. + +| Flag | Type | Default | Meaning | +| --- | --- | --- | --- | +| `-config` | string | see [Precedence](#precedence) | Config file path. | +| `-sink` | string | every sink | Send only to the sink with this name. | +| `-timeout` | duration | `30s` | Give up on a sink after this long, including retries. | +| `-socket` | string | platform default | tailscaled LocalAPI socket, used to read the collector identity. | + +Sinks are tried in parallel. One line is printed per sink, in config order: + +| Line | Meaning | +| --- | --- | +| ` ` | Delivered. `` is the 2xx response code. | +| ` ` | Failed. A permanent rejection such as `404` shows the status code. | +| ` error: timed out after retries (last: )` | Timeout reached while retrying. | +| ` error: ` | A secret could not be read, or the request could not be built. | + +Retryable failures (`408`, `429`, `5xx`, network errors) are retried with backoff +until `--timeout`. See [Event payload](payload.md) for the request format. + +## Exit codes + +| Code | Meaning | +| --- | --- | +| `0` | Success. For the collector: clean shutdown, or `--once` with every check passing. For `notify-test`: every selected sink answered 2xx. `--help` also exits 0. | +| `1` | Failure. Config or startup error, HTTP server error, a check failed under `--once`, or a sink failed under `notify-test`. | +| `2` | Usage error. Unknown flag, invalid `--log-level` or `TAIL_WATCH_LOG_LEVEL`, or `notify-test` with no sinks configured or an unknown `--sink`. | + +## HTTP endpoints + +Served on the listen address by the collector, not under `--once`. + +| Path | Response | +| --- | --- | +| `/` | Prober status page: each check's recent results, latency, and state. | +| `/metrics` | Prometheus text exposition. See [Metrics](metrics.md). | +| `/healthz` | `200` with body `ok`. Says the process is serving, not that targets are up. | +| `/ui/` | Monitor UI. `404` unless the UI is on. | +| `/api/v1/targets` | Monitor API: `GET` list, `POST` create. `404` unless the UI is on. | +| `/api/v1/targets/{name}` | Monitor API: `PUT` replace, `DELETE` remove. | +| `/api/v1/validate` | Monitor API: `POST` validate a monitor without saving it. | + +Write requests to `/api/v1/` must be `application/json` and must not carry an +`Origin` for another host. There is no authentication. See +[Manage monitors in the UI](../how-to/monitor-ui.md) and +[Monitor UI risks](../explanation/monitor-ui-risks.md). + +## Signals + +`SIGINT` and `SIGTERM` stop check scheduling and cancel running checks. The +collector then stops the notifier and shuts down the HTTP server, waiting up to the +longest target timeout plus 5 seconds. It exits 0. Under `--once`, a signal stops +the run and exits 0. diff --git a/docs/reference/config.md b/docs/reference/config.md new file mode 100644 index 0000000..8f75253 --- /dev/null +++ b/docs/reference/config.md @@ -0,0 +1,292 @@ +# Configuration reference + +tail-watch reads one YAML file. This page lists every key, its type, its default, +and the rules the loader enforces. Validate a file without contacting tailscaled: + +```bash +tail-watch check-config --config /etc/tail-watch/tail-watch.yaml +``` + +See [Command line](cli.md) for how the file path is chosen and [Checks](checks.md) +for the `checks` syntax. + +## File rules + +- The file must not be empty. +- Unknown keys are rejected at every level. +- Durations use Go syntax with a unit, such as `500ms`, `30s`, `15m`, `1h`. A + bare number is invalid. +- The loader reports every problem it finds in one pass, each prefixed with the + key path, such as `targets[2].checks[1]`. +- Secret values are never written in the file. Keys ending in `_env` name an + environment variable, and keys ending in `_file` name a file. + +## Top-level keys + +| Key | Type | Required | Default | +| --- | --- | --- | --- | +| `collector` | mapping | no | see below | +| `defaults` | mapping | no | see below | +| `targets` | list of targets | no | empty | +| `sinks` | list of sinks | no | empty | +| `heartbeat` | mapping | no | off | + +## Complete example + +Every key, set to its default where it has one. This file passes +`tail-watch check-config`. + +```yaml +collector: + listen: 127.0.0.1:9109 + state_dir: /var/lib/tail-watch + ui: false + +defaults: + interval: 30s + timeout: 5s + fail_threshold: 3 + recover_threshold: 2 + flap_limit: 4 + flap_window: 15m + +targets: + - name: nyc-router + addr: nyc-router + site: nyc + severity: page + network: tailnet + interval: 15s + timeout: 5s + fail_threshold: 3 + recover_threshold: 2 + flap_limit: 4 + flap_window: 15m + checks: [tsping] + + - name: nyc-nas + addr: 10.20.0.50 + site: nyc + severity: warn + parent: nyc-router + insecure_tls: true + checks: + - icmp + - tcp:445 + - "http:https://10.20.0.50:5001/|DSM" + + - name: lab-printer + addr: 192.168.1.50 + site: lab + network: local + checks: [icmp, tcp:9100] + +sinks: + - name: ops-webhook + type: webhook + url_env: OPS_WEBHOOK_URL + secret_env: OPS_WEBHOOK_SECRET + headers_env: + Authorization: OPS_WEBHOOK_AUTH + min_severity: page + events: [target.down, target.recovered] + match: + sites: [nyc] + targets: ["nyc-*"] + tags: [tag:router] + max_age: 1h + queue: disk + queue_size: 1000 + batch: true + group_wait: 10s + max_batch: 100 + + - name: ops-slack + type: slack + url_env: SLACK_WEBHOOK_URL + min_severity: warn + mention_on_page: "" + status_url: https://monitor-nyc.example.ts.net:9109 + group_wait: 10s + max_age: 1h + queue: memory + queue_size: 1000 + +heartbeat: + url_env: HEARTBEAT_URL + interval: 1m +``` + +## `collector` + +| Key | Type | Required | Default | Meaning | +| --- | --- | --- | --- | --- | +| `listen` | string | no | `127.0.0.1:9109` | HTTP listen address, `host:port`. `--listen` overrides it. | +| `state_dir` | string | no | unset | Directory for durable state. Created with mode `0700` when missing. | +| `ui` | bool | no | `false` | Serve the monitor UI and API. `--ui` overrides it, and `--ui=false` turns it off. | + +`state_dir` holds: + +- `notification-state.json`: open incidents, per-sink delivery state, and the + pending events of `queue: disk` sinks. Mode `0600`. Written whenever + `state_dir` is set and at least one sink exists. +- `targets.json`: monitors created in the UI. Mode `0600`. Read only when the UI + is on. + +`state_dir` is required when `ui` is on (startup exits with an error otherwise) +and when any sink sets `queue: disk`. `--once` ignores `ui`. + +## `defaults` + +Each key is the fallback for the same key on every target, including monitors +created in the UI. + +| Key | Type | Default | Constraint | +| --- | --- | --- | --- | +| `interval` | duration | `30s` | greater than 0 | +| `timeout` | duration | `5s` | greater than 0 and less than `interval` | +| `fail_threshold` | integer | `3` | at least 1 | +| `recover_threshold` | integer | `2` | at least 1 | +| `flap_limit` | integer | `4` | at least 1 | +| `flap_window` | duration | `15m` | greater than 0 | + +Constraints are checked on each target after defaults and overrides are merged. + +## `targets[]` + +| Key | Type | Required | Default | Meaning | +| --- | --- | --- | --- | --- | +| `name` | string | yes | | Unique identifier. Must match `[A-Za-z0-9._-]+`. Appears as the `target` metric label and in payloads. | +| `addr` | string | yes | | IP address or DNS name, including a MagicDNS short name. | +| `checks` | list of strings | yes | | Non-empty, no duplicates. See [Checks](checks.md). | +| `site` | string | no | empty | Free-form group label. Used for the `site` metric label and sink `match.sites`. | +| `severity` | string | no | `page` | `page` or `warn`. Severity of the `target.down` event. | +| `network` | string | no | `tailnet` | `tailnet` or `local`. | +| `parent` | string | no | empty | Name of another target whose only check is `icmp` or `tsping`. | +| `insecure_tls` | bool | no | `false` | Skip certificate verification for `http:` checks on this target. | +| `interval` | duration | no | `defaults.interval` | Time between runs of each check. | +| `timeout` | duration | no | `defaults.timeout` | Per-run cancellation time. | +| `fail_threshold` | integer | no | `defaults.fail_threshold` | Consecutive failures that mark the target `DOWN`. | +| `recover_threshold` | integer | no | `defaults.recover_threshold` | Consecutive successes that mark it `UP` again. | +| `flap_limit` | integer | no | `defaults.flap_limit` | State changes within `flap_window` that mark it flapping. | +| `flap_window` | duration | no | `defaults.flap_window` | Window for counting state changes. | + +Rules enforced across targets: + +- `tsping` is invalid when `network` is `local`. +- An `http:` check's URL host must equal `addr`. Comparison ignores case and a + trailing dot, and treats equivalent IP forms as equal. +- `parent` must name a target in the same list, and that target must have + exactly one check, either `icmp` or `tsping`. +- Parent cycles are rejected. +- `name` must be unique. + +For how the state, flap, and parent settings behave at run time, see +[Alert state](../explanation/alert-state.md) and +[Failure layers](../explanation/failure-layers.md). For `network`, see +[Monitor remote subnets and local LANs](../how-to/remote-subnets.md). + +## `sinks[]` + +A sink is a notification destination. For delivery behavior, see +[Incidents and queues](../explanation/incidents-and-queues.md), and for setup, see +[Send alerts to Slack](../how-to/slack-sink.md) and +[Send alerts to a webhook](../how-to/webhook-sink.md). + +### Common keys + +| Key | Type | Required | Default | Meaning | +| --- | --- | --- | --- | --- | +| `name` | string | yes | | Unique. Must match `[A-Za-z0-9._-]+`. `incidents`, `intake`, and `state` are reserved. | +| `type` | string | yes | | `webhook` or `slack`. | +| `url_env` | string | one of `url_env`, `url_file` | | Environment variable holding the destination URL. | +| `url_file` | string | one of `url_env`, `url_file` | | File holding the destination URL. | +| `min_severity` | string | no | `info` | `info`, `warn`, or `page`. Events below it are not sent. | +| `events` | list of strings | no | all types | Event types to send. See the list below. | +| `match` | mapping | no | no filter | Target filters. See below. | +| `max_age` | duration | no | `1h` | Age after which an undelivered event is dropped. Greater than 0. | +| `queue` | string | no | `memory` | `memory` or `disk`. `disk` requires `collector.state_dir`. | +| `queue_size` | integer | no | `1000` | Maximum events waiting. At least 1. | +| `group_wait` | duration | no | `10s` | How long to collect events into one request. Greater than 0. | + +Exactly one of `url_env` and `url_file` must be set. Values from a file have +surrounding whitespace trimmed. All secrets are read again on every send, so a +rotated secret needs no restart. + +`group_wait` groups events for a `slack` sink, and for a `webhook` sink only when +`batch` is `true`. + +Valid `events` values: + +| Value | Emitted | +| --- | --- | +| `target.down` | yes | +| `target.recovered` | yes | +| `target.flapping` | yes | +| `collector.started` | yes | +| `test` | yes, by `notify-test` only | +| `target.degraded` | never in this version; accepted and reserved | + +An event reaches a sink only when its type is listed (or `events` is unset), its +severity is at least `min_severity`, and every key in `match` matches. The `test` +event ignores all filters. An event with no target, such as `collector.started`, +ignores `match`. A recovery goes to the sinks that were selected for its down, +regardless of the current filters. + +### `match` + +All keys are optional. Within one key, any listed value may match. Across keys, +all must match. + +| Key | Type | Matches | +| --- | --- | --- | +| `sites` | list of strings | The target's `site`, exactly. | +| `targets` | list of strings | The target's `name`, as a glob (`*`, `?`, `[...]`). An invalid glob is rejected. | +| `tags` | list of strings | Any Tailscale tag on the node the target resolves to, such as `tag:router`. | + +### Webhook-only keys + +| Key | Type | Required | Default | Meaning | +| --- | --- | --- | --- | --- | +| `secret_env` | string | no | unset | Environment variable holding the HMAC key. | +| `secret_file` | string | no | unset | File holding the HMAC key. | +| `headers_env` | mapping | no | empty | Header name to the name of an environment variable holding its value. | +| `batch` | bool | no | `false` | Send several events per request. | +| `max_batch` | integer | no | `100` | Events per batch at most. At least 1. | + +Set at most one of `secret_env` and `secret_file`. Without a secret, requests are +unsigned and the collector logs a warning at startup. See +[Event payload](payload.md) for the signature and batch envelope. + +### Slack-only keys + +| Key | Type | Required | Default | Meaning | +| --- | --- | --- | --- | --- | +| `mention_on_page` | string | no | empty | Text prepended to a `target.down` of severity `page`, such as ``. | +| `status_url` | string | no | empty | Base URL. Each target line links to `/#`. | + +A `slack` sink rejects `batch`, `max_batch`, `secret_env`, `secret_file`, and +`headers_env`. A `webhook` sink rejects `mention_on_page` and `status_url`. + +## `heartbeat` + +Pings a dead-man's-switch URL with `GET`. The section is off when omitted. + +| Key | Type | Required | Default | Meaning | +| --- | --- | --- | --- | --- | +| `url_env` | string | one of `url_env`, `url_file` | | Environment variable holding the URL. | +| `url_file` | string | one of `url_env`, `url_file` | | File holding the URL. | +| `interval` | duration | no | `1m` | Time between pings. Greater than 0. | + +Pings start once every target and check has a first result. Each request times +out after 10s, and a non-2xx response or an error is counted in +`tailwatch_heartbeat_failures_total` and not retried early. See +[Watch the watcher with a heartbeat](../how-to/heartbeat.md). + +## Secret resolution + +At server startup, every `url_env`, `url_file`, `secret_env`, `secret_file`, +`headers_env` value, and the heartbeat URL must resolve to a non-empty value, or +startup exits with an error. `check-config` and `--once` do not resolve secrets. +`check-config` prints each sink's name, type, filters, and queue mode, and never a +secret. diff --git a/docs/reference/metrics.md b/docs/reference/metrics.md new file mode 100644 index 0000000..712d575 --- /dev/null +++ b/docs/reference/metrics.md @@ -0,0 +1,97 @@ +# Metrics reference + +The collector serves Prometheus text at `/metrics` on the listen address. Every +series starts with `tailwatch_`. The prefix has no hyphen because Prometheus metric +names cannot contain one. The default Go and process collectors are also exposed +and are not listed here. + +## Check series + +One set per target and check pair, registered by the scheduler when the check is +added and removed when the target is removed. Labels on every series: + +| Label | Value | +| --- | --- | +| `name` | `/`, such as `nyc-nas/tcp:445`. | +| `class` | Check kind: `icmp`, `tcp`, `http`, or `tsping`. | +| `target` | Target name. | +| `check` | Check string as configured, such as `tcp:445`. | +| `site` | Target `site`, empty when unset. | + +| Metric | Type | Extra labels | Meaning | +| --- | --- | --- | --- | +| `tailwatch_interval_secs` | gauge | | Probe interval in seconds. | +| `tailwatch_start_secs` | gauge | | Start time of the latest run, seconds since the epoch. | +| `tailwatch_end_secs` | gauge | | End time of the latest run, seconds since the epoch. | +| `tailwatch_latency_millis` | gauge | | Latency of the latest run in milliseconds. Absent when the latest run reported none. | +| `tailwatch_result` | gauge | | `1` if the latest run succeeded, `0` if it failed. | +| `tailwatch_in_flight` | gauge | | Runs in progress. | +| `tailwatch_attempts_total` | counter | `status` | Runs, by `status` `ok` or `fail`. | +| `tailwatch_seconds_total` | counter | `status` | Time spent running, in seconds, by `status`. | + +`tailwatch_result`, `tailwatch_end_secs`, and the two counters appear after the +first run finishes. These series come from the Tailscale `prober` package. + +## Classification and state series + +| Metric | Type | Labels | Meaning | +| --- | --- | --- | --- | +| `tailwatch_failure_layer_total` | counter | `target`, `layer` | Failed check results by assigned failure layer. Increments once per failed result. `layer` is `unrouted`, `control`, `tunnel`, `lan`, or `service`. | +| `tailwatch_path` | gauge | `target`, `kind` | Current `tsping` path kind. One series per `kind` (`direct`, `derp`, `peer_relay`); the active kind is `1` and the others are `0`. | +| `tailwatch_target_parent` | gauge | `target`, `parent` | `1` while `parent` is the target's current parent, configured or learned. Removed when the parent changes. | +| `tailwatch_target_state` | gauge | `target` | Alert state: `0` unknown, `1` up, `2` down, `3` suppressed, `4` flapping. | + +See [Failure layers](../explanation/failure-layers.md) and +[Alert state](../explanation/alert-state.md). + +## Notification series + +Registered only when at least one sink is configured, except +`tailwatch_heartbeat_failures_total`. The `sink` label is the sink `name`. + +| Metric | Type | Labels | Meaning | +| --- | --- | --- | --- | +| `tailwatch_notify_sent_total` | counter | `sink`, `type` | Notifications delivered, by event type. | +| `tailwatch_notify_failed_total` | counter | `sink`, `reason` | Failed delivery attempts. `reason` is `timeout`, `http_5xx`, `http_429`, `http_4xx_permanent`, or `network`. | +| `tailwatch_notify_dropped_total` | counter | `sink`, `reason` | Notifications dropped without delivery. `reason` is `queue_full`, `expired`, `permanent`, or `restart`. | +| `tailwatch_notify_persist_errors_total` | counter | `sink` | Failed writes of notification state. `sink` is always `state`. | +| `tailwatch_notify_queue_len` | gauge | `sink` | Events waiting in the sink's queue. | +| `tailwatch_notify_latency_seconds` | histogram | `sink` | Seconds from event creation to confirmed delivery. Buckets: 0.1, 0.5, 1, 5, 10, 30, 60, 300, 900, 3600. | +| `tailwatch_notify_batch_size` | histogram | `sink` | Events per request for batched webhook sinks. Buckets: 1, 2, 5, 10, 25, 50, 100, 250, 500. | +| `tailwatch_open_incidents` | gauge | | Incidents opened and not yet closed. Registered only when `collector.state_dir` is set. | + +`tailwatch_notify_dropped_total` also uses `sink="intake"` with `reason="queue_full"` +when the 4096-slot intake channel is full. The alerter's next reconcile, within a +second, repairs the lost transition. + +Drop reasons: + +| Reason | Cause | +| --- | --- | +| `queue_full` | The sink queue or the intake channel was full. | +| `expired` | The event was older than the sink's `max_age` when an attempt was due. | +| `permanent` | The receiver returned a non-retryable status, such as `404`. | +| `restart` | The event sat in a `memory` queue when the collector restarted. | + +See [Incidents and queues](../explanation/incidents-and-queues.md). + +## Heartbeat series + +| Metric | Type | Labels | Meaning | +| --- | --- | --- | --- | +| `tailwatch_heartbeat_failures_total` | counter | | Heartbeat pings that failed. Registered only when `heartbeat` is configured. | + +## Suggested alerts + +```yaml +- alert: TailWatchNotificationLoss + expr: increase(tailwatch_notify_dropped_total[15m]) > 0 +- alert: TailWatchQueueStuck + expr: min_over_time(tailwatch_notify_queue_len[15m]) > 0 +- alert: TargetDownQuorum + expr: count by (target) (tailwatch_target_state == 2) >= 2 + for: 1m +``` + +The first two mean an alert did not get out. The third pages only when two +collectors agree that a target is down. diff --git a/docs/reference/payload.md b/docs/reference/payload.md new file mode 100644 index 0000000..2c2019c --- /dev/null +++ b/docs/reference/payload.md @@ -0,0 +1,242 @@ +# Event payload reference + +Every notification is a JSON event. This page describes the version 1 payload, the +batch envelope, the webhook request, and the signature. For the Slack rendering of +the same events, see [Send alerts to Slack](../how-to/slack-sink.md). For how events +are produced and queued, see +[Incidents and queues](../explanation/incidents-and-queues.md). + +Within `version: 1`, changes only add fields. Receivers must ignore fields they do +not know. Fields that do not apply are omitted, never sent as `null`. + +## Event types + +| `type` | `severity` | Sent when | +| --- | --- | --- | +| `target.down` | The target's `severity`: `page` or `warn` | A target goes `DOWN` and has no open incident. | +| `target.recovered` | The severity of the down it resolves | A target with an open incident returns to `UP`, or the target is removed. | +| `target.flapping` | `warn` | A target crosses `flap_limit` state changes within `flap_window`. | +| `collector.started` | `info` | Once, after every target and check has a first result. | +| `test` | `info` | Only from `tail-watch notify-test`. | +| `target.degraded` | | Reserved. Never sent in this version. | + +Severity ranks `info` < `warn` < `page`. A sink's `min_severity` compares against it. +A start with every target `UP` sends only `collector.started`. A target already +failing at startup produces one `target.down`. + +## Fields + +| Field | Type | Present on | Meaning | +| --- | --- | --- | --- | +| `version` | integer | all | Always `1`. | +| `id` | string | all | Event id, see [Ids](#ids). | +| `type` | string | all | One of the types above. | +| `severity` | string | all | `info`, `warn`, or `page`. | +| `time` | string | all | Creation time, RFC 3339 in UTC with second precision. | +| `late` | boolean | any, when true | The first attempt happened more than one minute after `time`. The original `time` is kept. | +| `collector` | object | all | The sending collector. | +| `collector.name` | string | all | The node's hostname. | +| `collector.tailscale_ip` | string | when known | The node's first Tailscale IPv4 address, or IPv6 when it has none. | +| `target` | object | target events | The affected target. | +| `target.name` | string | target events | Target name. | +| `target.addr` | string | target events | Target address. | +| `target.site` | string | when set | Target site. | +| `target.parent` | string | when set | Parent target name. | +| `target.implicit` | boolean | when true | The target is a router tail-watch learned, not one from the config. | +| `check` | string | target events | The check that failed, such as `tcp:445`. | +| `layer` | string | down, flapping | Failure layer: `unrouted`, `control`, `tunnel`, `lan`, or `service`. | +| `error` | string | target events | The error from the failing check. | +| `since` | string | target events | When the target's outage began, RFC 3339 UTC. | +| `failures` | integer | target events | Consecutive failures that opened the incident. | +| `path` | object | when known | How the latest probe reached the target. | +| `path.kind` | string | when known | `direct`, `derp`, or `peer_relay`. | +| `path.endpoint` | string | direct paths | Remote UDP endpoint, `ip:port`. | +| `path.derp_region` | string | derp paths | DERP region code. | +| `path.latency_ms` | integer | when measured | Latency in milliseconds. | +| `dependents` | array of strings | down, recovered | Targets suppressed behind this one. Empty array when none. | +| `duration_seconds` | integer | recovered | Seconds from `since` to the recovery. | +| `down_event_id` | string | recovered | The `id` of the `target.down` it resolves. | +| `reason` | string | recovered, when set | `removed` when the target was deleted or edited while `DOWN`, rather than recovered. | + +`collector.started` and `test` carry only `version`, `id`, `type`, `severity`, +`time`, `collector`, and `late` when applicable. `target.recovered` has no `layer`. +`target.flapping` has no `dependents`. + +## Examples + +Full examples for each shape are in `cmd/tail-watch/testdata/payload/` in the +repository. + +### `target.down` + +```json +{ + "version": 1, + "id": "evt_01JAB3K9T6M4X2Q8R7N5W0YZCD", + "type": "target.down", + "severity": "page", + "time": "2026-04-02T14:02:11Z", + "collector": { + "name": "monitor-nyc", + "tailscale_ip": "100.101.102.103" + }, + "target": { + "name": "nyc-nas", + "addr": "10.20.0.50", + "site": "nyc", + "parent": "nyc-router" + }, + "check": "tcp:445", + "layer": "service", + "error": "connection refused", + "since": "2026-04-02T14:00:41Z", + "failures": 3, + "path": { + "kind": "direct", + "endpoint": "203.0.113.5:41641", + "latency_ms": 12 + }, + "dependents": [] +} +``` + +A learned router adds `"implicit": true` inside `target`. + +### `target.recovered` + +```json +{ + "version": 1, + "id": "evt_01JAB3K9T6M4X2Q8R7N5W0YZCG", + "type": "target.recovered", + "severity": "page", + "time": "2026-04-02T14:14:11Z", + "collector": { + "name": "monitor-nyc", + "tailscale_ip": "100.101.102.103" + }, + "target": { + "name": "nyc-nas", + "addr": "10.20.0.50", + "site": "nyc", + "parent": "nyc-router" + }, + "check": "tcp:445", + "error": "connection refused", + "since": "2026-04-02T14:00:41Z", + "failures": 3, + "path": { + "kind": "direct", + "endpoint": "203.0.113.5:41641", + "latency_ms": 12 + }, + "dependents": [ + "nyc-cam" + ], + "duration_seconds": 810, + "down_event_id": "evt_01JAB3K9T6M4X2Q8R7N5W0YZCD" +} +``` + +When the target was removed instead of recovering, the event also carries +`"reason": "removed"`. + +### `collector.started` and `test` + +```json +{ + "version": 1, + "id": "evt_01JAB3K9T6M4X2Q8R7N5W0YZCK", + "type": "collector.started", + "severity": "info", + "time": "2026-04-02T14:02:11Z", + "collector": { + "name": "monitor-nyc", + "tailscale_ip": "100.101.102.103" + } +} +``` + +## Ids + +An event `id` is `evt_` followed by 26 Crockford base32 characters +(`0123456789ABCDEFGHJKMNPQRSTVWXYZ`): 48 bits of Unix milliseconds followed by 80 +random bits. Ids sort by creation time across milliseconds. The id is the same +across every retry and every sink. + +A batch id is `bat_` followed by 26 characters of the same form. + +Delivery is at least once. Receivers must deduplicate on `id`, and must accept a +`target.recovered` whose `down_event_id` they never saw. + +## Webhook request + +Each request is a `POST` with these headers: + +| Header | Value | +| --- | --- | +| `Content-Type` | `application/json` | +| `User-Agent` | `tail-watch/` | +| `Tail-Watch-Event-Id` | The event `id`. Sent when `batch` is `false`. | +| `Tail-Watch-Batch-Id` | The `batch_id`. Sent when `batch` is `true`. | +| `Tail-Watch-Signature` | The signature. Sent when the sink has a secret. | +| any `headers_env` name | The value of the environment variable it names. | + +Redirects are not followed and each request times out after 10 seconds. + +| Response | Result | +| --- | --- | +| `2xx` | Delivered. | +| `408`, `5xx`, timeout, network error | Retried. | +| `429` | Retried, waiting at least `Retry-After` (seconds or an HTTP date). | +| Anything else, including `3xx` and other `4xx` | Dropped, logged once, counted as `permanent`. | + +The retry delay is `min(2^(attempt-1) seconds, 5 minutes)` scaled by 0.5 to 1.5 +times jitter, raised to `Retry-After` when longer, and never more than 5 minutes. +An event older than the sink's `max_age` when an attempt is due is dropped and +counted as `expired`. + +### Batch envelope + +With `batch: true`, the request body is: + +```json +{ + "version": 1, + "batch_id": "bat_01JAB3K9T6M4X2Q8R7N5W0YZCP", + "events": [] +} +``` + +`events` holds full event objects, in queue order. The sink waits `group_wait` after +the first queued event, or until `max_batch` events are waiting. A retry resends +identical body bytes and the same `batch_id`. Events that arrive meanwhile go into +the next batch. A permanent failure or expiry drops the whole batch, and every event +in it is counted. + +## Signature + +With `secret_env` or `secret_file`, each request carries: + +```text +Tail-Watch-Signature: t=,v1= +``` + +`v1` is the lowercase hex HMAC-SHA256, keyed with the secret, of the ASCII string +`.` followed by the exact request body bytes. For a batch the body is the whole +envelope. A retry sends the same body bytes with a fresh `t`. + +To verify, parse `t` and `v1`, reject a `t` outside your tolerance, recompute the +HMAC over `.`, and compare in constant time. Verify against the raw +bytes, not a re-serialized copy. + +```bash +{ printf '%s.' "$T"; cat body.json; } | openssl dgst -sha256 -hmac "$SECRET" -hex +``` + +## `late` + +The body of the first attempt is fixed when the request is first formed. If that +moment is more than one minute after the event's `time`, the event carries +`"late": true`, for example after a receiver outage or a collector restart. The +original `time` is never changed. diff --git a/docs/tutorial/first-monitor.md b/docs/tutorial/first-monitor.md new file mode 100644 index 0000000..57c03cc --- /dev/null +++ b/docs/tutorial/first-monitor.md @@ -0,0 +1,175 @@ +# Set up your first monitor + +In this tutorial you run tail-watch beside a Tailscale node, watch one device on your +tailnet, make that device fail, and see the alert arrive in Slack. It takes about 20 +minutes. + +You need: + +- Docker with Compose, on a machine that has `/dev/net/tun`. +- A tailnet where you can edit the policy file and create an auth key. +- A Slack workspace where you can add an incoming webhook. +- One device on your tailnet that you can reach by its MagicDNS name. This tutorial + calls it `my-device`. + +## 1. Allow the collector in your policy + +tail-watch runs as a node tagged `tag:monitor`. In your tailnet policy file, own the tag +and let it reach `my-device`. Replace `tag:server` with the tag your device has, or use +its user or IP instead: + +```jsonc +{ + "tagOwners": { "tag:monitor": ["autogroup:admin"] }, + "grants": [ + { + "src": ["tag:monitor"], + "dst": ["tag:server"], + "ip": ["*"] + } + ] +} +``` + +[Write the ACLs](../how-to/acls.md) explains each field. + +## 2. Create an auth key + +In the admin console, open **Settings → Keys** and generate an auth key. Tick +**Pre-approved** if your tailnet requires device approval, and choose `tag:monitor` under +**Tags**. Copy the key and export it: + +```bash +export TS_AUTHKEY=tskey-auth-... +``` + +## 3. Get a Slack webhook URL + +Follow [Get an incoming webhook URL](../how-to/slack-sink.md#get-an-incoming-webhook-url), +then export the URL: + +```bash +export SLACK_WEBHOOK_URL='https://hooks.slack.com/services/...' +``` + +## 4. Write a config + +Create `first-monitor.yaml` next to `examples/compose.yaml`. It has one target, the +tailnet ping of `my-device`, and one Slack sink: + +```yaml +collector: + listen: 127.0.0.1:9109 + state_dir: /var/lib/tail-watch + +defaults: + interval: 10s + timeout: 3s + fail_threshold: 2 + recover_threshold: 1 + +targets: + - name: my-device + addr: my-device + checks: [tsping] + +sinks: + - name: ops-slack + type: slack + url_env: SLACK_WEBHOOK_URL + group_wait: 5s +``` + +Check it. This reads the file and does not contact Tailscale or Slack: + +```bash +docker build -t tail-watch:local . +docker run --rm -v "$PWD/first-monitor.yaml:/first-monitor.yaml:ro" \ + -e SLACK_WEBHOOK_URL tail-watch:local \ + check-config --config /first-monitor.yaml +``` + +The output lists the target and the `ops-slack` sink. The shorter thresholds only make +the tutorial faster; the [defaults](../reference/config.md) are better for real use. + +## 5. Pass the webhook to the container + +`examples/compose.yaml` does not forward your environment to tail-watch. Add an override +file named `first-monitor.compose.yaml`: + +```yaml +services: + tail-watch: + environment: + SLACK_WEBHOOK_URL: ${SLACK_WEBHOOK_URL:?set SLACK_WEBHOOK_URL} +``` + +## 6. Start the stack + +```bash +TAIL_WATCH_CONFIG_FILE=./first-monitor.yaml \ + docker compose -f examples/compose.yaml -f first-monitor.compose.yaml up -d --build +``` + +The `tail-watch` node appears in your admin console with the `tag:monitor` tag. Check +the logs: + +```bash +docker compose -f examples/compose.yaml -f first-monitor.compose.yaml logs tail-watch +``` + +The last line is `listening` on `127.0.0.1:9109`. The `state_dir is readable by group +or others` warning is expected here, because the image creates that directory with mode +`0755`; it does not stop anything from working. Now send a test message through the real +Slack code: + +```bash +docker compose -f examples/compose.yaml -f first-monitor.compose.yaml \ + exec tail-watch /usr/local/bin/tail-watch notify-test \ + --config /etc/tail-watch/tail-watch.yaml +``` + +Slack shows a test message, and the command prints `ops-slack slack 200` and exits 0. +If it does not, see [Troubleshoot](../how-to/troubleshoot.md). + +You should also see `collector.started` arrive in the channel once every target has its +first result. + +## 7. Watch it work + +The status page runs on the node's loopback address, so open it from inside the +container namespace: + +```bash +docker compose -f examples/compose.yaml exec tailscale wget -qO- http://127.0.0.1:9109/ +``` + +`my-device` shows as up. + +## 8. Break it + +Stop Tailscale on `my-device`, or power the device off. Within about 30 seconds, +tail-watch marks the target `DOWN` and Slack shows a message that starts +`:red_circle: DOWN my-device`, with the failure layer in brackets. + +The layer says where the path broke. [Failure layers](../explanation/failure-layers.md) +explains why `control` and `tunnel` differ from `service` and `lan`. + +## 9. Fix it + +Start Tailscale on `my-device` again. After one successful check, tail-watch sends a +recovery message that names the original outage start and how long it lasted. + +## Clean up + +```bash +docker compose -f examples/compose.yaml -f first-monitor.compose.yaml down -v +``` + +Delete the `tail-watch` machine from the admin console if it does not expire on its own. + +## Next steps + +- [Deploy tail-watch](../how-to/deploy.md) on a host instead of Compose. +- [Monitor remote subnets](../how-to/remote-subnets.md) behind a router. +- [Configuration reference](../reference/config.md) for every key. diff --git a/openspec/changes/add-user-documentation/.openspec.yaml b/openspec/changes/archive/2026-09-29-add-user-documentation/.openspec.yaml similarity index 100% rename from openspec/changes/add-user-documentation/.openspec.yaml rename to openspec/changes/archive/2026-09-29-add-user-documentation/.openspec.yaml diff --git a/openspec/changes/add-user-documentation/design.md b/openspec/changes/archive/2026-09-29-add-user-documentation/design.md similarity index 88% rename from openspec/changes/add-user-documentation/design.md rename to openspec/changes/archive/2026-09-29-add-user-documentation/design.md index ab5fcd8..34320f0 100644 --- a/openspec/changes/add-user-documentation/design.md +++ b/openspec/changes/archive/2026-09-29-add-user-documentation/design.md @@ -34,16 +34,23 @@ sets in `main.go`), the archived specs under `openspec/specs/`, and docs/ README.md index, one line per page tutorial/first-monitor.md - how-to/ deploy.md acls.md remote-subnets.md - slack-sink.md webhook-sink.md troubleshoot.md + how-to/ deploy.md acls.md remote-subnets.md monitor-ui.md + slack-sink.md webhook-sink.md heartbeat.md + troubleshoot.md reference/ config.md checks.md cli.md metrics.md payload.md explanation/ failure-layers.md alert-state.md incidents-and-queues.md + monitor-ui-risks.md contributing/ live-notification-test.md releases.md ``` Alternative: a single long `docs/guide.md`. Rejected because it recreates the README problem and gives readers no stable links. +The monitor UI, heartbeat and egress path sections of the README have no page in the +first outline. They get `how-to/monitor-ui.md`, `explanation/monitor-ui-risks.md` and +`how-to/heartbeat.md`; the egress-path grant goes into the sink how-tos, and +`notify-test` goes into `reference/cli.md` with links from each sink page. + ### README becomes a landing page Keep the purpose paragraph, requirements summary, a five-line quickstart, and a link diff --git a/openspec/changes/add-user-documentation/proposal.md b/openspec/changes/archive/2026-09-29-add-user-documentation/proposal.md similarity index 100% rename from openspec/changes/add-user-documentation/proposal.md rename to openspec/changes/archive/2026-09-29-add-user-documentation/proposal.md diff --git a/openspec/changes/archive/2026-09-29-add-user-documentation/readme-map.md b/openspec/changes/archive/2026-09-29-add-user-documentation/readme-map.md new file mode 100644 index 0000000..aa0d37d --- /dev/null +++ b/openspec/changes/archive/2026-09-29-add-user-documentation/readme-map.md @@ -0,0 +1,43 @@ +# README migration map + +Working record for tasks 1.3 and 1.4. Not part of `docs/`. + +## Heading map + +| README section (before) | Target | +| ----------------------- | ------ | +| Intro (lines 1-9) | `README.md` overview | +| Requirements | `README.md` summary; `docs/how-to/deploy.md`; `docs/how-to/troubleshoot.md` | +| Remote subnets and local LANs | `docs/how-to/remote-subnets.md` | +| Build and run | `docs/how-to/deploy.md`; flags in `docs/reference/cli.md` | +| Docker | `docs/how-to/deploy.md` | +| Docker Compose without a dedicated host | `docs/how-to/deploy.md`; `docs/tutorial/first-monitor.md` | +| ACLs | `docs/how-to/acls.md` | +| Checks | `docs/reference/checks.md` | +| Failure layers | `docs/explanation/failure-layers.md` | +| Alert state | `docs/explanation/alert-state.md`; `docs/reference/metrics.md` | +| Monitor UI | `docs/how-to/monitor-ui.md` | +| Monitor UI / Reaching the UI | `docs/how-to/monitor-ui.md` | +| Monitor UI / Risks | `docs/explanation/monitor-ui-risks.md` | +| Notifications | `docs/reference/payload.md`; `docs/explanation/incidents-and-queues.md` | +| Notifications / Sink options | `docs/reference/config.md` | +| Notifications / Payload | `docs/reference/payload.md` | +| Notifications / Webhook sink | `docs/how-to/webhook-sink.md` | +| Notifications / Slack sink | `docs/how-to/slack-sink.md` | +| Notifications / Queues, `state_dir` and open incidents | `docs/explanation/incidents-and-queues.md`; metrics in `docs/reference/metrics.md` | +| Notifications / Watching the watcher | `docs/how-to/heartbeat.md` | +| Notifications / Trying it out | `docs/reference/cli.md`; linked from sink pages | +| Notifications / Egress path | `docs/how-to/heartbeat.md`; `docs/how-to/acls.md` | +| Notifications / Live notification test | `docs/contributing/live-notification-test.md` | +| Live notification test / Getting a Slack webhook URL | `docs/how-to/slack-sink.md` | +| Development and releases | `docs/contributing/releases.md` | + +## Inbound links + +A search for `README.md#` and `README#` found no anchor links outside +`openspec/changes/archive`. Other references to the README: + +| Location | Reference | Action | +| -------- | --------- | ------ | +| `scripts/live-stack.sh` line 4 | `README "Live notification test"` | Point at `docs/contributing/live-notification-test.md` | +| `.goreleaser.yaml` `archives.files` | ships `README.md` | Trimmed README links into `docs/`; see open question in the change summary | diff --git a/openspec/changes/add-user-documentation/tasks.md b/openspec/changes/archive/2026-09-29-add-user-documentation/tasks.md similarity index 53% rename from openspec/changes/add-user-documentation/tasks.md rename to openspec/changes/archive/2026-09-29-add-user-documentation/tasks.md index 5b1ee4b..fe4cba2 100644 --- a/openspec/changes/add-user-documentation/tasks.md +++ b/openspec/changes/archive/2026-09-29-add-user-documentation/tasks.md @@ -2,52 +2,55 @@ ## 1. Outline and scaffolding -- [ ] 1.1 Use the `documentation-writer` skill to confirm audience, goal and scope per page, and get the outline approved; verify the approved outline matches the tree in `design.md` -- [ ] 1.2 Create `docs/README.md` index and the empty quadrant directories; verify every planned page is listed and links resolve -- [ ] 1.3 Add a heading-by-heading map from current `README.md` sections to target pages (kept in the change directory, not committed to `docs/`); verify no README section is unassigned -- [ ] 1.4 Search the repo for `README.md#` anchors and inbound links (`CLAUDE.md`, `CHANGELOG.md`, `examples/`); verify each is listed with its new target +- [x] 1.1 Use the `documentation-writer` skill to confirm audience, goal and scope per page, and get the outline approved; verify the approved outline matches the tree in `design.md` +- [x] 1.2 Create `docs/README.md` index and the empty quadrant directories; verify every planned page is listed and links resolve +- [x] 1.3 Add a heading-by-heading map from current `README.md` sections to target pages (kept in the change directory, not committed to `docs/`); verify no README section is unassigned +- [x] 1.4 Search the repo for `README.md#` anchors and inbound links (`CLAUDE.md`, `CHANGELOG.md`, `examples/`); verify each is listed with its new target ## 2. Reference pages -- [ ] 2.1 Write `docs/reference/config.md` from `cmd/tail-watch/config.go` and `config_sinks.go`; verify every key in `examples/tail-watch.yaml` and every `sinks` key appears with type, default and required/optional -- [ ] 2.2 Save each full-config snippet from `docs/reference/config.md` and run `tail-watch check-config`; verify all pass -- [ ] 2.3 Write `docs/reference/cli.md` (flags, `TAIL_WATCH_CONFIG`, `TAIL_WATCH_LOG_LEVEL`, `check-config`, `notify-test`, exit codes) from `cmd/tail-watch/main.go` and `notifytest.go`; verify against `tail-watch --help` and `notify-test --help` -- [ ] 2.4 Write `docs/reference/metrics.md` from the metric registrations under `cmd/tail-watch/`; verify each documented `tailwatch_*` name appears in the source and each registered name is documented -- [ ] 2.5 Write `docs/reference/checks.md` (ICMP, TCP, HTTP, tsping options and semantics) from `checks.go`; verify against the `reachability-checks` spec -- [ ] 2.6 Write `docs/reference/payload.md` (event types, fields, severities, signature header) from `event.go` and `testdata/payload`; verify the field list matches the golden files +- [x] 2.1 Write `docs/reference/config.md` from `cmd/tail-watch/config.go` and `config_sinks.go`; verify every key in `examples/tail-watch.yaml` and every `sinks` key, `heartbeat` key and monitor UI key (`collector.ui`, `state_dir`) appears with type, default and required/optional +- [x] 2.2 Save each full-config snippet from `docs/reference/config.md` and run `tail-watch check-config`; verify all pass +- [x] 2.3 Write `docs/reference/cli.md` (flags, `TAIL_WATCH_CONFIG`, `TAIL_WATCH_LOG_LEVEL`, `check-config`, `notify-test`, exit codes) from `cmd/tail-watch/main.go` and `notifytest.go`; verify against `tail-watch --help` and `notify-test --help` +- [x] 2.4 Write `docs/reference/metrics.md` from the metric registrations under `cmd/tail-watch/`; verify each documented `tailwatch_*` name appears in the source and each registered name is documented +- [x] 2.5 Write `docs/reference/checks.md` (ICMP, TCP, HTTP, tsping options and semantics) from `checks.go`; verify against the `reachability-checks` spec +- [x] 2.6 Write `docs/reference/payload.md` (event types, fields, severities, signature header) from `event.go` and `testdata/payload`; verify the field list matches the golden files ## 3. Explanation pages -- [ ] 3.1 Write `docs/explanation/failure-layers.md` from the README "Failure layers" section and the `failure-classification` and `dependency-learning` specs; verify each layer name matches the `failure_layer` metric label values -- [ ] 3.2 Write `docs/explanation/alert-state.md` covering grace, flap suppression and dependency suppression, from the `alert-state-machine` and `dependency-suppression` specs; verify each state and timer name matches the config keys in `reference/config.md` -- [ ] 3.3 Write `docs/explanation/incidents-and-queues.md` covering `state_dir`, at-least-once delivery, replay and open incidents; verify against the `add-notifications` spec set once archived +- [x] 3.1 Write `docs/explanation/failure-layers.md` from the README "Failure layers" section and the `failure-classification` and `dependency-learning` specs; verify each layer name matches the `failure_layer` metric label values +- [x] 3.2 Write `docs/explanation/alert-state.md` covering grace, flap suppression and dependency suppression, from the `alert-state-machine` and `dependency-suppression` specs; verify each state and timer name matches the config keys in `reference/config.md` +- [x] 3.3 Write `docs/explanation/incidents-and-queues.md` covering `state_dir`, at-least-once delivery, replay and open incidents; verify against the `add-notifications` spec set once archived +- [x] 3.4 Write `docs/explanation/monitor-ui-risks.md` (no login, listen address, Origin and content-type guard, DNS rebinding, audit log) from README "Risks" and the `monitor-ui` and `monitor-management` specs; verify each claim against `monitor_guard.go` ## 4. How-to guides -- [ ] 4.1 Write `docs/how-to/deploy.md` (binary, Docker, Compose without a dedicated host) from README "Build and run" and "Docker"; verify the Docker build command runs and `--version` prints -- [ ] 4.2 Write `docs/how-to/acls.md` from README "ACLs"; verify the policy snippet is valid HuJSON -- [ ] 4.3 Write `docs/how-to/remote-subnets.md` from README "Remote subnets and local LANs"; verify each `network` value and startup failure named in it exists in `netenv.go` and `config.go` -- [ ] 4.4 Write `docs/how-to/slack-sink.md` including getting a webhook URL and using `*_env`; verify the config snippet passes `check-config` and `notify-test --sink` steps match `notifytest.go` -- [ ] 4.5 Write `docs/how-to/webhook-sink.md` (signing, headers, batching, retries); verify the signature verification example against `webhook_test.go` -- [ ] 4.6 Write `docs/how-to/troubleshoot.md` (userspace tailscaled refusal, missing `tag:monitor`, ICMP permissions, socket path, missing secrets, corrupt state); verify each symptom's message text against the source strings +- [x] 4.1 Write `docs/how-to/deploy.md` (binary, Docker, Compose without a dedicated host) from README "Build and run" and "Docker"; verify the Docker build command runs and `--version` prints +- [x] 4.2 Write `docs/how-to/acls.md` from README "ACLs"; verify the policy snippet is valid HuJSON +- [x] 4.3 Write `docs/how-to/remote-subnets.md` from README "Remote subnets and local LANs"; verify each `network` value and startup failure named in it exists in `netenv.go` and `config.go` +- [x] 4.4 Write `docs/how-to/slack-sink.md` including getting a webhook URL and using `*_env`; verify the config snippet passes `check-config` and `notify-test --sink` steps match `notifytest.go` +- [x] 4.5 Write `docs/how-to/webhook-sink.md` (signing, headers, batching, retries); verify the signature verification example against `webhook_test.go` +- [x] 4.6 Write `docs/how-to/troubleshoot.md` (userspace tailscaled refusal, missing `tag:monitor`, ICMP permissions, socket path, missing secrets, corrupt state); verify each symptom's message text against the source strings +- [x] 4.7 Write `docs/how-to/monitor-ui.md` (enable, `state_dir`, `targets.json`, reaching the UI over the tailnet, edit and delete semantics, screenshots from `docs/images/`); verify flags and behavior against `ui_serve.go`, `dynamic.go` and `monitor_api.go` +- [x] 4.8 Write `docs/how-to/heartbeat.md` (dead-man's-switch config, egress path and exit-node warning); verify against `heartbeat.go` ## 5. Tutorial -- [ ] 5.1 Write `docs/tutorial/first-monitor.md` starting from `examples/compose.yaml`, adding a failing target, and ending with a delivered alert; verify by following it on a clean checkout with `scripts/live-stack.sh` or the Compose file -- [ ] 5.2 Record any step that needed a fix while following it, and update the page; verify a second read-through needs no unwritten step +- [x] 5.1 Write `docs/tutorial/first-monitor.md` starting from `examples/compose.yaml`, adding a failing target, and ending with a delivered alert; verify by following it on a clean checkout with `scripts/live-stack.sh` or the Compose file +- [x] 5.2 Record any step that needed a fix while following it, and update the page; verify a second read-through needs no unwritten step ## 6. README and contributor split -- [ ] 6.1 Move "Live notification test" and "Development and releases" from `README.md` to `docs/contributing/`; verify commands still match `CLAUDE.md` and `scripts/live-stack.sh` -- [ ] 6.2 Trim `README.md` to overview, quickstart and links; verify against the 1.3 map that every removed section exists in a page -- [ ] 6.3 Fix or redirect the inbound links found in 1.4; verify a grep for the old anchors finds none -- [ ] 6.4 Add a `docs/` layout and "update docs with config or metric changes" note to `CLAUDE.md`; verify the note matches the final tree -- [ ] 6.5 Add a `CHANGELOG.md` entry if the release process lists documentation; verify against `release-please-config.json` +- [x] 6.1 Move "Live notification test" and "Development and releases" from `README.md` to `docs/contributing/`; verify commands still match `CLAUDE.md` and `scripts/live-stack.sh` +- [x] 6.2 Trim `README.md` to overview, quickstart and links; verify against the 1.3 map that every removed section exists in a page +- [x] 6.3 Fix or redirect the inbound links found in 1.4; verify a grep for the old anchors finds none +- [x] 6.4 Add a `docs/` layout and "update docs with config or metric changes" note to `CLAUDE.md`; verify the note matches the final tree +- [x] 6.5 Add a `CHANGELOG.md` entry if the release process lists documentation; verify against `release-please-config.json` ## 7. Validation -- [ ] 7.1 Run `mise exec -- go build ./...`, `go vet ./...` and `go test ./...`; verify all pass, confirming no code changed -- [ ] 7.2 Run `mise run lint` (golangci-lint) and `govulncheck ./...`; verify no findings -- [ ] 7.3 Run `markdownlint` over `README.md`, `docs/` and `openspec/changes/add-user-documentation/`; verify zero findings -- [ ] 7.4 Run a link check over `docs/` and `README.md` (local, relative links); verify none are broken -- [ ] 7.5 Run `mise exec -- openspec validate add-user-documentation --strict`; verify it passes +- [x] 7.1 Run `mise exec -- go build ./...`, `go vet ./...` and `go test ./...`; verify all pass, confirming no code changed +- [x] 7.2 Run `mise run lint` (golangci-lint) and `govulncheck ./...`; verify no findings +- [x] 7.3 Run `markdownlint` over `README.md`, `docs/` and `openspec/changes/add-user-documentation/`; verify zero findings +- [x] 7.4 Run a link check over `docs/` and `README.md` (local, relative links); verify none are broken +- [x] 7.5 Run `mise exec -- openspec validate add-user-documentation --strict`; verify it passes diff --git a/release-please-config.json b/release-please-config.json index 2f31bec..838ed51 100644 --- a/release-please-config.json +++ b/release-please-config.json @@ -4,7 +4,29 @@ ".": { "release-type": "go", "include-component-in-tag": false, - "bump-minor-pre-major": true + "bump-minor-pre-major": true, + "changelog-sections": [ + { + "type": "feat", + "section": "Features" + }, + { + "type": "fix", + "section": "Bug Fixes" + }, + { + "type": "perf", + "section": "Performance Improvements" + }, + { + "type": "revert", + "section": "Reverts" + }, + { + "type": "docs", + "section": "Documentation" + } + ] } } } diff --git a/scripts/live-stack.sh b/scripts/live-stack.sh index e0b4098..841ccf3 100755 --- a/scripts/live-stack.sh +++ b/scripts/live-stack.sh @@ -1,9 +1,9 @@ #!/usr/bin/env bash # Bring up (or tear down) only the tailnet side of the live notification test: # the collector's tailscale node plus one fixture node per check. It does not -# start the tail-watch process or run any scenario. See README "Live notification -# test". Credentials come from the pulumi env in TAIL_WATCH_PULUMI_ENV -# (default tailscale-phase-2/tailscale). +# start the tail-watch process or run any scenario. See +# docs/contributing/live-notification-test.md. Credentials come from the pulumi +# env in TAIL_WATCH_PULUMI_ENV (default tailscale-phase-2/tailscale). # # scripts/live-stack.sh up create ephemeral nodes and print their addresses # scripts/live-stack.sh down remove the containers and this stack's devices