Skip to content

cmd/tail-watch,openspec: add notification sinks and incident delivery - #1

Merged
rshade merged 2 commits into
mainfrom
add-notifications
Sep 29, 2026
Merged

rshade merged 2 commits into
mainfrom
add-notifications

Conversation

@rshade

@rshade rshade commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

Alert transitions reached nobody. Turn each one into a versioned JSON event and deliver it to webhook and Slack sinks without a probe ever waiting on a sink.

Each sink has its own worker and bounded queue, in-order delivery, a retry schedule with Retry-After and max_age, and a late flag. Webhooks are HMAC signed and can batch. Slack renders Block Kit with grouping, a one message per second limit, and page-only mentions. An open incident book keeps a recovery owed for every page, including across restarts: with state_dir one atomic notification-state.json holds incidents, per-sink delivery state and the pending events of queue: disk sinks, and is committed before any worker can send from it. A dropped intake slot is repaired from the alerter's dirty snapshots.

Also add the heartbeat dead-man's-switch ping and the notify-test subcommand, and export the tailwatch_notify_* metrics.

The alert state machine now lets a DOWN implicit target be removed, so authoritative topology reconciliation can close a restored learned router's incident as removed. Configured targets are still refused.

Add a live test and compose fixtures with one tailnet node per check, and scripts/live-stack.sh to bring up only those nodes.

Live verification, on a real tailnet with a Slack incoming webhook and a receiver that verifies every signature:

  • icmp, tsping, tcp, http and a LAN target behind a subnet router were each failed and healed: one down and one recovery apiece, with the original down_event_id, 15s outages, layers service, tunnel, service, service and lan. All requests verified; 28 heartbeats arrived.
  • Restarting the collector mid-outage with queue: disk did not page again and delivered the original recovery.
  • notify-test returned 200 for both sinks. The Slack channel showed the start, DOWN and RECOVERED messages.
  • The Slack sink uses a memory queue, so an alert still inside its group_wait window when the collector restarts is lost and counted as dropped{reason="restart"}. Use queue: disk to keep it.
  • With a tag:monitor collector, the tailnet policy must grant tcp:9000 and tcp:8080 or those fixtures are down from the start; validated with the policy test API, not applied. testdata/snippet.hujson has the grants.

Updates tailscale/tailscale#5967

Alert transitions reached nobody. Turn each one into a versioned JSON
event and deliver it to webhook and Slack sinks without a probe ever
waiting on a sink.

Each sink has its own worker and bounded queue, in-order delivery, a
retry schedule with Retry-After and max_age, and a late flag. Webhooks
are HMAC signed and can batch. Slack renders Block Kit with grouping, a
one message per second limit, and page-only mentions. An open incident
book keeps a recovery owed for every page, including across restarts:
with state_dir one atomic notification-state.json holds incidents,
per-sink delivery state and the pending events of queue: disk sinks, and
is committed before any worker can send from it. A dropped intake slot
is repaired from the alerter's dirty snapshots.

Also add the heartbeat dead-man's-switch ping and the notify-test
subcommand, and export the tailwatch_notify_* metrics.

The alert state machine now lets a DOWN implicit target be removed, so
authoritative topology reconciliation can close a restored learned
router's incident as removed. Configured targets are still refused.

Add a live test and compose fixtures with one tailnet node per check,
and scripts/live-stack.sh to bring up only those nodes.

Live verification, on a real tailnet with a Slack incoming webhook and a
receiver that verifies every signature:

- icmp, tsping, tcp, http and a LAN target behind a subnet router were
  each failed and healed: one down and one recovery apiece, with the
  original down_event_id, 15s outages, layers service, tunnel, service,
  service and lan. All requests verified; 28 heartbeats arrived.
- Restarting the collector mid-outage with queue: disk did not page
  again and delivered the original recovery.
- notify-test returned 200 for both sinks. The Slack channel showed the
  start, DOWN and RECOVERED messages.
- The Slack sink uses a memory queue, so an alert still inside its
  group_wait window when the collector restarts is lost and counted as
  dropped{reason="restart"}. Use queue: disk to keep it.
- With a tag:monitor collector, the tailnet policy must grant tcp:9000
  and tcp:8080 or those fixtures are down from the start; validated with
  the policy test API, not applied. testdata/snippet.hujson has the
  grants.

Updates tailscale/tailscale#5967

Signed-off-by: Richard Shade <rshade98@hotmail.com>
TestPresenceRunReconnect read the log right after it saw the bus marked
unseeded, but run clears the seeded state before it writes the
"presence bus disconnected" line, so the log could still be empty. Wait
for the line before counting it. This failed in CI and reproduced
locally within 400 runs under -race.

TestNotifyMetricsAfterScriptedRun waited for any new timer before
advancing the fake clock. With two sinks either could arm first, and the
failing sink's retry timer could be missed and never fire. Wait for both.

Updates tailscale/tailscale#5967

Signed-off-by: Richard Shade <rshade98@hotmail.com>
@rshade
rshade merged commit a339300 into main Sep 29, 2026
3 checks passed
@rshade
rshade deleted the add-notifications branch September 29, 2026 18:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant