Repository navigation
Conversation
Alert transitions reached nobody. Turn each one into a versioned JSON
event and deliver it to webhook and Slack sinks without a probe ever
waiting on a sink.
Each sink has its own worker and bounded queue, in-order delivery, a
retry schedule with Retry-After and max_age, and a late flag. Webhooks
are HMAC signed and can batch. Slack renders Block Kit with grouping, a
one message per second limit, and page-only mentions. An open incident
book keeps a recovery owed for every page, including across restarts:
with state_dir one atomic notification-state.json holds incidents,
per-sink delivery state and the pending events of queue: disk sinks, and
is committed before any worker can send from it. A dropped intake slot
is repaired from the alerter's dirty snapshots.
Also add the heartbeat dead-man's-switch ping and the notify-test
subcommand, and export the tailwatch_notify_* metrics.
The alert state machine now lets a DOWN implicit target be removed, so
authoritative topology reconciliation can close a restored learned
router's incident as removed. Configured targets are still refused.
Add a live test and compose fixtures with one tailnet node per check,
and scripts/live-stack.sh to bring up only those nodes.
Live verification, on a real tailnet with a Slack incoming webhook and a
receiver that verifies every signature:
- icmp, tsping, tcp, http and a LAN target behind a subnet router were
each failed and healed: one down and one recovery apiece, with the
original down_event_id, 15s outages, layers service, tunnel, service,
service and lan. All requests verified; 28 heartbeats arrived.
- Restarting the collector mid-outage with queue: disk did not page
again and delivered the original recovery.
- notify-test returned 200 for both sinks. The Slack channel showed the
start, DOWN and RECOVERED messages.
- The Slack sink uses a memory queue, so an alert still inside its
group_wait window when the collector restarts is lost and counted as
dropped{reason="restart"}. Use queue: disk to keep it.
- With a tag:monitor collector, the tailnet policy must grant tcp:9000
and tcp:8080 or those fixtures are down from the start; validated with
the policy test API, not applied. testdata/snippet.hujson has the
grants.
Updates tailscale/tailscale#5967
Signed-off-by: Richard Shade <rshade98@hotmail.com>
Add the add-user-documentation change. It proposes moving the README into a docs/ tree organized by Diataxis quadrant (tutorial, how-to, reference, explanation), leaving a short README as the landing page. New pages cover a first-monitor tutorial, the config, CLI and metrics references, and troubleshooting. Contributor material moves to docs/contributing/. The change sets skip_specs because it alters no system behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.