Skip to content

labs/threshold-alerts: host-metric threshold alerts, every host and one server group - #154

Merged
litkhai merged 1 commit into
mainfrom
threshold-alerts
Nov 11, 2026
Merged

litkhai merged 1 commit into
mainfrom
threshold-alerts

Conversation

@litkhai

@litkhai litkhai commented Nov 11, 2026

Copy link
Copy Markdown
Owner

New lab and guide: threshold alerts on host metrics, built with the chart builder on Managed ClickStack.

  • Every host: one line chart grouped by host.name, alert at least 0.9. Each host is notified and resolved on its own.
  • One server group: the same chart filtered on svc.group = 'game-a', alert at least 0.8. The other group's host at 0.93 is not notified.
  • Consecutive windows 3: two windows Pending, then Alert from the third. In this run it resolved one window after the plain alert.

Verified on our Seoul service (ClickHouse 26.8.1.654), 2026-11-11 04:57–05:08Z, one run. Synthetic system.memory.utilization for six hosts. Notifications were counted at a local receiver behind a tunnel; links and hashes are redacted in results/. The eight screens come from the same run, and every test object is deleted (--cleanup).

From the HyperDX source (read 2026-11-11), "Average" on a gauge uses each series' last_value in the window. The run matches this: the first window fired even though the breach started 9 seconds before the window ended.

The lab is not published as a notes-site card (no web: true); that is a separate decision.

Closes #153

🤖 Generated with Claude Code

…ne server group (Managed ClickStack)

Builder line charts grouped by host.name with tile alerts: every host >= 0.9, svc.group game-a
only >= 0.8, and every host with 3 consecutive windows. Synthetic system.memory.utilization for
six hosts in two groups, notifications counted at a local receiver behind a tunnel. One run on
the Seoul service (ClickHouse 26.8.1.654), 2026-11-11 04:57-05:08Z: per-host notifications and
resolves, the other group filtered out, consecutive windows two windows Pending first. English
and Korean step-by-step guide with eight screens from the same run.

Closes #153

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@litkhai
litkhai merged commit a7237ed into main Nov 11, 2026
5 checks passed
@litkhai
litkhai deleted the threshold-alerts branch November 11, 2026 05:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

labs/threshold-alerts: one metric alert for every host, and one per server group (Managed ClickStack)

1 participant