Skip to content

labs/threshold-alerts: host-metric threshold alerts, every host and one server group #154

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
litkhai merged 1 commit into from
Nov 11, 2026
Merged

labs/threshold-alerts: host-metric threshold alerts, every host and one server group #154

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ Observability on ClickHouse: OpenTelemetry ingestion, the ClickStack/HyperDX UI,
| [labs/promql](labs/promql/) | PromQL in HyperDX on a local ClickStack 2.42.0: turn on ClickHouse's `TimeSeries` engine and Prometheus HTTP API, fill the table from a refreshable MV, register a PromQL source, check rate and `histogram_quantile` with SQL and HTTP |
| [labs/process-down-alerts](labs/process-down-alerts/) | Does a ClickStack alert fire when one host's process series stops? Five tile alerts on local 2.42.0: a group-by alert stays silent, a SQL tile against an expected host list fires in 76 s (verified 2026-11-06) |
| [labs/common-fields](labs/common-fields/) | Adoption guide for the common fields (service taxonomy, host, environment): where apps and hosts set them, a gateway rename from EC2 tags, and ClickStack tables with them as columns in the sort key; schema tested in clickhouse local, rename in otelcol-contrib 0.155.0, demo on Managed ClickStack (2026-11-06) |
| [labs/threshold-alerts](labs/threshold-alerts/) | Host-metric threshold alerts in the chart builder: one rule for every host (a notification per host) and one for a server group only, plus consecutive windows; step-by-step with screens, verified on Managed ClickStack (2026-11-11) |
| [labs/hostmetrics-volume](labs/hostmetrics-volume/) | Linux and Windows EC2: data points and wire bytes per host at 1-minute hostmetrics, four ways to watch a Windows game-server process stop (process scraper, perfcounters, `windows_service`, a check script) and one gateway behind an endpoint service (verified 2026-11-06) |
| [labs/llm-serving-observability](labs/llm-serving-observability/) | Self-hosted LLM serving (vLLM behind LiteLLM) observed with ClickStack, runnable without a GPU: the OTel Collector configuration, an `nvidia-smi` → OTLP pusher, LiteLLM gateway-metric cardinality with and without the attribute allowlist, and the 32-tile HyperDX dashboard imported onto hand-made sources |
| [labs/vault-audit](labs/vault-audit/) | HashiCorp Vault OSS audit trail and HA metrics in ClickStack: a three-node Raft cluster's audit files (written by whichever node is active) through `file_log` and OTTL into OTel logs — TraceId per request, issuance joined to later use on HMAC'd accessors, no secret in clear — plus per-node metrics with `unauthenticated_metrics_access`; failover checked (verified locally and on Managed ClickStack, 2026-11-06) |
Expand Down Expand Up @@ -116,6 +117,7 @@ ClickHouse 기반 관측성 실습입니다. OpenTelemetry 수집, ClickStack/Hy
| [labs/promql](labs/promql/) | 로컬 ClickStack 2.42.0에서 HyperDX로 PromQL: ClickHouse `TimeSeries` 엔진과 Prometheus HTTP API를 켜고, 리프레시 MV로 테이블을 채우고, PromQL 소스를 등록해 rate와 `histogram_quantile`을 SQL과 HTTP로 확인 |
| [labs/process-down-alerts](labs/process-down-alerts/) | 호스트 하나의 프로세스 시계열이 끊기면 ClickStack 알림이 울리는가: 로컬 2.42.0에서 타일 알림 다섯 가지 — group by 알림은 울리지 않고, 기대 호스트 목록 대비 SQL 타일은 76초 뒤 울림 (2026-11-06 검증) |
| [labs/common-fields](labs/common-fields/) | 공통 필드(서비스 분류, 호스트, 환경) 적용 가이드: 앱과 호스트가 넣는 곳, EC2 태그를 gateway에서 표준 이름으로, 그 필드를 컬럼으로 정렬 키에 둔 ClickStack 테이블. 스키마는 clickhouse local, 이름 바꾸기는 otelcol-contrib 0.155.0, 데모는 Managed ClickStack에서 확인 (2026-11-06) |
| [labs/threshold-alerts](labs/threshold-alerts/) | 차트 빌더로 만드는 호스트 메트릭 임계치 알림: 규칙 하나로 전체 호스트(호스트마다 알림), 특정 서버군만, 연속 구간까지. 화면과 함께 단계별로, Managed ClickStack에서 확인 (2026-11-11) |
| [labs/hostmetrics-volume](labs/hostmetrics-volume/) | Linux·Windows EC2: 1분 hostmetrics의 호스트당 data point와 전송 바이트, Windows 게임 서버 프로세스 중지를 보는 네 방법(process scraper, perfcounters, `windows_service`, 확인 스크립트), endpoint service 뒤의 gateway 하나 (2026-11-06 검증) |
| [labs/llm-serving-observability](labs/llm-serving-observability/) | LiteLLM 뒤의 vLLM을 ClickStack으로 관측, GPU 없이 실행: OTel Collector 설정, `nvidia-smi` → OTLP 푸셔, 속성 허용 목록 유무에 따른 LiteLLM 게이트웨이 메트릭 카디널리티, 손으로 만든 소스에 가져온 HyperDX 대시보드(타일 32개) |
| [labs/vault-audit](labs/vault-audit/) | HashiCorp Vault OSS 감사 기록과 HA 메트릭을 ClickStack에서: Raft 3노드 클러스터의 감사 파일(그때 active인 노드가 씀)을 `file_log`와 OTTL로 OTel 로그로 — 요청마다 TraceId, HMAC 처리된 accessor로 발급과 이후 사용 연결, 평문 시크릿 없음 — 그리고 `unauthenticated_metrics_access`로 노드별 메트릭, 장애 전환 검사(2026-11-06 로컬과 Managed ClickStack에서 검증) |
Expand Down
Loading
Loading