diff --git a/README.md b/README.md index 973f000..4a1afb1 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,7 @@ Observability on ClickHouse: OpenTelemetry ingestion, the ClickStack/HyperDX UI, | [labs/promql](labs/promql/) | PromQL in HyperDX on a local ClickStack 2.42.0: turn on ClickHouse's `TimeSeries` engine and Prometheus HTTP API, fill the table from a refreshable MV, register a PromQL source, check rate and `histogram_quantile` with SQL and HTTP | | [labs/process-down-alerts](labs/process-down-alerts/) | Does a ClickStack alert fire when one host's process series stops? Five tile alerts on local 2.42.0: a group-by alert stays silent, a SQL tile against an expected host list fires in 76 s (verified 2026-11-06) | | [labs/common-fields](labs/common-fields/) | Adoption guide for the common fields (service taxonomy, host, environment): where apps and hosts set them, a gateway rename from EC2 tags, and ClickStack tables with them as columns in the sort key; schema tested in clickhouse local, rename in otelcol-contrib 0.155.0, demo on Managed ClickStack (2026-11-06) | +| [labs/threshold-alerts](labs/threshold-alerts/) | Host-metric threshold alerts in the chart builder: one rule for every host (a notification per host) and one for a server group only, plus consecutive windows; step-by-step with screens, verified on Managed ClickStack (2026-11-11) | | [labs/hostmetrics-volume](labs/hostmetrics-volume/) | Linux and Windows EC2: data points and wire bytes per host at 1-minute hostmetrics, four ways to watch a Windows game-server process stop (process scraper, perfcounters, `windows_service`, a check script) and one gateway behind an endpoint service (verified 2026-11-06) | | [labs/llm-serving-observability](labs/llm-serving-observability/) | Self-hosted LLM serving (vLLM behind LiteLLM) observed with ClickStack, runnable without a GPU: the OTel Collector configuration, an `nvidia-smi` → OTLP pusher, LiteLLM gateway-metric cardinality with and without the attribute allowlist, and the 32-tile HyperDX dashboard imported onto hand-made sources | | [labs/vault-audit](labs/vault-audit/) | HashiCorp Vault OSS audit trail and HA metrics in ClickStack: a three-node Raft cluster's audit files (written by whichever node is active) through `file_log` and OTTL into OTel logs — TraceId per request, issuance joined to later use on HMAC'd accessors, no secret in clear — plus per-node metrics with `unauthenticated_metrics_access`; failover checked (verified locally and on Managed ClickStack, 2026-11-06) | @@ -116,6 +117,7 @@ ClickHouse 기반 관측성 실습입니다. OpenTelemetry 수집, ClickStack/Hy | [labs/promql](labs/promql/) | 로컬 ClickStack 2.42.0에서 HyperDX로 PromQL: ClickHouse `TimeSeries` 엔진과 Prometheus HTTP API를 켜고, 리프레시 MV로 테이블을 채우고, PromQL 소스를 등록해 rate와 `histogram_quantile`을 SQL과 HTTP로 확인 | | [labs/process-down-alerts](labs/process-down-alerts/) | 호스트 하나의 프로세스 시계열이 끊기면 ClickStack 알림이 울리는가: 로컬 2.42.0에서 타일 알림 다섯 가지 — group by 알림은 울리지 않고, 기대 호스트 목록 대비 SQL 타일은 76초 뒤 울림 (2026-11-06 검증) | | [labs/common-fields](labs/common-fields/) | 공통 필드(서비스 분류, 호스트, 환경) 적용 가이드: 앱과 호스트가 넣는 곳, EC2 태그를 gateway에서 표준 이름으로, 그 필드를 컬럼으로 정렬 키에 둔 ClickStack 테이블. 스키마는 clickhouse local, 이름 바꾸기는 otelcol-contrib 0.155.0, 데모는 Managed ClickStack에서 확인 (2026-11-06) | +| [labs/threshold-alerts](labs/threshold-alerts/) | 차트 빌더로 만드는 호스트 메트릭 임계치 알림: 규칙 하나로 전체 호스트(호스트마다 알림), 특정 서버군만, 연속 구간까지. 화면과 함께 단계별로, Managed ClickStack에서 확인 (2026-11-11) | | [labs/hostmetrics-volume](labs/hostmetrics-volume/) | Linux·Windows EC2: 1분 hostmetrics의 호스트당 data point와 전송 바이트, Windows 게임 서버 프로세스 중지를 보는 네 방법(process scraper, perfcounters, `windows_service`, 확인 스크립트), endpoint service 뒤의 gateway 하나 (2026-11-06 검증) | | [labs/llm-serving-observability](labs/llm-serving-observability/) | LiteLLM 뒤의 vLLM을 ClickStack으로 관측, GPU 없이 실행: OTel Collector 설정, `nvidia-smi` → OTLP 푸셔, 속성 허용 목록 유무에 따른 LiteLLM 게이트웨이 메트릭 카디널리티, 손으로 만든 소스에 가져온 HyperDX 대시보드(타일 32개) | | [labs/vault-audit](labs/vault-audit/) | HashiCorp Vault OSS 감사 기록과 HA 메트릭을 ClickStack에서: Raft 3노드 클러스터의 감사 파일(그때 active인 노드가 씀)을 `file_log`와 OTTL로 OTel 로그로 — 요청마다 TraceId, HMAC 처리된 accessor로 발급과 이후 사용 연결, 평문 시크릿 없음 — 그리고 `unauthenticated_metrics_access`로 노드별 메트릭, 장애 전환 검사(2026-11-06 로컬과 Managed ClickStack에서 검증) | diff --git a/labs/threshold-alerts/README.md b/labs/threshold-alerts/README.md new file mode 100644 index 0000000..4eeccf1 --- /dev/null +++ b/labs/threshold-alerts/README.md @@ -0,0 +1,244 @@ +# Threshold alerts for host metrics in ClickStack: every host, or one server group + +[English](#english) | [한국어](#한국어) + +## English + +Two alerts an infra team asks for first, built in the ClickStack chart builder: + +1. **One threshold for every server** — for example memory used at least 90%, with a separate + notification for each host that crosses it. +2. **A threshold for one server group only** — the hosts whose `svc.group` resource attribute + (set from an EC2 tag, see [`labs/common-fields`](../common-fields/README.md)) is `game-a`. + +> **Verified on** Managed ClickStack, our ClickHouse Cloud Seoul service (ClickHouse 26.8.1.654), +> 2026-11-11 04:57–05:08Z, one run. [`managed/threshold_alerts.py`](managed/threshold_alerts.py) +> created the metric source, dashboard, builder charts, alerts and a generic webhook through the +> ClickHouse Cloud API, sent synthetic `system.memory.utilization` for six hosts every 20 s, and +> counted the notifications at a local receiver behind a tunnel. Screens are from the same run. +> Report: [`results/2026-11-11-managed.md`](results/2026-11-11-managed.md). + +### Result + +Six hosts, two groups (`game-a-01..03`, `game-b-01..03`), all at 0.50; for six minutes +game-a-01 at 0.95, game-a-02 at 0.85, game-b-02 at 0.93; then all back at 0.50. + +| Alert (builder line chart, Average, group by `host.name`, every 1 m) | Notified | Not notified | First window | Resolved | +|---|---|---|---|---| +| A1 every host, **≥ 0.9** | game-a-01, game-b-02 (6 each) | game-a-02 (0.85) and the rest | 05:00–05:01 | 05:06–05:07, per host | +| A2 `svc.group = 'game-a'` only, **≥ 0.8** | game-a-01, game-a-02 (6 each) | game-b-02 (0.93, other group) | 05:00–05:01 | 05:06–05:07, per host | +| A3 as A1 with **3 consecutive windows** | game-a-01, game-b-02 (5 each) | as A1 | 05:02–05:03 (2 windows Pending) | 05:07–05:08, per host | + +What this means: + +1. **One rule covers every host.** Group the chart by `host.name`; the alert then evaluates each + host as its own group and notifies (and later resolves) each host on its own. The alert's + history shows the groups of every window (screen 7). +2. **A server group is a filter on the same chart.** Add the group attribute to the chart's + Where. A different threshold per group is one chart and alert per group. +3. **A breaching host is notified every window** (6 notifications in a 6-minute breach). + Consecutive windows delays the first notification (here 2 windows Pending, then Alert) and + still repeats after that; with many hosts, also look at the interval and how the receiving + channel groups messages. + +### Step by step + +**1. A chart grouped by host.** Dashboards → Add → Time Series. Data Source: your metric source. +Average of `system.memory.utilization`, Where (SQL) `Attributes['state'] = 'used'`, Group By +`ResourceAttributes['host.name']`. The hostmetrics memory scraper sends one series per state +(used, free); the filter keeps one series per host. + +![Chart editor: Average of system.memory.utilization, Where Attributes['state'] = 'used', Group By ResourceAttributes['host.name'], threshold band above 0.9](images/02-every-host-chart.webp) + +**2. The alert on that chart.** Edit alert → Static threshold, "is at least (≥)" 0.9, Send to a +webhook (Slack or generic). The message template may use `{{group}}` and `{{value}}`. + +![Alert panel: Static threshold, is at least 0.9, send to webhook threshold-alerts lab](images/03-alert-settings.webp) + +The panel showed "over 5 minute" for these alerts, created with `interval: 1m` through the API; +the alert's own page says "Evaluates every 1m", and notifications came every minute. Pick the +interval you want when you create the alert in the UI. + +**3. One server group.** The same chart with the group in Where: +`Attributes['state'] = 'used' AND ResourceAttributes['svc.group'] = 'game-a'`, alert at 0.8. + +![Chart editor with Where Attributes['state'] = 'used' AND ResourceAttributes['svc.group'] = 'game-a': only game-a hosts, threshold band above 0.8](images/04-group-filter.webp) + +**4. Fewer notifications for short spikes.** Alert → Advanced Settings → Consecutive windows. + +![Advanced Settings: Consecutive windows 3](images/05-consecutive-windows.webp) + +**5. Where to see it.** The dashboard (the bell turns red while an alert fires), the Alerts page, +and each alert's page with one row per window and, expanded, one row per host. + +![Dashboard with the three charts, 1-minute granularity: game-a-01, game-b-02 and game-a-02 rise from 05:00 to 05:06](images/01-dashboard.webp) + +![Alerts page: the three alerts with their history bars; A3 shows orange (Pending) bars before red](images/06-alerts-list.webp) + +![A1 evaluation history: 2:00–2:05 PM KST "2/6 groups firing"; expanded, game-a-01 0.95 and game-b-02 0.93 Alert, game-a-02 0.85 OK](images/07-every-host-history.webp) + +![A3 evaluation history: Pending at 2:00 and 2:01 PM KST, Alert from 2:02 to 2:06 PM](images/08-consecutive-history.webp) + +The alert pages show local time (KST here); the notifications and the report use UTC. + +### What to know before rolling it out + +- **A gauge is read as the last value in each window.** "Average" on a gauge metric takes each + series' last value in the window, then averages across series (`last_value(Value)` in + `packages/common-utils/src/core/renderChartConfig.ts`, HyperDX source read 2026-11-11). Grouped + by host with one series per host, that is the host's latest sample. Seen here: the breach + started at 05:00:51 and the 05:00–05:01 window already fired. With the hostmetrics default of + one sample per minute, each window holds one sample anyway. +- **The notification title rounds the value** to one decimal: 0.93 arrived as + "0.9 meets or exceeds 0.9". `{{value}}` in the message carried 0.93. "is at least" is ≥. +- **The group label is the expression**, as received: + `Group: "arrayElement(ResourceAttributes, 'host.name'):game-b-02"`. +- **A host that stops sending does not fire a threshold alert** — there is no value to compare. + That needs the expected-host-list alert in [`labs/process-down-alerts`](../process-down-alerts/README.md). +- **Consecutive windows, as measured once:** A3 fired from its third window and resolved one + window after A1 (05:07–05:08 against 05:06–05:07), on the same chart query. + +### Run it + +```bash +ngrok http 9310 # or any tunnel to 127.0.0.1:9310; optional +managed/threshold_alerts.py --webhook-url https:// [--keep] # ~14 min +managed/threshold_alerts.py --cleanup # after --keep +managed/threshold_alerts.py --report # report again from results/ +``` + +Env files: `CH_HOST`, `CH_USER`, `CH_PASSWORD` (`--ch-env`) and `CHC_KEY`, `CHC_SECRET` +(`--api-env`). Everything it creates is named `threshold-alerts lab` and lives in the database +`threshold_alerts_lab`; `--cleanup` removes it. Without `--webhook-url` the webhook points at +example.com and only alert states are recorded (the Cloud API reports one state per alert, not +per host). Links and per-alert hashes are redacted before anything is written to `results/`. + +| File | What | +|---|---| +| `managed/threshold_alerts.py` | source, dashboard, alerts, webhook; data sender; receiver; report | +| `results/2026-11-11-managed.md` | alert states every 30 s and every notification | +| `results/2026-11-11-webhook.jsonl`, `-managed-states.jsonl`, `-phases.json` | the raw records | +| `images/` | the screens above, from the same run | + +Not covered: a Slack channel (the generic webhook carries the same text), CPU +(`system.cpu.utilization` has one series per CPU and state, so its filter and threshold differ — +not run here), one rule with a different threshold per group, silences, real hosts (the data is +synthetic in the hostmetrics shape), and the local OSS build. + +--- + +## 한국어 + +인프라 팀이 가장 먼저 찾는 알림 두 가지를 ClickStack 차트 빌더로 만듭니다. + +1. **전체 서버에 같은 임계치** — 예: 메모리 사용률 90% 이상. 넘은 호스트마다 알림이 따로 갑니다. +2. **특정 서버군만 임계치** — 리소스 속성 `svc.group`이 `game-a`인 호스트만. 이 속성은 EC2 태그에서 + 넣습니다([`labs/common-fields`](../common-fields/README.md) 참고). + +> **검증:** Managed ClickStack, 우리 ClickHouse Cloud 서울 서비스(ClickHouse 26.8.1.654), +> 2026-11-11 04:57–05:08Z, 1회. [`managed/threshold_alerts.py`](managed/threshold_alerts.py)가 ClickHouse +> Cloud API로 메트릭 소스, 대시보드, 빌더 차트, 알림, generic webhook을 만들고, 호스트 6대의 합성 +> `system.memory.utilization`을 20초마다 보내고, 터널 뒤 로컬 수신기에서 알림을 셌습니다. 화면도 같은 +> 실행에서 찍었습니다. 보고: [`results/2026-11-11-managed.md`](results/2026-11-11-managed.md). + +### 결과 + +호스트 6대, 서버군 2개(`game-a-01..03`, `game-b-01..03`), 모두 0.50에서 시작합니다. 6분 동안 +game-a-01은 0.95, game-a-02는 0.85, game-b-02는 0.93으로 올리고, 그 뒤 모두 0.50으로 되돌렸습니다. + +| 알림 (빌더 line 차트, Average, `host.name`으로 group by, 1분마다) | 알림 받음 | 알림 없음 | 첫 구간 | 해소 | +|---|---|---|---|---| +| A1 전체 호스트, **0.9 이상** | game-a-01, game-b-02 (각 6건) | game-a-02(0.85)와 나머지 | 05:00–05:01 | 05:06–05:07, 호스트별 | +| A2 `svc.group = 'game-a'`만, **0.8 이상** | game-a-01, game-a-02 (각 6건) | game-b-02(0.93, 다른 서버군) | 05:00–05:01 | 05:06–05:07, 호스트별 | +| A3 A1과 같고 **연속 3구간** | game-a-01, game-b-02 (각 5건) | A1과 같음 | 05:02–05:03 (2구간 Pending) | 05:07–05:08, 호스트별 | + +뜻: + +1. **규칙 하나로 모든 호스트를 봅니다.** 차트를 `host.name`으로 group by 하면 알림이 호스트마다 따로 + 평가하고, 호스트마다 따로 알리고 해소합니다. 알림 이력에서 구간마다 호스트별 상태를 볼 수 있습니다 + (화면 7). +2. **서버군은 같은 차트에 거는 필터입니다.** 차트의 Where에 서버군 속성을 더합니다. 서버군마다 임계치가 + 다르면 서버군마다 차트와 알림을 하나씩 둡니다. +3. **넘은 호스트는 구간마다 다시 알립니다**(6분 초과에 6건). 연속 구간은 첫 알림을 늦출 뿐(여기서는 + 2구간 Pending 뒤 Alert) 그 뒤에는 똑같이 반복합니다. 호스트가 많으면 평가 간격과 받는 채널의 + 메시지 묶기도 함께 정하세요. + +### 따라 하기 + +**1. 호스트별로 묶은 차트.** Dashboards → Add → Time Series. Data Source는 메트릭 소스. +`system.memory.utilization`의 Average, Where(SQL) `Attributes['state'] = 'used'`, Group By +`ResourceAttributes['host.name']`. hostmetrics memory scraper는 상태(used, free)마다 시계열을 보내므로, +이 필터로 호스트당 시계열 하나만 남깁니다. + +![차트 편집기: system.memory.utilization의 Average, Where Attributes['state'] = 'used', Group By ResourceAttributes['host.name'], 0.9 위 임계치 띠](images/02-every-host-chart.webp) + +**2. 그 차트에 알림.** Edit alert → Static threshold, "is at least (≥)" 0.9, Send to에 webhook(Slack +또는 generic). 메시지 템플릿에 `{{group}}`, `{{value}}`를 쓸 수 있습니다. + +![알림 패널: Static threshold, is at least 0.9, webhook threshold-alerts lab으로 보냄](images/03-alert-settings.webp) + +API로 `interval: 1m`을 넣어 만든 이 알림들은 패널에 "over 5 minute"로 보였습니다. 알림 페이지에는 +"Evaluates every 1m"로 나오고, 알림도 1분마다 왔습니다. UI에서 만들 때는 원하는 간격을 직접 고르세요. + +**3. 서버군 하나.** 같은 차트의 Where에 서버군을 더합니다: +`Attributes['state'] = 'used' AND ResourceAttributes['svc.group'] = 'game-a'`, 알림은 0.8. + +![Where Attributes['state'] = 'used' AND ResourceAttributes['svc.group'] = 'game-a'인 차트 편집기: game-a 호스트만, 0.8 위 임계치 띠](images/04-group-filter.webp) + +**4. 짧은 튐에 알림 줄이기.** 알림 → Advanced Settings → Consecutive windows. + +![Advanced Settings: Consecutive windows 3](images/05-consecutive-windows.webp) + +**5. 보는 곳.** 대시보드(알림이 울리는 동안 종 아이콘이 빨간색), Alerts 페이지, 알림마다의 페이지(구간마다 +한 줄, 펼치면 호스트마다 한 줄). + +![세 차트가 있는 대시보드, 1분 단위: game-a-01, game-b-02, game-a-02가 05:00부터 05:06까지 올라감](images/01-dashboard.webp) + +![Alerts 페이지: 세 알림과 이력 막대, A3는 빨간색 앞에 주황색(Pending) 막대](images/06-alerts-list.webp) + +![A1 평가 이력: 오후 2:00–2:05 KST "2/6 groups firing", 펼치면 game-a-01 0.95와 game-b-02 0.93은 Alert, game-a-02 0.85는 OK](images/07-every-host-history.webp) + +![A3 평가 이력: 오후 2:00, 2:01 KST Pending, 2:02부터 2:06까지 Alert](images/08-consecutive-history.webp) + +알림 페이지는 현지 시각(여기서는 KST), 알림 본문과 보고서는 UTC입니다. + +### 적용 전에 알아 둘 것 + +- **gauge는 구간의 마지막 값으로 읽습니다.** gauge 메트릭의 "Average"는 시계열마다 구간 안의 마지막 + 값을 잡은 뒤 시계열끼리 평균합니다(HyperDX 소스 `packages/common-utils/src/core/renderChartConfig.ts`의 + `last_value(Value)`, 2026-11-11에 읽음). 호스트로 group by 하고 호스트당 시계열이 하나면 그 호스트의 + 마지막 샘플입니다. 이번 실행에서도 05:00:51에 올렸는데 05:00–05:01 구간이 이미 울렸습니다. + hostmetrics 기본값(1분에 샘플 하나)이면 어차피 구간마다 샘플이 하나입니다. +- **알림 제목의 값은 소수 한 자리로 반올림됩니다.** 0.93이 "0.9 meets or exceeds 0.9"로 왔습니다. + 메시지의 `{{value}}`에는 0.93이 그대로 들어갔습니다. "is at least"는 ≥입니다. +- **그룹 이름은 식 그대로 옵니다:** `Group: "arrayElement(ResourceAttributes, 'host.name'):game-b-02"`. +- **데이터를 보내지 않게 된 호스트는 임계치 알림이 울리지 않습니다** — 비교할 값이 없습니다. 이 경우는 + [`labs/process-down-alerts`](../process-down-alerts/README.md)의 기대 호스트 목록 알림을 씁니다. +- **연속 구간(1회 측정):** A3는 세 번째 구간부터 울렸고, 같은 차트 쿼리인 A1보다 한 구간 늦게 + 해소됐습니다(05:07–05:08 대 05:06–05:07). + +### 실행 + +```bash +ngrok http 9310 # 127.0.0.1:9310으로 가는 터널이면 무엇이든; 선택 +managed/threshold_alerts.py --webhook-url https:// [--keep] # 약 14분 +managed/threshold_alerts.py --cleanup # --keep 뒤에 +managed/threshold_alerts.py --report # results/에서 보고서 다시 만들기 +``` + +환경 파일: `CH_HOST`, `CH_USER`, `CH_PASSWORD`(`--ch-env`)와 `CHC_KEY`, `CHC_SECRET`(`--api-env`). +만드는 것은 모두 이름이 `threshold-alerts lab`이고 데이터베이스는 `threshold_alerts_lab`이며, +`--cleanup`이 지웁니다. `--webhook-url`이 없으면 webhook이 example.com을 가리키고 알림 상태만 +기록합니다(Cloud API는 알림마다 상태 하나만 주고 호스트별로는 주지 않습니다). 링크와 알림별 해시는 +`results/`에 쓰기 전에 가립니다. + +| 파일 | 내용 | +|---|---| +| `managed/threshold_alerts.py` | 소스, 대시보드, 알림, webhook; 데이터 전송; 수신기; 보고서 | +| `results/2026-11-11-managed.md` | 30초마다의 알림 상태와 모든 알림 | +| `results/2026-11-11-webhook.jsonl`, `-managed-states.jsonl`, `-phases.json` | 원본 기록 | +| `images/` | 위 화면, 같은 실행 | + +다루지 않은 것: Slack 채널(generic webhook에 같은 본문이 옴), CPU(`system.cpu.utilization`은 CPU와 +상태마다 시계열이 있어 필터와 임계치가 다름 — 여기서는 실행하지 않음), 서버군마다 다른 임계치를 규칙 +하나로, silence, 실제 호스트(데이터는 hostmetrics 모양의 합성 데이터), 로컬 OSS 빌드. diff --git a/labs/threshold-alerts/images/01-dashboard.webp b/labs/threshold-alerts/images/01-dashboard.webp new file mode 100644 index 0000000..11870a3 Binary files /dev/null and b/labs/threshold-alerts/images/01-dashboard.webp differ diff --git a/labs/threshold-alerts/images/02-every-host-chart.webp b/labs/threshold-alerts/images/02-every-host-chart.webp new file mode 100644 index 0000000..dfc501e Binary files /dev/null and b/labs/threshold-alerts/images/02-every-host-chart.webp differ diff --git a/labs/threshold-alerts/images/03-alert-settings.webp b/labs/threshold-alerts/images/03-alert-settings.webp new file mode 100644 index 0000000..b49ff3d Binary files /dev/null and b/labs/threshold-alerts/images/03-alert-settings.webp differ diff --git a/labs/threshold-alerts/images/04-group-filter.webp b/labs/threshold-alerts/images/04-group-filter.webp new file mode 100644 index 0000000..b6caff4 Binary files /dev/null and b/labs/threshold-alerts/images/04-group-filter.webp differ diff --git a/labs/threshold-alerts/images/05-consecutive-windows.webp b/labs/threshold-alerts/images/05-consecutive-windows.webp new file mode 100644 index 0000000..ddc1c67 Binary files /dev/null and b/labs/threshold-alerts/images/05-consecutive-windows.webp differ diff --git a/labs/threshold-alerts/images/06-alerts-list.webp b/labs/threshold-alerts/images/06-alerts-list.webp new file mode 100644 index 0000000..d630d2b Binary files /dev/null and b/labs/threshold-alerts/images/06-alerts-list.webp differ diff --git a/labs/threshold-alerts/images/07-every-host-history.webp b/labs/threshold-alerts/images/07-every-host-history.webp new file mode 100644 index 0000000..7402ed5 Binary files /dev/null and b/labs/threshold-alerts/images/07-every-host-history.webp differ diff --git a/labs/threshold-alerts/images/08-consecutive-history.webp b/labs/threshold-alerts/images/08-consecutive-history.webp new file mode 100644 index 0000000..0ca81f2 Binary files /dev/null and b/labs/threshold-alerts/images/08-consecutive-history.webp differ diff --git a/labs/threshold-alerts/lab.yaml b/labs/threshold-alerts/lab.yaml new file mode 100644 index 0000000..8e3111b --- /dev/null +++ b/labs/threshold-alerts/lab.yaml @@ -0,0 +1,6 @@ +# Runner keys (tools/lab.schema.md). Not run by hol: needs a ClickHouse Cloud service with +# Managed ClickStack and, for per-host evidence, a tunnel to the local webhook receiver. +target: cloud +tier: T1 +services: [clickstack] +verified_on: "Managed ClickStack (ClickHouse Cloud 26.8.1.654), 2026-11-11" diff --git a/labs/threshold-alerts/managed/threshold_alerts.py b/labs/threshold-alerts/managed/threshold_alerts.py new file mode 100644 index 0000000..f0563fd --- /dev/null +++ b/labs/threshold-alerts/managed/threshold_alerts.py @@ -0,0 +1,369 @@ +#!/usr/bin/env python3 +"""Threshold alerts for host metrics on Managed ClickStack (ClickHouse Cloud) — stdlib only. + +Two questions an infra team asks first, answered with the builder (the chart editor a user +clicks through), created here through the ClickHouse Cloud API so the run can be repeated: + + A1 every host, memory used above 0.9, group by host.name -> one notification per host + A2 server group game-a only, memory used above 0.8, group by host -> game-a hosts only + A3 as A1, but numConsecutiveWindows = 3 -> fires after 3 windows + +Data: six hosts (game-a-01..03, game-b-01..03, resource attribute svc.group) send +`system.memory.utilization` (state used / free, as the hostmetrics memory scraper does) every +20 s into a database of their own, read by a ClickStack metric source of its own. +Phases (min): P0 all at 0.50 for 3 · P1 breach for 6 (game-a-01 0.95, game-a-02 0.85, +game-b-02 0.93) · P2 all at 0.50 for 4. + +Evidence: each alert's `state` from the Cloud API every 30 s, and every webhook POST at a local +receiver (`--webhook-url` must forward to it, e.g. `ngrok http 9310`). Without --webhook-url the +webhook points at example.com and only the states are recorded. + + threshold_alerts.py [--webhook-url https://] [--port 9310] [--keep] + [--ch-env ~/.config/khai-harness/secrets/clickhouse-seoul.env] + [--api-env ~/.config/khai-harness/secrets/chc-api.env] + threshold_alerts.py --cleanup delete what a --keep run left, drop the database + +--keep leaves dashboard, alerts, source, webhook and data for screen captures (P2 values keep +being sent until Ctrl-C, so charts stay live); run --cleanup after. + -> ../results/-managed.md, -managed-states.jsonl, -webhook.jsonl + +CH env: CH_HOST, CH_USER, CH_PASSWORD. API env: CHC_KEY, CHC_SECRET. Values are never printed. +""" +import base64 +import json +import os +import re +import secrets +import sys +import threading +import time +import urllib.error +import urllib.parse +import urllib.request +from http.server import BaseHTTPRequestHandler, HTTPServer + +HERE = os.path.dirname(os.path.abspath(__file__)) +RESULTS = os.path.join(HERE, "..", "results") +DB = "threshold_alerts_lab" +NAME = "threshold-alerts lab" +API = "https://api.clickhouse.cloud/v1" +METRIC = "system.memory.utilization" +HOSTS = {f"game-{g}-0{i}": f"game-{g}" for g in ("a", "b") for i in (1, 2, 3)} +BREACH = {"game-a-01": 0.95, "game-a-02": 0.85, "game-b-02": 0.93} +PHASES = [("P0 all normal", {}, 3), ("P1 breach", BREACH, 6), ("P2 all normal", {}, 4)] +HOST = "ResourceAttributes['host.name']" +GROUP = "ResourceAttributes['svc.group']" +USED = "Attributes['state'] = 'used'" +# (tile title, where, alert threshold, numConsecutiveWindows) +TILES = [ + ("A1 memory used, every host", USED, 0.9, None), + ("A2 memory used, game-a", f"{USED} AND {GROUP} = 'game-a'", 0.8, None), + ("A3 memory used, every host, 3 windows", USED, 0.9, 3), +] + +SCHEMA = f"""CREATE TABLE IF NOT EXISTS {DB}.otel_metrics_gauge ( + ResourceAttributes Map(LowCardinality(String), String) CODEC(ZSTD(1)), + ResourceSchemaUrl String CODEC(ZSTD(1)), + ScopeName String CODEC(ZSTD(1)), + ScopeVersion String CODEC(ZSTD(1)), + ScopeAttributes Map(LowCardinality(String), String) CODEC(ZSTD(1)), + ScopeDroppedAttrCount UInt32 CODEC(ZSTD(1)), + ScopeSchemaUrl String CODEC(ZSTD(1)), + ServiceName LowCardinality(String) CODEC(ZSTD(1)), + MetricName String CODEC(ZSTD(1)), + MetricDescription String CODEC(ZSTD(1)), + MetricUnit String CODEC(ZSTD(1)), + Attributes Map(LowCardinality(String), String) CODEC(ZSTD(1)), + StartTimeUnix DateTime64(9) CODEC(Delta, ZSTD(1)), + TimeUnix DateTime64(9) CODEC(Delta, ZSTD(1)), + Value Float64 CODEC(ZSTD(1)), + Flags UInt32 CODEC(ZSTD(1)), + Exemplars Nested (FilteredAttributes Map(LowCardinality(String), String), TimeUnix DateTime64(9), + Value Float64, SpanId String, TraceId String) CODEC(ZSTD(1)) +) ENGINE = MergeTree +PARTITION BY toDate(TimeUnix) +ORDER BY (ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix))""" + + +def load_env(path): + env = {} + for line in open(os.path.expanduser(path)): + line = line.strip() + if line and not line.startswith("#") and "=" in line: + k, v = line.split("=", 1) + env[k.removeprefix("export ").strip()] = v.strip().strip('"').strip("'") + return env + + +def basic(u, p): + return "Basic " + base64.b64encode(f"{u}:{p}".encode()).decode() + + +class CH: + def __init__(self, env): + self.host = env["CH_HOST"] + self.auth = basic(env["CH_USER"], env["CH_PASSWORD"]) + + def q(self, sql, **settings): + url = f"https://{self.host}:8443/?" + urllib.parse.urlencode(settings) + req = urllib.request.Request(url, data=sql.encode(), headers={"Authorization": self.auth}) + with urllib.request.urlopen(req, timeout=120) as r: + return r.read().decode() + + +class Cloud: + def __init__(self, env, ch_host): + self.auth = basic(env["CHC_KEY"], env["CHC_SECRET"]) + self.org = self.call("GET", "/organizations")[0]["id"] + svc = next(s for s in self.call("GET", f"/organizations/{self.org}/services") + if any(e.get("host") == ch_host for e in s.get("endpoints", []))) + self.base = f"/organizations/{self.org}/services/{svc['id']}/clickstack" + + def call(self, method, path, body=None): + data = json.dumps(body).encode() if body is not None else None + req = urllib.request.Request(API + path, data=data, method=method, + headers={"Authorization": self.auth, "Content-Type": "application/json"}) + try: + with urllib.request.urlopen(req, timeout=60) as r: + raw = r.read() + except urllib.error.HTTPError as e: + raise SystemExit(f"{method} {path}: HTTP {e.code}: {e.read()[:800].decode(errors='replace')}") + d = json.loads(raw) if raw else {} + res = d.get("result", d) + return res.get("data", res) if isinstance(res, dict) and "data" in res and len(res) <= 2 else res + + def cs(self, method, path, body=None): + return self.call(method, self.base + path, body) + + +def redact(obj): + """Links carry the service and dashboard ids; the trailing hash is per alert. Neither is evidence.""" + if isinstance(obj, dict): + return {k: redact(v) for k, v in obj.items() if k not in ("teamId",)} + if isinstance(obj, list): + return [redact(v) for v in obj] + if isinstance(obj, str): + obj = re.sub(r"https://[^ |\n]+", "", obj) + return re.sub(r"\b[0-9a-f]{40}\b", "", obj) + return obj + + +def receiver(port, token, path): + class H(BaseHTTPRequestHandler): + def do_POST(self): + n = int(self.headers.get("content-length") or 0) + raw = self.rfile.read(n).decode(errors="replace") + if self.path.split("?")[0] != f"/{token}": + self.send_response(404) + self.end_headers() + return + try: + body = json.loads(raw) + except ValueError: + body = raw + with open(path, "a") as f: + f.write(json.dumps({"received_at": time.time(), "body": redact(body)}) + "\n") + self.send_response(200) + self.end_headers() + self.wfile.write(b"ok") + + def log_message(self, *a): + pass + + srv = HTTPServer(("127.0.0.1", port), H) + threading.Thread(target=srv.serve_forever, daemon=True).start() + return srv + + +def insert(ch, values): + now = time.time() + ts = time.strftime("%Y-%m-%d %H:%M:%S", time.gmtime(now)) + f".{int(now * 1000) % 1000:03d}" + rows = [] + for host, group in HOSTS.items(): + used = values.get(host, 0.50) + for state, v in (("used", used), ("free", round(1 - used, 4))): + rows.append(json.dumps({ + "TimeUnix": ts, "StartTimeUnix": ts, "MetricName": METRIC, "MetricUnit": "1", + "MetricDescription": "Percentage of memory bytes in use.", "Value": v, + "ScopeName": "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver/internal/scraper/memoryscraper", + "Attributes": {"state": state}, + "ResourceAttributes": {"host.name": host, "svc.group": group, "os.type": "linux"}})) + ch.q(f"INSERT INTO {DB}.otel_metrics_gauge FORMAT JSONEachRow\n" + "\n".join(rows), + date_time_input_format="best_effort") + + +def cleanup(ch, cloud): + dash = [d for d in cloud.cs("GET", "/dashboards") if d.get("name") == NAME] + ids = {d["id"] for d in dash} + for a in cloud.cs("GET", "/alerts"): + if a.get("dashboardId") in ids: + cloud.cs("DELETE", f"/alerts/{a['id']}") + for d in dash: + cloud.cs("DELETE", f"/dashboards/{d['id']}") + for w in cloud.cs("GET", "/webhooks"): + if w.get("name") == NAME: + cloud.cs("DELETE", f"/webhooks/{w['id']}") + for s in cloud.cs("GET", "/sources"): + if s.get("name") == NAME: + cloud.cs("DELETE", f"/sources/{s['id']}") + ch.q(f"DROP DATABASE IF EXISTS {DB}") + print(f"cleaned up: {len(dash)} dashboard(s) and their alerts, webhook, source, database {DB}") + + +def builder_tile(src_id, where): + return {"displayType": "line", "sourceId": src_id, + "select": [{"aggFn": "avg", "valueExpression": "Value", "metricType": "gauge", "metricName": METRIC, + "where": where, "whereLanguage": "sql", "alias": "memory used"}], + "groupBy": HOST} + + +def main(): + args = sys.argv[1:] + opt = lambda k, d: args[args.index(k) + 1] if k in args else d # noqa: E731 + ch = CH(load_env(opt("--ch-env", "~/.config/khai-harness/secrets/clickhouse-seoul.env"))) + cloud = Cloud(load_env(opt("--api-env", "~/.config/khai-harness/secrets/chc-api.env")), ch.host) + if "--cleanup" in args: + return cleanup(ch, cloud) + day = time.strftime("%Y-%m-%d", time.gmtime()) + os.makedirs(RESULTS, exist_ok=True) + states_path = os.path.join(RESULTS, f"{day}-managed-states.jsonl") + hook_path = os.path.join(RESULTS, f"{day}-webhook.jsonl") + phases_path = os.path.join(RESULTS, f"{day}-phases.json") + open(states_path, "w").close() + open(hook_path, "w").close() + token = secrets.token_urlsafe(16) + base_url = opt("--webhook-url", "") + if base_url: + receiver(int(opt("--port", "9310")), token, hook_path) + hook_url = f"{base_url.rstrip('/')}/{token}" if base_url else "https://example.com/threshold-alerts-lab" + server = ch.q("SELECT version()").strip() + print("server:", server) + + cleanup(ch, cloud) + ch.q(f"CREATE DATABASE IF NOT EXISTS {DB}") + ch.q(SCHEMA) + insert(ch, {}) + conn = next(s for s in cloud.cs("GET", "/sources") if s.get("connection"))["connection"] + src = cloud.cs("POST", "/sources", { + "name": NAME, "kind": "metric", "connection": conn, "querySettings": [], "disabled": False, + "from": {"databaseName": DB, "tableName": ""}, "timestampValueExpression": "TimeUnix", + "resourceAttributesExpression": "ResourceAttributes", "metricTables": {"gauge": "otel_metrics_gauge"}}) + hook = cloud.cs("POST", "/webhooks", {"name": NAME, "service": "generic", "url": hook_url, + "description": "labs/threshold-alerts (synthetic data)"}) + dash = cloud.cs("POST", "/dashboards", {"name": NAME, "tags": ["threshold-alerts"], "tiles": [ + {"name": title, "x": 0, "y": i * 4, "w": 24, "h": 4, "config": builder_tile(src["id"], where)} + for i, (title, where, _, _) in enumerate(TILES)]}) + tile_id = {t["name"]: t["id"] for t in dash["tiles"]} + alerts = {} + for title, _, threshold, windows in TILES: + body = {"dashboardId": dash["id"], "tileId": tile_id[title], "source": "tile", "name": title, + "thresholdType": "above", "threshold": threshold, "interval": "1m", + "channel": {"type": "webhook", "webhookId": hook["id"]}, + "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}"} + if windows: + body["numConsecutiveWindows"] = windows + a = cloud.cs("POST", "/alerts", body) + alerts[a["id"]] = title + print(f"created source, webhook ({'tunnel' if base_url else 'example.com'}), dashboard, {len(alerts)} alerts") + + phases = [] + try: + for name, values, minutes in PHASES: + t0 = time.time() + phases.append({"phase": name, "values": values, "start": t0}) + json.dump(phases, open(phases_path, "w")) + print(time.strftime("%H:%M:%S", time.gmtime()), name, values or "all 0.50", flush=True) + last_poll = 0 + while time.time() - t0 < minutes * 60: + insert(ch, values) + if time.time() - last_poll >= 30: + last_poll = time.time() + st = [a for a in cloud.cs("GET", "/alerts") if a["id"] in alerts] + with open(states_path, "a") as f: + f.write(json.dumps({"t": last_poll, "phase": name, "alerts": redact(st)}) + "\n") + print(" ", " · ".join(f"{alerts[a['id']][:2]} {a.get('state')}" for a in st), flush=True) + time.sleep(20) + time.sleep(70) # the last window's evaluation and its notifications + finally: + report(day, server, phases, states_path, hook_path, alerts, bool(base_url)) + if "--keep" in args: + print("--keep: sending P2 values until Ctrl-C (then run --cleanup)") + try: + while True: + insert(ch, {}) + time.sleep(20) + except KeyboardInterrupt: + pass + else: + cleanup(ch, cloud) + + +def report(day, server, phases, states_path, hook_path, alerts, tunnel): + hms = lambda t: time.strftime("%H:%M:%S", time.gmtime(t)) # noqa: E731 + phase_at = lambda t: next((p["phase"] for p in reversed(phases) if p["start"] <= t), "-") # noqa: E731 + since = lambda t: int(t - next((p["start"] for p in reversed(phases) if p["start"] <= t), t)) # noqa: E731 + lines = [f"# Threshold alerts on Managed ClickStack — run {day}", "", + f"Server: ClickHouse {server} (Managed ClickStack). Synthetic `{METRIC}` for {len(HOSTS)} hosts every 20 s.", + "Phases: " + " · ".join(f"{p['phase']} {hms(p['start'])}Z" for p in phases), "", + "## Alert state (Cloud API, every 30 s)", "", "| time (UTC) | phase | " + " | ".join( + t[:2] for t in alerts.values()) + " |", "|---|---|" + "---|" * len(alerts)] + for line in open(states_path): + r = json.loads(line) + st = {a["id"]: a.get("state") for a in r["alerts"]} + lines.append(f"| {hms(r['t'])} | {r['phase']} | " + " | ".join(st.get(i, "-") for i in alerts) + " |") + lines += ["", "## Notifications at the webhook receiver", ""] + if not tunnel: + lines.append("No tunnel: the webhook pointed at example.com, nothing was received.") + else: + notes = [parse(json.loads(line)) for line in open(hook_path)] + lines += ["Per alert and host: the evaluation windows (UTC) that notified, and the resolve.", "", + "| alert | host | fired for windows | notifications | resolved (window) |", "|---|---|---|---|---|"] + for title in alerts.values(): + key = title[:2] + for host in sorted({n["host"] for n in notes if n["alert"] == key}): + mine = [n for n in notes if n["alert"] == key and n["host"] == host] + fired = [n["window"] for n in mine if n["state"] == "ALERT"] + ok = [n["window"] for n in mine if n["state"] == "OK"] + lines.append(f"| {key} | {host} | {fired[0]} – {fired[-1]} | {len(fired)} | {ok[0] if ok else '-'} |") + lines += ["", "Every notification:", "", "| received (UTC) | phase | s into phase | alert | host | state | window | title value | {{value}} |", + "|---|---|---|---|---|---|---|---|---|"] + for n in notes: + t = n["received_at"] + lines.append(f"| {hms(t)} | {phase_at(t)} | {since(t)} | {n['alert']} | {n['host']} | {n['state']} | " + f"{n['window']} | {n['title_value']} | {n['value']} |") + lines += ["", "First notification text, as received (links redacted):", "", "```text", + json.loads(open(hook_path).readline())["body"]["text"].strip(), "```"] + path = os.path.join(RESULTS, f"{day}-managed.md") + open(path, "w").write("\n".join(lines) + "\n") + print("report:", path) + + +def parse(rec): + """One generic-webhook POST -> alert key, host, state, window, values (HyperDX default text).""" + text = rec["body"]["text"] if isinstance(rec["body"], dict) else rec["body"] + host = re.search(r"host\.name'\):([\w.-]+)", text) + window = re.search(r"\[\w+ \d+ ([\d:]+ [AP]M) - \w+ \d+ ([\d:]+ [AP]M)\)", text) + title_value = re.search(r"\n([\d.]+) (?:meets or exceeds|exceeds|falls below)", text) + value = re.search(r"= ([\d.]+)", text) + hhmm = lambda s: time.strftime("%H:%M", time.strptime(s, "%I:%M:%S %p")) # noqa: E731 + return {"received_at": rec["received_at"], "alert": text.split("|")[0].strip()[2:].strip()[:2], + "host": host.group(1) if host else "-", "state": "OK" if text.startswith("✅") else "ALERT", + "window": f"{hhmm(window.group(1))}–{hhmm(window.group(2))}" if window else "-", + "title_value": title_value.group(1) if title_value else "-", "value": value.group(1) if value else "-"} + + +def rebuild(day): + """--report : the report again from the saved files (no Cloud or ClickHouse calls).""" + states_path = os.path.join(RESULTS, f"{day}-managed-states.jsonl") + first = json.loads(open(states_path).readline()) + alerts = {a["id"]: a["name"] for a in first["alerts"]} + phases = json.load(open(os.path.join(RESULTS, f"{day}-phases.json"))) + server = open(os.path.join(RESULTS, f"{day}-managed.md")).read().split("Server: ClickHouse ")[1].split(" ")[0] + report(day, server, phases, states_path, os.path.join(RESULTS, f"{day}-webhook.jsonl"), alerts, True) + + +if __name__ == "__main__": + if "--report" in sys.argv: + rebuild(sys.argv[sys.argv.index("--report") + 1]) + else: + main() diff --git a/labs/threshold-alerts/results/2026-11-11-managed-states.jsonl b/labs/threshold-alerts/results/2026-11-11-managed-states.jsonl new file mode 100644 index 0000000..d2b822a --- /dev/null +++ b/labs/threshold-alerts/results/2026-11-11-managed-states.jsonl @@ -0,0 +1,20 @@ +{"t": 1794373062.700381, "phase": "P0 all normal", "alerts": [{"id": "6af3f5c1496f6a67baa8f490", "name": "A1 memory used, every host", "displayName": "threshold-alerts lab - A1 memory used, every host", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fd9", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "createdAt": "2026-11-11T04:57:37.812Z", "updatedAt": "2026-11-11T04:57:37.812Z"}, {"id": "6af3f5c4d2f443adda91ffd0", "name": "A2 memory used, game-a", "displayName": "threshold-alerts lab - A2 memory used, game-a", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.8, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fda", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "createdAt": "2026-11-11T04:57:40.149Z", "updatedAt": "2026-11-11T04:57:40.149Z"}, {"id": "6af3f5c65df9a4c17d4d9fe8", "name": "A3 memory used, every host, 3 windows", "displayName": "threshold-alerts lab - A3 memory used, every host, 3 windows", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": 3, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fdb", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "createdAt": "2026-11-11T04:57:42.459Z", "updatedAt": "2026-11-11T04:57:42.459Z"}]} +{"t": 1794373104.5155368, "phase": "P0 all normal", "alerts": [{"id": "6af3f5c1496f6a67baa8f490", "name": "A1 memory used, every host", "displayName": "threshold-alerts lab - A1 memory used, every host", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fd9", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:37.812Z", "updatedAt": "2026-11-11T04:58:05.078Z"}, {"id": "6af3f5c4d2f443adda91ffd0", "name": "A2 memory used, game-a", "displayName": "threshold-alerts lab - A2 memory used, game-a", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.8, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fda", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:40.149Z", "updatedAt": "2026-11-11T04:58:05.078Z"}, {"id": "6af3f5c65df9a4c17d4d9fe8", "name": "A3 memory used, every host, 3 windows", "displayName": "threshold-alerts lab - A3 memory used, every host, 3 windows", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": 3, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fdb", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:42.459Z", "updatedAt": "2026-11-11T04:58:05.078Z"}]} +{"t": 1794373146.4181418, "phase": "P0 all normal", "alerts": [{"id": "6af3f5c1496f6a67baa8f490", "name": "A1 memory used, every host", "displayName": "threshold-alerts lab - A1 memory used, every host", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fd9", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:37.812Z", "updatedAt": "2026-11-11T04:59:02.834Z"}, {"id": "6af3f5c4d2f443adda91ffd0", "name": "A2 memory used, game-a", "displayName": "threshold-alerts lab - A2 memory used, game-a", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.8, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fda", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:40.149Z", "updatedAt": "2026-11-11T04:59:02.834Z"}, {"id": "6af3f5c65df9a4c17d4d9fe8", "name": "A3 memory used, every host, 3 windows", "displayName": "threshold-alerts lab - A3 memory used, every host, 3 windows", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": 3, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fdb", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:42.459Z", "updatedAt": "2026-11-11T04:59:02.834Z"}]} +{"t": 1794373188.3184981, "phase": "P0 all normal", "alerts": [{"id": "6af3f5c1496f6a67baa8f490", "name": "A1 memory used, every host", "displayName": "threshold-alerts lab - A1 memory used, every host", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fd9", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:37.812Z", "updatedAt": "2026-11-11T04:59:02.834Z"}, {"id": "6af3f5c4d2f443adda91ffd0", "name": "A2 memory used, game-a", "displayName": "threshold-alerts lab - A2 memory used, game-a", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.8, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fda", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:40.149Z", "updatedAt": "2026-11-11T04:59:02.834Z"}, {"id": "6af3f5c65df9a4c17d4d9fe8", "name": "A3 memory used, every host, 3 windows", "displayName": "threshold-alerts lab - A3 memory used, every host, 3 windows", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": 3, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fdb", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:42.459Z", "updatedAt": "2026-11-11T04:59:02.834Z"}]} +{"t": 1794373230.2046, "phase": "P0 all normal", "alerts": [{"id": "6af3f5c1496f6a67baa8f490", "name": "A1 memory used, every host", "displayName": "threshold-alerts lab - A1 memory used, every host", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fd9", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:37.812Z", "updatedAt": "2026-11-11T05:00:06.273Z"}, {"id": "6af3f5c4d2f443adda91ffd0", "name": "A2 memory used, game-a", "displayName": "threshold-alerts lab - A2 memory used, game-a", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.8, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fda", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:40.149Z", "updatedAt": "2026-11-11T05:00:06.273Z"}, {"id": "6af3f5c65df9a4c17d4d9fe8", "name": "A3 memory used, every host, 3 windows", "displayName": "threshold-alerts lab - A3 memory used, every host, 3 windows", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": 3, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fdb", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:42.459Z", "updatedAt": "2026-11-11T05:00:06.273Z"}]} +{"t": 1794373252.995701, "phase": "P1 breach", "alerts": [{"id": "6af3f5c1496f6a67baa8f490", "name": "A1 memory used, every host", "displayName": "threshold-alerts lab - A1 memory used, every host", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fd9", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:37.812Z", "updatedAt": "2026-11-11T05:00:06.273Z"}, {"id": "6af3f5c4d2f443adda91ffd0", "name": "A2 memory used, game-a", "displayName": "threshold-alerts lab - A2 memory used, game-a", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.8, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fda", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:40.149Z", "updatedAt": "2026-11-11T05:00:06.273Z"}, {"id": "6af3f5c65df9a4c17d4d9fe8", "name": "A3 memory used, every host, 3 windows", "displayName": "threshold-alerts lab - A3 memory used, every host, 3 windows", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": 3, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fdb", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:42.459Z", "updatedAt": "2026-11-11T05:00:06.273Z"}]} +{"t": 1794373295.145914, "phase": "P1 breach", "alerts": [{"id": "6af3f5c1496f6a67baa8f490", "name": "A1 memory used, every host", "displayName": "threshold-alerts lab - A1 memory used, every host", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "ALERT", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fd9", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:37.812Z", "updatedAt": "2026-11-11T05:01:02.622Z"}, {"id": "6af3f5c4d2f443adda91ffd0", "name": "A2 memory used, game-a", "displayName": "threshold-alerts lab - A2 memory used, game-a", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.8, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "ALERT", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fda", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:40.149Z", "updatedAt": "2026-11-11T05:01:02.622Z"}, {"id": "6af3f5c65df9a4c17d4d9fe8", "name": "A3 memory used, every host, 3 windows", "displayName": "threshold-alerts lab - A3 memory used, every host, 3 windows", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": 3, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "PENDING", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fdb", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:42.459Z", "updatedAt": "2026-11-11T05:01:02.622Z"}]} +{"t": 1794373337.194668, "phase": "P1 breach", "alerts": [{"id": "6af3f5c1496f6a67baa8f490", "name": "A1 memory used, every host", "displayName": "threshold-alerts lab - A1 memory used, every host", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "ALERT", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fd9", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:37.812Z", "updatedAt": "2026-11-11T05:02:01.955Z"}, {"id": "6af3f5c4d2f443adda91ffd0", "name": "A2 memory used, game-a", "displayName": "threshold-alerts lab - A2 memory used, game-a", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.8, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "ALERT", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fda", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:40.149Z", "updatedAt": "2026-11-11T05:02:01.955Z"}, {"id": "6af3f5c65df9a4c17d4d9fe8", "name": "A3 memory used, every host, 3 windows", "displayName": "threshold-alerts lab - A3 memory used, every host, 3 windows", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": 3, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "PENDING", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fdb", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:42.459Z", "updatedAt": "2026-11-11T05:02:01.955Z"}]} +{"t": 1794373379.2789388, "phase": "P1 breach", "alerts": [{"id": "6af3f5c1496f6a67baa8f490", "name": "A1 memory used, every host", "displayName": "threshold-alerts lab - A1 memory used, every host", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "ALERT", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fd9", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:37.812Z", "updatedAt": "2026-11-11T05:02:01.955Z"}, {"id": "6af3f5c4d2f443adda91ffd0", "name": "A2 memory used, game-a", "displayName": "threshold-alerts lab - A2 memory used, game-a", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.8, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "ALERT", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fda", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:40.149Z", "updatedAt": "2026-11-11T05:02:01.955Z"}, {"id": "6af3f5c65df9a4c17d4d9fe8", "name": "A3 memory used, every host, 3 windows", "displayName": "threshold-alerts lab - A3 memory used, every host, 3 windows", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": 3, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "PENDING", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fdb", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:42.459Z", "updatedAt": "2026-11-11T05:02:01.955Z"}]} +{"t": 1794373421.2193658, "phase": "P1 breach", "alerts": [{"id": "6af3f5c1496f6a67baa8f490", "name": "A1 memory used, every host", "displayName": "threshold-alerts lab - A1 memory used, every host", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "ALERT", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fd9", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:37.812Z", "updatedAt": "2026-11-11T05:03:02.349Z"}, {"id": "6af3f5c4d2f443adda91ffd0", "name": "A2 memory used, game-a", "displayName": "threshold-alerts lab - A2 memory used, game-a", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.8, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "ALERT", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fda", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:40.149Z", "updatedAt": "2026-11-11T05:03:02.349Z"}, {"id": "6af3f5c65df9a4c17d4d9fe8", "name": "A3 memory used, every host, 3 windows", "displayName": "threshold-alerts lab - A3 memory used, every host, 3 windows", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": 3, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "ALERT", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fdb", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:42.459Z", "updatedAt": "2026-11-11T05:03:02.349Z"}]} +{"t": 1794373464.640749, "phase": "P1 breach", "alerts": [{"id": "6af3f5c1496f6a67baa8f490", "name": "A1 memory used, every host", "displayName": "threshold-alerts lab - A1 memory used, every host", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "ALERT", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fd9", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:37.812Z", "updatedAt": "2026-11-11T05:04:01.260Z"}, {"id": "6af3f5c4d2f443adda91ffd0", "name": "A2 memory used, game-a", "displayName": "threshold-alerts lab - A2 memory used, game-a", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.8, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "ALERT", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fda", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:40.149Z", "updatedAt": "2026-11-11T05:04:01.260Z"}, {"id": "6af3f5c65df9a4c17d4d9fe8", "name": "A3 memory used, every host, 3 windows", "displayName": "threshold-alerts lab - A3 memory used, every host, 3 windows", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": 3, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "ALERT", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fdb", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:42.459Z", "updatedAt": "2026-11-11T05:04:01.260Z"}]} +{"t": 1794373506.5744371, "phase": "P1 breach", "alerts": [{"id": "6af3f5c1496f6a67baa8f490", "name": "A1 memory used, every host", "displayName": "threshold-alerts lab - A1 memory used, every host", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "ALERT", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fd9", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:37.812Z", "updatedAt": "2026-11-11T05:05:01.311Z"}, {"id": "6af3f5c4d2f443adda91ffd0", "name": "A2 memory used, game-a", "displayName": "threshold-alerts lab - A2 memory used, game-a", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.8, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "ALERT", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fda", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:40.149Z", "updatedAt": "2026-11-11T05:05:01.311Z"}, {"id": "6af3f5c65df9a4c17d4d9fe8", "name": "A3 memory used, every host, 3 windows", "displayName": "threshold-alerts lab - A3 memory used, every host, 3 windows", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": 3, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "ALERT", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fdb", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:42.459Z", "updatedAt": "2026-11-11T05:05:01.311Z"}]} +{"t": 1794373550.103929, "phase": "P1 breach", "alerts": [{"id": "6af3f5c1496f6a67baa8f490", "name": "A1 memory used, every host", "displayName": "threshold-alerts lab - A1 memory used, every host", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "ALERT", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fd9", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:37.812Z", "updatedAt": "2026-11-11T05:05:01.311Z"}, {"id": "6af3f5c4d2f443adda91ffd0", "name": "A2 memory used, game-a", "displayName": "threshold-alerts lab - A2 memory used, game-a", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.8, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "ALERT", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fda", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:40.149Z", "updatedAt": "2026-11-11T05:05:01.311Z"}, {"id": "6af3f5c65df9a4c17d4d9fe8", "name": "A3 memory used, every host, 3 windows", "displayName": "threshold-alerts lab - A3 memory used, every host, 3 windows", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": 3, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "ALERT", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fdb", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:42.459Z", "updatedAt": "2026-11-11T05:05:01.311Z"}]} +{"t": 1794373591.963058, "phase": "P1 breach", "alerts": [{"id": "6af3f5c1496f6a67baa8f490", "name": "A1 memory used, every host", "displayName": "threshold-alerts lab - A1 memory used, every host", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "ALERT", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fd9", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:37.812Z", "updatedAt": "2026-11-11T05:06:02.318Z"}, {"id": "6af3f5c4d2f443adda91ffd0", "name": "A2 memory used, game-a", "displayName": "threshold-alerts lab - A2 memory used, game-a", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.8, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "ALERT", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fda", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:40.149Z", "updatedAt": "2026-11-11T05:06:02.318Z"}, {"id": "6af3f5c65df9a4c17d4d9fe8", "name": "A3 memory used, every host, 3 windows", "displayName": "threshold-alerts lab - A3 memory used, every host, 3 windows", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": 3, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "ALERT", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fdb", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:42.459Z", "updatedAt": "2026-11-11T05:06:02.318Z"}]} +{"t": 1794373613.757588, "phase": "P2 all normal", "alerts": [{"id": "6af3f5c1496f6a67baa8f490", "name": "A1 memory used, every host", "displayName": "threshold-alerts lab - A1 memory used, every host", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "ALERT", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fd9", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:37.812Z", "updatedAt": "2026-11-11T05:06:02.318Z"}, {"id": "6af3f5c4d2f443adda91ffd0", "name": "A2 memory used, game-a", "displayName": "threshold-alerts lab - A2 memory used, game-a", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.8, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "ALERT", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fda", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:40.149Z", "updatedAt": "2026-11-11T05:06:02.318Z"}, {"id": "6af3f5c65df9a4c17d4d9fe8", "name": "A3 memory used, every host, 3 windows", "displayName": "threshold-alerts lab - A3 memory used, every host, 3 windows", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": 3, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "ALERT", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fdb", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:42.459Z", "updatedAt": "2026-11-11T05:06:02.318Z"}]} +{"t": 1794373655.654197, "phase": "P2 all normal", "alerts": [{"id": "6af3f5c1496f6a67baa8f490", "name": "A1 memory used, every host", "displayName": "threshold-alerts lab - A1 memory used, every host", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fd9", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:37.812Z", "updatedAt": "2026-11-11T05:07:01.888Z"}, {"id": "6af3f5c4d2f443adda91ffd0", "name": "A2 memory used, game-a", "displayName": "threshold-alerts lab - A2 memory used, game-a", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.8, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fda", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:40.149Z", "updatedAt": "2026-11-11T05:07:01.888Z"}, {"id": "6af3f5c65df9a4c17d4d9fe8", "name": "A3 memory used, every host, 3 windows", "displayName": "threshold-alerts lab - A3 memory used, every host, 3 windows", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": 3, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "ALERT", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fdb", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:42.459Z", "updatedAt": "2026-11-11T05:07:01.888Z"}]} +{"t": 1794373697.488943, "phase": "P2 all normal", "alerts": [{"id": "6af3f5c1496f6a67baa8f490", "name": "A1 memory used, every host", "displayName": "threshold-alerts lab - A1 memory used, every host", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fd9", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:37.812Z", "updatedAt": "2026-11-11T05:08:01.766Z"}, {"id": "6af3f5c4d2f443adda91ffd0", "name": "A2 memory used, game-a", "displayName": "threshold-alerts lab - A2 memory used, game-a", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.8, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fda", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:40.149Z", "updatedAt": "2026-11-11T05:08:01.766Z"}, {"id": "6af3f5c65df9a4c17d4d9fe8", "name": "A3 memory used, every host, 3 windows", "displayName": "threshold-alerts lab - A3 memory used, every host, 3 windows", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": 3, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fdb", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:42.459Z", "updatedAt": "2026-11-11T05:08:01.766Z"}]} +{"t": 1794373739.454431, "phase": "P2 all normal", "alerts": [{"id": "6af3f5c1496f6a67baa8f490", "name": "A1 memory used, every host", "displayName": "threshold-alerts lab - A1 memory used, every host", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fd9", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:37.812Z", "updatedAt": "2026-11-11T05:08:01.766Z"}, {"id": "6af3f5c4d2f443adda91ffd0", "name": "A2 memory used, game-a", "displayName": "threshold-alerts lab - A2 memory used, game-a", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.8, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fda", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:40.149Z", "updatedAt": "2026-11-11T05:08:01.766Z"}, {"id": "6af3f5c65df9a4c17d4d9fe8", "name": "A3 memory used, every host, 3 windows", "displayName": "threshold-alerts lab - A3 memory used, every host, 3 windows", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": 3, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fdb", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:42.459Z", "updatedAt": "2026-11-11T05:08:01.766Z"}]} +{"t": 1794373781.34599, "phase": "P2 all normal", "alerts": [{"id": "6af3f5c1496f6a67baa8f490", "name": "A1 memory used, every host", "displayName": "threshold-alerts lab - A1 memory used, every host", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fd9", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:37.812Z", "updatedAt": "2026-11-11T05:09:01.859Z"}, {"id": "6af3f5c4d2f443adda91ffd0", "name": "A2 memory used, game-a", "displayName": "threshold-alerts lab - A2 memory used, game-a", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.8, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fda", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:40.149Z", "updatedAt": "2026-11-11T05:09:01.859Z"}, {"id": "6af3f5c65df9a4c17d4d9fe8", "name": "A3 memory used, every host, 3 windows", "displayName": "threshold-alerts lab - A3 memory used, every host, 3 windows", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": 3, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fdb", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:42.459Z", "updatedAt": "2026-11-11T05:09:01.859Z"}]} +{"t": 1794373823.273008, "phase": "P2 all normal", "alerts": [{"id": "6af3f5c1496f6a67baa8f490", "name": "A1 memory used, every host", "displayName": "threshold-alerts lab - A1 memory used, every host", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fd9", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:37.812Z", "updatedAt": "2026-11-11T05:10:04.971Z"}, {"id": "6af3f5c4d2f443adda91ffd0", "name": "A2 memory used, game-a", "displayName": "threshold-alerts lab - A2 memory used, game-a", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.8, "interval": "1m", "numConsecutiveWindows": null, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fda", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:40.149Z", "updatedAt": "2026-11-11T05:10:04.971Z"}, {"id": "6af3f5c65df9a4c17d4d9fe8", "name": "A3 memory used, every host, 3 windows", "displayName": "threshold-alerts lab - A3 memory used, every host, 3 windows", "tags": ["threshold-alerts"], "message": "labs/threshold-alerts (synthetic) {{group}} = {{value}}", "note": null, "threshold": 0.9, "interval": "1m", "numConsecutiveWindows": 3, "thresholdType": "above", "detectionMode": "threshold", "source": "tile", "state": "OK", "channel": {"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}, "channels": [{"type": "webhook", "webhookId": "6af3f5bdd168394233f41166", "webhookService": "generic"}], "tileId": "6af3f5bf5df9a4c17d4d9fdb", "dashboardId": "6af3f5bf5df9a4c17d4d9fdc", "executionErrors": [], "createdAt": "2026-11-11T04:57:42.459Z", "updatedAt": "2026-11-11T05:10:04.971Z"}]} diff --git a/labs/threshold-alerts/results/2026-11-11-managed.md b/labs/threshold-alerts/results/2026-11-11-managed.md new file mode 100644 index 0000000..d2faa33 --- /dev/null +++ b/labs/threshold-alerts/results/2026-11-11-managed.md @@ -0,0 +1,96 @@ +# Threshold alerts on Managed ClickStack — run 2026-11-11 + +Server: ClickHouse 26.8.1.654 (Managed ClickStack). Synthetic `system.memory.utilization` for 6 hosts every 20 s. +Phases: P0 all normal 04:57:42Z · P1 breach 05:00:51Z · P2 all normal 05:06:53Z + +## Alert state (Cloud API, every 30 s) + +| time (UTC) | phase | A1 | A2 | A3 | +|---|---|---|---|---| +| 04:57:42 | P0 all normal | OK | OK | OK | +| 04:58:24 | P0 all normal | OK | OK | OK | +| 04:59:06 | P0 all normal | OK | OK | OK | +| 04:59:48 | P0 all normal | OK | OK | OK | +| 05:00:30 | P0 all normal | OK | OK | OK | +| 05:00:52 | P1 breach | OK | OK | OK | +| 05:01:35 | P1 breach | ALERT | ALERT | PENDING | +| 05:02:17 | P1 breach | ALERT | ALERT | PENDING | +| 05:02:59 | P1 breach | ALERT | ALERT | PENDING | +| 05:03:41 | P1 breach | ALERT | ALERT | ALERT | +| 05:04:24 | P1 breach | ALERT | ALERT | ALERT | +| 05:05:06 | P1 breach | ALERT | ALERT | ALERT | +| 05:05:50 | P1 breach | ALERT | ALERT | ALERT | +| 05:06:31 | P1 breach | ALERT | ALERT | ALERT | +| 05:06:53 | P2 all normal | ALERT | ALERT | ALERT | +| 05:07:35 | P2 all normal | OK | OK | ALERT | +| 05:08:17 | P2 all normal | OK | OK | OK | +| 05:08:59 | P2 all normal | OK | OK | OK | +| 05:09:41 | P2 all normal | OK | OK | OK | +| 05:10:23 | P2 all normal | OK | OK | OK | + +## Notifications at the webhook receiver + +Per alert and host: the evaluation windows (UTC) that notified, and the resolve. + +| alert | host | fired for windows | notifications | resolved (window) | +|---|---|---|---|---| +| A1 | game-a-01 | 05:00–05:01 – 05:05–05:06 | 6 | 05:06–05:07 | +| A1 | game-b-02 | 05:00–05:01 – 05:05–05:06 | 6 | 05:06–05:07 | +| A2 | game-a-01 | 05:00–05:01 – 05:05–05:06 | 6 | 05:06–05:07 | +| A2 | game-a-02 | 05:00–05:01 – 05:05–05:06 | 6 | 05:06–05:07 | +| A3 | game-a-01 | 05:02–05:03 – 05:06–05:07 | 5 | 05:07–05:08 | +| A3 | game-b-02 | 05:02–05:03 – 05:06–05:07 | 5 | 05:07–05:08 | + +Every notification: + +| received (UTC) | phase | s into phase | alert | host | state | window | title value | {{value}} | +|---|---|---|---|---|---|---|---|---| +| 05:01:02 | P1 breach | 11 | A1 | game-b-02 | ALERT | 05:00–05:01 | 0.9 | 0.93 | +| 05:01:02 | P1 breach | 11 | A1 | game-a-01 | ALERT | 05:00–05:01 | 1.0 | 0.95 | +| 05:01:02 | P1 breach | 11 | A2 | game-a-01 | ALERT | 05:00–05:01 | 1.0 | 0.95 | +| 05:01:02 | P1 breach | 11 | A2 | game-a-02 | ALERT | 05:00–05:01 | 0.9 | 0.85 | +| 05:02:02 | P1 breach | 71 | A1 | game-a-01 | ALERT | 05:01–05:02 | 1.0 | 0.95 | +| 05:02:02 | P1 breach | 71 | A1 | game-b-02 | ALERT | 05:01–05:02 | 0.9 | 0.93 | +| 05:02:02 | P1 breach | 71 | A2 | game-a-01 | ALERT | 05:01–05:02 | 1.0 | 0.95 | +| 05:02:02 | P1 breach | 71 | A2 | game-a-02 | ALERT | 05:01–05:02 | 0.9 | 0.85 | +| 05:03:02 | P1 breach | 131 | A3 | game-b-02 | ALERT | 05:02–05:03 | 0.9 | 0.93 | +| 05:03:02 | P1 breach | 131 | A3 | game-a-01 | ALERT | 05:02–05:03 | 1.0 | 0.95 | +| 05:03:02 | P1 breach | 131 | A2 | game-a-01 | ALERT | 05:02–05:03 | 1.0 | 0.95 | +| 05:03:02 | P1 breach | 131 | A1 | game-b-02 | ALERT | 05:02–05:03 | 0.9 | 0.93 | +| 05:03:02 | P1 breach | 131 | A2 | game-a-02 | ALERT | 05:02–05:03 | 0.9 | 0.85 | +| 05:03:02 | P1 breach | 131 | A1 | game-a-01 | ALERT | 05:02–05:03 | 1.0 | 0.95 | +| 05:04:01 | P1 breach | 190 | A1 | game-a-01 | ALERT | 05:03–05:04 | 1.0 | 0.95 | +| 05:04:01 | P1 breach | 190 | A1 | game-b-02 | ALERT | 05:03–05:04 | 0.9 | 0.93 | +| 05:04:01 | P1 breach | 190 | A2 | game-a-02 | ALERT | 05:03–05:04 | 0.9 | 0.85 | +| 05:04:01 | P1 breach | 190 | A2 | game-a-01 | ALERT | 05:03–05:04 | 1.0 | 0.95 | +| 05:04:01 | P1 breach | 190 | A3 | game-a-01 | ALERT | 05:03–05:04 | 1.0 | 0.95 | +| 05:04:01 | P1 breach | 190 | A3 | game-b-02 | ALERT | 05:03–05:04 | 0.9 | 0.93 | +| 05:05:01 | P1 breach | 250 | A3 | game-b-02 | ALERT | 05:04–05:05 | 0.9 | 0.93 | +| 05:05:01 | P1 breach | 250 | A3 | game-a-01 | ALERT | 05:04–05:05 | 1.0 | 0.95 | +| 05:05:01 | P1 breach | 250 | A1 | game-a-01 | ALERT | 05:04–05:05 | 1.0 | 0.95 | +| 05:05:01 | P1 breach | 250 | A1 | game-b-02 | ALERT | 05:04–05:05 | 0.9 | 0.93 | +| 05:05:01 | P1 breach | 250 | A2 | game-a-02 | ALERT | 05:04–05:05 | 0.9 | 0.85 | +| 05:05:01 | P1 breach | 250 | A2 | game-a-01 | ALERT | 05:04–05:05 | 1.0 | 0.95 | +| 05:06:02 | P1 breach | 311 | A1 | game-a-01 | ALERT | 05:05–05:06 | 1.0 | 0.95 | +| 05:06:02 | P1 breach | 311 | A3 | game-a-01 | ALERT | 05:05–05:06 | 1.0 | 0.95 | +| 05:06:02 | P1 breach | 311 | A3 | game-b-02 | ALERT | 05:05–05:06 | 0.9 | 0.93 | +| 05:06:02 | P1 breach | 311 | A2 | game-a-01 | ALERT | 05:05–05:06 | 1.0 | 0.95 | +| 05:06:02 | P1 breach | 311 | A1 | game-b-02 | ALERT | 05:05–05:06 | 0.9 | 0.93 | +| 05:06:02 | P1 breach | 311 | A2 | game-a-02 | ALERT | 05:05–05:06 | 0.9 | 0.85 | +| 05:07:01 | P2 all normal | 8 | A1 | game-a-01 | OK | 05:06–05:07 | - | 0.5 | +| 05:07:01 | P2 all normal | 8 | A1 | game-b-02 | OK | 05:06–05:07 | - | 0.5 | +| 05:07:01 | P2 all normal | 8 | A2 | game-a-01 | OK | 05:06–05:07 | - | 0.5 | +| 05:07:01 | P2 all normal | 8 | A2 | game-a-02 | OK | 05:06–05:07 | - | 0.5 | +| 05:07:02 | P2 all normal | 9 | A3 | game-a-01 | ALERT | 05:06–05:07 | 1.0 | 0.95 | +| 05:07:02 | P2 all normal | 9 | A3 | game-b-02 | ALERT | 05:06–05:07 | 0.9 | 0.93 | +| 05:08:01 | P2 all normal | 68 | A3 | game-b-02 | OK | 05:07–05:08 | - | 0.5 | +| 05:08:02 | P2 all normal | 69 | A3 | game-a-01 | OK | 05:07–05:08 | - | 0.5 | + +First notification text, as received (links redacted): + +```text +🚨 A1 memory used, every host | Group: "arrayElement(ResourceAttributes, 'host.name'):game-b-02" +0.9 meets or exceeds 0.9 +Time Range (UTC): [Nov 11 5:00:00 AM - Nov 11 5:01:00 AM) +labs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-b-02 = 0.93 | | ALERT | 1794373200000 | 1794373260000 | +``` diff --git a/labs/threshold-alerts/results/2026-11-11-phases.json b/labs/threshold-alerts/results/2026-11-11-phases.json new file mode 100644 index 0000000..46c2481 --- /dev/null +++ b/labs/threshold-alerts/results/2026-11-11-phases.json @@ -0,0 +1 @@ +[{"phase": "P0 all normal", "values": {}, "start": 1794373062}, {"phase": "P1 breach", "values": {"game-a-01": 0.95, "game-a-02": 0.85, "game-b-02": 0.93}, "start": 1794373251}, {"phase": "P2 all normal", "values": {}, "start": 1794373613}] \ No newline at end of file diff --git a/labs/threshold-alerts/results/2026-11-11-webhook.jsonl b/labs/threshold-alerts/results/2026-11-11-webhook.jsonl new file mode 100644 index 0000000..00395f4 --- /dev/null +++ b/labs/threshold-alerts/results/2026-11-11-webhook.jsonl @@ -0,0 +1,40 @@ +{"received_at": 1794373262.694857, "body": {"text": "\ud83d\udea8 A1 memory used, every host | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-b-02\"\n0.9 meets or exceeds 0.9\nTime Range (UTC): [Nov 11 5:00:00 AM - Nov 11 5:01:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-b-02 = 0.93 | | ALERT | 1794373200000 | 1794373260000 | "}} +{"received_at": 1794373262.729749, "body": {"text": "\ud83d\udea8 A1 memory used, every host | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-01\"\n1.0 meets or exceeds 0.9\nTime Range (UTC): [Nov 11 5:00:00 AM - Nov 11 5:01:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-01 = 0.95 | | ALERT | 1794373200000 | 1794373260000 | "}} +{"received_at": 1794373262.733508, "body": {"text": "\ud83d\udea8 A2 memory used, game-a | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-01\"\n1.0 meets or exceeds 0.8\nTime Range (UTC): [Nov 11 5:00:00 AM - Nov 11 5:01:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-01 = 0.95 | | ALERT | 1794373200000 | 1794373260000 | "}} +{"received_at": 1794373262.734022, "body": {"text": "\ud83d\udea8 A2 memory used, game-a | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-02\"\n0.9 meets or exceeds 0.8\nTime Range (UTC): [Nov 11 5:00:00 AM - Nov 11 5:01:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-02 = 0.85 | | ALERT | 1794373200000 | 1794373260000 | "}} +{"received_at": 1794373322.044545, "body": {"text": "\ud83d\udea8 A1 memory used, every host | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-01\"\n1.0 meets or exceeds 0.9\nTime Range (UTC): [Nov 11 5:01:00 AM - Nov 11 5:02:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-01 = 0.95 | | ALERT | 1794373260000 | 1794373320000 | "}} +{"received_at": 1794373322.0522182, "body": {"text": "\ud83d\udea8 A1 memory used, every host | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-b-02\"\n0.9 meets or exceeds 0.9\nTime Range (UTC): [Nov 11 5:01:00 AM - Nov 11 5:02:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-b-02 = 0.93 | | ALERT | 1794373260000 | 1794373320000 | "}} +{"received_at": 1794373322.080189, "body": {"text": "\ud83d\udea8 A2 memory used, game-a | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-01\"\n1.0 meets or exceeds 0.8\nTime Range (UTC): [Nov 11 5:01:00 AM - Nov 11 5:02:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-01 = 0.95 | | ALERT | 1794373260000 | 1794373320000 | "}} +{"received_at": 1794373322.0822752, "body": {"text": "\ud83d\udea8 A2 memory used, game-a | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-02\"\n0.9 meets or exceeds 0.8\nTime Range (UTC): [Nov 11 5:01:00 AM - Nov 11 5:02:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-02 = 0.85 | | ALERT | 1794373260000 | 1794373320000 | "}} +{"received_at": 1794373382.4352572, "body": {"text": "\ud83d\udea8 A3 memory used, every host, 3 windows | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-b-02\"\n0.9 meets or exceeds 0.9\nTime Range (UTC): [Nov 11 5:02:00 AM - Nov 11 5:03:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-b-02 = 0.93 | | ALERT | 1794373320000 | 1794373380000 | "}} +{"received_at": 1794373382.55295, "body": {"text": "\ud83d\udea8 A3 memory used, every host, 3 windows | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-01\"\n1.0 meets or exceeds 0.9\nTime Range (UTC): [Nov 11 5:02:00 AM - Nov 11 5:03:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-01 = 0.95 | | ALERT | 1794373320000 | 1794373380000 | "}} +{"received_at": 1794373382.553335, "body": {"text": "\ud83d\udea8 A2 memory used, game-a | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-01\"\n1.0 meets or exceeds 0.8\nTime Range (UTC): [Nov 11 5:02:00 AM - Nov 11 5:03:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-01 = 0.95 | | ALERT | 1794373320000 | 1794373380000 | "}} +{"received_at": 1794373382.553709, "body": {"text": "\ud83d\udea8 A1 memory used, every host | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-b-02\"\n0.9 meets or exceeds 0.9\nTime Range (UTC): [Nov 11 5:02:00 AM - Nov 11 5:03:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-b-02 = 0.93 | | ALERT | 1794373320000 | 1794373380000 | "}} +{"received_at": 1794373382.55391, "body": {"text": "\ud83d\udea8 A2 memory used, game-a | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-02\"\n0.9 meets or exceeds 0.8\nTime Range (UTC): [Nov 11 5:02:00 AM - Nov 11 5:03:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-02 = 0.85 | | ALERT | 1794373320000 | 1794373380000 | "}} +{"received_at": 1794373382.554139, "body": {"text": "\ud83d\udea8 A1 memory used, every host | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-01\"\n1.0 meets or exceeds 0.9\nTime Range (UTC): [Nov 11 5:02:00 AM - Nov 11 5:03:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-01 = 0.95 | | ALERT | 1794373320000 | 1794373380000 | "}} +{"received_at": 1794373441.2984362, "body": {"text": "\ud83d\udea8 A1 memory used, every host | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-01\"\n1.0 meets or exceeds 0.9\nTime Range (UTC): [Nov 11 5:03:00 AM - Nov 11 5:04:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-01 = 0.95 | | ALERT | 1794373380000 | 1794373440000 | "}} +{"received_at": 1794373441.3360782, "body": {"text": "\ud83d\udea8 A1 memory used, every host | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-b-02\"\n0.9 meets or exceeds 0.9\nTime Range (UTC): [Nov 11 5:03:00 AM - Nov 11 5:04:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-b-02 = 0.93 | | ALERT | 1794373380000 | 1794373440000 | "}} +{"received_at": 1794373441.3461802, "body": {"text": "\ud83d\udea8 A2 memory used, game-a | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-02\"\n0.9 meets or exceeds 0.8\nTime Range (UTC): [Nov 11 5:03:00 AM - Nov 11 5:04:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-02 = 0.85 | | ALERT | 1794373380000 | 1794373440000 | "}} +{"received_at": 1794373441.35536, "body": {"text": "\ud83d\udea8 A2 memory used, game-a | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-01\"\n1.0 meets or exceeds 0.8\nTime Range (UTC): [Nov 11 5:03:00 AM - Nov 11 5:04:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-01 = 0.95 | | ALERT | 1794373380000 | 1794373440000 | "}} +{"received_at": 1794373441.3588932, "body": {"text": "\ud83d\udea8 A3 memory used, every host, 3 windows | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-01\"\n1.0 meets or exceeds 0.9\nTime Range (UTC): [Nov 11 5:03:00 AM - Nov 11 5:04:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-01 = 0.95 | | ALERT | 1794373380000 | 1794373440000 | "}} +{"received_at": 1794373441.37168, "body": {"text": "\ud83d\udea8 A3 memory used, every host, 3 windows | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-b-02\"\n0.9 meets or exceeds 0.9\nTime Range (UTC): [Nov 11 5:03:00 AM - Nov 11 5:04:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-b-02 = 0.93 | | ALERT | 1794373380000 | 1794373440000 | "}} +{"received_at": 1794373501.4934678, "body": {"text": "\ud83d\udea8 A3 memory used, every host, 3 windows | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-b-02\"\n0.9 meets or exceeds 0.9\nTime Range (UTC): [Nov 11 5:04:00 AM - Nov 11 5:05:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-b-02 = 0.93 | | ALERT | 1794373440000 | 1794373500000 | "}} +{"received_at": 1794373501.4953861, "body": {"text": "\ud83d\udea8 A3 memory used, every host, 3 windows | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-01\"\n1.0 meets or exceeds 0.9\nTime Range (UTC): [Nov 11 5:04:00 AM - Nov 11 5:05:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-01 = 0.95 | | ALERT | 1794373440000 | 1794373500000 | "}} +{"received_at": 1794373501.528291, "body": {"text": "\ud83d\udea8 A1 memory used, every host | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-01\"\n1.0 meets or exceeds 0.9\nTime Range (UTC): [Nov 11 5:04:00 AM - Nov 11 5:05:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-01 = 0.95 | | ALERT | 1794373440000 | 1794373500000 | "}} +{"received_at": 1794373501.5288792, "body": {"text": "\ud83d\udea8 A1 memory used, every host | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-b-02\"\n0.9 meets or exceeds 0.9\nTime Range (UTC): [Nov 11 5:04:00 AM - Nov 11 5:05:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-b-02 = 0.93 | | ALERT | 1794373440000 | 1794373500000 | "}} +{"received_at": 1794373501.5292318, "body": {"text": "\ud83d\udea8 A2 memory used, game-a | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-02\"\n0.9 meets or exceeds 0.8\nTime Range (UTC): [Nov 11 5:04:00 AM - Nov 11 5:05:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-02 = 0.85 | | ALERT | 1794373440000 | 1794373500000 | "}} +{"received_at": 1794373501.529608, "body": {"text": "\ud83d\udea8 A2 memory used, game-a | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-01\"\n1.0 meets or exceeds 0.8\nTime Range (UTC): [Nov 11 5:04:00 AM - Nov 11 5:05:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-01 = 0.95 | | ALERT | 1794373440000 | 1794373500000 | "}} +{"received_at": 1794373562.529997, "body": {"text": "\ud83d\udea8 A1 memory used, every host | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-01\"\n1.0 meets or exceeds 0.9\nTime Range (UTC): [Nov 11 5:05:00 AM - Nov 11 5:06:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-01 = 0.95 | | ALERT | 1794373500000 | 1794373560000 | "}} +{"received_at": 1794373562.53047, "body": {"text": "\ud83d\udea8 A3 memory used, every host, 3 windows | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-01\"\n1.0 meets or exceeds 0.9\nTime Range (UTC): [Nov 11 5:05:00 AM - Nov 11 5:06:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-01 = 0.95 | | ALERT | 1794373500000 | 1794373560000 | "}} +{"received_at": 1794373562.530654, "body": {"text": "\ud83d\udea8 A3 memory used, every host, 3 windows | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-b-02\"\n0.9 meets or exceeds 0.9\nTime Range (UTC): [Nov 11 5:05:00 AM - Nov 11 5:06:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-b-02 = 0.93 | | ALERT | 1794373500000 | 1794373560000 | "}} +{"received_at": 1794373562.530826, "body": {"text": "\ud83d\udea8 A2 memory used, game-a | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-01\"\n1.0 meets or exceeds 0.8\nTime Range (UTC): [Nov 11 5:05:00 AM - Nov 11 5:06:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-01 = 0.95 | | ALERT | 1794373500000 | 1794373560000 | "}} +{"received_at": 1794373562.530978, "body": {"text": "\ud83d\udea8 A1 memory used, every host | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-b-02\"\n0.9 meets or exceeds 0.9\nTime Range (UTC): [Nov 11 5:05:00 AM - Nov 11 5:06:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-b-02 = 0.93 | | ALERT | 1794373500000 | 1794373560000 | "}} +{"received_at": 1794373562.531165, "body": {"text": "\ud83d\udea8 A2 memory used, game-a | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-02\"\n0.9 meets or exceeds 0.8\nTime Range (UTC): [Nov 11 5:05:00 AM - Nov 11 5:06:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-02 = 0.85 | | ALERT | 1794373500000 | 1794373560000 | "}} +{"received_at": 1794373621.8985631, "body": {"text": "\u2705 A1 memory used, every host | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-01\" - The alert has been resolved.\nTime Range (UTC): [Nov 11 5:06:00 AM - Nov 11 5:07:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-01 = 0.5 | | OK | 1794373560000 | 1794373620000 | "}} +{"received_at": 1794373621.9335449, "body": {"text": "\u2705 A1 memory used, every host | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-b-02\" - The alert has been resolved.\nTime Range (UTC): [Nov 11 5:06:00 AM - Nov 11 5:07:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-b-02 = 0.5 | | OK | 1794373560000 | 1794373620000 | "}} +{"received_at": 1794373621.935159, "body": {"text": "\u2705 A2 memory used, game-a | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-01\" - The alert has been resolved.\nTime Range (UTC): [Nov 11 5:06:00 AM - Nov 11 5:07:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-01 = 0.5 | | OK | 1794373560000 | 1794373620000 | "}} +{"received_at": 1794373621.936034, "body": {"text": "\u2705 A2 memory used, game-a | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-02\" - The alert has been resolved.\nTime Range (UTC): [Nov 11 5:06:00 AM - Nov 11 5:07:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-02 = 0.5 | | OK | 1794373560000 | 1794373620000 | "}} +{"received_at": 1794373622.088864, "body": {"text": "\ud83d\udea8 A3 memory used, every host, 3 windows | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-01\"\n1.0 meets or exceeds 0.9\nTime Range (UTC): [Nov 11 5:06:00 AM - Nov 11 5:07:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-01 = 0.95 | | ALERT | 1794373560000 | 1794373620000 | "}} +{"received_at": 1794373622.126268, "body": {"text": "\ud83d\udea8 A3 memory used, every host, 3 windows | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-b-02\"\n0.9 meets or exceeds 0.9\nTime Range (UTC): [Nov 11 5:06:00 AM - Nov 11 5:07:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-b-02 = 0.93 | | ALERT | 1794373560000 | 1794373620000 | "}} +{"received_at": 1794373681.964888, "body": {"text": "\u2705 A3 memory used, every host, 3 windows | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-b-02\" - The alert has been resolved.\nTime Range (UTC): [Nov 11 5:07:00 AM - Nov 11 5:08:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-b-02 = 0.5 | | OK | 1794373620000 | 1794373680000 | "}} +{"received_at": 1794373682.000363, "body": {"text": "\u2705 A3 memory used, every host, 3 windows | Group: \"arrayElement(ResourceAttributes, 'host.name'):game-a-01\" - The alert has been resolved.\nTime Range (UTC): [Nov 11 5:07:00 AM - Nov 11 5:08:00 AM)\nlabs/threshold-alerts (synthetic) arrayElement(ResourceAttributes, 'host.name'):game-a-01 = 0.5 | | OK | 1794373620000 | 1794373680000 | "}}