diff --git a/labs/common-fields/README.md b/labs/common-fields/README.md index a6db8ba..95a36db 100644 --- a/labs/common-fields/README.md +++ b/labs/common-fields/README.md @@ -10,10 +10,11 @@ set, how the collector normalises the names, and the ClickStack tables that keep with the configs as files and every piece checked. > **Verified on:** `schema/schema.sql` in `clickhouse local` 26.9.6.6 (`tests/test_schema.py`: -> every table created, the columns fill, a filter on them uses the primary key) and on ClickHouse -> Cloud 26.8 with Managed ClickStack (`managed/demo.py`, the screens below), 2026-11-06. +> every table created, the columns fill, a filter on them uses the primary key; the negative checks +> and the injected fault in [Tests](#tests)), 2026-11-07, and on ClickHouse Cloud 26.8.1.608 with +> Managed ClickStack (`managed/demo.py`, the screens below), 2026-11-07. > `collector/standard-fields.yaml` in otelcol-contrib 0.155.0 (`tests/test_collector.sh`: the -> rename and its precedence), 2026-11-06. The read-volume numbers come from +> rename, its precedence, the negative check and the injected fault), 2026-11-07. The read-volume numbers come from > [`labs/hostmetrics-volume`](../hostmetrics-volume/README.md) (logs and gauge metrics, settled > run, 2026-11-06). Traces and the other metric types follow the same rule and were not measured. @@ -29,8 +30,8 @@ with the configs as files and every piece checked. All are **resource attributes**: they describe who sent the data, so they are the same on every span, log line and metric point of that process, and they land in the `ResourceAttributes` -column. Keep the standard to these few (20–30 % of the fields anyone filters on); everything else -stays free-form in the attribute maps — slower to filter, still searchable. +column. Keep the standard to these few; everything else stays free-form in the attribute maps — +slower to filter, still searchable. ### 1. Apps (development teams) @@ -98,7 +99,7 @@ clickhouse client --database otel --multiquery < schema/schema.sql Apply it into an empty database before the first export. ClickStack needs no change: a search on the map key (`svc.group:"payments"`) is rewritten to the column. -What it buys, measured on ClickHouse Cloud with synthetic data the size of 4,000 hosts +What it buys, measured on ClickHouse Cloud 26.8.1.608 with synthetic data the size of 4,000 hosts ([`labs/hostmetrics-volume`](../hostmetrics-volume/README.md), 2026-11-06, settled tables): | Query | Exporter default | This schema | @@ -115,11 +116,11 @@ finds the materialized column and rewrites a search on the map key to it. ![ClickStack source settings: cf demo logs, database common_fields_demo, table otel_logs](images/01-source-settings.webp) -On the demo (2 M log rows an hour, 200 services in 20 groups), `ResourceAttributes.svc.group:"g7"` -over 15 minutes became `svc_group = 'g7'` and read 49,152 rows — six granules — for 21,856 +On the demo (ClickHouse Cloud 26.8.1.608, 2026-11-07; 2 M log rows an hour, 200 services in 20 +groups), `ResourceAttributes.svc.group:"g7"` over 15 minutes became `svc_group = 'g7'` and read 49,152 rows — six granules — for 24,836 results: -![Exact match svc.group:"g7": 21,856 results, Scanned Rows 49,152](images/02-search-exact.webp) +![Exact match svc.group:"g7": 24,836 results, Scanned Rows 49,152](images/02-search-exact.webp) Without quotes, `svc.group:g7` is a **substring** match ("contains"), sent as `svc_group ILIKE '%g7%'`. On this schema it read the same 49,152 rows, because the column's set @@ -137,8 +138,10 @@ id) and the common fields under Resource Attributes: ![A g7 error span with its four log lines in the waterfall; Resource Attributes show svc.domain d3, svc.group g7, host.name, deployment.environment.name](images/04-trace-with-logs.webp) -Screens: Managed ClickStack (HyperDX 2.42.0) on ClickHouse Cloud 26.8.1.608, Seoul, 2026-11-06, -`managed/demo.py`; rows read confirmed in `system.query_log`. +Screens: Managed ClickStack on ClickHouse Cloud 26.8.1.608, Seoul, 2026-11-07, `managed/demo.py`, +the 15 minutes 12:39–12:54 KST. `system.query_log` shows both searches' queries +(`svc_group = 'g7'` and `svc_group ILIKE '%g7%'`) reading 49,152 rows, and a direct `count()` on +the window gives the 24,836 results. ### Run the demo on Managed ClickStack @@ -158,6 +161,15 @@ python3 -m unittest discover -s tests # schema.sql in clickhouse local (skip tests/test_collector.sh # standard-fields.yaml in otelcol-contrib 0.155.0 (Docker) ``` +Each test has a positive check, a negative check and a fault it injects into a copy of the file +and must catch (2026-11-07, `clickhouse local` 26.9.6.6 · otelcol-contrib 0.155.0: 5/5 tests, +6/6 checks, both faults caught): + +| Test | Positive | Negative | Injected fault | +|---|---|---|---| +| `test_schema.py` | `svc_group = 'g7'` (logs) and `svc_group` + `host` (gauge) prune primary-key granules | `ResourceAttributes['svc.group'] = 'g7'` does not use the primary key (ClickHouse does not rewrite the map lookup; ClickStack's search does); a row without `svc.*` gets `''` | `svc_group` dropped from the logs sort key: the pruning check fails | +| `test_collector.sh` | EC2 tags become `svc.domain` / `svc.group`; an app's own `svc.group` is kept; `ec2.tag.svc.*` removed | a log with neither tags nor `svc.*` gets no `svc.*` key | the `svc.group` rule without its "app value first" guard: "app: its own svc.group kept" fails | + ## 한국어 플릿을 조회할 때는 매번 같은 몇 개 필드로 거릅니다. 어느 서비스 도메인·그룹인지, 어느 호스트인지, @@ -166,9 +178,10 @@ tests/test_collector.sh # standard-fields.yaml in otelcol-con 두고, 하나씩 확인했습니다. > **검증:** `schema/schema.sql`은 `clickhouse local` 26.9.6.6(`tests/test_schema.py`: 테이블이 모두 -> 만들어지고, 컬럼이 채워지고, 그 컬럼 조건이 primary key를 씀)과 ClickHouse Cloud 26.8의 Managed -> ClickStack(`managed/demo.py`, 아래 화면)에서, 2026-11-06. `collector/standard-fields.yaml`은 -> otelcol-contrib 0.155.0(`tests/test_collector.sh`: 이름 바꾸기와 우선순위), 2026-11-06. 읽는 양 +> 만들어지고, 컬럼이 채워지고, 그 컬럼 조건이 primary key를 씀. 반대 확인과 일부러 넣은 오류는 +> [테스트](#테스트) 참고)에서 2026-11-07, ClickHouse Cloud 26.8.1.608의 Managed ClickStack(`managed/demo.py`, +> 아래 화면)에서 2026-11-07. `collector/standard-fields.yaml`은 otelcol-contrib 0.155.0(`tests/test_collector.sh`: +> 이름 바꾸기, 우선순위, 반대 확인, 일부러 넣은 오류), 2026-11-07. 읽는 양 > 수치는 [`labs/hostmetrics-volume`](../hostmetrics-volume/README.md)(로그와 gauge 메트릭, 병합이 끝난 > 뒤 실행, 2026-11-06)에서 왔습니다. trace와 나머지 메트릭 종류는 같은 원칙을 따르지만 재지 않았습니다. @@ -183,8 +196,7 @@ tests/test_collector.sh # standard-fields.yaml in otelcol-con | `service.version` | 배포 버전 | 앱: `OTEL_RESOURCE_ATTRIBUTES`(빌드에서) | 모두 **resource attribute**입니다. 데이터를 보낸 주체를 설명하므로 그 프로세스의 모든 span, 로그 줄, -메트릭 point에 똑같이 붙고, `ResourceAttributes` 컬럼에 들어갑니다. 표준은 이 몇 개(사람들이 거르는 -필드의 20–30%)로 둡니다. 나머지는 attribute map에 자유롭게 둡니다. 거를 때 느릴 뿐 검색은 됩니다. +메트릭 point에 똑같이 붙고, `ResourceAttributes` 컬럼에 들어갑니다. 표준은 이 몇 개로 둡니다. 나머지는 attribute map에 자유롭게 둡니다. 거를 때 느릴 뿐 검색은 됩니다. ### 1. 앱(개발팀) @@ -249,7 +261,7 @@ clickhouse client --database otel --multiquery < schema/schema.sql 첫 전송 전에 빈 데이터베이스에 적용합니다. ClickStack은 바꿀 것이 없습니다. map 키 검색 (`svc.group:"payments"`)을 그 컬럼으로 바꿔 씁니다. -효과는 4,000대 규모의 합성 데이터로 ClickHouse Cloud에서 쟀습니다([`labs/hostmetrics-volume`](../hostmetrics-volume/README.md), +효과는 4,000대 규모의 합성 데이터로 ClickHouse Cloud 26.8.1.608에서 쟀습니다([`labs/hostmetrics-volume`](../hostmetrics-volume/README.md), 2026-11-06, 병합이 끝난 테이블). | 조회 | exporter 기본 | 이 스키마 | @@ -266,10 +278,10 @@ materialized 컬럼을 찾아 map 키 검색을 그 컬럼으로 바꿔 씁니 ![ClickStack 소스 설정: cf demo logs, 데이터베이스 common_fields_demo, 테이블 otel_logs](images/01-source-settings.webp) -데모(시간당 로그 200만 행, 서비스 200개·그룹 20개)에서 15분 구간의 `ResourceAttributes.svc.group:"g7"`은 -`svc_group = 'g7'`으로 바뀌었고, granule 여섯 개인 49,152행을 읽어 결과 21,856건을 냈습니다. +데모(ClickHouse Cloud 26.8.1.608, 2026-11-07. 시간당 로그 200만 행, 서비스 200개·그룹 20개)에서 15분 구간의 `ResourceAttributes.svc.group:"g7"`은 +`svc_group = 'g7'`으로 바뀌었고, granule 여섯 개인 49,152행을 읽어 결과 24,836건을 냈습니다. -![정확 일치 svc.group:"g7": 결과 21,856건, Scanned Rows 49,152](images/02-search-exact.webp) +![정확 일치 svc.group:"g7": 결과 24,836건, Scanned Rows 49,152](images/02-search-exact.webp) 따옴표가 없는 `svc.group:g7`은 **부분 일치**("contains")로, `svc_group ILIKE '%g7%'`로 나갑니다. 이 스키마에서는 같은 49,152행을 읽었습니다. 컬럼의 set 인덱스가 granule마다 값을 확인할 수 있기 때문입니다. 그 인덱스 없이 @@ -285,8 +297,9 @@ trace ID가 들어 있음), Resource Attributes에 공통 필드가 보입니다 ![g7의 에러 span과 로그 네 줄이 한 waterfall에, Resource Attributes에 svc.domain d3, svc.group g7, host.name, deployment.environment.name](images/04-trace-with-logs.webp) -화면: Managed ClickStack(HyperDX 2.42.0), ClickHouse Cloud 26.8.1.608, 서울, 2026-11-06, `managed/demo.py`. -읽은 행은 `system.query_log`로 확인했습니다. +화면: Managed ClickStack, ClickHouse Cloud 26.8.1.608, 서울, 2026-11-07, `managed/demo.py`, 12:39–12:54 KST의 +15분. `system.query_log`에서 두 검색의 쿼리(`svc_group = 'g7'`, `svc_group ILIKE '%g7%'`)가 모두 49,152행을 읽었고, +같은 구간을 직접 `count()`하면 결과 24,836건이 나옵니다. ### Managed ClickStack에서 데모 돌리기 @@ -304,3 +317,11 @@ python3 managed/demo.py --cleanup python3 -m unittest discover -s tests # clickhouse local의 schema.sql(바이너리가 없으면 건너뜀) tests/test_collector.sh # otelcol-contrib 0.155.0의 standard-fields.yaml(Docker) ``` + +테스트마다 맞는 경우 확인, 반대 확인, 그리고 파일 사본에 일부러 넣어 잡아내야 하는 오류가 하나씩 있습니다 +(2026-11-07, `clickhouse local` 26.9.6.6 · otelcol-contrib 0.155.0: 테스트 5/5, 확인 6/6, 오류 둘 다 잡음). + +| 테스트 | 맞는 경우 | 반대 확인 | 일부러 넣은 오류 | +|---|---|---|---| +| `test_schema.py` | `svc_group = 'g7'`(로그), `svc_group` + `host`(gauge)가 primary key granule을 줄임 | `ResourceAttributes['svc.group'] = 'g7'`은 primary key를 쓰지 않음(ClickHouse는 map 조회를 컬럼으로 바꾸지 않고, ClickStack 검색이 바꿈). `svc.*`가 없는 행은 `''` | 로그 정렬 키에서 `svc_group`을 뺌: granule 줄임 확인이 실패 | +| `test_collector.sh` | EC2 태그가 `svc.domain` / `svc.group`이 됨, 앱이 넣은 `svc.group`은 유지, `ec2.tag.svc.*` 삭제 | 태그도 `svc.*`도 없는 로그에는 `svc.*` 키가 생기지 않음 | `svc.group` 규칙에서 "앱 값 우선" 조건을 뺌: "app: its own svc.group kept"가 실패 | diff --git a/labs/common-fields/images/01-source-settings.webp b/labs/common-fields/images/01-source-settings.webp index f1ad5f3..6397ea6 100644 Binary files a/labs/common-fields/images/01-source-settings.webp and b/labs/common-fields/images/01-source-settings.webp differ diff --git a/labs/common-fields/images/02-search-exact.webp b/labs/common-fields/images/02-search-exact.webp index d6ad992..2994da2 100644 Binary files a/labs/common-fields/images/02-search-exact.webp and b/labs/common-fields/images/02-search-exact.webp differ diff --git a/labs/common-fields/images/03-search-substring.webp b/labs/common-fields/images/03-search-substring.webp index 130ee85..9ed5f66 100644 Binary files a/labs/common-fields/images/03-search-substring.webp and b/labs/common-fields/images/03-search-substring.webp differ diff --git a/labs/common-fields/images/04-trace-with-logs.webp b/labs/common-fields/images/04-trace-with-logs.webp index aa32aeb..fc04f2e 100644 Binary files a/labs/common-fields/images/04-trace-with-logs.webp and b/labs/common-fields/images/04-trace-with-logs.webp differ diff --git a/labs/common-fields/lab.yaml b/labs/common-fields/lab.yaml index bf4e64f..b375c40 100644 --- a/labs/common-fields/lab.yaml +++ b/labs/common-fields/lab.yaml @@ -3,4 +3,4 @@ target: cloud tier: T1 services: [clickstack] -verified_on: "clickhouse local 26.9.6.6, otelcol-contrib 0.155.0 and Managed ClickStack (ClickHouse Cloud 26.8), 2026-11-06" +verified_on: "clickhouse local 26.9.6.6 and otelcol-contrib 0.155.0, 2026-11-07; Managed ClickStack (ClickHouse Cloud 26.8.1.608), 2026-11-07" diff --git a/labs/common-fields/managed/demo.py b/labs/common-fields/managed/demo.py index 4061774..f0427b7 100644 --- a/labs/common-fields/managed/demo.py +++ b/labs/common-fields/managed/demo.py @@ -103,19 +103,33 @@ def statements(text): if(number % 50 = 0, concat('timeout calling payment-gateway after ', toString(1000 + number % 4000), ' ms'), concat('GET /api/v1/items/', toString(number % 100000), ' completed in ', toString(number % 90), ' ms')), {RES}, map('http.route', '/api/v1/items/{{id}}') -FROM (SELECT number, h % 200 AS s, h FROM (SELECT number, cityHash64(intDiv(number, 4)) % 4000 AS h FROM numbers(2000000)))""" +FROM (SELECT number, h % 200 AS s, h FROM (SELECT number, cityHash64(intDiv(number, 4)) % 4000 AS h FROM numbers(@START, @N)))""" TRACES = f"""INSERT INTO otel_traces (Timestamp, TraceId, SpanId, ParentSpanId, SpanName, SpanKind, ServiceName, ResourceAttributes, SpanAttributes, Duration, StatusCode) SELECT now64(9) - toIntervalSecond(3600) + toIntervalMillisecond(intDiv(number * 4 * 3600000, 2000000)), lower(hex(cityHash64(number))) || lower(hex(cityHash64(number, 1))), lower(hex(cityHash64(number, 2))), '', 'GET /api/v1/items/{{id}}', 'Server', concat('svc-', toString(s)), {RES}, map('http.route', '/api/v1/items/{{id}}'), (5 + number % 200) * 1000000, if(number % 50 = 0, 'Error', 'Ok') -FROM (SELECT number, h % 200 AS s, h FROM (SELECT number, cityHash64(number) % 4000 AS h FROM numbers(500000)))""" +FROM (SELECT number, h % 200 AS s, h FROM (SELECT number, cityHash64(number) % 4000 AS h FROM numbers(@START, @N)))""" METRICS = f"""INSERT INTO otel_metrics_gauge (ResourceAttributes, ServiceName, MetricName, MetricUnit, Attributes, TimeUnix, Value) SELECT {RES}, '', m, '1', map(), toStartOfMinute(now64(9)) - toIntervalMinute(60 - t), (cityHash64(h, t, m) % 1000) / 1000 -FROM (SELECT number % 4000 AS h, h % 200 AS s, intDiv(number, 4000) AS t FROM numbers(4000 * 60)) +FROM (SELECT number % 4000 AS h, h % 200 AS s, intDiv(number, 4000) AS t FROM numbers(@START, @N)) ARRAY JOIN ['system.cpu.utilization', 'system.memory.utilization'] AS m""" +# (statement, rows, rows per INSERT). As one INSERT ... SELECT, the 2 M log rows hit the Seoul +# service's total memory limit on 2026-11-07 (MEMORY_LIMIT_EXCEEDED, maximum 6.89 GiB). The chunks +# keep `number`, so the rows are the same as one INSERT's. +CHUNKS = ((LOGS, 2000000, 250000), (TRACES, 500000, 125000), (METRICS, 4000 * 60, 60000)) + + +def inserts(base): + """The INSERTs, in chunks, with one base time (epoch ms) for all of them.""" + for sql, total, step in CHUNKS: + sql = sql.replace("now64(9)", f"fromUnixTimestamp64Milli({base}, 'UTC')") + for start in range(0, total, step): + yield sql.replace("@START", str(start)).replace("@N", str(min(step, total - start))) + + def sources(cloud, create): for s in cloud.cs("GET", "/sources"): if s.get("name") in NAMES.values(): @@ -165,8 +179,8 @@ def main(): # One base time for all three inserts, so a trace and its log lines share a timestamp # (now64() in each INSERT would differ by however long the previous insert took). base = ch.q("SELECT toUnixTimestamp64Milli(now64(3))").strip() - for sql in (LOGS, TRACES, METRICS): - ch.q(sql.replace("now64(9)", f"fromUnixTimestamp64Milli({base}, 'UTC')"), DB) + for sql in inserts(base): + ch.q(sql, DB) counts = ch.q("SELECT table, sum(rows) FROM system.parts WHERE database = currentDatabase() AND active " "GROUP BY table ORDER BY table FORMAT TSV", DB).strip() sources(cloud, True) diff --git a/labs/common-fields/tests/test_collector.sh b/labs/common-fields/tests/test_collector.sh index 3c9f878..2d69a82 100755 --- a/labs/common-fields/tests/test_collector.sh +++ b/labs/common-fields/tests/test_collector.sh @@ -1,26 +1,37 @@ #!/usr/bin/env bash # Runs ../collector/standard-fields.yaml in otelcol-contrib (Docker) and checks the rename: -# a log from a host with only EC2 tags -> svc.domain / svc.group set, ec2.tag.svc.* gone -# a log from an app that sets svc.group -> its own value kept -# Exit 0 when both hold. Needs Docker; pulls nothing beyond the collector image. +# positive a log from a host with only EC2 tags -> svc.domain / svc.group set, ec2.tag.svc.* gone +# a log from an app that sets svc.group -> its own value kept +# negative a log with neither tags nor svc.* -> no svc.* key made up +# fault the same checks on a copy of the file whose svc.group rule lost its "app value +# first" guard must FAIL; the script fails if they pass +# Exit 0 when the real file passes every check and the faulted copy fails. Needs Docker; pulls +# nothing beyond the collector image. set -euo pipefail here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" IMAGE="${IMAGE:-otel/opentelemetry-collector-contrib:0.155.0}" -out="$(mktemp -d)"; chmod 777 "$out" -name="cf-test-$$" -trap 'docker rm -f "$name" >/dev/null 2>&1 || true; rm -rf "$out"' EXIT -docker run -d --name "$name" -p 127.0.0.1::4318 -v "$here/..:/cf:ro" -v "$out:/out" "$IMAGE" \ - --config /cf/collector/standard-fields.yaml --config /cf/tests/collector-test.yaml >/dev/null -port="$(docker port "$name" 4318/tcp | head -1 | sed 's/.*://')" -for _ in $(seq 1 30); do curl -s -o /dev/null "http://127.0.0.1:$port/v1/logs" -X POST -H 'Content-Type: application/json' -d '{}' && break; sleep 1; done -send() { - curl -sf "http://127.0.0.1:$port/v1/logs" -H 'Content-Type: application/json' -d "{\"resourceLogs\":[{\"resource\":{\"attributes\":$1}, - \"scopeLogs\":[{\"logRecords\":[{\"timeUnixNano\":\"$(date +%s)000000000\",\"body\":{\"stringValue\":\"$2\"}}]}]}]}" >/dev/null -} -send '[{"key":"host.name","value":{"stringValue":"host-1"}},{"key":"ec2.tag.svc.domain","value":{"stringValue":"d1"}},{"key":"ec2.tag.svc.group","value":{"stringValue":"g7"}}]' tagged -send '[{"key":"service.name","value":{"stringValue":"api"}},{"key":"svc.group","value":{"stringValue":"g3"}},{"key":"ec2.tag.svc.group","value":{"stringValue":"g9"}}]' app -for _ in $(seq 1 20); do [ "$(grep -c resourceLogs "$out/out.json" 2>/dev/null || true)" -ge 2 ] && break; sleep 1; done -python3 - "$out/out.json" <<'PY' +work="$(mktemp -d)"; chmod 755 "$work" +names=() +trap 'for n in "${names[@]}"; do docker rm -f "$n" >/dev/null 2>&1 || true; done; rm -rf "$work"' EXIT + +# run_checks : prints PASS/FAIL lines, returns 1 on any FAIL +run_checks() { + local cfg="$1" out name port + out="$(mktemp -d "$work/out.XXXX")"; chmod 777 "$out" + name="cf-test-$$-${#names[@]}"; names+=("$name") + docker run -d --name "$name" -p 127.0.0.1::4318 -v "$cfg:/cf/collector:ro" -v "$here:/cf/tests:ro" -v "$out:/out" \ + "$IMAGE" --config /cf/collector/standard-fields.yaml --config /cf/tests/collector-test.yaml >/dev/null + port="$(docker port "$name" 4318/tcp | head -1 | sed 's/.*://')" + for _ in $(seq 1 30); do curl -s -o /dev/null "http://127.0.0.1:$port/v1/logs" -X POST -H 'Content-Type: application/json' -d '{}' && break; sleep 1; done + send() { + curl -sf "http://127.0.0.1:$port/v1/logs" -H 'Content-Type: application/json' -d "{\"resourceLogs\":[{\"resource\":{\"attributes\":$1}, + \"scopeLogs\":[{\"logRecords\":[{\"timeUnixNano\":\"$(date +%s)000000000\",\"body\":{\"stringValue\":\"$2\"}}]}]}]}" >/dev/null + } + send '[{"key":"host.name","value":{"stringValue":"host-1"}},{"key":"ec2.tag.svc.domain","value":{"stringValue":"d1"}},{"key":"ec2.tag.svc.group","value":{"stringValue":"g7"}}]' tagged + send '[{"key":"service.name","value":{"stringValue":"api"}},{"key":"svc.group","value":{"stringValue":"g3"}},{"key":"ec2.tag.svc.group","value":{"stringValue":"g9"}}]' app + send '[{"key":"host.name","value":{"stringValue":"host-2"}}]' untagged + for _ in $(seq 1 20); do [ "$(grep -c resourceLogs "$out/out.json" 2>/dev/null || true)" -ge 3 ] && break; sleep 1; done + python3 - "$out/out.json" <<'PY' import json, sys res = {} for line in open(sys.argv[1]): @@ -28,15 +39,31 @@ for line in open(sys.argv[1]): attrs = {a["key"]: a["value"].get("stringValue") for a in rl["resource"].get("attributes", [])} body = rl["scopeLogs"][0]["logRecords"][0]["body"]["stringValue"] res[body] = attrs -t, a = res["tagged"], res["app"] +t, a, u = res["tagged"], res["app"], res["untagged"] checks = [ ("tagged host: svc.domain from the tag", t.get("svc.domain") == "d1"), ("tagged host: svc.group from the tag", t.get("svc.group") == "g7"), ("tagged host: ec2.tag.svc.* removed", not any(k.startswith("ec2.tag.svc.") for k in t)), ("app: its own svc.group kept", a.get("svc.group") == "g3"), ("app: ec2.tag.svc.* removed", not any(k.startswith("ec2.tag.svc.") for k in a)), + ("untagged (negative): no svc.* key added", not any(k.startswith("svc.") for k in u)), ] for name, ok in checks: print(("PASS " if ok else "FAIL ") + name) sys.exit(0 if all(ok for _, ok in checks) else 1) PY +} + +echo "== standard-fields.yaml" +run_checks "$here/../collector" + +echo "== fault: svc.group rule without its 'app value first' guard (must FAIL)" +mkdir "$work/fault" +guard=' where attributes["svc.group"] == nil and attributes["ec2.tag.svc.group"] != nil' +src="$here/../collector/standard-fields.yaml" +[ "$(grep -cF "$guard" "$src")" = 1 ] || { echo "the guard this fault removes is not in $src"; exit 1; } +python3 -c 'import sys; s=open(sys.argv[1]).read(); open(sys.argv[2],"w").write(s.replace(sys.argv[3], " where attributes[\"ec2.tag.svc.group\"] != nil"))' \ + "$src" "$work/fault/standard-fields.yaml" "$guard" +chmod 755 "$work/fault"; chmod 644 "$work/fault/standard-fields.yaml" +if run_checks "$work/fault"; then echo "FAIL the faulted copy passed: the checks do not catch a lost guard"; exit 1; fi +echo "PASS the faulted copy failed, as it must" diff --git a/labs/common-fields/tests/test_schema.py b/labs/common-fields/tests/test_schema.py index 8fca9dd..4ac15b4 100644 --- a/labs/common-fields/tests/test_schema.py +++ b/labs/common-fields/tests/test_schema.py @@ -2,6 +2,12 @@ fill from ResourceAttributes, and a filter on them uses the primary key. Skipped without the `clickhouse` binary. No network. + positive a filter on svc_group (logs) and svc_group + host (gauge) prunes primary-key granules + negative the same filter written on the map, ResourceAttributes['svc.group'], does not use the + primary key; a row without the svc.* keys gets '' in the columns, not a made-up value + fault test_fault_fires drops svc_group from the logs sort key and asserts the positive check + then fails, so a schema that loses the key cannot pass + python3 -m unittest discover -s tests """ import os @@ -27,14 +33,19 @@ INSERT INTO otel_traces (Timestamp, TraceId, SpanId, SpanName, ServiceName, Duration, ResourceAttributes) SELECT now64(9) - toIntervalSecond(number % 3600), hex(number), hex(number), 'GET /items', 'svc-1', 1000, map('svc.group', 'g1') FROM numbers(1000); +INSERT INTO otel_logs (Timestamp, ServiceName, Body, ResourceAttributes) +VALUES (now64(9), 'untagged', 'no svc keys', map('host.name', 'host-x')); """ +LOGS_KEY = "ORDER BY (svc_group, ServiceName, toStartOfFiveMinutes(Timestamp), Timestamp)" @unittest.skipUnless(shutil.which("clickhouse"), "clickhouse binary not on PATH") class Schema(unittest.TestCase): - def run_sql(self, tail): - with open(SCHEMA) as f: - script = "CREATE DATABASE otel; USE otel;\n" + f.read() + SEED + tail + def run_sql(self, tail, schema=None): + if schema is None: + with open(SCHEMA) as f: + schema = f.read() + script = "CREATE DATABASE otel; USE otel;\n" + schema + SEED + tail with tempfile.NamedTemporaryFile("w", suffix=".sql", delete=False) as f: f.write(script) try: @@ -48,19 +59,43 @@ def test_tables(self): self.assertEqual(set(out.split()), TABLES) def test_columns_fill(self): - out = self.run_sql("SELECT countIf(svc_group = '' OR svc_domain = '' OR host = '') FROM otel_logs;" - "SELECT count() FROM otel_traces_trace_id_ts;") - self.assertEqual(out.split(), ["0", "1000"]) + out = self.run_sql("SELECT countIf(svc_group = '' OR svc_domain = '' OR host = '') FROM otel_logs " + "WHERE ServiceName != 'untagged';" + "SELECT count() FROM otel_traces_trace_id_ts;" + "SELECT concat('[', svc_domain, '|', svc_group, '|', host, ']') FROM otel_logs " + "WHERE ServiceName = 'untagged';") + self.assertEqual(out.split(), ["0", "1000", "[||host-x]"]) + + def primary_key(self, sql, schema=None): + """(condition, granules read, granules total) of the PrimaryKey step of EXPLAIN indexes = 1.""" + out = self.run_sql(f"EXPLAIN indexes = 1 {sql};", schema) + pk = out.split("PrimaryKey", 1)[1].split("Skip", 1)[0] + cond = pk.split("Condition: ", 1)[1].split("\n", 1)[0].strip() + got, total = (int(x) for x in pk.split("Granules: ", 1)[1].split()[0].split("/")) + return cond, got, total + + def assert_pruned(self, sql, schema=None): + cond, got, total = self.primary_key(sql, schema) + self.assertIn("svc_group in", cond, sql) + self.assertLess(got, total, sql) def test_primary_key_used(self): - for sql in ("SELECT count() FROM otel_logs WHERE svc_group = 'g7'", - "SELECT count() FROM otel_metrics_gauge WHERE MetricName = 'system.cpu.utilization' " - "AND svc_group = 'g7' AND host = 'host-1234'"): - out = self.run_sql(f"EXPLAIN indexes = 1 {sql};") - pk = out.split("PrimaryKey", 1)[1].split("Skip", 1)[0] - self.assertIn("svc_group in", pk, sql) - got, total = (int(x) for x in pk.split("Granules: ", 1)[1].split()[0].split("/")) - self.assertLess(got, total, sql) + self.assert_pruned("SELECT count() FROM otel_logs WHERE svc_group = 'g7'") + self.assert_pruned("SELECT count() FROM otel_metrics_gauge WHERE MetricName = 'system.cpu.utilization' " + "AND svc_group = 'g7' AND host = 'host-1234'") + + def test_map_filter_skips_primary_key(self): + # Negative: ClickHouse does not rewrite the map lookup to the column; ClickStack's search does. + cond, got, total = self.primary_key("SELECT count() FROM otel_logs WHERE ResourceAttributes['svc.group'] = 'g7'") + self.assertEqual((cond, got), ("true", total)) + + def test_fault_fires(self): + with open(SCHEMA) as f: + schema = f.read() + self.assertEqual(schema.count(LOGS_KEY), 1, "the logs sort key this fault edits is gone") + broken = schema.replace(LOGS_KEY, "ORDER BY (ServiceName, toStartOfFiveMinutes(Timestamp), Timestamp)") + with self.assertRaises(AssertionError): + self.assert_pruned("SELECT count() FROM otel_logs WHERE svc_group = 'g7'", broken) if __name__ == "__main__":