diff --git a/README.md b/README.md index 4a1afb1..de98850 100644 --- a/README.md +++ b/README.md @@ -28,6 +28,7 @@ Observability on ClickHouse: OpenTelemetry ingestion, the ClickStack/HyperDX UI, | [labs/llm-serving-observability](labs/llm-serving-observability/) | Self-hosted LLM serving (vLLM behind LiteLLM) observed with ClickStack, runnable without a GPU: the OTel Collector configuration, an `nvidia-smi` โ†’ OTLP pusher, LiteLLM gateway-metric cardinality with and without the attribute allowlist, and the 32-tile HyperDX dashboard imported onto hand-made sources | | [labs/vault-audit](labs/vault-audit/) | HashiCorp Vault OSS audit trail and HA metrics in ClickStack: a three-node Raft cluster's audit files (written by whichever node is active) through `file_log` and OTTL into OTel logs โ€” TraceId per request, issuance joined to later use on HMAC'd accessors, no secret in clear โ€” plus per-node metrics with `unauthenticated_metrics_access`; failover checked (verified locally and on Managed ClickStack, 2026-11-06) | | [labs/managed-rbac](labs/managed-rbac/) | How finely Managed ClickStack splits access per game, user and column: ClickStack roles (resource, name/tag/ID, the API's beta row conditions), the per-user JWT identity search uses and the one shared identity alerts use, row and masking policies on each โ€” and where alerts and Map keys leak (verified on Managed ClickStack with an Admin and a test user, 2026-11-07) | +| [labs/vector-route-dual-write](labs/vector-route-dual-write/) | Vector already ships every Kubernetes log to Loki: route two Kong gateways and one API into their own ClickHouse tables from the same Vector, leave Loki as it is, and see what each sink buffer does to Loki while ClickHouse is down (`block` stalls the whole source, `drop_newest` drops ClickHouse rows, a disk buffer keeps both); one bad row drops a whole batch (verified on local k3d, Vector 0.61.0, ClickStack 2.44.0, 2026-11-13) | ### ๐ŸŽ“ Workshops (`workshops/`) @@ -122,6 +123,7 @@ ClickHouse ๊ธฐ๋ฐ˜ ๊ด€์ธก์„ฑ ์‹ค์Šต์ž…๋‹ˆ๋‹ค. OpenTelemetry ์ˆ˜์ง‘, ClickStack/Hy | [labs/llm-serving-observability](labs/llm-serving-observability/) | LiteLLM ๋’ค์˜ vLLM์„ ClickStack์œผ๋กœ ๊ด€์ธก, GPU ์—†์ด ์‹คํ–‰: OTel Collector ์„ค์ •, `nvidia-smi` โ†’ OTLP ํ‘ธ์…”, ์†์„ฑ ํ—ˆ์šฉ ๋ชฉ๋ก ์œ ๋ฌด์— ๋”ฐ๋ฅธ LiteLLM ๊ฒŒ์ดํŠธ์›จ์ด ๋ฉ”ํŠธ๋ฆญ ์นด๋””๋„๋ฆฌํ‹ฐ, ์†์œผ๋กœ ๋งŒ๋“  ์†Œ์Šค์— ๊ฐ€์ ธ์˜จ HyperDX ๋Œ€์‹œ๋ณด๋“œ(ํƒ€์ผ 32๊ฐœ) | | [labs/vault-audit](labs/vault-audit/) | HashiCorp Vault OSS ๊ฐ์‚ฌ ๊ธฐ๋ก๊ณผ HA ๋ฉ”ํŠธ๋ฆญ์„ ClickStack์—์„œ: Raft 3๋…ธ๋“œ ํด๋Ÿฌ์Šคํ„ฐ์˜ ๊ฐ์‚ฌ ํŒŒ์ผ(๊ทธ๋•Œ active์ธ ๋…ธ๋“œ๊ฐ€ ์”€)์„ `file_log`์™€ OTTL๋กœ OTel ๋กœ๊ทธ๋กœ โ€” ์š”์ฒญ๋งˆ๋‹ค TraceId, HMAC ์ฒ˜๋ฆฌ๋œ accessor๋กœ ๋ฐœ๊ธ‰๊ณผ ์ดํ›„ ์‚ฌ์šฉ ์—ฐ๊ฒฐ, ํ‰๋ฌธ ์‹œํฌ๋ฆฟ ์—†์Œ โ€” ๊ทธ๋ฆฌ๊ณ  `unauthenticated_metrics_access`๋กœ ๋…ธ๋“œ๋ณ„ ๋ฉ”ํŠธ๋ฆญ, ์žฅ์•  ์ „ํ™˜ ๊ฒ€์‚ฌ(2026-11-06 ๋กœ์ปฌ๊ณผ Managed ClickStack์—์„œ ๊ฒ€์ฆ) | | [labs/managed-rbac](labs/managed-rbac/) | Managed ClickStack ์ ‘๊ทผ ๊ถŒํ•œ์„ ๊ฒŒ์ž„ยท์‚ฌ์šฉ์žยท์—ด ๋‹จ์œ„๋กœ ์–ด๋””๊นŒ์ง€ ๋‚˜๋ˆŒ ์ˆ˜ ์žˆ๋‚˜: ClickStack ์—ญํ• (๋ฆฌ์†Œ์Šค, ์ด๋ฆ„ยทํƒœ๊ทธยทID, API์˜ beta ํ–‰ ์กฐ๊ฑด), ๊ฒ€์ƒ‰์ด ์“ฐ๋Š” ์‚ฌ์šฉ์ž๋ณ„ JWT ์‹ ์›๊ณผ ์•Œ๋ฆผ์ด ์“ฐ๋Š” ๊ณต์šฉ ์‹ ์› ํ•˜๋‚˜, ๊ฐ๊ฐ์— ๋Œ€ํ•œ rowยทmasking policy, ๊ทธ๋ฆฌ๊ณ  ์•Œ๋ฆผ๊ณผ Map ํ‚ค๋กœ ์ƒˆ๋Š” ์ง€์  (Admin๊ณผ ์‹œํ—˜ ์‚ฌ์šฉ์ž๋กœ Managed ClickStack์—์„œ ๊ฒ€์ฆ, 2026-11-07) | +| [labs/vector-route-dual-write](labs/vector-route-dual-write/) | ์ด๋ฏธ ๋ชจ๋“  Kubernetes ๋กœ๊ทธ๋ฅผ Loki๋กœ ๋ณด๋‚ด๋Š” Vector์—์„œ Kong gateway ๋‘˜๊ณผ API ํ•˜๋‚˜๋งŒ ClickHouse ์ „์šฉ ํ…Œ์ด๋ธ”๋กœ ๋ถ„๊ธฐํ•˜๊ณ , Loki๋Š” ๊ทธ๋Œ€๋กœ ๋‘๋ฉฐ, ClickHouse๊ฐ€ ๋‚ด๋ ค๊ฐ€ ์žˆ์„ ๋•Œ sink ๋ฒ„ํผ๋ณ„๋กœ Loki๊ฐ€ ์–ด๋–ป๊ฒŒ ๋˜๋Š”์ง€ ํ™•์ธ(`block`์€ source ์ „์ฒด๋ฅผ ๋ฉˆ์ถ”๊ณ , `drop_newest`๋Š” ClickHouse ํ–‰์„ ๋ฒ„๋ฆฌ๊ณ , ๋””์Šคํฌ ๋ฒ„ํผ๋Š” ๋‘˜ ๋‹ค ์ง€ํ‚ด). ์ž˜๋ชป๋œ ํ–‰ ํ•˜๋‚˜๊ฐ€ ๋ฐฐ์น˜ ์ „์ฒด๋ฅผ ๋ฒ„๋ฆฌ๊ฒŒ ํ•จ (๋กœ์ปฌ k3d, Vector 0.61.0, ClickStack 2.44.0์—์„œ ๊ฒ€์ฆ, 2026-11-13) | ### ๐ŸŽ“ ์›Œํฌ์ˆ (`workshops/`) diff --git a/labs/vector-route-dual-write/.gitignore b/labs/vector-route-dual-write/.gitignore new file mode 100644 index 0000000..b157018 --- /dev/null +++ b/labs/vector-route-dual-write/.gitignore @@ -0,0 +1 @@ +.ui-account diff --git a/labs/vector-route-dual-write/README.md b/labs/vector-route-dual-write/README.md new file mode 100644 index 0000000..7109407 --- /dev/null +++ b/labs/vector-route-dual-write/README.md @@ -0,0 +1,175 @@ +# Vector โ†’ ClickStack next to an existing Loki pipeline + +[English](#english) | [ํ•œ๊ตญ์–ด](#ํ•œ๊ตญ์–ด) + +## English + +Verified on Vector 0.61.0 (Helm chart 0.61.0), clickstack-all-in-one 2.44.0 (ClickHouse 26.8.36.3), Loki 3.7.9, Kong 3.9.4, on k3d (k3s v1.35.5), 2026-11-13. + +**The question.** Vector already ships every Kubernetes log to Loki. Can the same Vector also send *some* workloads to ClickHouse, into their own tables, while the Loki path stays as it is? What happens to Loki while ClickHouse is down? Here two Kong gateways (`kong-external`, `kong-internal`) and one API deployment are the workloads to route. A traffic generator stands in for everything else. + +### Layout + +``` + kubernetes_logs (k8s) โ”€โ”€ every pod on the node + โ”‚ โ”‚ + โ”‚ (existing) โ”‚ (added) + โ–ผ โ–ผ + loki_labels โ”€โ–ถ loki by_workload (route) + โ”œโ”€ kong_external โ”€โ” + โ”œโ”€ kong_internal โ”€โ”ดโ–ถ kong_rows (remap) โ”€โ–ถ ch_kong โ”€โ–ถ logs.kong_access + โ”œโ”€ api โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ถ api_rows (remap) โ”€โ–ถ ch_app โ”€โ–ถ logs.app_logs + โ””โ”€ _unmatched (no consumer: Loki only) +``` + +- [`vector/values.yaml`](vector/values.yaml): the whole Vector config. The Loki half stays as it was. The route, the two remaps and the two ClickHouse sinks are the additions. +- [`vector/mode-*.yaml`](vector/): the ClickHouse sinks' buffer. `block` is the default `when_full`; `drop` uses `drop_newest`; `disk` uses a 256 MiB disk buffer with `drop_newest`. The Loki sink keeps Vector's default buffer in every mode. +- [`sql/schema.sql`](sql/schema.sql): `logs.kong_access` (gateway fields as columns, `Gateway` = external/internal, `Cluster`) and `logs.app_logs` (JSON unpacked, a non-JSON line kept whole in `Body`). +- [`k8s/`](k8s/): Loki, ClickStack (one PVC for ClickHouse and the UI's MongoDB), two Kong Deployments with the same DB-less config told apart by the pod label `app`, and the API. +- [`bin/lab.py`](bin/lab.py): `up`, `mode`, `run ` (N requests to each gateway, tagged `run=`), `counts`, `outage`, `down`. + +### Run + +```bash +bin/lab.py up # k3d cluster, backends, workloads, schema, Vector (block) +bin/lab.py run R3 300 && sleep 20 && bin/lab.py counts R3 +bin/lab.py outage block B3 3000 # ClickStack scaled to 0, 3,000 requests, 2 min, back up +bin/lab.py outage drop D2 3000 +bin/lab.py outage disk K1 3000 +bin/lab.py down +``` + +The UI is at `kubectl --context k3d-vector-lab -n clickstack port-forward svc/clickstack 8080:8080`. Use port 8080: after login the UI redirects to `localhost:8080`. + +### Results (2026-11-13) + +**Routing and dual write** ([`results/routing-R3.txt`](results/routing-R3.txt)): 300 requests to each gateway. + +| | ClickHouse | Loki | +|---|---|---| +| kong-external | 300 (`Gateway = external`) | 300 | +| kong-internal | 300 (`Gateway = internal`) | 300 | +| API (one line per request) | 600 (12 `ERROR`, 588 `INFO`) | 600 | +| traffic generator (unmatched) | 0 in either table | 300 | + +Each gateway had 6 responses with status 500, as designed (every 50th request). + +**ClickHouse down for about 2 minutes**: 3,000 requests to each gateway, so 6,000 rows per ClickHouse sink. Logs: [`results/outage-*.txt`](results/). + +| ClickHouse sink buffer | Loki during the outage | ClickHouse after recovery | Discarded | +|---|---|---|---| +| memory 500, `block` (default) | **stopped** at 1,959 of 3,000 for 124 s. The unrelated generator's logs stopped too | 6,000 / 6,000 kong, 6,000 / 6,000 API, 31 s after restart | 0 | +| memory 500, `drop_newest` | complete (3,000 / 3,000) within 20 s | 2,816 / 6,000 kong, 2,500 / 6,000 API | 3,184 + 3,500, exactly the shortfall | +| disk 256 MiB, `drop_newest` | complete within 20 s | 6,000 / 6,000 kong, 6,000 / 6,000 API, 31 s after restart | 0 | + +What this shows: + +1. **`block` stops the whole source, not just the ClickHouse branch.** Loki got nothing new for as long as ClickHouse was down, including logs from pods that never go to ClickHouse. Nothing was lost in this run, because `kubernetes_logs` resumes from its file checkpoints. The cost is a Loki gap for the length of the outage. Log rotation during a long outage was not tested. +2. **`drop_newest` keeps Loki whole and sacrifices ClickHouse.** `vector_buffer_discarded_events_total` counted exactly what ClickHouse was missing. +3. **A disk buffer gives both** for as long as it lasts. The 6,000 gateway rows took 13.4 MB in the buffer (about 2.2 KB per row, raw line included), so 256 MiB holds about 120,000 such rows. That figure is computed, not measured: hold time = buffer size รท the routed rate. + +**One bad row drops the whole batch.** On the first run, Kong 3.9.4 logged `latencies.proxy = -3` for one request. The table had `UInt32`, so ClickHouse rejected the 1,000-row insert (`CANNOT_PARSE_NUMBER`, HTTP 400). Vector treats a 400 as not retriable and dropped all 1,000 rows (`vector_component_discarded_events_total{intentional="false"}`, [`results/outage-drop-D1-uint-schema.txt`](results/outage-drop-D1-uint-schema.txt)). The latency columns are now `Int32`. Watch that counter, not only the buffer counters. + +Other things seen on the way: + +- A new pod's log file can take up to `glob_minimum_cooldown_ms` (default 60,000 ms) to be picked up; the lab sets 5,000. +- Helm passes `customConfig` through `tpl`, so Vector's `{{ field }}` templates must be written `{{ "{{" }} field {{ "}}" }}`. +- `async_insert` is on by default in this ClickHouse build (`system.settings`: `async_insert = 1`, `changed = 0`). Vector's 1,000-row batches went through it. + +**ClickStack.** Both tables are ordinary log sources. `app_logs` (OTel-like column names) was inferred automatically. For `kong_access`, the timestamp, service (`Service`), body and log level (`if(Status >= 500, 'error', 'info')`) were set by hand. + +![Team settings: Kong access (logs.kong_access) and API app logs (logs.app_logs) next to the default sources](images/01-sources.webp) + +![Kong access search filtered to Gateway:internal Status:500, with Gateway, Service, Method, Path, Status and LatencyMs as columns](images/03-kong-internal-500.webp) + +![API app logs search filtered to SeverityText:ERROR](images/04-api-error.webp) + +### Not covered + +GCP Private Service Connect, ClickHouse Cloud regions and cross-region latency, a dedicated insert-only user (the lab uses the image's `api` user), and log rotation during long outages. + +## ํ•œ๊ตญ์–ด + +2026-11-13 k3d(k3s v1.35.5)์—์„œ Vector 0.61.0(Helm chart 0.61.0), clickstack-all-in-one 2.44.0(ClickHouse 26.8.36.3), Loki 3.7.9, Kong 3.9.4๋กœ ๊ฒ€์ฆํ–ˆ์Šต๋‹ˆ๋‹ค. + +**์งˆ๋ฌธ.** Vector๊ฐ€ ์ด๋ฏธ ๋ชจ๋“  Kubernetes ๋กœ๊ทธ๋ฅผ Loki๋กœ ๋ณด๋‚ด๊ณ  ์žˆ์Šต๋‹ˆ๋‹ค. ๊ฐ™์€ Vector๊ฐ€ Loki ๊ฒฝ๋กœ๋Š” ๊ทธ๋Œ€๋กœ ๋‘” ์ฑ„ *์ผ๋ถ€* ์›Œํฌ๋กœ๋“œ๋งŒ ClickHouse์˜ ์ „์šฉ ํ…Œ์ด๋ธ”๋กœ๋„ ๋ณด๋‚ผ ์ˆ˜ ์žˆ์„๊นŒ์š”? ClickHouse๊ฐ€ ๋‚ด๋ ค๊ฐ€ ์žˆ๋Š” ๋™์•ˆ Loki๋Š” ์–ด๋–ป๊ฒŒ ๋ ๊นŒ์š”? ์—ฌ๊ธฐ์„œ๋Š” Kong gateway ๋‘ ๊ฐœ(`kong-external`, `kong-internal`)์™€ API Deployment ํ•˜๋‚˜๋ฅผ ๋ถ„๊ธฐ ๋Œ€์ƒ์œผ๋กœ ๋‘ก๋‹ˆ๋‹ค. ๊ทธ ๋ฐ–์˜ ๋ชจ๋“  ์›Œํฌ๋กœ๋“œ๋Š” ํŠธ๋ž˜ํ”ฝ ์ƒ์„ฑ๊ธฐ๊ฐ€ ๋Œ€์‹ ํ•ฉ๋‹ˆ๋‹ค. + +### ๊ตฌ์„ฑ + +``` + kubernetes_logs (k8s) โ”€โ”€ ๋…ธ๋“œ์˜ ๋ชจ๋“  ํŒŒ๋“œ + โ”‚ โ”‚ + โ”‚ (๊ธฐ์กด) โ”‚ (์ถ”๊ฐ€) + โ–ผ โ–ผ + loki_labels โ”€โ–ถ loki by_workload (route) + โ”œโ”€ kong_external โ”€โ” + โ”œโ”€ kong_internal โ”€โ”ดโ–ถ kong_rows (remap) โ”€โ–ถ ch_kong โ”€โ–ถ logs.kong_access + โ”œโ”€ api โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ถ api_rows (remap) โ”€โ–ถ ch_app โ”€โ–ถ logs.app_logs + โ””โ”€ _unmatched (์†Œ๋น„์ž ์—†์Œ: Loki์—๋งŒ ๊ฐ) +``` + +- [`vector/values.yaml`](vector/values.yaml): Vector ์„ค์ • ์ „์ฒด. Loki ์ชฝ์€ ์›๋ž˜ ๊ทธ๋Œ€๋กœ์ž…๋‹ˆ๋‹ค. route, remap ๋‘˜, ClickHouse sink ๋‘˜์ด ์ถ”๊ฐ€ํ•œ ๋ถ€๋ถ„์ž…๋‹ˆ๋‹ค. +- [`vector/mode-*.yaml`](vector/): ClickHouse sink์˜ ๋ฒ„ํผ. `block`์€ ๊ธฐ๋ณธ `when_full`์ด๊ณ , `drop`์€ `drop_newest`, `disk`๋Š” 256 MiB ๋””์Šคํฌ ๋ฒ„ํผ์— `drop_newest`์ž…๋‹ˆ๋‹ค. Loki sink๋Š” ์–ด๋А ๋ชจ๋“œ์—์„œ๋‚˜ Vector ๊ธฐ๋ณธ ๋ฒ„ํผ๋ฅผ ์”๋‹ˆ๋‹ค. +- [`sql/schema.sql`](sql/schema.sql): `logs.kong_access`(gateway ํ•„๋“œ๋ฅผ ์ปฌ๋Ÿผ์œผ๋กœ, `Gateway` = external/internal, `Cluster`)์™€ `logs.app_logs`(JSON์„ ํŽผ์นจ, JSON์ด ์•„๋‹Œ ์ค„์€ `Body`์— ๊ทธ๋Œ€๋กœ). +- [`k8s/`](k8s/): Loki, ClickStack(ClickHouse์™€ UI์˜ MongoDB๊ฐ€ PVC ํ•˜๋‚˜๋ฅผ ์”€), ๊ฐ™์€ DB-less ์„ค์ •์„ ์“ฐ๊ณ  ํŒŒ๋“œ ๋ผ๋ฒจ `app`์œผ๋กœ๋งŒ ๊ตฌ๋ถ„๋˜๋Š” Kong Deployment ๋‘˜, API. +- [`bin/lab.py`](bin/lab.py): `up`, `mode`, `run `(gateway๋งˆ๋‹ค ์š”์ฒญ N๊ฐœ, `run=` ํ‘œ์‹œ), `counts`, `outage`, `down`. + +### ์‹คํ–‰ + +```bash +bin/lab.py up # k3d ํด๋Ÿฌ์Šคํ„ฐ, ๋ฐฑ์—”๋“œ, ์›Œํฌ๋กœ๋“œ, ์Šคํ‚ค๋งˆ, Vector(block) +bin/lab.py run R3 300 && sleep 20 && bin/lab.py counts R3 +bin/lab.py outage block B3 3000 # ClickStack์„ 0์œผ๋กœ ์ค„์ด๊ณ  ์š”์ฒญ 3,000๊ฐœ, 2๋ถ„ ๋’ค ๋‹ค์‹œ ์˜ฌ๋ฆผ +bin/lab.py outage drop D2 3000 +bin/lab.py outage disk K1 3000 +bin/lab.py down +``` + +UI๋Š” `kubectl --context k3d-vector-lab -n clickstack port-forward svc/clickstack 8080:8080`์œผ๋กœ ์—ฝ๋‹ˆ๋‹ค. ํฌํŠธ๋Š” 8080์ด์–ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ๋กœ๊ทธ์ธ ๋’ค UI๊ฐ€ `localhost:8080`์œผ๋กœ ๋ณด๋ƒ…๋‹ˆ๋‹ค. + +### ๊ฒฐ๊ณผ (2026-11-13) + +**๋ถ„๊ธฐ์™€ ์ด์ค‘ ์ ์žฌ** ([`results/routing-R3.txt`](results/routing-R3.txt)): gateway๋งˆ๋‹ค ์š”์ฒญ 300๊ฐœ. + +| | ClickHouse | Loki | +|---|---|---| +| kong-external | 300 (`Gateway = external`) | 300 | +| kong-internal | 300 (`Gateway = internal`) | 300 | +| API (์š”์ฒญ๋งˆ๋‹ค ํ•œ ์ค„) | 600 (`ERROR` 12, `INFO` 588) | 600 | +| ํŠธ๋ž˜ํ”ฝ ์ƒ์„ฑ๊ธฐ(๋ถ„๊ธฐ ์•ˆ ๋จ) | ๋‘ ํ…Œ์ด๋ธ” ๋ชจ๋‘ 0 | 300 | + +gateway๋งˆ๋‹ค ์ƒํƒœ 500 ์‘๋‹ต์ด 6๊ฐœ์”ฉ์ด์—ˆ์Šต๋‹ˆ๋‹ค. ์„ค๊ณ„๋Œ€๋กœ 50๋ฒˆ์งธ ์š”์ฒญ๋งˆ๋‹ค ํ•˜๋‚˜์ž…๋‹ˆ๋‹ค. + +**ClickHouse๋ฅผ ์•ฝ 2๋ถ„ ๋‚ด๋ฆผ**: gateway๋งˆ๋‹ค ์š”์ฒญ 3,000๊ฐœ, ์ฆ‰ ClickHouse sink๋งˆ๋‹ค 6,000ํ–‰. ๋กœ๊ทธ: [`results/outage-*.txt`](results/). + +| ClickHouse sink ๋ฒ„ํผ | ์žฅ์•  ์ค‘ Loki | ๋ณต๊ตฌ ๋’ค ClickHouse | ๋ฒ„๋ฆฐ ๊ฑด์ˆ˜ | +|---|---|---|---| +| memory 500, `block`(๊ธฐ๋ณธ๊ฐ’) | 3,000 ์ค‘ 1,959์—์„œ 124์ดˆ ๋™์•ˆ **๋ฉˆ์ถค**. ๊ด€๊ณ„์—†๋Š” ์ƒ์„ฑ๊ธฐ ๋กœ๊ทธ๋„ ํ•จ๊ป˜ ๋ฉˆ์ถค | ๋‹ค์‹œ ์˜ฌ๋ฆฐ ๋’ค 31์ดˆ ๋งŒ์— kong 6,000 / 6,000, API 6,000 / 6,000 | 0 | +| memory 500, `drop_newest` | 20์ดˆ ์•ˆ์— ์ „๋Ÿ‰(3,000 / 3,000) | kong 2,816 / 6,000, API 2,500 / 6,000 | 3,184 + 3,500, ๋ชจ์ž๋ž€ ๋งŒํผ๊ณผ ์ •ํ™•ํžˆ ๊ฐ™์Œ | +| disk 256 MiB, `drop_newest` | 20์ดˆ ์•ˆ์— ์ „๋Ÿ‰ | ๋‹ค์‹œ ์˜ฌ๋ฆฐ ๋’ค 31์ดˆ ๋งŒ์— kong 6,000 / 6,000, API 6,000 / 6,000 | 0 | + +์•Œ ์ˆ˜ ์žˆ๋Š” ๊ฒƒ: + +1. **`block`์€ ClickHouse ๋ถ„๊ธฐ๋งŒ์ด ์•„๋‹ˆ๋ผ source ์ „์ฒด๋ฅผ ๋ฉˆ์ถฅ๋‹ˆ๋‹ค.** ClickHouse๊ฐ€ ๋‚ด๋ ค๊ฐ€ ์žˆ๋Š” ๋™์•ˆ Loki์— ์ƒˆ ๋กœ๊ทธ๊ฐ€ ํ•˜๋‚˜๋„ ๋“ค์–ด๊ฐ€์ง€ ์•Š์•˜๊ณ , ClickHouse๋กœ ๊ฐ€์ง€ ์•Š๋Š” ํŒŒ๋“œ์˜ ๋กœ๊ทธ๋„ ๋งˆ์ฐฌ๊ฐ€์ง€์˜€์Šต๋‹ˆ๋‹ค. ์ด๋ฒˆ ์‹คํ–‰์—์„œ ์œ ์‹ค์€ ์—†์—ˆ์Šต๋‹ˆ๋‹ค. `kubernetes_logs`๊ฐ€ ํŒŒ์ผ ์ฒดํฌํฌ์ธํŠธ์—์„œ ์ด์–ด ์ฝ๊ธฐ ๋•Œ๋ฌธ์ž…๋‹ˆ๋‹ค. ๋Œ€๊ฐ€๋Š” ์žฅ์• ๊ฐ€ ์ด์–ด์ง„ ์‹œ๊ฐ„๋งŒํผ Loki์— ์ƒ๊ธฐ๋Š” ๊ณต๋ฐฑ์ž…๋‹ˆ๋‹ค. ์žฅ์• ๊ฐ€ ๊ธธ์–ด ๋กœ๊ทธ๊ฐ€ rotation๋˜๋Š” ๊ฒฝ์šฐ๋Š” ์‹œํ—˜ํ•˜์ง€ ์•Š์•˜์Šต๋‹ˆ๋‹ค. +2. **`drop_newest`๋Š” Loki๋ฅผ ์ง€ํ‚ค๊ณ  ClickHouse ์ชฝ์„ ๋ฒ„๋ฆฝ๋‹ˆ๋‹ค.** `vector_buffer_discarded_events_total`์ด ClickHouse์—์„œ ๋ชจ์ž๋ž€ ๊ฑด์ˆ˜๋ฅผ ์ •ํ™•ํžˆ ์…Œ์Šต๋‹ˆ๋‹ค. +3. **๋””์Šคํฌ ๋ฒ„ํผ๋Š” ๋ฒ„ํ‹ฐ๋Š” ๋™์•ˆ ๋‘˜ ๋‹ค ์ง€ํ‚ต๋‹ˆ๋‹ค.** gateway 6,000ํ–‰์ด ๋ฒ„ํผ์—์„œ 13.4 MB๋ฅผ ์ฐจ์ง€ํ–ˆ์Šต๋‹ˆ๋‹ค(์›๋ณธ ์ค„ ํฌํ•จ ํ–‰๋‹น ์•ฝ 2.2 KB). ๊ทธ๋Ÿฌ๋ฉด 256 MiB์—๋Š” ์ด๋Ÿฐ ํ–‰์ด ์•ฝ 12๋งŒ ๊ฐœ ๋“ค์–ด๊ฐ‘๋‹ˆ๋‹ค. ์ด ๊ฐ’์€ ์žฐ ๊ฒƒ์ด ์•„๋‹ˆ๋ผ ๊ณ„์‚ฐํ•œ ๊ฒƒ์ž…๋‹ˆ๋‹ค. ๋ฒ„ํ‹ฐ๋Š” ์‹œ๊ฐ„ = ๋ฒ„ํผ ํฌ๊ธฐ รท ๋ถ„๊ธฐ๋˜๋Š” ์ดˆ๋‹น ์–‘. + +**์ž˜๋ชป๋œ ํ–‰ ํ•˜๋‚˜๊ฐ€ ๋ฐฐ์น˜ ์ „์ฒด๋ฅผ ๋ฒ„๋ฆฌ๊ฒŒ ํ•ฉ๋‹ˆ๋‹ค.** ์ฒซ ์‹คํ–‰์—์„œ Kong 3.9.4๊ฐ€ ์š”์ฒญ ํ•˜๋‚˜์— `latencies.proxy = -3`์„ ๋‚จ๊ฒผ์Šต๋‹ˆ๋‹ค. ํ…Œ์ด๋ธ” ์ปฌ๋Ÿผ์ด `UInt32`๋ผ ClickHouse๊ฐ€ 1,000ํ–‰ insert๋ฅผ ๊ฑฐ๋ถ€ํ–ˆ์Šต๋‹ˆ๋‹ค(`CANNOT_PARSE_NUMBER`, HTTP 400). Vector๋Š” 400์„ ์žฌ์‹œ๋„ํ•˜์ง€ ์•Š๋Š” ์˜ค๋ฅ˜๋กœ ๋ณด๊ณ  1,000ํ–‰์„ ๋ชจ๋‘ ๋ฒ„๋ ธ์Šต๋‹ˆ๋‹ค(`vector_component_discarded_events_total{intentional="false"}`, [`results/outage-drop-D1-uint-schema.txt`](results/outage-drop-D1-uint-schema.txt)). ์ง€๊ธˆ์€ ์ง€์—ฐ ์ปฌ๋Ÿผ์„ `Int32`๋กœ ๋ฐ”๊ฟจ์Šต๋‹ˆ๋‹ค. ๋ฒ„ํผ ์ง€ํ‘œ๋งŒ ๋ณด์ง€ ๋ง๊ณ  ์ด ์นด์šดํ„ฐ๋„ ๋ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. + +๊ทธ ๋ฐ–์— ํ™•์ธํ•œ ๊ฒƒ: + +- ์ƒˆ ํŒŒ๋“œ์˜ ๋กœ๊ทธ ํŒŒ์ผ์€ `glob_minimum_cooldown_ms`(๊ธฐ๋ณธ 60,000 ms)๋งŒํผ ๋Šฆ๊ฒŒ ์ฝํž ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์‹ค์Šต์—์„œ๋Š” 5,000์œผ๋กœ ์ค„์˜€์Šต๋‹ˆ๋‹ค. +- Helm์€ `customConfig`๋ฅผ `tpl`๋กœ ์ฒ˜๋ฆฌํ•ฉ๋‹ˆ๋‹ค. ๊ทธ๋ž˜์„œ Vector์˜ `{{ field }}` ํ…œํ”Œ๋ฆฟ์„ `{{ "{{" }} field {{ "}}" }}`๋กœ ์จ์•ผ ํ•ฉ๋‹ˆ๋‹ค. +- ์ด ClickHouse ๋นŒ๋“œ๋Š” `async_insert`๊ฐ€ ๊ธฐ๋ณธ์œผ๋กœ ์ผœ์ ธ ์žˆ์Šต๋‹ˆ๋‹ค(`system.settings`: `async_insert = 1`, `changed = 0`). Vector์˜ 1,000ํ–‰ ๋ฐฐ์น˜๋„ ์ด ๊ฒฝ๋กœ๋กœ ๋“ค์–ด๊ฐ”์Šต๋‹ˆ๋‹ค. + +**ClickStack.** ๋‘ ํ…Œ์ด๋ธ” ๋ชจ๋‘ ์ผ๋ฐ˜ ๋กœ๊ทธ ์†Œ์Šค๋กœ ๋“ฑ๋ก๋ฉ๋‹ˆ๋‹ค. `app_logs`๋Š” OTel๊ณผ ๋น„์Šทํ•œ ์ปฌ๋Ÿผ ์ด๋ฆ„์ด๋ผ ์„ค์ •์ด ์ž๋™์œผ๋กœ ์ฑ„์›Œ์กŒ์Šต๋‹ˆ๋‹ค. `kong_access`๋Š” timestamp, service(`Service`), body, log level(`if(Status >= 500, 'error', 'info')`)์„ ์ง์ ‘ ๋„ฃ์—ˆ์Šต๋‹ˆ๋‹ค. + +![Team settings: ๊ธฐ๋ณธ ์†Œ์Šค ์˜†์˜ Kong access(logs.kong_access)์™€ API app logs(logs.app_logs)](images/01-sources.webp) + +![Gateway:internal Status:500์œผ๋กœ ๊ฑฐ๋ฅธ Kong access ๊ฒ€์ƒ‰. Gateway, Service, Method, Path, Status, LatencyMs๊ฐ€ ์ปฌ๋Ÿผ์œผ๋กœ ๋ณด์ธ๋‹ค](images/03-kong-internal-500.webp) + +![SeverityText:ERROR๋กœ ๊ฑฐ๋ฅธ API app logs ๊ฒ€์ƒ‰](images/04-api-error.webp) + +### ๋‹ค๋ฃจ์ง€ ์•Š์€ ๊ฒƒ + +GCP Private Service Connect, ClickHouse Cloud ๋ฆฌ์ „๊ณผ ๋ฆฌ์ „ ๊ฐ„ ์ง€์—ฐ, insert ์ „์šฉ ์‚ฌ์šฉ์ž(์‹ค์Šต์€ ์ด๋ฏธ์ง€์˜ `api` ์‚ฌ์šฉ์ž๋ฅผ ์”€), ์žฅ์• ๊ฐ€ ๊ธธ ๋•Œ์˜ ๋กœ๊ทธ rotation. diff --git a/labs/vector-route-dual-write/bin/lab.py b/labs/vector-route-dual-write/bin/lab.py new file mode 100755 index 0000000..6688126 --- /dev/null +++ b/labs/vector-route-dual-write/bin/lab.py @@ -0,0 +1,226 @@ +#!/usr/bin/env python3 +"""Driver for labs/vector-route-dual-write (local k3d only). + + lab.py up cluster, backends, workloads, schema, Vector (mode block) + lab.py mode re-deploy Vector with that ClickHouse sink buffer + lab.py run N requests to each gateway, tagged run=; waits + lab.py counts ClickHouse and Loki counts for one run + lab.py outage ClickStack down, run, sample Loki, ClickStack up, final counts + lab.py sql "" clickhouse client in the ClickStack pod + lab.py metrics Vector buffer / sent / discarded counters + lab.py down delete the cluster + +Needs docker, k3d, kubectl, helm. Writes nothing outside this lab and the +k3d cluster. +""" +import json +import pathlib +import subprocess +import sys +import time +import urllib.parse + +LAB = pathlib.Path(__file__).resolve().parent.parent +CLUSTER = "vector-lab" +CTX = f"k3d-{CLUSTER}" +CHART = "0.61.0" + + +def sh(*args, check=True, capture=False, stdin=None): + r = subprocess.run(args, check=check, text=True, input=stdin, + stdout=subprocess.PIPE if capture else None) + return r.stdout if capture else r.returncode + + +def kubectl(*args, **kw): + return sh("kubectl", "--context", CTX, *args, **kw) + + +def sql(query): + return kubectl("-n", "clickstack", "exec", "deploy/clickstack", "--", + "clickhouse", "client", "-q", query, capture=True).strip() + + +def up(): + have = sh("k3d", "cluster", "list", "-o", "json", capture=True) + if CLUSTER not in [c["name"] for c in json.loads(have)]: + sh("k3d", "cluster", "create", CLUSTER, "--wait") + kubectl("apply", "-f", str(LAB / "k8s/backends.yaml")) + kubectl("apply", "-f", str(LAB / "k8s/workloads.yaml")) + for ns, d in [("loki", "loki"), ("clickstack", "clickstack"), ("kong", "kong-external"), + ("kong", "kong-internal"), ("api", "api")]: + kubectl("-n", ns, "rollout", "status", f"deploy/{d}", "--timeout=300s") + wait_clickhouse() + kubectl("-n", "clickstack", "exec", "-i", "deploy/clickstack", "--", + "clickhouse", "client", "--multiquery", stdin=(LAB / "sql/schema.sql").read_text()) + mode("block") + + +def wait_clickhouse(timeout=300): + t0 = time.time() + while time.time() - t0 < timeout: + try: + if sql("SELECT 1") == "1": + return + except subprocess.CalledProcessError: + pass + time.sleep(3) + sys.exit("ClickHouse did not come up") + + +def mode(m): + sh("helm", "repo", "add", "vector", "https://helm.vector.dev", "--force-update", capture=True) + sh("helm", "--kube-context", CTX, "upgrade", "--install", "vector", "vector/vector", + "--version", CHART, "-n", "vector", "--create-namespace", + "-f", str(LAB / "vector/values.yaml"), "-f", str(LAB / f"vector/mode-{m}.yaml")) + kubectl("-n", "vector", "rollout", "status", "ds/vector", "--timeout=180s") + + +JOB = """apiVersion: batch/v1 +kind: Job +metadata: {{name: loadgen-{name}, namespace: loadgen}} +spec: + backoffLimit: 0 + template: + metadata: {{labels: {{app: loadgen}}}} + spec: + restartPolicy: Never + containers: + - name: loadgen + image: busybox:1.37 + command: ["sh", "-c"] + args: + - | + for i in $(seq 1 {n}); do + wget -q -O /dev/null "http://kong-external.kong:8000/orders?run={run}&i=$i" || true + wget -q -O /dev/null "http://kong-internal.kong:8000/users?run={run}&i=$i" || true + echo "loadgen run={run} i=$i sent" + done +""" + + +def run(run_id, n, wait=True): + kubectl("apply", "-f", "-", stdin=JOB.format(name=run_id.lower(), run=run_id, n=n), capture=True) + if wait: + kubectl("-n", "loadgen", "wait", f"job/loadgen-{run_id.lower()}", + "--for=condition=complete", "--timeout=600s") + + +def loki_count(app, needle): + q = f'sum(count_over_time({{cluster="lab-cluster-1", app="{app}"}} |= `{needle}` [3h]))' + path = ("/api/v1/namespaces/loki/services/loki:3100/proxy/loki/api/v1/query?" + + urllib.parse.urlencode({"query": q})) + out = json.loads(kubectl("get", "--raw", path, capture=True)) + res = out["data"]["result"] + return int(float(res[0]["value"][1])) if res else 0 + + +def counts(run_id, quiet=False): + c = { + "ch.kong_access.external": int(sql(f"SELECT count() FROM logs.kong_access WHERE Gateway='external' AND Path LIKE '%run={run_id}&%'")), + "ch.kong_access.internal": int(sql(f"SELECT count() FROM logs.kong_access WHERE Gateway='internal' AND Path LIKE '%run={run_id}&%'")), + "ch.app_logs.api": int(sql(f"SELECT count() FROM logs.app_logs WHERE LogAttributes['run']='{run_id}'")), + "ch.any_table.loadgen": int(sql( + f"SELECT (SELECT count() FROM logs.kong_access WHERE Body LIKE '%loadgen run={run_id} %')" + f" + (SELECT count() FROM logs.app_logs WHERE Body LIKE '%loadgen run={run_id} %')")), + "loki.kong-external": loki_count("kong-external", f"run={run_id}&"), + "loki.kong-internal": loki_count("kong-internal", f"run={run_id}&"), + "loki.api": loki_count("api", f'"run": "{run_id}"'), + "loki.loadgen": loki_count("loadgen", f"loadgen run={run_id} "), + } + if not quiet: + for k, v in c.items(): + print(f"{k:28} {v}") + return c + + +def metrics(): + pod = kubectl("-n", "vector", "get", "pod", "-l", "app.kubernetes.io/name=vector", + "-o", "jsonpath={.items[0].metadata.name}", capture=True) + text = kubectl("get", "--raw", f"/api/v1/namespaces/vector/pods/{pod}:9598/proxy/metrics", capture=True) + want = ("vector_buffer_size_events", "vector_buffer_discarded_events_total", + "vector_component_sent_events_total", "vector_buffer_size_bytes") + out = {} + for line in text.splitlines(): + if line.startswith(want) and any(f'component_id="{c}"' in line for c in ("ch_kong", "ch_app", "loki")): + name, val = line.rsplit("} ", 1)[0] + "}", line.rsplit("} ", 1)[1].split()[0] + out[name] = float(val) + return out + + +def outage(m, run_id, n): + mode(m) + time.sleep(10) + log = [] + def note(msg): + line = f"{time.strftime('%H:%M:%S')} {msg}" + print(line, flush=True) + log.append(line) + note(f"mode={m} run={run_id} n={n}: scale ClickStack to 0") + kubectl("-n", "clickstack", "scale", "deploy/clickstack", "--replicas=0") + kubectl("-n", "clickstack", "wait", "pod", "-l", "app=clickstack", "--for=delete", "--timeout=120s") + t0 = time.time() + run(run_id, n, wait=False) + note("loadgen started") + while time.time() - t0 < 120: + time.sleep(10) + lk = loki_count("kong-external", f"run={run_id}&") + lg = loki_count("loadgen", f"loadgen run={run_id} ") + mt = metrics() + buf = {k.split("{")[0].replace("vector_", "") + ":" + k.split('component_id="')[1].split('"')[0]: v + for k, v in mt.items() if "buffer" in k and ("ch_kong" in k)} + note(f"t+{int(time.time()-t0):3}s loki kong-external={lk}/{n} loadgen={lg}/{n} {buf}") + note("scale ClickStack to 1") + kubectl("-n", "clickstack", "scale", "deploy/clickstack", "--replicas=1") + kubectl("-n", "clickstack", "rollout", "status", "deploy/clickstack", "--timeout=300s") + wait_clickhouse() + t1 = time.time() + seen, same = None, 0 + want = (n, n, 2 * n) + while time.time() - t1 < 300: # until complete, or unchanged for 60 s + time.sleep(15) + c = counts(run_id, quiet=True) + key = (c["ch.kong_access.external"], c["ch.kong_access.internal"], c["ch.app_logs.api"]) + note(f"t+{int(time.time()-t1):3}s after recovery ch(external, internal, api)={key} loki kong-external={c['loki.kong-external']}") + same = same + 1 if key == seen else 0 + seen = key + if key == want or same >= 4: + break + final = counts(run_id) + mt = metrics() + disc = {k: v for k, v in mt.items() if "discarded" in k} + note(f"final {json.dumps(final)}") + note(f"discarded {json.dumps(disc)}") + out = LAB / "results" / f"outage-{m}-{run_id}.txt" + out.write_text("\n".join(log) + "\n") + print(f"written {out.relative_to(LAB)}") + return log + + +def main(argv): + if not argv: + sys.exit(__doc__) + cmd, args = argv[0], argv[1:] + if cmd == "up": + up() + elif cmd == "mode": + mode(args[0]) + elif cmd == "run": + run(args[0], int(args[1])) + elif cmd == "counts": + counts(args[0]) + elif cmd == "outage": + outage(args[0], args[1], int(args[2])) + elif cmd == "sql": + print(sql(args[0])) + elif cmd == "metrics": + for k, v in sorted(metrics().items()): + print(f"{v:>12.0f} {k}") + elif cmd == "down": + sh("k3d", "cluster", "delete", CLUSTER) + else: + sys.exit(__doc__) + + +if __name__ == "__main__": + main(sys.argv[1:]) diff --git a/labs/vector-route-dual-write/images/01-sources.webp b/labs/vector-route-dual-write/images/01-sources.webp new file mode 100644 index 0000000..a95cb3c Binary files /dev/null and b/labs/vector-route-dual-write/images/01-sources.webp differ diff --git a/labs/vector-route-dual-write/images/03-kong-internal-500.webp b/labs/vector-route-dual-write/images/03-kong-internal-500.webp new file mode 100644 index 0000000..a033177 Binary files /dev/null and b/labs/vector-route-dual-write/images/03-kong-internal-500.webp differ diff --git a/labs/vector-route-dual-write/images/04-api-error.webp b/labs/vector-route-dual-write/images/04-api-error.webp new file mode 100644 index 0000000..0d09f14 Binary files /dev/null and b/labs/vector-route-dual-write/images/04-api-error.webp differ diff --git a/labs/vector-route-dual-write/k8s/backends.yaml b/labs/vector-route-dual-write/k8s/backends.yaml new file mode 100644 index 0000000..ea89c47 --- /dev/null +++ b/labs/vector-route-dual-write/k8s/backends.yaml @@ -0,0 +1,122 @@ +# The two destinations Vector writes to: Loki (the pipeline that already +# exists) and ClickStack (the one being added). Lab only: no auth, local +# storage, one replica each. +apiVersion: v1 +kind: Namespace +metadata: {name: loki} +--- +apiVersion: v1 +kind: ConfigMap +metadata: {name: loki-config, namespace: loki} +data: + config.yaml: | + auth_enabled: false + server: + http_listen_port: 3100 + grpc_listen_port: 9096 + common: + instance_addr: 127.0.0.1 + path_prefix: /loki + replication_factor: 1 + ring: + kvstore: + store: inmemory + storage: + filesystem: + chunks_directory: /loki/chunks + rules_directory: /loki/rules + schema_config: + configs: + - from: 2024-01-01 + store: tsdb + object_store: filesystem + schema: v13 + index: + prefix: index_ + period: 24h + limits_config: + reject_old_samples: false + ingestion_rate_mb: 64 + ingestion_burst_size_mb: 128 +--- +apiVersion: apps/v1 +kind: Deployment +metadata: {name: loki, namespace: loki} +spec: + replicas: 1 + selector: {matchLabels: {app: loki}} + template: + metadata: {labels: {app: loki}} + spec: + containers: + - name: loki + image: grafana/loki:3.7.9 + args: ["-config.file=/etc/loki/config.yaml"] + ports: [{containerPort: 3100}] + readinessProbe: + httpGet: {path: /ready, port: 3100} + periodSeconds: 5 + volumeMounts: + - {name: config, mountPath: /etc/loki} + - {name: data, mountPath: /loki} + volumes: + - name: config + configMap: {name: loki-config} + - name: data + emptyDir: {} +--- +apiVersion: v1 +kind: Service +metadata: {name: loki, namespace: loki} +spec: + selector: {app: loki} + ports: [{name: http, port: 3100, targetPort: 3100}] +--- +apiVersion: v1 +kind: Namespace +metadata: {name: clickstack} +--- +# One volume for ClickHouse (/var/lib/clickhouse) and the UI's own store +# (MongoDB in /data/db), so scaling the Deployment to zero (the outage in `lab.py +# outage`) keeps tables and sources. +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: {name: clickstack-data, namespace: clickstack} +spec: + accessModes: [ReadWriteOnce] + resources: {requests: {storage: 5Gi}} +--- +apiVersion: apps/v1 +kind: Deployment +metadata: {name: clickstack, namespace: clickstack} +spec: + replicas: 1 + strategy: {type: Recreate} + selector: {matchLabels: {app: clickstack}} + template: + metadata: {labels: {app: clickstack}} + spec: + containers: + - name: clickstack + image: clickhouse/clickstack-all-in-one:2.44.0 + ports: + - {name: ui, containerPort: 8080} + - {name: ch-http, containerPort: 8123} + readinessProbe: + httpGet: {path: /ping, port: 8123} + periodSeconds: 5 + volumeMounts: + - {name: data, mountPath: /data/db, subPath: mongo} + - {name: data, mountPath: /var/lib/clickhouse, subPath: clickhouse} + volumes: + - name: data + persistentVolumeClaim: {claimName: clickstack-data} +--- +apiVersion: v1 +kind: Service +metadata: {name: clickstack, namespace: clickstack} +spec: + selector: {app: clickstack} + ports: + - {name: ch-http, port: 8123, targetPort: 8123} + - {name: ui, port: 8080, targetPort: 8080} diff --git a/labs/vector-route-dual-write/k8s/workloads.yaml b/labs/vector-route-dual-write/k8s/workloads.yaml new file mode 100644 index 0000000..7c78af3 --- /dev/null +++ b/labs/vector-route-dual-write/k8s/workloads.yaml @@ -0,0 +1,181 @@ +# The workloads whose logs are routed: two Kong gateways (external and +# internal, same DB-less config, told apart only by the pod label `app`) +# in front of one API deployment. The two Deployments differ only in name +# and label. Kong writes one JSON line per request +# (file-log plugin to stdout); its own nginx access log is off so a request +# is logged once. The API writes one JSON line per request. +apiVersion: v1 +kind: Namespace +metadata: {name: kong} +--- +apiVersion: v1 +kind: ConfigMap +metadata: {name: kong-declarative, namespace: kong} +data: + kong.yml: | + _format_version: "3.0" + services: + - name: orders + url: http://api.api.svc:8080/orders + routes: + - name: orders + paths: [/orders] + - name: users + url: http://api.api.svc:8080/users + routes: + - name: users + paths: [/users] + plugins: + - name: file-log + config: + path: /dev/stdout +--- +apiVersion: apps/v1 +kind: Deployment +metadata: {name: kong-external, namespace: kong} +spec: + replicas: 1 + selector: {matchLabels: {app: kong-external}} + template: + metadata: {labels: {app: kong-external}} + spec: + containers: + - name: kong + image: kong:3.9.4 + env: + - {name: KONG_DATABASE, value: "off"} + - {name: KONG_DECLARATIVE_CONFIG, value: /kong/kong.yml} + - {name: KONG_PROXY_LISTEN, value: "0.0.0.0:8000"} + - {name: KONG_ADMIN_LISTEN, value: "off"} + - {name: KONG_PROXY_ACCESS_LOG, value: "off"} + - {name: KONG_PROXY_ERROR_LOG, value: /dev/stderr} + - {name: KONG_NGINX_WORKER_PROCESSES, value: "1"} + ports: [{containerPort: 8000}] + readinessProbe: + tcpSocket: {port: 8000} + periodSeconds: 5 + volumeMounts: + - {name: declarative, mountPath: /kong} + volumes: + - name: declarative + configMap: {name: kong-declarative} +--- +apiVersion: apps/v1 +kind: Deployment +metadata: {name: kong-internal, namespace: kong} +spec: + replicas: 1 + selector: {matchLabels: {app: kong-internal}} + template: + metadata: {labels: {app: kong-internal}} + spec: + containers: + - name: kong + image: kong:3.9.4 + env: + - {name: KONG_DATABASE, value: "off"} + - {name: KONG_DECLARATIVE_CONFIG, value: /kong/kong.yml} + - {name: KONG_PROXY_LISTEN, value: "0.0.0.0:8000"} + - {name: KONG_ADMIN_LISTEN, value: "off"} + - {name: KONG_PROXY_ACCESS_LOG, value: "off"} + - {name: KONG_PROXY_ERROR_LOG, value: /dev/stderr} + - {name: KONG_NGINX_WORKER_PROCESSES, value: "1"} + ports: [{containerPort: 8000}] + readinessProbe: + tcpSocket: {port: 8000} + periodSeconds: 5 + volumeMounts: + - {name: declarative, mountPath: /kong} + volumes: + - name: declarative + configMap: {name: kong-declarative} +--- +apiVersion: v1 +kind: Service +metadata: {name: kong-external, namespace: kong} +spec: + selector: {app: kong-external} + ports: [{port: 8000, targetPort: 8000}] +--- +apiVersion: v1 +kind: Service +metadata: {name: kong-internal, namespace: kong} +spec: + selector: {app: kong-internal} + ports: [{port: 8000, targetPort: 8000}] +--- +apiVersion: v1 +kind: Namespace +metadata: {name: api} +--- +apiVersion: v1 +kind: ConfigMap +metadata: {name: api-app, namespace: api} +data: + app.py: | + # One JSON log line per request. Every 50th request (i % 50 == 0) + # answers 500 and logs ERROR, so severity is not constant. + import json, sys, time + from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + from urllib.parse import urlparse, parse_qs + + class H(BaseHTTPRequestHandler): + def do_GET(self): + u = urlparse(self.path) + q = parse_qs(u.query) + i = int(q.get("i", ["0"])[0]) + status = 500 if i and i % 50 == 0 else 200 + body = json.dumps({"path": u.path, "ok": status == 200}).encode() + self.send_response(status) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + print(json.dumps({ + "ts": time.time(), "level": "ERROR" if status == 500 else "INFO", + "msg": f"{u.path.strip('/')} request handled", + "method": "GET", "path": u.path, "status": status, + "run": q.get("run", [""])[0], "i": i, + }), flush=True) + + def log_message(self, *a): + pass + + print("api listening on 8080", flush=True) # not JSON: kept as a plain body + ThreadingHTTPServer(("0.0.0.0", 8080), H).serve_forever() +--- +apiVersion: apps/v1 +kind: Deployment +metadata: {name: api, namespace: api} +spec: + replicas: 1 + selector: {matchLabels: {app: api}} + template: + metadata: {labels: {app: api}} + spec: + containers: + - name: api + image: python:3.12-alpine + command: ["python", "-u", "/app/app.py"] + ports: [{containerPort: 8080}] + readinessProbe: + tcpSocket: {port: 8080} + periodSeconds: 5 + volumeMounts: + - {name: app, mountPath: /app} + volumes: + - name: app + configMap: {name: api-app} +--- +apiVersion: v1 +kind: Service +metadata: {name: api, namespace: api} +spec: + selector: {app: api} + ports: [{port: 8080, targetPort: 8080}] +--- +# Traffic comes from Jobs in this namespace (`lab.py run`). Their own log +# lines name the run too, and must reach Loki only. +apiVersion: v1 +kind: Namespace +metadata: {name: loadgen} diff --git a/labs/vector-route-dual-write/lab.yaml b/labs/vector-route-dual-write/lab.yaml new file mode 100644 index 0000000..26259cd --- /dev/null +++ b/labs/vector-route-dual-write/lab.yaml @@ -0,0 +1,5 @@ +# Runner keys (tools/lab.schema.md). Not run by hol: needs a local k3d cluster (bin/lab.py). +target: oss +tier: T1 +services: [clickstack, loki, vector, kong] +verified_on: "k3d (k3s v1.35.5), Vector 0.61.0 (chart 0.61.0), clickstack-all-in-one 2.44.0 (ClickHouse 26.8.36.3), Loki 3.7.9, Kong 3.9.4, 2026-11-13" diff --git a/labs/vector-route-dual-write/results/outage-block-B3.txt b/labs/vector-route-dual-write/results/outage-block-B3.txt new file mode 100644 index 0000000..800644c --- /dev/null +++ b/labs/vector-route-dual-write/results/outage-block-B3.txt @@ -0,0 +1,19 @@ +16:04:05 mode=block run=B3 n=3000: scale ClickStack to 0 +16:04:36 loadgen started +16:04:46 t+ 10s loki kong-external=1959/3000 loadgen=0/3000 {'buffer_size_bytes:ch_kong': 1515214.0, 'buffer_size_events:ch_kong': 500.0} +16:04:57 t+ 20s loki kong-external=1959/3000 loadgen=0/3000 {'buffer_size_bytes:ch_kong': 1515214.0, 'buffer_size_events:ch_kong': 500.0} +16:05:07 t+ 31s loki kong-external=1959/3000 loadgen=0/3000 {'buffer_size_bytes:ch_kong': 1515214.0, 'buffer_size_events:ch_kong': 500.0} +16:05:17 t+ 41s loki kong-external=1959/3000 loadgen=0/3000 {'buffer_size_bytes:ch_kong': 1515214.0, 'buffer_size_events:ch_kong': 500.0} +16:05:28 t+ 51s loki kong-external=1959/3000 loadgen=0/3000 {'buffer_size_bytes:ch_kong': 1515214.0, 'buffer_size_events:ch_kong': 500.0} +16:05:38 t+ 61s loki kong-external=1959/3000 loadgen=0/3000 {'buffer_size_bytes:ch_kong': 1515214.0, 'buffer_size_events:ch_kong': 500.0} +16:05:48 t+ 72s loki kong-external=1959/3000 loadgen=0/3000 {'buffer_size_bytes:ch_kong': 1515214.0, 'buffer_size_events:ch_kong': 500.0} +16:05:59 t+ 82s loki kong-external=1959/3000 loadgen=0/3000 {'buffer_size_bytes:ch_kong': 1515214.0, 'buffer_size_events:ch_kong': 500.0} +16:06:09 t+ 93s loki kong-external=1959/3000 loadgen=0/3000 {'buffer_size_bytes:ch_kong': 1515214.0, 'buffer_size_events:ch_kong': 500.0} +16:06:20 t+103s loki kong-external=1959/3000 loadgen=0/3000 {'buffer_size_bytes:ch_kong': 1515214.0, 'buffer_size_events:ch_kong': 500.0} +16:06:30 t+113s loki kong-external=1959/3000 loadgen=0/3000 {'buffer_size_bytes:ch_kong': 1515214.0, 'buffer_size_events:ch_kong': 500.0} +16:06:40 t+124s loki kong-external=1959/3000 loadgen=0/3000 {'buffer_size_bytes:ch_kong': 1515214.0, 'buffer_size_events:ch_kong': 500.0} +16:06:40 scale ClickStack to 1 +16:07:00 t+ 15s after recovery ch(external, internal, api)=(1995, 1995, 0) loki kong-external=2140 +16:07:16 t+ 31s after recovery ch(external, internal, api)=(3000, 3000, 6000) loki kong-external=3000 +16:07:17 final {"ch.kong_access.external": 3000, "ch.kong_access.internal": 3000, "ch.app_logs.api": 6000, "ch.any_table.loadgen": 0, "loki.kong-external": 3000, "loki.kong-internal": 3000, "loki.api": 6000, "loki.loadgen": 3000} +16:07:17 discarded {} diff --git a/labs/vector-route-dual-write/results/outage-disk-K1.txt b/labs/vector-route-dual-write/results/outage-disk-K1.txt new file mode 100644 index 0000000..eb26b5a --- /dev/null +++ b/labs/vector-route-dual-write/results/outage-disk-K1.txt @@ -0,0 +1,19 @@ +16:12:00 mode=disk run=K1 n=3000: scale ClickStack to 0 +16:12:31 loadgen started +16:12:41 t+ 10s loki kong-external=2866/3000 loadgen=3000/3000 {'buffer_size_bytes:ch_kong': 11502040.0, 'buffer_size_events:ch_kong': 5163.0} +16:12:52 t+ 20s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_size_bytes:ch_kong': 13378560.0, 'buffer_size_events:ch_kong': 6000.0} +16:13:02 t+ 31s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_size_bytes:ch_kong': 13378560.0, 'buffer_size_events:ch_kong': 6000.0} +16:13:13 t+ 41s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_size_bytes:ch_kong': 13378560.0, 'buffer_size_events:ch_kong': 6000.0} +16:13:23 t+ 52s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_size_bytes:ch_kong': 13378560.0, 'buffer_size_events:ch_kong': 6000.0} +16:13:33 t+ 62s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_size_bytes:ch_kong': 13378560.0, 'buffer_size_events:ch_kong': 6000.0} +16:13:44 t+ 72s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_size_bytes:ch_kong': 13378560.0, 'buffer_size_events:ch_kong': 6000.0} +16:13:54 t+ 82s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_size_bytes:ch_kong': 13378560.0, 'buffer_size_events:ch_kong': 6000.0} +16:14:04 t+ 93s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_size_bytes:ch_kong': 13378560.0, 'buffer_size_events:ch_kong': 6000.0} +16:14:15 t+103s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_size_bytes:ch_kong': 13378560.0, 'buffer_size_events:ch_kong': 6000.0} +16:14:25 t+113s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_size_bytes:ch_kong': 13378560.0, 'buffer_size_events:ch_kong': 6000.0} +16:14:35 t+124s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_size_bytes:ch_kong': 13378560.0, 'buffer_size_events:ch_kong': 6000.0} +16:14:35 scale ClickStack to 1 +16:14:54 t+ 15s after recovery ch(external, internal, api)=(0, 0, 6000) loki kong-external=3000 +16:15:10 t+ 31s after recovery ch(external, internal, api)=(3000, 3000, 6000) loki kong-external=3000 +16:15:10 final {"ch.kong_access.external": 3000, "ch.kong_access.internal": 3000, "ch.app_logs.api": 6000, "ch.any_table.loadgen": 0, "loki.kong-external": 3000, "loki.kong-internal": 3000, "loki.api": 6000, "loki.loadgen": 3000} +16:15:10 discarded {} diff --git a/labs/vector-route-dual-write/results/outage-drop-D1-uint-schema.txt b/labs/vector-route-dual-write/results/outage-drop-D1-uint-schema.txt new file mode 100644 index 0000000..217a345 --- /dev/null +++ b/labs/vector-route-dual-write/results/outage-drop-D1-uint-schema.txt @@ -0,0 +1,32 @@ +15:57:48 mode=drop run=D1 n=3000: scale ClickStack to 0 +15:58:20 loadgen started +15:58:30 t+ 10s loki kong-external=2336/3000 loadgen=2336/3000 {'buffer_discarded_events_total:ch_kong': 1166.0, 'buffer_size_bytes:ch_kong': 1525203.0, 'buffer_size_events:ch_kong': 500.0} +15:58:40 t+ 20s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_discarded_events_total:ch_kong': 3395.0, 'buffer_size_bytes:ch_kong': 1525203.0, 'buffer_size_events:ch_kong': 500.0} +15:58:51 t+ 31s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_discarded_events_total:ch_kong': 3395.0, 'buffer_size_bytes:ch_kong': 1525203.0, 'buffer_size_events:ch_kong': 500.0} +15:59:01 t+ 41s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_discarded_events_total:ch_kong': 3395.0, 'buffer_size_bytes:ch_kong': 1525203.0, 'buffer_size_events:ch_kong': 500.0} +15:59:12 t+ 52s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_discarded_events_total:ch_kong': 3395.0, 'buffer_size_bytes:ch_kong': 1525203.0, 'buffer_size_events:ch_kong': 500.0} +15:59:22 t+ 62s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_discarded_events_total:ch_kong': 3395.0, 'buffer_size_bytes:ch_kong': 1525203.0, 'buffer_size_events:ch_kong': 500.0} +15:59:32 t+ 72s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_discarded_events_total:ch_kong': 3395.0, 'buffer_size_bytes:ch_kong': 1525203.0, 'buffer_size_events:ch_kong': 500.0} +15:59:43 t+ 83s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_discarded_events_total:ch_kong': 3395.0, 'buffer_size_bytes:ch_kong': 1525203.0, 'buffer_size_events:ch_kong': 500.0} +15:59:53 t+ 93s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_discarded_events_total:ch_kong': 3395.0, 'buffer_size_bytes:ch_kong': 1525203.0, 'buffer_size_events:ch_kong': 500.0} +16:00:03 t+103s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_discarded_events_total:ch_kong': 3395.0, 'buffer_size_bytes:ch_kong': 1525203.0, 'buffer_size_events:ch_kong': 500.0} +16:00:14 t+114s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_discarded_events_total:ch_kong': 3395.0, 'buffer_size_bytes:ch_kong': 1525203.0, 'buffer_size_events:ch_kong': 500.0} +16:00:24 t+124s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_discarded_events_total:ch_kong': 3395.0, 'buffer_size_bytes:ch_kong': 1525203.0, 'buffer_size_events:ch_kong': 500.0} +16:00:24 scale ClickStack to 1 +16:00:44 t+ 15s after recovery ch(external, internal, api)=(1302, 303, 2501) loki kong-external=3000 +16:01:00 t+ 31s after recovery ch(external, internal, api)=(1302, 303, 2501) loki kong-external=3000 +16:01:16 t+ 47s after recovery ch(external, internal, api)=(1302, 303, 2501) loki kong-external=3000 +16:01:31 t+ 63s after recovery ch(external, internal, api)=(1302, 303, 2501) loki kong-external=3000 +16:01:48 t+ 79s after recovery ch(external, internal, api)=(1302, 303, 2501) loki kong-external=3000 +16:01:49 final {"ch.kong_access.external": 1302, "ch.kong_access.internal": 303, "ch.app_logs.api": 2501, "ch.any_table.loadgen": 0, "loki.kong-external": 3000, "loki.kong-internal": 3000, "loki.api": 6000, "loki.loadgen": 3000} +16:01:49 discarded {"vector_buffer_discarded_events_total{buffer_id=\"ch_kong\",buffer_type=\"memory\",component_id=\"ch_kong\",component_kind=\"sink\",component_type=\"clickhouse\",host=\"vector-bj4t7\",intentional=\"true\",stage=\"0\"}": 3395.0, "vector_buffer_discarded_events_total{buffer_id=\"ch_app\",buffer_type=\"memory\",component_id=\"ch_app\",component_kind=\"sink\",component_type=\"clickhouse\",host=\"vector-bj4t7\",intentional=\"true\",stage=\"0\"}": 3499.0} + +# First drop run, while kong_access still had UInt32 latency columns. +# kong: 1302 + 303 = 1605 rows arrived, 3395 discarded by the buffer, 1000 missing besides. +# Vector metrics after the run (pod vector-bj4t7): +# vector_component_discarded_events_total{component_id="ch_kong",intentional="false"} 1000 +# vector_component_errors_total{component_id="ch_kong",error_type="request_failed",stage="sending"} 1 +# Vector log, 07:00:37Z: "Not retriable; dropping the request. reason=\"Default retry strategy: Bad Request\"" ... "Events dropped intentional=false count=1000" +# ClickHouse system.query_log, 07:00:37, user api, ExceptionWhileProcessing: +# Code: 72. DB::Exception: Unsigned type must not contain '-' symbol: (while reading the value of key UpstreamLatencyMs): (at row 137) +# Loki: one Kong line with latencies {"request": 0, "kong": 0, "proxy": -3, "receive": 0}, kong-internal, /users?run=D1&i=137 diff --git a/labs/vector-route-dual-write/results/outage-drop-D2.txt b/labs/vector-route-dual-write/results/outage-drop-D2.txt new file mode 100644 index 0000000..6c5ae49 --- /dev/null +++ b/labs/vector-route-dual-write/results/outage-drop-D2.txt @@ -0,0 +1,23 @@ +16:07:30 mode=drop run=D2 n=3000: scale ClickStack to 0 +16:08:01 loadgen started +16:08:11 t+ 10s loki kong-external=2866/3000 loadgen=2865/3000 {'buffer_discarded_events_total:ch_kong': 3114.0, 'buffer_size_bytes:ch_kong': 1524783.0, 'buffer_size_events:ch_kong': 500.0} +16:08:22 t+ 20s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_discarded_events_total:ch_kong': 3184.0, 'buffer_size_bytes:ch_kong': 1524783.0, 'buffer_size_events:ch_kong': 500.0} +16:08:32 t+ 30s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_discarded_events_total:ch_kong': 3184.0, 'buffer_size_bytes:ch_kong': 1524783.0, 'buffer_size_events:ch_kong': 500.0} +16:08:42 t+ 41s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_discarded_events_total:ch_kong': 3184.0, 'buffer_size_bytes:ch_kong': 1524783.0, 'buffer_size_events:ch_kong': 500.0} +16:08:52 t+ 51s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_discarded_events_total:ch_kong': 3184.0, 'buffer_size_bytes:ch_kong': 1524783.0, 'buffer_size_events:ch_kong': 500.0} +16:09:03 t+ 61s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_discarded_events_total:ch_kong': 3184.0, 'buffer_size_bytes:ch_kong': 1524783.0, 'buffer_size_events:ch_kong': 500.0} +16:09:13 t+ 72s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_discarded_events_total:ch_kong': 3184.0, 'buffer_size_bytes:ch_kong': 1524783.0, 'buffer_size_events:ch_kong': 500.0} +16:09:23 t+ 82s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_discarded_events_total:ch_kong': 3184.0, 'buffer_size_bytes:ch_kong': 1524783.0, 'buffer_size_events:ch_kong': 500.0} +16:09:34 t+ 92s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_discarded_events_total:ch_kong': 3184.0, 'buffer_size_bytes:ch_kong': 1524783.0, 'buffer_size_events:ch_kong': 500.0} +16:09:44 t+103s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_discarded_events_total:ch_kong': 3184.0, 'buffer_size_bytes:ch_kong': 1524783.0, 'buffer_size_events:ch_kong': 500.0} +16:09:55 t+113s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_discarded_events_total:ch_kong': 3184.0, 'buffer_size_bytes:ch_kong': 1524783.0, 'buffer_size_events:ch_kong': 500.0} +16:10:05 t+124s loki kong-external=3000/3000 loadgen=3000/3000 {'buffer_discarded_events_total:ch_kong': 3184.0, 'buffer_size_bytes:ch_kong': 1524783.0, 'buffer_size_events:ch_kong': 500.0} +16:10:05 scale ClickStack to 1 +16:10:25 t+ 16s after recovery ch(external, internal, api)=(0, 0, 2500) loki kong-external=3000 +16:10:41 t+ 32s after recovery ch(external, internal, api)=(1408, 1408, 2500) loki kong-external=3000 +16:10:58 t+ 48s after recovery ch(external, internal, api)=(1408, 1408, 2500) loki kong-external=3000 +16:11:14 t+ 64s after recovery ch(external, internal, api)=(1408, 1408, 2500) loki kong-external=3000 +16:11:31 t+ 81s after recovery ch(external, internal, api)=(1408, 1408, 2500) loki kong-external=3000 +16:11:47 t+ 97s after recovery ch(external, internal, api)=(1408, 1408, 2500) loki kong-external=3000 +16:11:48 final {"ch.kong_access.external": 1408, "ch.kong_access.internal": 1408, "ch.app_logs.api": 2500, "ch.any_table.loadgen": 0, "loki.kong-external": 3000, "loki.kong-internal": 3000, "loki.api": 6000, "loki.loadgen": 3000} +16:11:48 discarded {"vector_buffer_discarded_events_total{buffer_id=\"ch_app\",buffer_type=\"memory\",component_id=\"ch_app\",component_kind=\"sink\",component_type=\"clickhouse\",host=\"vector-n4ddk\",intentional=\"true\",stage=\"0\"}": 3500.0, "vector_buffer_discarded_events_total{buffer_id=\"ch_kong\",buffer_type=\"memory\",component_id=\"ch_kong\",component_kind=\"sink\",component_type=\"clickhouse\",host=\"vector-n4ddk\",intentional=\"true\",stage=\"0\"}": 3184.0} diff --git a/labs/vector-route-dual-write/results/routing-R3.txt b/labs/vector-route-dual-write/results/routing-R3.txt new file mode 100644 index 0000000..12d7f7c --- /dev/null +++ b/labs/vector-route-dual-write/results/routing-R3.txt @@ -0,0 +1,19 @@ +ch.kong_access.external 300 +ch.kong_access.internal 300 +ch.app_logs.api 600 +ch.any_table.loadgen 0 +loki.kong-external 300 +loki.kong-internal 300 +loki.api 600 +loki.loadgen 300 + +# column checks, run R3 (expected: 6 x 500 per gateway, 12 ERROR in the API) +โ”Œโ”€Gatewayโ”€โ”€โ”ฌโ”€Serviceโ”€โ”ฌโ”€Routeโ”€โ”€โ”ฌโ”€s500โ”€โ”ฌโ”€โ”€โ”€nโ”€โ”ฌโ”€any(Cluster)โ”€โ”€โ” +1. โ”‚ external โ”‚ orders โ”‚ orders โ”‚ 6 โ”‚ 300 โ”‚ lab-cluster-1 โ”‚ +2. โ”‚ internal โ”‚ users โ”‚ users โ”‚ 6 โ”‚ 300 โ”‚ lab-cluster-1 โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +โ”Œโ”€ServiceNameโ”€โ”ฌโ”€SeverityTextโ”€โ”ฌโ”€count()โ”€โ” +1. โ”‚ api โ”‚ ERROR โ”‚ 12 โ”‚ +2. โ”‚ api โ”‚ INFO โ”‚ 588 โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + diff --git a/labs/vector-route-dual-write/sql/schema.sql b/labs/vector-route-dual-write/sql/schema.sql new file mode 100644 index 0000000..b2205dc --- /dev/null +++ b/labs/vector-route-dual-write/sql/schema.sql @@ -0,0 +1,52 @@ +-- Two tables, one per kind of log. Created before Vector starts: the Vector +-- ClickHouse sink inserts JSONEachRow and never creates tables. + +CREATE DATABASE IF NOT EXISTS logs; + +-- Gateway access logs (both gateways). Fixed fields become columns, so +-- status/latency aggregations read a few narrow columns. Gateway tells +-- external from internal; Cluster tells the Kubernetes clusters apart. +CREATE TABLE IF NOT EXISTS logs.kong_access +( + Timestamp DateTime64(3) CODEC(Delta, ZSTD(1)), + Cluster LowCardinality(String), + Gateway LowCardinality(String), + Service LowCardinality(String), + Route LowCardinality(String), + Method LowCardinality(String), + Path String CODEC(ZSTD(1)), + Status UInt16, + -- Signed on purpose: Kong 3.9.4 logged latencies.proxy = -3 once in this + -- lab (results/outage-drop-D1-uint-schema.txt). With UInt32 that one row + -- failed the whole 1000-row insert and Vector dropped the batch. + LatencyMs Int32, + KongLatencyMs Int32, + UpstreamLatencyMs Int32, + ClientIp String, + Pod LowCardinality(String), + Body String CODEC(ZSTD(1)) +) +ENGINE = MergeTree +PARTITION BY toDate(Timestamp) +ORDER BY (Gateway, Service, Timestamp) +TTL toDateTime(Timestamp) + INTERVAL 90 DAY; + +-- Application logs. JSON lines are unpacked; a line that is not JSON is +-- kept whole in Body. +CREATE TABLE IF NOT EXISTS logs.app_logs +( + Timestamp DateTime64(9) CODEC(Delta, ZSTD(1)), + Cluster LowCardinality(String), + Namespace LowCardinality(String), + Pod LowCardinality(String), + Container LowCardinality(String), + ServiceName LowCardinality(String), + SeverityText LowCardinality(String), + Body String CODEC(ZSTD(1)), + LogAttributes Map(LowCardinality(String), String), + INDEX idx_body Body TYPE tokenbf_v1(32768, 3, 0) GRANULARITY 8 +) +ENGINE = MergeTree +PARTITION BY toDate(Timestamp) +ORDER BY (ServiceName, SeverityText, Timestamp) +TTL toDateTime(Timestamp) + INTERVAL 90 DAY; diff --git a/labs/vector-route-dual-write/vector/mode-block.yaml b/labs/vector-route-dual-write/vector/mode-block.yaml new file mode 100644 index 0000000..20edb4a --- /dev/null +++ b/labs/vector-route-dual-write/vector/mode-block.yaml @@ -0,0 +1,9 @@ +# Default when_full (block), small memory buffer so an outage fills it within a run. +# Applied on top of values.yaml to both ClickHouse sinks only; the Loki +# sink keeps Vector's default buffer. +customConfig: + sinks: + ch_kong: + buffer: {type: memory, max_events: 500, when_full: block} + ch_app: + buffer: {type: memory, max_events: 500, when_full: block} diff --git a/labs/vector-route-dual-write/vector/mode-disk.yaml b/labs/vector-route-dual-write/vector/mode-disk.yaml new file mode 100644 index 0000000..6144b01 --- /dev/null +++ b/labs/vector-route-dual-write/vector/mode-disk.yaml @@ -0,0 +1,9 @@ +# Disk buffer at its documented minimum size (268435488 bytes), drop_newest once full. +# Applied on top of values.yaml to both ClickHouse sinks only; the Loki +# sink keeps Vector's default buffer. +customConfig: + sinks: + ch_kong: + buffer: {type: disk, max_size: 268435488, when_full: drop_newest} + ch_app: + buffer: {type: disk, max_size: 268435488, when_full: drop_newest} diff --git a/labs/vector-route-dual-write/vector/mode-drop.yaml b/labs/vector-route-dual-write/vector/mode-drop.yaml new file mode 100644 index 0000000..a5d310c --- /dev/null +++ b/labs/vector-route-dual-write/vector/mode-drop.yaml @@ -0,0 +1,9 @@ +# Same buffer, but a full buffer drops new events instead of pushing back. +# Applied on top of values.yaml to both ClickHouse sinks only; the Loki +# sink keeps Vector's default buffer. +customConfig: + sinks: + ch_kong: + buffer: {type: memory, max_events: 500, when_full: drop_newest} + ch_app: + buffer: {type: memory, max_events: 500, when_full: drop_newest} diff --git a/labs/vector-route-dual-write/vector/values.yaml b/labs/vector-route-dual-write/vector/values.yaml new file mode 100644 index 0000000..4f5d7d5 --- /dev/null +++ b/labs/vector-route-dual-write/vector/values.yaml @@ -0,0 +1,175 @@ +# Vector Agent (DaemonSet) โ€” helm chart vector/vector 0.61.0. +# +# The existing pipeline is `k8s -> loki_labels -> loki`: every pod's logs. +# What is added: `by_workload` (route) splits the same source into three +# named outputs, two remaps shape them into table rows, and two ClickHouse +# sinks insert them. Nothing on the Loki side changes; the source simply has +# one more consumer. +# +# The ClickHouse sinks' buffer is set by one of mode-*.yaml (block, drop, +# disk) on top of this file โ€” see README "Buffer modes". +role: Agent + +env: + - name: CLUSTER_NAME + value: lab-cluster-1 + +service: + ports: + - name: prom + port: 9598 + protocol: TCP + +customConfig: + data_dir: /vector-data-dir + api: + enabled: false + + sources: + k8s: + type: kubernetes_logs + # Default 60000: a new pod's log file can take up to a minute to be + # picked up. Shortened so a lab run is read within seconds. + glob_minimum_cooldown_ms: 5000 + vector_metrics: + type: internal_metrics + + transforms: + # --- existing pipeline ------------------------------------------------- + loki_labels: + type: remap + inputs: [k8s] + source: | + .loki_app = .kubernetes.pod_labels.app || "none" + .loki_cluster = get_env_var("CLUSTER_NAME") ?? "unknown" + + # --- added: route by workload ----------------------------------------- + by_workload: + type: route + inputs: [k8s] + route: + kong_external: '.kubernetes.pod_labels.app == "kong-external"' + kong_internal: '.kubernetes.pod_labels.app == "kong-internal"' + api: '.kubernetes.pod_namespace == "api" && .kubernetes.container_name == "api"' + # everything else -> by_workload._unmatched, which nothing consumes + + kong_rows: + type: remap + inputs: [by_workload.kong_external, by_workload.kong_internal] + drop_on_abort: true + drop_on_error: true + source: | + parsed, err = parse_json(.message) + if err != null || !is_object(parsed) { abort } + p = object!(parsed) + if !exists(p.request) { abort } # Kong's own startup/error lines + gateway = if .kubernetes.pod_labels.app == "kong-external" { "external" } else { "internal" } + . = { + "Timestamp": from_unix_timestamp!(int!(p.started_at), unit: "milliseconds"), + "Cluster": get_env_var("CLUSTER_NAME") ?? "unknown", + "Gateway": gateway, + "Service": p.service.name, + "Route": p.route.name, + "Method": p.request.method, + "Path": p.request.uri, + "Status": p.response.status, + "LatencyMs": p.latencies.request, + "KongLatencyMs": p.latencies.kong, + "UpstreamLatencyMs": p.latencies.proxy, + "ClientIp": p.client_ip, + "Pod": .kubernetes.pod_name, + "Body": .message + } + + api_rows: + type: remap + inputs: [by_workload.api] + drop_on_error: true + source: | + row = { + "Timestamp": .timestamp, + "Cluster": get_env_var("CLUSTER_NAME") ?? "unknown", + "Namespace": .kubernetes.pod_namespace, + "Pod": .kubernetes.pod_name, + "Container": .kubernetes.container_name, + "ServiceName": .kubernetes.pod_labels.app || "unknown", + "SeverityText": "INFO", + "Body": .message, + "LogAttributes": {} + } + parsed, err = parse_json(.message) + if err == null && is_object(parsed) { + p = object!(parsed) + row.SeverityText = p.level || "INFO" + row.Body = p.msg || .message + row.LogAttributes = { + "method": to_string(p.method) ?? "", + "path": to_string(p.path) ?? "", + "status": to_string(p.status) ?? "", + "run": to_string(p.run) ?? "", + "i": to_string(p.i) ?? "" + } + } + . = row + + sinks: + # --- existing pipeline ------------------------------------------------- + loki: + type: loki + inputs: [loki_labels] + endpoint: http://loki.loki.svc:3100 + encoding: + codec: text + # Helm runs customConfig through tpl, so Vector's {{ field }} templates + # are escaped as {{ "{{" }} field {{ "}}" }}. + labels: + cluster: '{{ "{{" }} loki_cluster {{ "}}" }}' + namespace: '{{ "{{" }} kubernetes.pod_namespace {{ "}}" }}' + app: '{{ "{{" }} loki_app {{ "}}" }}' + container: '{{ "{{" }} kubernetes.container_name {{ "}}" }}' + + # --- added: ClickHouse --------------------------------------------------- + ch_kong: + type: clickhouse + inputs: [kong_rows] + endpoint: http://clickstack.clickstack.svc:8123 + database: logs + table: kong_access + format: json_each_row + skip_unknown_fields: true + date_time_best_effort: true + # The all-in-one image's own HTTP user (users.d/default-user.xml, the + # same one ClickStack's Vector guide uses). On ClickHouse Cloud: a + # user with INSERT on these two tables only, password from a Secret. + auth: + strategy: basic + user: api + password: api + batch: + max_events: 1000 + timeout_secs: 2 + + ch_app: + type: clickhouse + inputs: [api_rows] + endpoint: http://clickstack.clickstack.svc:8123 + database: logs + table: app_logs + format: json_each_row + skip_unknown_fields: true + date_time_best_effort: true + # The all-in-one image's own HTTP user (users.d/default-user.xml, the + # same one ClickStack's Vector guide uses). On ClickHouse Cloud: a + # user with INSERT on these two tables only, password from a Secret. + auth: + strategy: basic + user: api + password: api + batch: + max_events: 1000 + timeout_secs: 2 + + prom: + type: prometheus_exporter + inputs: [vector_metrics] + address: 0.0.0.0:9598