Dev ch tests

README.md

@@ -29,7 +29,7 @@ After the test, the result is merged into a single per-(hardware, subject) JSON
 
 ## Available tests
 
-### Performance tests (25)
+### Performance tests (26)
 
 | Test | What it does |
 | --- | --- |
@@ -58,6 +58,7 @@ After the test, the result is merged into a single per-(hardware, subject) JSON
 | `tcp_to_sns_performance` | TCP in, SNS topic out, observed via an SQS subscription (LocalStack emulator) |
 | `tcp_to_kinesis_performance` | TCP in, Kinesis stream out across 4 shards (LocalStack emulator) |
 | `tcp_to_cloudwatch_performance` | TCP in, CloudWatch Logs out (LocalStack emulator) |
+| `tcp_to_clickhouse_performance` | TCP in, batch-INSERT into a real ClickHouse server — native protocol (vmetric, otel) or HTTP JSONEachRow (Vector, Fluent Bit, Cribl); receiver counts committed rows |
 
 ### Correctness tests (17)
 
@@ -90,11 +91,17 @@ Two caveats when reading the numbers:
 - **Compare across subjects, not across case types.** The emulators (Python/Node single-process services) cap throughput far below the raw TCP cases. Every subject faces the same ceiling, so rankings are meaningful; absolute EPS is not comparable to `tcp_to_tcp_performance`.
 - **Out of scope:** Azure Event Hubs, Service Bus, Sentinel and Data Explorer have no usable local emulator; AWS Firehose and DynamoDB delivery are LocalStack pro-tier. Cases for those services would silently measure a mock, so they don't exist.
 
+### ClickHouse ingest case
+
+`tcp_to_clickhouse_performance` measures **sustained rows/s committed to a real `clickhouse-server` (24.8)** through each subject's native ClickHouse integration. Unlike the emulator cases the target is the real database, and unlike the TCP cases the receiver isn't in the data path — it polls ClickHouse for the committed row count and the harness derives EPS from that timeline. Coverage is the subjects with a first-party ClickHouse path: **vmetric** and the **OpenTelemetry Collector** batch-INSERT over the native TCP protocol (`:9000`); **Vector**, **Fluent Bit** (via its `http` output), and **Cribl Stream** POST JSONEachRow over the HTTP interface (`:8123`).
+
+Reading the numbers: the generator runs unbounded, so this is a **peak** test. Tools that backpressure cleanly on TCP throttle the generator and deliver nearly everything (low `loss%`); tools that can't shed load once ClickHouse can't keep up, which shows as high `loss%`. Weigh **EPS alongside CPU/memory and loss** — a high raw EPS bought with an order of magnitude more RAM and a large fraction of events dropped is a very different result than the same EPS at a fraction of the resources with almost no loss.
+
 ## Subjects
 
 | Name | Image | Version |
 | --- | --- | --- |
-| VirtualMetric DataStream | `vmetric/director` | `2.0.6` |
+| VirtualMetric DataStream | `vmetric/director` | `2.0.9` |
 | Vector | `timberio/vector` | `0.54.0-alpine` |
 | Fluent Bit | `fluent/fluent-bit` | `5.0` |
 | Fluentd | `fluent/fluentd` | `v1.17-debian-1` |
@@ -127,7 +134,7 @@ PipeBench/
     receiver/            Receives output, counts lines, validates correctness
     collector/           Polls Docker stats API, writes metrics CSV
     vmetric/             Dockerfile + pre-built binary for the VirtualMetric Director subject
-  cases/                 51 test cases (27 performance + 24 correctness), each with per-subject configs
+  cases/                 60 test cases (32 performance + 28 correctness), each with per-subject configs
   web/                   Static PipeBench UI (single HTML + per-(hardware, subject) JSON under web/results/)
 ```
 

cases/tcp_to_clickhouse_performance/case.yaml

@@ -0,0 +1,63 @@
+name: tcp_to_clickhouse_performance
+type: performance
+description: "Measures throughput of a TCP-in to ClickHouse-out pipeline against a real clickhouse-server (24.8). Unlimited generator rate — pushes peak. Each subject ingests into ClickHouse over its native integration: vmetric and the OpenTelemetry Collector batch-INSERT over the native TCP protocol (:9000, the lowest-overhead wire path); Vector, Fluent Bit, and Cribl Stream POST JSONEachRow over the HTTP interface (:8123). This isolates how efficiently each tool batches and pipelines log ingest into ClickHouse — not the TCP receiver, which is out of the data path here. The receiver polls ClickHouse for the committed row count every 500ms (summing active MergeTree parts across the whole database, so it counts whatever table a subject created) and the harness derives EPS from the row-count timeline."
+
+duration: 60s
+# ClickHouse-server needs a few seconds to open its ports; the receiver creates
+# the bench database + table during warmup, before any data reaches the subject.
+warmup: 30s
+# Sink batches flush on interval/size — give the tail time to land in ClickHouse.
+drain_grace: 45s
+
+endpoints:
+  - name: clickhouse
+    image: clickhouse/clickhouse-server:24.8
+    env:
+      CLICKHOUSE_USER: "bench"
+      CLICKHOUSE_PASSWORD: "benchpass"
+      CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: "1"
+    # Gate the subject on ClickHouse being up. Most subjects retry their sink
+    # lazily, but the otel-collector clickhouse exporter creates its schema
+    # eagerly at startup and crashes if the server isn't accepting connections
+    # yet — so the subject waits for this healthcheck to pass.
+    healthcheck:
+      test: "clickhouse-client --query 'SELECT 1'"
+      interval: 2s
+      timeout: 3s
+      retries: 45
+      start_period: 3s
+
+generator:
+  mode: tcp
+  target: "subject:9000"
+  rate: 0           # unlimited — push as fast as possible
+  line_size: 256
+  format: raw
+  # Some subjects (Fluent Bit, Cribl) can't backpressure cleanly on TCP when
+  # ClickHouse can't keep up at peak — they reset the socket instead. Reconnect
+  # so those resets are absorbed and still yield a measured throughput, rather
+  # than aborting the whole run. Subjects that backpressure cleanly (vmetric,
+  # Vector, otel) never trigger it.
+  reconnect: true
+
+receiver:
+  mode: clickhouse
+  env:
+    RECEIVER_CH_ENDPOINT: "http://clickhouse:8123"
+    RECEIVER_CH_USER: "bench"
+    RECEIVER_CH_PASSWORD: "benchpass"
+    # Poll the row count twice a second for a tight EPS window.
+    RECEIVER_CLOUD_POLL_MS: "500"
+
+requires: [clickhouse_sink]
+
+subjects:
+  - vmetric
+  - vector
+  - fluent-bit
+  - otel-collector
+  - cribl-stream
+
+configurations:
+  default:
+    description: "Raw TCP to a ClickHouse table — no transforms, unlimited rate"

cases/tcp_to_clickhouse_performance/configs/cribl-stream/cribl.yml

@@ -0,0 +1,13 @@
+api:
+  host: 0.0.0.0
+  port: 9999
+  disabled: false
+auth:
+  type: local
+system:
+  installType: standalone
+  intercom: false
+workers:
+  count: -2
+  minimum: 1
+  memory: 2048

cases/tcp_to_clickhouse_performance/configs/cribl-stream/inputs.yml

@@ -0,0 +1,16 @@
+inputs:
+  bench_tcp_in:
+    type: tcp
+    host: 0.0.0.0
+    port: 9000
+    disabled: false
+    tls:
+      disabled: true
+    enableProxyHeader: false
+    staleChannelFlushMs: 10000
+    enableHeader: false
+    preprocess:
+      disabled: true
+    sendToRoutes: true
+    pqEnabled: false
+    authType: manual

cases/tcp_to_clickhouse_performance/configs/cribl-stream/messages.yml

@@ -0,0 +1 @@
+messages: {}

cases/tcp_to_clickhouse_performance/configs/cribl-stream/outputs.yml

@@ -0,0 +1,17 @@
+outputs:
+  default:
+    defaultId: bench_clickhouse_out
+    type: default
+  bench_clickhouse_out:
+    type: click_house
+    url: http://clickhouse:8123
+    database: bench
+    tableName: logs
+    format: json-each-row
+    mappingType: automatic
+    authType: basic
+    username: bench
+    password: benchpass
+    asyncInserts: false
+    onBackpressure: block
+    compress: false

cases/tcp_to_clickhouse_performance/configs/cribl-stream/pipelines/ch_prep.yml

@@ -0,0 +1,16 @@
+id: ch_prep
+conf:
+  output: default
+  groups: {}
+  asyncFuncTimeout: 1000
+  functions:
+    - id: eval
+      filter: "true"
+      disabled: false
+      conf:
+        add:
+          - name: message
+            value: _raw
+        remove:
+          - "_raw"
+          - "_time"

cases/tcp_to_clickhouse_performance/configs/cribl-stream/pipelines/route.yml

@@ -0,0 +1,12 @@
+id: default
+groups: {}
+comments: []
+routes:
+  - id: bench_route
+    name: bench_clickhouse
+    final: true
+    disabled: false
+    pipeline: ch_prep
+    filter: "true"
+    output: bench_clickhouse_out
+    description: "Map _raw to message, then insert into ClickHouse"

cases/tcp_to_clickhouse_performance/configs/fluent-bit.conf

@@ -0,0 +1,43 @@
+# Fluent Bit has no native ClickHouse output; the stock `http` output POSTs
+# JSONEachRow to ClickHouse's HTTP interface (:8123) via an INSERT … FORMAT
+# JSONEachRow query. The raw TCP line arrives under the `log` key (tcp input,
+# Format none); a modify filter renames it to `message` so it lands in the
+# `message` column of bench.logs. json_date_key is disabled so the record is a
+# clean {"message": "..."} object.
+
+[SERVICE]
+    Flush         1
+    Log_Level     info
+    Daemon        off
+
+[INPUT]
+    Name          tcp
+    Listen        0.0.0.0
+    Port          9000
+    Format        none
+    Chunk_Size    512
+    Buffer_Size   1024
+    # Bound the in-memory buffer so a slower ClickHouse insert path pauses the
+    # TCP input (backpressure to the generator) instead of growing until the
+    # container is OOM-killed at unlimited rate.
+    Mem_Buf_Limit 256M
+
+[FILTER]
+    Name          modify
+    Match         *
+    Rename        log message
+
+[OUTPUT]
+    Name          http
+    Match         *
+    Host          clickhouse
+    Port          8123
+    URI           /?query=INSERT+INTO+bench.logs+FORMAT+JSONEachRow&input_format_skip_unknown_fields=1
+    Format        json_lines
+    Json_Date_Key false
+    Header        Content-Type application/json
+    http_user     bench
+    http_passwd   benchpass
+    # Parallel POST workers so the HTTP insert path isn't a single-stream
+    # bottleneck against ClickHouse.
+    workers       4

cases/tcp_to_clickhouse_performance/configs/http/vmetric.yml

@@ -0,0 +1,30 @@
+# vmetric variant: ClickHouse target over the HTTP interface (:8123) with
+# INSERT … FORMAT JSONEachRow + gzip (use_compression) — the same high-throughput
+# wire path Vector/Fluent Bit/Cribl use here, for an apples-to-apples comparison
+# against the native-protocol default config. Selected with `-c http`.
+devices:
+  - id: 1
+    name: tcp-in
+    type: tcp
+    properties:
+      address: "0.0.0.0"
+      port: 9000
+
+targets:
+  - name: ch-out
+    type: clickhouse
+    properties:
+      address: "clickhouse"
+      port: 8123
+      database: "bench"
+      table: "logs"
+      username: "bench"
+      password: "benchpass"
+      use_compression: true
+
+routes:
+  - name: tcp-to-ch
+    devices:
+      - name: tcp-in
+    targets:
+      - name: ch-out

cases/tcp_to_clickhouse_performance/configs/otel-collector.yaml

@@ -0,0 +1,40 @@
+# OpenTelemetry Collector (contrib): tcplog receiver in, native `clickhouse`
+# exporter out. The exporter INSERTs over the native protocol (:9000) and
+# manages its own schema (create_schema) — it writes to its own bench.otel_logs
+# table, not the shared bench.logs. The perf receiver counts rows database-wide,
+# so otel_logs is measured on equal footing. The body of each log record is the
+# raw TCP line.
+
+receivers:
+  tcplog:
+    listen_address: "0.0.0.0:9000"
+
+processors:
+  batch:
+    send_batch_size: 10000
+    timeout: 1s
+
+exporters:
+  clickhouse:
+    endpoint: tcp://clickhouse:9000?dial_timeout=10s
+    database: bench
+    username: bench
+    password: benchpass
+    logs_table_name: otel_logs
+    create_schema: true
+    ttl: 0
+    timeout: 10s
+    retry_on_failure:
+      enabled: true
+    sending_queue:
+      queue_size: 1000
+
+service:
+  telemetry:
+    metrics:
+      level: none
+  pipelines:
+    logs:
+      receivers: [tcplog]
+      processors: [batch]
+      exporters: [clickhouse]

cases/tcp_to_clickhouse_performance/configs/parquet/vmetric.yml

@@ -0,0 +1,29 @@
+# vmetric variant: ClickHouse target, columnar Parquet POSTed over the HTTP REST
+# interface (:8123). parquet-go encodes a single `message` column file; ClickHouse
+# maps it by name. Parquet self-compresses (no gzip). Selected with `-c parquet`.
+devices:
+  - id: 1
+    name: tcp-in
+    type: tcp
+    properties:
+      address: "0.0.0.0"
+      port: 9000
+
+targets:
+  - name: ch-out
+    type: clickhouse
+    properties:
+      address: "clickhouse"
+      port: 8123
+      database: "bench"
+      table: "logs"
+      username: "bench"
+      password: "benchpass"
+      format: "parquet"
+
+routes:
+  - name: tcp-to-ch
+    devices:
+      - name: tcp-in
+    targets:
+      - name: ch-out

cases/tcp_to_clickhouse_performance/configs/profile/vmetric.yml

@@ -0,0 +1,35 @@
+# Native ClickHouse target + pprof profiler enabled — used to capture CPU/heap
+# profiles of the native ingestion path. Selected with `-c profile`. Reach the
+# profiler at http://127.0.0.1:6061/debug/pprof/ from inside the subject
+# container (docker exec + wget).
+profiler:
+  status: true
+  remote: true
+  director_profile_port: 6061
+
+devices:
+  - id: 1
+    name: tcp-in
+    type: tcp
+    properties:
+      address: "0.0.0.0"
+      port: 9000
+
+targets:
+  - name: ch-out
+    type: clickhouse
+    properties:
+      address: "clickhouse"
+      port: 9000
+      database: "bench"
+      table: "logs"
+      username: "bench"
+      password: "benchpass"
+      format: "native"
+
+routes:
+  - name: tcp-to-ch
+    devices:
+      - name: tcp-in
+    targets:
+      - name: ch-out