Dev ch tests

[yusufozturk]

Summary by CodeRabbit

• New Features

  • Added a TCP-to-ClickHouse performance benchmark measuring end-to-end committed-row throughput.
  • Added benchmark coverage for multiple ingestion formats and data pipeline implementations.
  • Added endpoint readiness checks and optional reconnect handling for interrupted TCP connections.

• Documentation

  • Updated the performance test matrix and project test count.
  • Documented ClickHouse ingestion methodology and throughput calculations.
  • Updated the VirtualMetric DataStream version information.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: a0e09600-c201-49fb-8849-f37bb3a77ca3

📥 Commits

Reviewing files that changed from the base of the PR and between 90f6973 and 14ad51d.

📒 Files selected for processing (23)

• README.md
• cases/tcp_to_clickhouse_performance/case.yaml
• cases/tcp_to_clickhouse_performance/configs/cribl-stream/cribl.inited
• cases/tcp_to_clickhouse_performance/configs/cribl-stream/cribl.yml
• cases/tcp_to_clickhouse_performance/configs/cribl-stream/inputs.yml
• cases/tcp_to_clickhouse_performance/configs/cribl-stream/messages.yml
• cases/tcp_to_clickhouse_performance/configs/cribl-stream/outputs.yml
• cases/tcp_to_clickhouse_performance/configs/cribl-stream/pipelines/ch_prep.yml
• cases/tcp_to_clickhouse_performance/configs/cribl-stream/pipelines/route.yml
• cases/tcp_to_clickhouse_performance/configs/fluent-bit.conf
• cases/tcp_to_clickhouse_performance/configs/http/vmetric.yml
• cases/tcp_to_clickhouse_performance/configs/otel-collector.yaml
• cases/tcp_to_clickhouse_performance/configs/parquet/vmetric.yml
• cases/tcp_to_clickhouse_performance/configs/profile/vmetric.yml
• cases/tcp_to_clickhouse_performance/configs/tsv/vmetric.yml
• cases/tcp_to_clickhouse_performance/configs/vector.toml
• cases/tcp_to_clickhouse_performance/configs/vmetric.yml
• containers/generator/main.go
• containers/receiver/clickhouse.go
• containers/receiver/main.go
• internal/config/case.go
• internal/config/subject.go
• internal/orchestrator/docker.go

Disabled knowledge base sources:

• Jira integration is disabled

You can enable these sources in your CodeRabbit configuration.

---

Walkthrough

Adds a TCP-to-ClickHouse performance benchmark with committed-row measurement, endpoint healthchecks, generator reconnect support, configurations for multiple subjects, and updated documentation.

Changes

ClickHouse benchmark

Layer / File(s)	Summary
Benchmark contract and compose wiring cases/tcp_to_clickhouse_performance/case.yaml, internal/config/case.go, internal/config/subject.go, internal/orchestrator/docker.go	Defines ClickHouse benchmark settings, endpoint readiness checks, reconnect configuration, subject capabilities, and Docker Compose wiring.
ClickHouse receiver measurement containers/receiver/clickhouse.go, containers/receiver/main.go	Initializes ClickHouse schema, polls committed row counts over HTTP, records row deltas, and exposes the clickhouse receiver mode.
Generator reconnect loop containers/generator/main.go	Adds deadline-bounded TCP/TLS dialing and reconnect-and-resume behavior for parallel workers.
Subject ingestion configurations cases/tcp_to_clickhouse_performance/configs/*	Adds TCP-to-ClickHouse configurations for Cribl Stream, Fluent Bit, Vector, OpenTelemetry Collector, and VirtualMetric variants.
Benchmark documentation README.md	Documents the new performance case, committed-row EPS measurement, supported subjects, and updated counts and versions.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Generator
  participant Subject
  participant ClickHouse
  participant Receiver
  Generator->>Subject: send raw TCP lines
  Subject->>ClickHouse: insert transformed or raw records
  Receiver->>ClickHouse: poll committed row count
  ClickHouse-->>Receiver: return current count
  Receiver->>Receiver: record committed-row delta

Loading

Possibly related PRs

• VirtualMetric/PipeBench#4: Shares generator configuration and orchestration integration points.
• VirtualMetric/PipeBench#5: Extends the endpoint model and Docker Compose rendering used for healthchecks.
• VirtualMetric/PipeBench#19: Also modifies TCP generator reconnect behavior.

Poem

I’m a rabbit with packets to chase,
Through ClickHouse rows at a hopping pace.
Reconnect, retry, and count each flow,
While benchmarks measure what sinks below.
Hop hop hooray—the pipes now glow!

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch dev-ch-tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[cloudflare-workers-and-pages]

Deploying pipebench with    Cloudflare Pages

Latest commit:	14ad51d
Status:	⚡️  Build in progress...

View logs