diff --git a/BENCHMARKS.md b/BENCHMARKS.md index 83449beb..79341576 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -1,227 +1,21 @@ -# CPU Cost Guide +# Benchmarks -**Baseline benchmarks for dfe-loader performance optimization.** +The criterion benches in `benches/` measure the hot path's CPU cost. Run them on the machine you are comparing, since a figure from another host or an older build says nothing about this one. -Last updated: 2025-12-28 -Platform: AMD64 Linux (Fedora 42) -Rust: 1.83.0 (release profile) - ---- - -## Summary: Per-Message Pipeline Cost - -| Stage | Cost/msg | Notes | -|-------|----------|-------| -| **JSON Parse (sonic-rs)** | ~150-200 ns | SIMD-accelerated | -| **Routing** | ~45-170 ns | route_value fastest (45ns), bytes slower (170ns) | -| **Flatten** | ~130-825 ns | Depth-dependent (3 fields: 130ns, 6 levels: 825ns) | -| **Transform** | ~600-900 ns | Full transform with logjson capture | -| **GeoIP (cache hit)** | ~30 ns/IP | Private IP fast path: ~27 ns | -| **Reputation (cache hit)** | ~36 ns/IP | O(1) hash lookup | -| **Risk Scoring** | ~33-100 ns | Integer-only math | -| **Full Enrichment** | ~100 ns/IP | GeoIP + Reputation + Risk combined | - -**Estimated total per message: ~1.5-2.5 µs** (without ClickHouse I/O) - ---- - -## Detailed Benchmarks - -### 1. JSON Parsing (sonic-rs) - -| Benchmark | Time | Throughput | -|-----------|------|------------| -| Parse small payload (52 bytes) | 177-195 ns | 255-282 MiB/s | -| Parse medium payload (262 bytes) | 784-798 ns | 329-335 MiB/s | -| Parse large payload (636 bytes) | 1.73-1.82 µs | 350-368 MiB/s | -| Field extraction (top-level) | 61-65 ns | - | -| Field extraction (nested) | 158-167 ns | - | -| Field extraction (3 levels deep) | 193-206 ns | - | - -### 2. Routing - -| Benchmark | Time | Throughput | -|-----------|------|------------| -| route_bytes (raw JSON) | 159-179 ns | 5.6-6.3 M/s | -| route_cow (Cow return) | 147-159 ns | 6.3-6.8 M/s | -| **route_value (pre-parsed)** | **44-46 ns** | **21.9-22.9 M/s** | - -**Recommendation**: Parse once, route with `route_value()` for 3-4x speedup. - -### 3. Flattening - -| Benchmark | Time | Notes | -|-----------|------|-------| -| Shallow (3 fields) | 127-137 ns | Simple objects | -| Medium nested | 591-624 ns | 2-3 levels | -| Deep (6 levels) | 814-837 ns | Deep nesting | -| Wide (50 fields) | 4.45-4.97 µs | Many top-level fields | - -**Cost model**: ~100-150 ns base + ~100 ns per nesting level + ~80 ns per field. - -### 4. Transform Pipeline - -| Benchmark | Time | Notes | -|-----------|------|-------| -| Transform small | 594-650 ns | With logjson capture | -| Transform medium | 2.24-2.49 µs | Typical event | -| Transform large | 5.85-6.54 µs | Complex event | - -### 5. Full E2E Pipeline (no I/O) - -| Benchmark | Time | Throughput | -|-----------|------|------------| -| Parse + Route + Flatten + Transform (medium) | 2.66-2.74 µs | 95-99 MiB/s | - ---- - -## Enrichment Benchmarks - -### GeoIP Lookup - -| Benchmark | Time (10 IPs) | Per-IP | Notes | -|-----------|---------------|--------|-------| -| Private IP fast path | 269-279 ns | **~27 ns** | RFC1918/loopback detection | -| Cache hit | 295-350 ns | **~30-35 ns** | RwLock read | -| Cache miss (no DB) | 358-1,140 ns | ~36-114 ns | Varies with contention | - -### Reputation Lookup - -| Benchmark | Time (10 IPs) | Per-IP | Notes | -|-----------|---------------|--------|-------| -| Empty blocklist | 349-405 ns | **~35-40 ns** | Cache + lock overhead | -| Cache hit | 354-402 ns | **~35-40 ns** | O(1) FxHashMap | -| 1000 IPs + 100 CIDRs | 359-426 ns | **~36-43 ns** | Hash still O(1) | -| Prefix match | 356-3,500 ns | ~36-350 ns | CIDR scan cost varies | - -### Risk Scoring - -| Benchmark | Time | Notes | -|-----------|------|-------| -| Minimal input | 32-42 ns | Empty risk factors | -| Full input (all threats) | 81-104 ns | Botnet + malicious + abuse score | -| High security preset | 40-42 ns | Preset lookup O(1) | -| From enrichment results | 47-52 ns | Struct copy overhead | - -### Full Enrichment Pipeline - -| Benchmark | Time (10 IPs) | Per-IP | -|-----------|---------------|--------| -| GeoIP + Reputation + Risk | 950-1,020 ns | **~95-102 ns** | - ---- - -## Optimization Priorities - -### Hot Path (optimize first) - -1. **JSON parsing** - Already SIMD (sonic-rs), ~200 ns/msg -2. **Routing** - Use `route_value()` after parsing, ~45 ns -3. **Transform** - ~600-900 ns, most time in flatten - -### Cold Path (defer) - -1. **Schema fetch** - Once per table, cached -2. **Blocklist reload** - Background task -3. **MMDB reload** - On file change - -### Memory Allocation - -| Operation | Allocations | Notes | -|-----------|-------------|-------| -| Flatten shallow | 0 | Reuses existing Map | -| Flatten deep | ~N | One per nested level | -| Transform | 1-2 | logjson + _tags | -| Routing | 0 | Returns borrowed str | - ---- - -## Scaling Estimates - -### Single-threaded throughput - -| Scenario | Est. msgs/sec | Notes | -|----------|---------------|-------| -| Parse only | ~5-6 M/s | Just JSON parsing | -| Parse + Route | ~4-5 M/s | With routing | -| Full transform | ~400-600 K/s | All processing | -| With enrichment | ~350-500 K/s | +GeoIP/Reputation/Risk | - -### Bottleneck analysis - -1. **CPU-bound**: Transform dominates at ~2.5 µs/msg -2. **Network I/O**: ClickHouse insert batches amortize -3. **Memory**: ~1-2 KB per in-flight message - ---- - -## Benchmark Commands +| Bench | Measures | +|---|---| +| `json_parsing` | sonic-rs parsing and field lookups | +| `pipeline` | every per-message stage from parse to transform, with no ClickHouse | +| `bakeoff` | hot-path throughput on flat 10-field and 30-field events and a two-level nested event | +| `extractor` | `HeaderExtractor::extract` per row | +| `transform` | the legacy flatten and transform path | +| `enrichment` | GeoIP, reputation and risk scoring lookups | +| `json_depth` | the nesting-depth guard on the payloads `JSON_DEPTH_CORPUS` names | +| `insert_bakeoff` | JSONEachRow against RowBinary inserts, against the ClickHouse its header's variables name | ```bash -# Run all benchmarks -cargo bench - -# Specific benchmark suites -cargo bench --bench json_parsing -cargo bench --bench transform cargo bench --bench pipeline -cargo bench --bench enrichment - -# HTML reports -cargo bench -- --noplot # faster, no gnuplot -open target/criterion/report/index.html +cargo bench --bench json_parsing -- --noplot # skip the gnuplot step ``` ---- - -## Mison Structural Index (Experimental) - -Mison-style structural indexing for schema-guided extraction. Based on the VLDB 2017 paper. - -### Benchmarks at v0.11.0 - -| Benchmark | Mison | Traditional | Ratio | -|-----------|-------|-------------|-------| -| Index build (simple) | ~165 ns | - | - | -| Single field extract | ~70-165 ns | - | - | -| Route (2 fields) | 1.1 µs | 543 ns | 2x slower | -| Schema extract (6 fields) | 834 ns | - | - | -| Full single | 3.1 µs | 653 ns | 5x slower | -| Batch 100 | 179 µs | 57 µs | 3x slower | -| sonic_get_from_slice (2 fields) | - | 141 ns | **Fastest** | - -### Analysis - -Current implementation is **slower** than sonic-rs for these reasons: - -1. **Scalar fallback**: AVX2/SSE4.2 not enabled at compile-time (needs runtime detection) -2. **Per-bit scanning**: Leveled bitmap uses O(n) bit-by-bit loop vs O(1) SIMD ops -3. **Index overhead**: Building full index for 2 fields is wasteful -4. **String comparisons**: Key matching not optimized - -### When Mison Would Win - -Mison approach is designed for: - -- **Many field extractions** (10+ fields per message) -- **Same schema for many messages** (index can be partially reused) -- **Avoiding DOM allocation** (current impl still has overhead) - -### Next Steps - -1. Enable runtime SIMD detection (is_x86_feature_detected!) -2. Use popcount + bit manipulation instead of per-bit loops -3. Add "fast path" for schema-only extraction (skip full index) -4. Consider simdjson integration for better baseline - -**Recommendation**: Use `sonic_rs::get_from_slice()` for routing (141 ns/2 fields). - ---- - -## Version History - -| Version | Date | Changes | -|---------|------|---------| -| 0.11.0 | 2025-12-28 | Mison structural index implementation (experimental) | -| 0.10.0 | 2025-12-28 | Initial enrichment benchmarks | -| 0.9.0 | 2025-12-25 | Hot path round 4 (FxHashMap, CompactString) | +Reports land under `target/criterion/`. The parser and insert-encoding bake-offs behind the loader's design, with their figures, are in [docs/performance/history.md](docs/performance/history.md). diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 06b31cf7..9b2f700a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -31,7 +31,7 @@ versioning and changelog generation. All commits must follow this format: | `build` | Changes to build system or dependencies | None | | `ci` | Changes to CI configuration | None | | `chore` | Other changes that don't modify src or test files | None | -| `revert` | Reverts a previous commit | Varies | +| `revert` | Reverts a previous commit | Patch (0.0.X) | ### Breaking Changes @@ -67,17 +67,16 @@ affected (e.g., `api`, `auth`, `core`, `cli`, `docs`). This project follows [Semantic Versioning 2.0.0](https://semver.org/): -- **MAJOR** (X.0.0): Breaking changes that require users to modify their code -- **MINOR** (0.X.0): New features that are backwards-compatible -- **PATCH** (0.0.X): Bug fixes and minor improvements +- MAJOR (X.0.0): breaking changes that require users to modify their code +- MINOR (0.X.0): new features that are backwards-compatible +- PATCH (0.0.X): bug fixes and minor improvements Versions are automatically determined by semantic-release based on commit messages. Do not manually update version numbers. ## Developer Certificate of Origin -This project uses the Developer Certificate of Origin (DCO) to ensure that -contributors have the right to submit their contributions. +This project uses the Developer Certificate of Origin (DCO), in which each contributor certifies the right to submit their contribution. By making a contribution to this project, you certify that: @@ -133,11 +132,13 @@ anniversary of its release. ## How to Contribute -1. **Fork the repository** and create your branch from `main` -2. **Make your changes** following the commit message format above -3. **Sign off your commits** with the DCO -4. **Test your changes** to ensure they work as expected -5. **Submit a pull request** with a clear description of what you've done +1. Fork the repository and create your branch from `main` +2. Make your changes following the commit message format above +3. Sign off your commits with the DCO +4. Test your changes to ensure they work as expected +5. Submit a pull request with a clear description of what you've done + +If you have questions about contributing, please open an issue or contact us. ### Pull Request Checklist @@ -165,25 +166,18 @@ surviving CI. ## CI/CD Workflow -When your pull request is merged to `main`: - -1. **semantic-release** analyses commit messages since the last release -2. Determines the next version number based on commit types -3. Generates/updates the CHANGELOG -4. Creates a new GitHub release with release notes -5. Publishes the package (if applicable) - -This happens automatically - no manual intervention required. +A merge to `main` releases nothing on its own. CI validates the merge, and publishes only when the merge commit carries a `Release: true` trailer (`hyperi-ci push --release` adds it) or someone dispatches a release of `main` (`hyperi-ci release`). A release then runs in order: -## Questions - -If you have questions about contributing, please open an issue or contact us. +1. semantic-release analyses the commit messages since the last release +2. It picks the next version from the commit types +3. It renders CHANGELOG.md and tags HEAD +4. hyperi-ci creates the GitHub release and publishes the image and chart ## Context ### What this is -dfe-loader writes rows into ClickHouse -- the last hop of the DFE core data path and the only component that writes there. +dfe-loader writes the data path's records into ClickHouse, the last hop of the DFE core data path. NOT a transform stage, NOT a schema manager. Its only outbound topic is the DLQ. dfe-engine creates the destination tables, and the loader holds records pending schema rather than issuing DDL. The dynamic insert comes from `dfe-clickhouse`'s `dynamic` module, because the `clickhouse` crate keeps insertion compile-time typed. @@ -193,9 +187,10 @@ NOT a transform stage, NOT a schema manager. Its only outbound topic is the DLQ. |------|-------| | `src/main.rs`, `src/config/` | CLI entry, config structs, `Config::validate()`, flat env | | `src/pipeline/` | Orchestrator and per-message processor -- the hot path | -| `src/routing/`, `src/transform/`, `src/enrich/` | Routing, coercion and capture, enrichment | +| `src/routing/`, `src/transform/`, `src/enrich/` | Routing, the header extractor, field mapping, computed columns and timestamp checks, and the enrichment lookups | +| `src/payload/`, `src/kafka/`, `src/column_meta/`, `src/schema/`, `src/metrics/` | The JSON gate and batch split, the transport adapters, column directives, table tags, metrics | | `src/buffer/`, `src/clickhouse/` | Per-table buffers, and the query client, schema cache and inserter | -| `tests/` | `smoke.rs`, `integration/`, `e2e/` -- see `tests/TESTING.md` | +| `tests/` | `smoke.rs`, `integration_tests.rs` (`integration/`, `e2e/`, `unit/`) -- see `tests/TESTING.md` | ### Commands that prove a change @@ -203,11 +198,11 @@ NOT a transform stage, NOT a schema manager. Its only outbound topic is the DLQ. make check # hyperi-ci check -- the full local gate cargo nextest run --lib # unit, no infra cargo nextest run --test smoke # startup smoke, no infra -cargo nextest run --test integration_tests # integration, some cases need ClickHouse -cargo nextest run -- --ignored # e2e, needs Kafka AND ClickHouse +cargo nextest run --features testcontainers # core, Docker for the shared ClickHouse and Kafka +cargo nextest run --features testcontainers --run-ignored all --ignore-default-filter # full ``` -Green lies three ways: `skip_if_no_clickhouse!()` and `skip_if_no_kafka!()` return early and PASS when the datastore or broker is absent, the e2e suite is `#[ignore]` so a plain run never touches the real insert path, and `default = []` compiles a feature-gated test out while CI still reports green. `ci.yml` sets `paths-ignore` for `docs/**` and `**.md`, so a docs-only push runs no jobs. +Green lies three ways. `require_clickhouse!()` returns early, and the test passes, when no ClickHouse is named and the build lacks `--features testcontainers`, or when a named one does not answer outside CI. The e2e and other real-server tests run only in the full tier. And `default = []` compiles a feature-gated test out while the run still reports green. `ci.yml` sets `paths-ignore` for `docs/**` and `**.md`, so a docs-only push runs no jobs, though its pull request still runs CI. ### What tends to bite @@ -215,22 +210,23 @@ Green lies three ways: `skip_if_no_clickhouse!()` and `skip_if_no_kafka!()` retu |-------|----|-----| | Bump scalo and leave `release.helm.library` behind | Move `release.helm.library` in `.hyperi-ci.yaml` to the same scalo version | The release assembles the chart from the emitted contract on that scalo-service version, and a scalo-service release ships the schema for only the contract version its scalo release writes | | Gate a test behind a cargo feature without adding it to `.hyperi-ci.yaml` | Add `default,` to the feature-set list | `default = []` (`Cargo.toml`), so it compiles out of every CI run and CI still reports green. `helm_contract.rs:64` asserts `transport-memory` and `testcontainers` stay listed | -| Inject a nested config key as `DFE_LOADER_SECTION_FIELD` | `DFE_LOADER__SECTION__FIELD` | figment strips exactly `DFE_LOADER_` and the double underscore separates nested keys. A single-underscore nested name matches no field and is dropped, which `tests/integration/config_reachability.rs` checks | -| Set `clickhouse.protocol: native` with `insert_format: json_each_row` | `rowbinary` over native, or `http` for JSONEachRow | JSONEachRow is an HTTP body format. `validate()` rejects the pair by name | +| Inject a nested config key as `DFE_LOADER_SECTION_FIELD` | `DFE_LOADER__SECTION__FIELD` or `DFE_LOADER_SECTION__FIELD` | `__` separates nested keys. A name with single underscores reaches only a key the flat list names (`apply_flat_env` in `src/config/loader.rs`), and anything else matches no field and is dropped. `tests/integration/config_reachability.rs` checks both nested spellings | +| Set `clickhouse.protocol: native` with `insert_format: json_each_row` | `rowbinary` over native, or `http` for JSONEachRow | JSONEachRow is an HTTP body format. `ClickHouseConfig::validate()` refuses the pair by name when the pipeline starts, and `config-check` does not catch it | | Name `clickhouse.tls.ca_cert_file`, `cert_file`, `key_file` or `skip_verify` | Set `tls.enabled`, mount the CA into the trust store | Only `enabled` reaches a client, and `validate()` rejects the certificate-file and `skip_verify` keys by name | -| Widen the `cel` range past scalo's | Keep it on `>=0.14.5, <0.15` | `cel::Program` crosses the scalo boundary. Wider resolves two semver-incompatible `cel` crates and `Program` stops being the same type (`Cargo.toml:46-51`) | +| Widen the `cel` range past scalo's | Keep it on scalo's range, beside the `cel` entry in `Cargo.toml` | `cel::Program` crosses the scalo boundary. Wider resolves two semver-incompatible `cel` crates and `Program` stops being the same type | | Take `clickhouse` from git or a `[patch]` | The crates.io release | `deny.toml` allows no git source, and `dfe-clickhouse` builds against the published `clickhouse` only | -| Mechanically sync dfe-engine's loader validation to `Config::validate()` | Read both, keep the divergence | dfe-engine scopes the broker check to the Kafka transport and adds a `grpc.listen` check this side lacks. A blind sync rejects valid gRPC-only configs at author time | +| Change a config key and leave `docs/config-schema.*` behind | `dfe-loader config-schema --dir docs` | `test_config_artifacts_do_not_drift` fails on a stale schema, and dfe-engine authors loader configs from the schema the contract carries | ### Where this sits Inbound, declared in `dfe-infra/suite.yaml`: -- **scalo-rs -> dfe-loader** (`cargo-dep`) -- `Cargo.toml:37` takes `scalo` by range for transports, config cascade, CLI, metrics, deployment contract and DLQ. A scalo release reaches this repo here. -- **scalo-rs -> dfe-loader** (`generated-file`, lockstep) -- `Dockerfile` is emitted by `scalo::deployment::generate_dockerfile()` at schema version 4. Regenerate and commit the diff. +- scalo-rs -> dfe-loader (`cargo-dep`): `Cargo.toml` takes `scalo` by range for transports, config cascade, CLI, metrics, deployment contract and DLQ. A scalo release reaches this repo here. +- scalo-rs -> dfe-loader (`generated-file`, lockstep): `Dockerfile` is emitted by `scalo::deployment::generate_dockerfile()` at the schema version its header names. Regenerate with `dfe-loader --emit-dockerfile` and commit the diff. + Outbound, so what a change here can break: -- **dfe-loader -> dfe-infra** (`image-pin`, lockstep) -- `dfe-infra/versions.yaml` pins this image and the thin chart released here, and Argo deploys that published chart. -- **dfe-loader -> dfe-engine** (`mirrored-logic`) -- dfe-engine hand-reimplements this repo's config validation in `plugins_builtin/loader.py` to catch errors when it authors a config. Nothing is copied, so no script compares them. +- dfe-loader -> dfe-infra (`image-pin`, lockstep): `dfe-infra/versions.yaml` pins this image and the thin chart released here, and Argo deploys that published chart. +- dfe-loader -> dfe-engine: dfe-engine takes a loader config's defaults, types and descriptions from the `config-schema.json` this repo's contract carries (dfe-engine `src/dfe_engine/appmgmt/contract.py`). -Runtime, not build edges: dfe-engine's `dfe-schema` entry point creates every destination table before the loader can insert, and producers reach the loader over Kafka (`*_land` / `*_load` topics) or direct gRPC (default topic `main_land`) -- in the suite that producer is dfe-receiver. +Runtime, not build edges: dfe-engine applies the dfe-schemas manifest at boot, creating every destination table before the loader can insert. Producers reach the loader over Kafka or direct gRPC (default topic `main_land`): in the suite dfe-receiver and dfe-fetcher produce `*_land` topics, the transforms produce `*_load`, and dfe-receiver and the transforms push over gRPC. diff --git a/README.md b/README.md index ab345dd1..2b1c435c 100644 --- a/README.md +++ b/README.md @@ -13,13 +13,15 @@ [![Build Status](https://github.com/hyperi-io/dfe-loader/actions/workflows/ci.yml/badge.svg)](https://github.com/hyperi-io/dfe-loader/actions) [![License](https://img.shields.io/badge/license-BUSL--1.1-blue)](LICENSE) -Loads Kafka and gRPC streams into ClickHouse, batching small writes into large inserts. In a DFE deployment it is the last hop of the core data path, taking what dfe-receiver sends over Kafka or direct gRPC and landing it in the tables dfe-schemas defines. +Loads Kafka and gRPC streams into ClickHouse in large batched inserts. + +In a DFE deployment it is the last hop of the core data path, taking what dfe-receiver, dfe-fetcher and the transforms send over Kafka, or what dfe-receiver and the transforms push over direct gRPC, and landing it in the tables dfe-schemas defines. ## What it gives you - Routing to `db.table` from fields in each record, so one loader can feed many tables. - Columns read from the live table's `system.columns`: matching fields become typed columns and the rest stays in `_json`, with no schema compiled in. -- RowBinary inserts by default, so ClickHouse does no JSON parsing on the way in. +- RowBinary inserts by default, so ClickHouse parses no row text and infers no types. A JSON column's value, `_json` included, still arrives as JSON text for the server to parse. - Per-table buffers flushed on row count, byte size or age, whichever comes first. - At-least-once delivery by default: Kafka offsets commit once per flush, never past a row not yet in ClickHouse or the DLQ. `kafka.acknowledgements.enabled: false` commits at receipt instead, and a row only a DLQ could take is dropped and counted in `dfe_loader_rows_lost_total` when no DLQ is working. - A batch with a bad row is split until only the bad rows are dead-lettered, and a schema drift error refreshes the cached schema and retries. @@ -36,14 +38,7 @@ cargo build That prints `[ok] configuration is valid` and the resolved config, and opens no connections. `--config` is a global option, so it goes BEFORE the subcommand -- `config-check --config ...` is rejected by the argument parser. -To move real data, bring up a local broker and datastore: - -```bash -docker compose -f docker-compose.dev.yaml up -d -cargo run -- --config config.dev.yaml -``` - -Metrics land on `http://localhost:9090/metrics`, with `/livez` and `/readyz` alongside. `docker compose -f docker-compose.dev.yaml down -v` takes it down. +Moving real data needs a broker or a gRPC sender, and the DFE tables, which the loader never creates. [dfe-docker](https://github.com/hyperi-io/dfe-docker) brings up the whole stack with the tables in place. A running loader serves metrics on `http://localhost:9090/metrics`, with `/livez` and `/readyz` alongside. ## What it does @@ -56,16 +51,16 @@ dfe-loader reads records off Kafka or a direct gRPC listener, works out which Cl | Promote | Schema read from `system.columns`, matching fields become typed columns, the rest stays in `_json` | | Enrich | Optional GeoIP, reputation and risk columns | | Buffer | Per-table, flushed on row count, byte size or age | -| Insert | RowBinary by default, so ClickHouse skips JSON parsing. JSONEachRow is the fallback | +| Insert | RowBinary by default, so ClickHouse parses no row text. JSONEachRow is the fallback | | Commit | Kafka offsets commit once per flush cycle, never past a row not yet in ClickHouse or the DLQ -- at-least-once | -JSON is the only payload format. MessagePack, supported in DFE/XDR 2.0 and 2.1, is deprecated in DFE 2.2 and no longer accepted: the JSON path (SIMD parsing with sonic-rs, zstd on the wire) is fast enough that MessagePack gave no CPU saving. +JSON is the only payload format. MessagePack, which DFE/XDR 2.0 and 2.1 accepted, was removed in DFE 2.2, and a MessagePack record is dead-lettered as not JSON: the JSON path (SIMD parsing with sonic-rs, zstd on the wire) is fast enough that MessagePack gave no CPU saving. Failures degrade rather than stall: a batch with a bad row is binary-split so only the bad rows are dead-lettered, a batch whose insert fails is held and retried with backoff while the commit stays below it, and a schema drift error invalidates the cached schema and retries. ## Configuration essentials -One YAML file, named by `--config`. The chart mounts it at `/etc/dfe/loader.yaml`. `config.example.yaml` is the annotated reference, and every setting has a default, so an empty config boots. +One YAML file, named by `--config` or `DFE_LOADER_CONFIG`. With neither set the loader reads `./config.yaml` or `./config.yml` if one exists, but only a named file is hot-reloaded. The chart mounts it at `/etc/dfe/loader.yaml`. `config.example.yaml` is an annotated example, `docs/config-schema.yaml` lists every key with its default, and every setting has a default, so an empty config boots. Environment overrides use the `DFE_LOADER` prefix in two forms, and the difference matters: `DFE_LOADER_KAFKA_GROUP_ID` is the flat single-underscore set, a fixed contract with dfe-engine, while `DFE_LOADER__CLICKHOUSE__DATABASE` uses double underscores to nest into any config key. @@ -77,7 +72,7 @@ Environment overrides use the `DFE_LOADER` prefix in two forms, and the differen | `clickhouse.insert_format` | `rowbinary` | `jsoneachrow` is the fallback | | `clickhouse.tls.enabled` | unset | The ONLY honoured TLS key. Mount a private CA into the container trust store rather than naming cert files | | `kafka.group` | `dfe-loader` | | -| `routing.default_db` / `default_table` | `dfe` / `main` | Where a record lands when no routing field matches | +| `routing.default_db` / `default_table` | `dfe` / `main` | Where a record lands when neither a routing field nor its topic names a table, or the table it names does not exist | | `buffer.flush_rows` / `flush_bytes` / `flush_age_secs` | `20000` / `67108864` (64 MiB) / `5` | Per-table flush triggers, each greater than 0 | | `metrics.address` | `0.0.0.0:9090` | | @@ -94,7 +89,7 @@ Full cascade and per-setting behaviour: [docs/configuration.md](docs/configurati | Running, metrics, hot-reload, DLQ | [docs/operations.md](docs/operations.md) | | Insert formats, schema cache, types, TLS, DDL directives | [docs/clickhouse/](docs/clickhouse/) | | Routing, coercion, enrichment, capture modes | [docs/pipeline/](docs/pipeline/) | -| Kafka, gRPC, the common header | [docs/transport/](docs/transport/) | +| gRPC, the common header | [docs/transport/](docs/transport/) | | Container and chart publishing | [docs/deployment/](docs/deployment/) | | Test layout and what needs infra | [tests/TESTING.md](tests/TESTING.md) | diff --git a/docs/README.md b/docs/README.md index f7c51092..96cf454d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -10,10 +10,7 @@ # dfe-loader docs -High-performance loader from message transports (Kafka, gRPC, Memory) into -ClickHouse. Point it at a topic, give it a routing rule, and it parses, routes, -promotes schema columns, enriches, buffers per table, and inserts -- RowBinary -by default so ClickHouse skips JSON parsing, with at-least-once Kafka delivery. +Loads records from Kafka or a gRPC push into ClickHouse. Point it at a topic, give it a routing rule, and it parses, routes, promotes schema columns, enriches, buffers per table, and inserts -- RowBinary by default so ClickHouse parses no row text, with at-least-once Kafka delivery. This is the index. Read [architecture.md](architecture.md) for the 10,000-foot view, [clickhouse/](clickhouse/) for the insert layer, and @@ -25,11 +22,11 @@ view, [clickhouse/](clickhouse/) for the insert layer, and | You give it | It handles | You don't write | |-------------|------------|-----------------| -| A Kafka topic or a gRPC push | JSON parse (sonic-rs, SIMD), zero-copy `Arc<[u8]>`, anything not JSON dead-lettered | A consumer, a parser, an input gate | +| A Kafka topic or a gRPC push | JSON parse (sonic-rs, SIMD), the payload kept once as `Arc<[u8]>`, anything not JSON dead-lettered | A consumer, a parser, an input gate | | The routing fields (`table_fields`, `org_routes`) | Pre-flatten routing to `db.table`, dot-notation nested access, the default table on a miss | A router, a dispatch table | | A ClickHouse table | Schema reflected from `system.columns`, fields promoted to typed columns, the rest kept in `_json` | A schema mapping, an ORM, a migration | -| `insert_format` (default RowBinary) | Dynamic `Map` -> RowBinary over the HTTP `Client`; JSONEachRow fallback | A serialiser, a type encoder, a wire format | -| `capture_mode` (full/raw_only/json_only/extracted_only) | `_json` and `_raw` population per table, overridable per-table and per-DDL | Conditional capture plumbing | +| `insert_format` (default RowBinary) | Dynamic `Map` -> RowBinary over HTTP, or Native blocks over the native protocol, with a JSONEachRow fallback over HTTP | A serialiser, a type encoder, a wire format | +| `capture_mode` (full/raw_only/json_only/extracted_only) | `_json` and `_raw` population per table, overridable per-table and per-DDL | Per-table capture logic | | Enrichment toggles (geoip/reputation/risk) | Flat enriched columns injected on the promoted row only | Lookup, caching, CIDR matching | | Nothing extra | Per-table buffering, batch salvage, held-batch retry, schema-cache recovery, one offset commit per flush cycle | The resilience layer | @@ -44,7 +41,6 @@ flowchart TB subgraph Ingress["Transport (scalo)"] K["Kafka"] G["gRPC"] - M["Memory"] end subgraph Hot["Hot path (per message)"] @@ -57,19 +53,21 @@ flowchart TB subgraph Sink["ClickHouse insert (dfe_clickhouse::dynamic::DynamicInsert)"] CE["DynamicRow encoder
Map + schema -> binary"] - HTTP["HTTP: insert_formatted_with
FORMAT RowBinaryWithDefaults"] + HTTP["HTTP: insert_formatted_with
FORMAT RowBinaryWithNamesAndTypes"] + NAT["native: insert_native
FORMAT Native"] JF["HTTP: insert_formatted_with
FORMAT JSONEachRow (fallback)"] end - CH[("ClickHouse")] + CH[(ClickHouse)] DLQ["DLQ (Kafka primary, file fallback)"] - K & G & M --> P --> R --> X --> E --> B + K & G --> P --> R --> X --> E --> B B -->|RowBinary default| CE CE --> HTTP --> CH + CE --> NAT --> CH B -->|insert_format=json_each_row| JF --> CH - B -.salvaged bad rows.-> DLQ - CH -.success.-> Commit["One Kafka commit per flush cycle"] + B -.->|salvaged bad rows| DLQ + CH -.->|success| Commit["One Kafka commit per flush cycle"] ``` --- @@ -84,7 +82,7 @@ flowchart TB | Build features | [feature-flags.md](feature-flags.md) | Cargo features and the ClickHouse client crates | | Insert layer | [clickhouse/](clickhouse/) | RowBinary vs JSONEachRow, the dynamic insert, schema cache, type handling, TLS, DDL directives | | Hot path | [pipeline/](pipeline/) | Routing, extraction, enrichment, capture modes, buffering | -| Transports | [transport/](transport/) | Kafka / gRPC / Memory, the common header | +| Transports | [transport/](transport/) | The gRPC listener, the common header | | Deployment | [deployment/](deployment/) | Container + chart publishing, schemas | | Performance | [performance/](performance/) | Hot-path optimisations, parser-selection history, PGO/BOLT | diff --git a/docs/architecture.md b/docs/architecture.md index be5f8af7..df65a2da 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -13,7 +13,7 @@ dfe-loader is a per-message hot path with a per-table batching tail. A message arrives as bytes, gets routed and promoted into a typed row, accumulates in a per-table buffer, and flushes to ClickHouse as a batch. Kafka offsets commit once per flush cycle, never past a row not placed yet -- at-least-once. Everything on the hot path follows one rule: do the cheap thing per -message, defer the expensive thing to flush. Parse once (SIMD, zero-copy), +message, defer the expensive thing to flush. Parse once (SIMD), promote only schema columns, keep the full payload as a reference-counted `Arc<[u8]>`, and splice it as `_json` only at serialise time. @@ -35,7 +35,7 @@ flowchart TB INS["Inserter
format dispatch, retry, salvage"] end subgraph L1["L1 -- ClickHouse layer"] - QC["ClickHouseQueryClient
DDL, schema, health"] + QC["ClickHouseQueryClient
schema, comments, health"] EXT["dfe_clickhouse::dynamic
ParsedType, encoder, DynamicInsert"] SC["SchemaCache"] end @@ -77,7 +77,7 @@ flowchart LR end end subgraph client["clickhouse::Client"] - IFW["insert_formatted_with
FORMAT RowBinaryWithDefaults | JSONEachRow (HTTP)"] + IFW["insert_formatted_with
FORMAT RowBinaryWithNamesAndTypes | JSONEachRow (HTTP)"] end INS --> DI @@ -87,10 +87,7 @@ flowchart LR INS -->|JSONEachRow fallback| IFW ``` -Over HTTP every insert goes through `insert_formatted_with`: one -`FORMAT RowBinaryWithDefaults` INSERT per batch, or `JSONEachRow`. With -`clickhouse.protocol: native` the rows go out as `FORMAT Native` blocks through -`TcpClient::insert_native`, and `JSONEachRow` is refused at startup. See [clickhouse/clickhouse-ext.md](clickhouse/clickhouse-ext.md). +Over HTTP every insert goes through `insert_formatted_with`: `FORMAT RowBinaryWithNamesAndTypes`, one INSERT for each set of defaulted columns the batch's rows supply, or `JSONEachRow`. With `clickhouse.protocol: native` the rows go out as `FORMAT Native` blocks through `TcpClient::insert_native`, and `JSONEachRow` is refused at startup. See [clickhouse/clickhouse-ext.md](clickhouse/clickhouse-ext.md). ## Insert dispatch @@ -98,7 +95,7 @@ Over HTTP every insert goes through `insert_formatted_with`: one flowchart TB F{"insert_format?"} RB["DynamicInsert.write_map(s)"] - HTTP["Client.insert_formatted_with
FORMAT RowBinaryWithDefaults (HTTP)"] + HTTP["Client.insert_formatted_with
FORMAT RowBinaryWithNamesAndTypes (HTTP)"] JE["Client.insert_formatted_with
FORMAT JSONEachRow (HTTP)"] SM{"SchemaMismatch?"} REC["invalidate schema cache
re-fetch + retry"] @@ -107,11 +104,11 @@ flowchart TB F -->|json_each_row| JE HTTP --> SM SM -->|yes| REC --> RB - SM -->|no| OK["offsets join the flush cycle's one commit"] + SM -->|no| OK["offsets join the one commit of the flush cycle"] ``` -The RowBinary path ships `FORMAT RowBinaryWithDefaults` through -`insert_formatted_with`. A refusal is the rows' own only once a fresh read shows +The RowBinary path ships `FORMAT RowBinaryWithNamesAndTypes` through +`insert_formatted_with`, and the header's types make the server refuse rows encoded against a type the column no longer has. A refusal is the rows' own only once a fresh read shows the table unchanged since they were encoded, and otherwise the retry encodes against the current schema. Over native the block declares the types. @@ -131,20 +128,16 @@ per-table config > global default. |------|---------|--------|-----| | `full` (default) | full payload (JSON type) | the record's own raw field, else NULL | path queries + text search | | `json_only` | full payload (JSON type) | NULL | path queries without the `_raw` overhead | -| `raw_only` | NULL | full Kafka payload (String) | CH CPU saving, no JSON-type overhead | -| `extracted_only` | NULL | NULL | promoted columns only | +| `raw_only` | NULL, unless a directive fills it | full Kafka payload (String) | CH CPU saving, no JSON-type overhead | +| `extracted_only` | NULL, unless a directive fills it | NULL | promoted columns only | -All four still promote schema columns; the difference is whether (and where) -the full payload is kept. Under `full` a JSON record is stored once, in `_json`: `_raw` holds only raw data the source sent with it, such as a syslog line the receiver captured. +All four still promote schema columns, and differ only in whether, and where, +the full payload is kept. A directive on `_json` makes it a promoted column under every mode ([pipeline/capture-modes.md](pipeline/capture-modes.md)). Under `full` a JSON record is stored once, in `_json`: `_raw` holds only raw data the source sent with it, such as a syslog line the receiver captured. ## Resilience -- **Batch salvage** -- on a data error, binary-split the batch to isolate the - bad row(s), DLQ only those, keep the good ones. -- **Sink-down gate** -- a flush cycle in which every insert failed opens the - circuit gate on the KEDA scaling signal, and the next successful insert closes - it. A failed insert never sends its batch to the DLQ. -- **Schema-cache recovery** -- on `SchemaMismatch` (e.g. `ALTER ... ADD COLUMN`), - invalidate and re-fetch, then retry. -- **One commit per flush cycle** -- offsets commit once per cycle, and on each partition stop below the lowest offset not placed yet. A failed batch or a dead letter the DLQ refused is held and retried with jittered backoff, and holds its partition's commit until it lands. A dead letter no DLQ backend can ever hold is dropped and counted instead (`pipeline_dead_letters_dropped_total{reason}`). A held batch whose table ClickHouse has since reported absent is retried against the default table. -- **Partition leases** -- rows of a partition a rebalance has moved away are discarded before any write, since the partition's next owner reads them again from the committed offset (`transport_revoke_discarded_total{stage="buffer"}`). See [operations.md](operations.md#rebalances). +- A data error binary-splits the batch down to the bad rows, which go to the DLQ, and the rest land. +- A flush cycle in which every insert failed opens the circuit gate on the KEDA scaling signal, and the next successful insert closes it. A failed insert never sends its batch to the DLQ. +- Drift, a column dropped or retyped by `ALTER`, drops the cached schema, and the retry encodes against a fresh read. An added column breaks no insert, because the INSERT names its columns, so it takes its default until the cached schema refreshes. +- Offsets commit once per cycle, and on each partition stop below the lowest offset not placed yet. A failed batch or a dead letter the DLQ refused is held and retried with jittered backoff, and holds its partition's commit until it lands. A dead letter no DLQ backend can ever hold is dropped and counted instead (`pipeline_dead_letters_dropped_total{reason}`). A held batch whose table ClickHouse has since reported absent is retried against the default table. +- Rows of a partition a rebalance has moved away are discarded before any write, since the partition's next owner reads them again from the committed offset (`transport_revoke_discarded_total{stage="buffer"}`). See [operations.md](operations.md#rebalances). diff --git a/docs/clickhouse/clickhouse-ext.md b/docs/clickhouse/clickhouse-ext.md index 7f63255f..c6220e96 100644 --- a/docs/clickhouse/clickhouse-ext.md +++ b/docs/clickhouse/clickhouse-ext.md @@ -23,7 +23,7 @@ The official `clickhouse` crate keeps insertion typed (`T: Row`). dfe-loader ins | `DynamicInsert` | `write_map` / `write_map_with_raw`, then `end` | | `shape_for_type` | The same value rules applied to a row map, for the JSONEachRow path | -Over HTTP an insert is one `INSERT ... FORMAT RowBinaryWithDefaults`: a row that omits a defaulted column takes the server's default, so one INSERT carries rows with differing columns, and rows go out in 1 MiB chunks as they are written. Over the native protocol rows go out as `FORMAT Native` blocks, and a row that supplies a different set of defaulted columns starts a new INSERT. JSON columns are written as text with `input_format_binary_read_json_as_string=1`. +Over HTTP an insert is `INSERT ... FORMAT RowBinaryWithNamesAndTypes`, whose header declares each column's cached type, sent in 1 MiB chunks as rows are written. An INSERT names the required columns and the defaulted ones its rows supply, so rows that supply a different set go to an INSERT of their own, up to four open at once. Over the native protocol rows go out as `FORMAT Native` blocks, and a row that supplies a different set of defaulted columns starts a new INSERT. JSON columns are written as text, with `input_format_binary_read_json_as_string=1` over HTTP. The coercions a source value gets on the way: a zoned timestamp lands at the instant it names, a numeric DateTime64 epoch is read by magnitude and rescaled to the column's precision, an IPv4 literal lands in an IPv6 column as `::ffff:a.b.c.d`, a JSON column wraps a root array under `list` and a scalar or non-JSON text under `value`, a `Point` takes every Elasticsearch geo_point form inside its coordinate ranges, and one value lands in an `Array` as a one-element array. A value the column cannot hold (an integer past its width, a decimal past its backing, an undeclared enum value) fails the row rather than being written wrong. `tests/integration/insert_coercions.rs` runs each case through the loader's `Inserter` on every protocol the shared ClickHouse offers. diff --git a/docs/clickhouse/ddl-directives.md b/docs/clickhouse/ddl-directives.md index ec333a08..10d0214c 100644 --- a/docs/clickhouse/ddl-directives.md +++ b/docs/clickhouse/ddl-directives.md @@ -12,10 +12,7 @@ dfe-loader reads the schema from ClickHouse, not from a config file -- but a plain `CREATE TABLE` does not say where each column's value comes from. That is -what these `@`-comment directives do. You write them as comments above (or -beside, in CSV) each column, and the data-prep stage reads the ones it acts on -to decide whether a column is read from a source field path or computed from a -CEL expression. +what these `@`-comment directives do. They live in each column's `COMMENT`, which the loader reads from `system.columns`: a SQL `--` line comment never reaches the server. dfe-schemas renders a column's `expr` into that `COMMENT`, and the data-prep stage reads the directives it acts on to decide whether a column is read from a source field path or computed from a CEL expression. Five directives change loader behaviour: `@skip`, `@default`, `@source`, `@renamed` and `@computed` (`DIRECTIVE_NAMES` in @@ -31,7 +28,7 @@ elsewhere. This page is the full reference -- grammar, the directive list, what a `@computed` expression may use, and table-level tags. -For the column types these directives target, see [types.md](types.md); for how +For the column types these directives target, see [types.md](types.md). For how capture interacts with `_json` and `_raw`, see capture modes in [../architecture.md](../architecture.md). @@ -42,7 +39,7 @@ flowchart TB COL["Column in reflected schema
+ @-comment directive"] K{"Which directive?"} SRC["@source: field
read input, optional fallback"] - REN["@renamed: field
same source-path mapping as @source
no-op if destination exists"] + REN["@renamed: field
same source-path mapping as @source
read in place of the column name"] CMP["@computed: expression
evaluate one CEL expression"] OUT["Value written to output column"] GEN["@generated: expr
(descriptive label only)"] @@ -50,18 +47,18 @@ flowchart TB DEF["ClickHouse DEFAULT/MATERIALIZED/ALIAS
on the column itself"] COL --> K - K -->|"@source"| SRC - K -->|"@renamed"| REN - K -->|"@computed"| CMP + K -->|source| SRC + K -->|renamed| REN + K -->|computed| CMP SRC --> OUT REN --> OUT CMP --> OUT - GEN -. describes .-> DEF + GEN -.->|describes| DEF DEF -.-> SKIP["field omitted from insert"] ``` `@generated` and `@config` sit outside the loader's directive dispatch: the -loader parses neither. They are notes an author leaves in the comment; the +loader parses neither. They are notes an author leaves in the comment, and the real mechanism sits elsewhere, covered under each heading below. ## Overview @@ -88,21 +85,16 @@ Map the column to a source field path during data prep. | Syntax | Description | |--------|-------------| | `@source: field` | Read from source field, NULL if missing | -| `@source: field \| now()` | Read from source, use `now()` if missing/invalid | -| `@source: field \| default_value` | Read from source with custom fallback | +| `@source: field \| now()` | Read from source, use `now()` if missing or null | +| `@source: field \| "value"` | Read from source with a literal fallback | | `@source: first(a/b/c)` | First non-null from ordered list (/ separator) | **Examples:** ```sql --- @source: timestamp | now() -`event_time` DateTime64(3), - --- @source: first(user_id/uid/id) -`user_id` String, - --- @source: status_code -`http_status` UInt16, +`event_time` DateTime64(3) COMMENT '@source: timestamp | now()', +`user_id` String COMMENT '@source: first(user_id/uid/id)', +`http_status` UInt16 COMMENT '@source: status_code', ``` ### @generated @@ -116,8 +108,7 @@ The loader still omits the field from the insert, but by a different route: it reads `default_kind` for the column from `system.columns`, and any column whose kind is `DEFAULT`, `MATERIALIZED`, `ALIAS` or `EPHEMERAL` is marked `has_default` and can be left out of the row it sends (the dynamic insert in -`dfe-clickhouse`'s `dynamic` module). The -applier (dfe-engine's schema module) writes that DEFAULT/MATERIALIZED/ALIAS +`dfe-clickhouse`'s `dynamic` module). dfe-schemas' renderer, whose DDL dfe-engine applies, writes that DEFAULT/MATERIALIZED/ALIAS clause from the schema's own `default:` and `attribute:` fields -- not by reading the `@generated:` text -- so keep the two in agreement by hand. @@ -134,11 +125,8 @@ reading the `@generated:` text -- so keep the two in agreement by hand. **Examples:** ```sql --- @generated: now64(3) -`inserted_at` DateTime64(3) DEFAULT now64(3), - --- @generated: generateUUIDv7() -`id` UUID DEFAULT generateUUIDv7(), +`inserted_at` DateTime64(3) DEFAULT now64(3) COMMENT '@generated: now64(3)', +`id` UUID DEFAULT generateUUIDv7() COMMENT '@generated: generateUUIDv7()', ``` ### Capture is not a column directive @@ -156,13 +144,12 @@ writes to `_raw` and `_json`. Not a directive the loader parses either -- same drop-as-unknown path as `@generated`. It is a note that the field beside it is not hardcoded: some loader config key decides the actual mapping. Nothing reads the comment text -to resolve that path; the config key is wired up separately, by the column's +to resolve that path: the config key is wired up separately, by the column's special-cased name. `_org_id`, for example, is filled from whichever field -`routing.org_id_field` names, and `_tags` from the first field in -`metadata.tags_fields` -- both real, live config +`routing.org_id_field` names, whatever its directive says, and `_tags` from its own `@source` directive, else the first field in `metadata.tags_fields` (dfe-loader `src/config/pipeline.rs`, `src/routing/router.rs`, `src/transform/extractor.rs`, `src/transform/transformer.rs`). Change the -config path and the mapping changes; change the `@config:` comment on its own +config path and the mapping changes. Change the `@config:` comment on its own and nothing does. ```text @@ -173,16 +160,11 @@ and nothing does. **Examples:** ```sql --- @source: _org_id --- @config: routing.org_id_field (default: "_org_id") -`_org_id` String, - --- @source: first(tags/_tags/meta) --- @config: metadata.tags_fields -`_tags` JSON, +`_org_id` String COMMENT '@source: _org_id @config: routing.org_id_field (default: "_org_id")', +`_tags` JSON COMMENT '@source: first(_tags/tags/meta) @config: metadata.tags_fields', ``` -### @computed +## @computed A value computed from the record's other fields during data prep, as one CEL expression. @@ -219,14 +201,9 @@ risk come from the loader's `geoip` and `enrichment` config, not from **Examples:** ```sql --- @computed: risk_score > 80 ? "high" : risk_score > 40 ? "medium" : "low" -`risk_label` LowCardinality(String), - --- @computed: src_ip.startsWith("10.") || src_ip.startsWith("192.168.") -`is_private_src` Bool, - --- @computed: bytes_in + bytes_out -`bytes_total` UInt64, +`risk_label` LowCardinality(String) COMMENT '@computed: risk_score > 80 ? "high" : risk_score > 40 ? "medium" : "low"', +`is_private_src` Bool COMMENT '@computed: src_ip.startsWith("10.") || src_ip.startsWith("192.168.")', +`bytes_total` UInt64 COMMENT '@computed: bytes_in + bytes_out', ``` **Semantics:** @@ -247,7 +224,7 @@ The `computed_columns` config section sets or overrides the same expressions: one. A per-table override beats the global entry, which beats the column comment. -### @renamed +## @renamed The hand-written form of `@source`: the same source-path mapping, resolved into the same `renamed` list, with no transformation of the value. @@ -264,31 +241,23 @@ the same `renamed` list, with no transformation of the value. **Semantics:** -- **No-op if destination exists**: If the destination field is already - present in the data, the mapping is skipped. This prevents overwriting - upstream-populated fields. -- **First-match wins**: With `first()`, fields are tried in order and the first - present one is used. -- **Not on `_org_id` or `_timestamp`**: both read fixed fields and ignore the directive, so no column comment can point the tenant at a field a sender writes. +- The listed paths are read in place of the column's own name, so a record field named like the column is not read while the directive names others. Only the field-mapping step, which runs after, skips a destination already present. +- With `first()`, fields are tried in order and the first present, non-null one is used. +- `_org_id` and `_timestamp` read fixed fields and ignore the directive, so no column comment can point the tenant at a field a sender writes. **Examples:** ```sql --- @renamed: timestamp -`event_time` DateTime64(3), - --- @renamed: org -`org_name` String, - --- @renamed: first(logoriginal/_raw/raw/raw_log/message) -`_raw` Nullable(String) CODEC(ZSTD(3)), +`event_time` DateTime64(3) COMMENT '@renamed: timestamp', +`org_name` String COMMENT '@renamed: org', +`_raw` Nullable(String) CODEC(ZSTD(3)) COMMENT '@renamed: first(logoriginal/_raw/raw/raw_log/message)', ``` ## Operators ### Fallback operator (`|`) -Provides a fallback when the source is missing or invalid: +Provides a fallback when every source path is missing or null: ```text @source: field | fallback_expression @@ -331,145 +300,51 @@ Separates alternatives in `first()`: - Uses first non-null value - Returns NULL if all missing -## CSV schema format - -```csv -column,type,default,nullable,codec,source,comment -``` - -The `source` column contains the expression: - -```csv -column,type,default,nullable,codec,source,comment -event_time,DateTime64(3),,false,Delta/ZSTD(1),@source: timestamp | now(),When event occurred -id,UUID,generateUUIDv7(),false,,@generated: generateUUIDv7(),Unique identifier -tags,JSON,,true,ZSTD(3),@source: first(tags/_tags/meta),Event metadata -``` - -## SQL DDL format - -Expressions appear as comments above the column definition: - -```sql -CREATE TABLE events ( - -- @generated: now64(3) - `inserted_at` DateTime64(3) DEFAULT now64(3), - - -- @source: timestamp | now() - `event_time` DateTime64(3), - - -- @source: user_id - -- @config: routing.user_id_field - `user_id` String, - - -- @source: first(tags/_tags/metadata.tags) - -- @config: metadata.tags_fields - `tags` JSON -) -``` - ## Data prep behaviour -### Field extraction - -1. **@source** fields are read from input, optionally transformed, written to - output -2. **@generated** fields are omitted, but not because data prep reads the - directive -- the column carries a ClickHouse DEFAULT/MATERIALIZED/ALIAS and - the loader reads that from `system.columns` instead. See the @generated - section above. -3. **@computed** fields are calculated during transformation -4. **@renamed** fields are read the same way as **@source** (no-op if the - destination is already present) - -### Input vs output names - -Expressions document the **input** field name. The column name is the **output**: +dfe-schemas renders the directives into the column `COMMENT`, with the human description after ` - `, which the parser trims: ```sql --- @source: timestamp <- reads "timestamp" from input -`_timestamp` DateTime64(3) <- writes to "_timestamp" in output +CREATE TABLE dfe.main ( + `_timestamp_load` DateTime64(3,'UTC') DEFAULT now64(3) COMMENT '@generated: now64(3) - Insertion timestamp (ms precision)', + `_timestamp` DateTime64(3,'UTC') COMMENT '@source: timestamp | now() - Event timestamp from source data', + `_source` LowCardinality(Nullable(String)) COMMENT '@source: first(_source) | topic_name - Data source label (e.g. beats, syslog, crowdstrike-edr)', + ... +) ``` -### Field removal - -By default, routing/extracted fields can be removed after extraction to avoid -duplication. This is configurable per deployment. - -## Implementation reference - -The data prep stage (transformer) implements these expressions: +On the default `json_primary` path the header extractor (`src/transform/extractor.rs`) resolves them, and on `legacy_flatten` the transformer and the field mapping do: | Expression | Implementation | |------------|----------------| | `@source: field`, `@renamed: field` | One entry in the column's `renamed` source-path list | -| `@source: field \| now()` | Fallback to `Utc::now()` if missing/invalid | +| `@source: field \| now()` | Fallback to `Utc::now()` when every path is missing or null | | `@source: first(a/b)`, `@renamed: first(a/b)` | Try each path in order, first non-null wins | | `@computed: expr` | Compiled once with `scalo::expression::compile`, evaluated per record | -`@generated: expr` is not in this table because the transformer does not -implement it -- it is dropped as an unknown directive. The column is still -omitted from the insert, by the `system.columns` `default_kind` route -described under @generated above. `@config: path` is absent for the same -reason: the transformer never reads it. +`@generated` and `@config` are dropped as unknown directives. A `@generated` column is still left out of the insert, by the `system.columns` `default_kind` route described under @generated above. A directive names the input field, and the column name is the output: `@source: timestamp` on `_timestamp` reads `timestamp`. With `metadata.remove_routing_fields` (default `true`) the routing fields are removed after extraction. ## Table-level tags -Tables support metadata tags stored in the ClickHouse `COMMENT` field, using the -same `@tag: key=value` syntax as column expressions. - -### Syntax - -Tags are stored in the table COMMENT, separated by ` | `: +A table `COMMENT` carries tags in the same `@tag: value` syntax, separated by ` | `. The loader reads one, `@capture_mode`, which overrides the configured capture mode for that table (see [capture-modes.md](../pipeline/capture-modes.md)) and ignores the rest. dfe-schemas writes `@profile` and `@profile_version`, naming the common-header profile the table was built from: ```sql -CREATE TABLE common.events (...) -ENGINE = MergeTree() +CREATE TABLE dfe.main (...) ... -COMMENT '@schema_source: core | @schema_version: 2 | @created_by: dfe-loader' +COMMENT '@profile: timeseries | @profile_version: | @capture_mode: json_only' ``` -### Standard tags - -| Tag | Values | Description | -|-----|--------|-------------| -| `@schema_source` | `core`, `user` | `core` = pre-supplied by dfe-loader, absent/`user` = user-created | -| `@schema_version` | `1`, `2`, ... | Schema version for migrations | -| `@created_by` | string | Tool/user that created the table | - -### Querying by tag - ```sql --- Find all core schemas -SELECT database, name, comment -FROM system.tables -WHERE comment LIKE '%@schema_source: core%'; - --- Find tables by version SELECT database, name, comment FROM system.tables -WHERE comment LIKE '%@schema_version: 2%'; - --- Extract tag value using regex -SELECT - database, - name, - extractAll(comment, '@schema_source: ([^|]+)')[1] AS schema_source -FROM system.tables -WHERE comment LIKE '%@schema_source:%'; +WHERE comment LIKE '%@capture_mode:%'; ``` -### Rust API +`TableTags` (`src/schema/mod.rs`) parses and builds them: ```rust use dfe_loader::schema::TableTags; -// Build a tag set -let tags = TableTags::new() - .with("schema_source", "core") - .with("schema_version", "2"); - -// Parse tags from an existing table comment -let tags = TableTags::from_comment("@schema_source: core | @schema_version: 2"); -assert_eq!(tags.get("schema_source"), Some("core")); +let tags = TableTags::from_comment("@profile: timeseries | @capture_mode: json_only"); +assert_eq!(tags.get("capture_mode"), Some("json_only")); ``` diff --git a/docs/clickhouse/insert-formats.md b/docs/clickhouse/insert-formats.md index a414df62..2bc6f59a 100644 --- a/docs/clickhouse/insert-formats.md +++ b/docs/clickhouse/insert-formats.md @@ -12,7 +12,7 @@ dfe-loader has two knobs for how rows reach ClickHouse: `clickhouse.protocol` (how the bytes travel) and `clickhouse.insert_format` (how the rows are -encoded). The default `http + row_binary` is the fast path; `json_each_row` is +encoded). The default `http + row_binary` is the fast path, and `json_each_row` is a deliberate fallback. `protocol: native` runs the schema fetch, the queries and `row_binary` inserts over the native protocol through `dfe_clickhouse::TcpClient`. @@ -21,11 +21,11 @@ flowchart TB ROW["Buffered rows
Map<String,Value> + raw _json"] F{"insert_format"} T{"protocol"} - RBH["insert_formatted_with
FORMAT RowBinaryWithDefaults (HTTP)"] + RBH["insert_formatted_with
FORMAT RowBinaryWithNamesAndTypes (HTTP)"] RBT["TcpClient::insert_native
FORMAT Native blocks (TCP)"] JE["insert_formatted_with
FORMAT JSONEachRow (HTTP)"] - REJ["rejected at startup"] - CH[("ClickHouse")] + REJ["refused when the pipeline starts"] + CH[(ClickHouse)] ROW --> F F -->|row_binary| T @@ -39,26 +39,16 @@ flowchart TB | `insert_format` | `protocol = http` | `protocol = native` | |-----------------|--------------------|----------------------| -| `row_binary` (default) | `FORMAT RowBinaryWithDefaults` via `insert_formatted_with` | `FORMAT Native` blocks via `TcpClient::insert_native` | -| `json_each_row` | `FORMAT JSONEachRow` via `insert_formatted_with` | rejected at `config-check` | +| `row_binary` (default) | `FORMAT RowBinaryWithNamesAndTypes` via `insert_formatted_with` | `FORMAT Native` blocks via `TcpClient::insert_native` | +| `json_each_row` | `FORMAT JSONEachRow` via `insert_formatted_with` | refused when the pipeline starts, not by `config-check` | -The rejection is deliberate and surfaced early: JSONEachRow is an HTTP body -format, and a native client has no HTTP insert endpoint for it. The guard lives -in `ClickHouseConfig::validate()` and runs at orchestrator startup, so a bad -combination fails the boot with a clear message rather than at first insert. +The rejection is deliberate: JSONEachRow is an HTTP body format, and a native client has no HTTP insert endpoint for it. The guard lives in `ClickHouseConfig::validate()`, which the orchestrator runs before the readiness probe, so a bad combination stops the loader with a clear message rather than at first insert. `config-check` runs `Config::validate` only and passes it. -A native block declares each column's type, so the server casts a value to a -changed column type instead of refusing the batch the way the HTTP header check -does (code 117). A column whose type `dfe_clickhouse` cannot declare in a block -fails the insert and the batch is held. As of `dfe-clickhouse` 0.1.5 that -includes a parameterised `JSON(max_dynamic_paths = N)` column. +Both protocols check the cached schema against the table. Over HTTP the header declares each column's type and the server refuses a type the column no longer has (code 117). Over native the server answers the INSERT with the table's types, and the insert refuses a difference before sending a row. Either refusal reads as drift. A column whose type `dfe_clickhouse` cannot write fails the insert and the batch is held. ## Why RowBinary is the default -RowBinary ships pre-typed bytes. The server does no JSON parse, no type -inference -- it reads the column values straight off the wire in -`system.columns` order. For a high-throughput loader that is the difference -between ClickHouse spending CPU on ingest formatting and spending it on merges. +RowBinary ships pre-typed bytes. The server parses no row text and infers no types for the typed columns, reading their values off the wire in the order the INSERT names them. A JSON column is the exception: its value travels as JSON text the server parses. For a high-throughput loader that is the difference between ClickHouse spending CPU on ingest formatting and spending it on merges. JSONEachRow is kept because it is self-describing and forgiving: the server coerces types and tolerates schema drift. That makes it the right tool for @@ -68,16 +58,7 @@ the throughput choice. ## How RowBinary is produced -The dynamic encoder in `dfe-clickhouse`'s `dynamic` module turns one -`Map` plus its reflected `ColumnDef` schema into RowBinary, sent -in 1 MiB chunks as one `INSERT INTO db.table (cols) FORMAT -RowBinaryWithDefaults` per batch. The INSERT names every column the server takes -a value for. A row that leaves out a defaulted column flags it, and the server -fills its default (`_uuid`, `_timestamp_load`), so rows with differing columns share one -INSERT. Over native a row that supplies a different set of defaulted columns -starts a new INSERT. - -## JSON columns on the wire +The dynamic encoder in `dfe-clickhouse`'s `dynamic` module turns one `Map` plus its reflected `ColumnDef` schema into RowBinary, sent in 1 MiB chunks as `INSERT INTO db.table (cols) FORMAT RowBinaryWithNamesAndTypes`. An INSERT names the required columns and the defaulted ones its rows supply, and the server fills the rest from their DEFAULT expressions (`_uuid`, `_timestamp_load`). Rows that supply a different set of defaulted columns go to an INSERT of their own: HTTP keeps up to four open at once, and over native a new set commits the rows before it. A row that supplies no column at all goes as `RowBinaryWithDefaults`. A JSON-typed column (including `_json`) is written as a length-prefixed string. On the RowBinary path the loader sets @@ -89,9 +70,8 @@ and [types.md](types.md) for per-type encoding. ## When the schema drifts -An insert can fail because the cached schema no longer matches the table (an -`ALTER ... ADD COLUMN`, a type change). The crate classifies the server -error: a drift signal (`TYPE_MISMATCH`, `NO_SUCH_COLUMN`, `CANNOT_PARSE`, code +An insert can fail because the cached schema no longer matches the table: a column dropped or retyped by `ALTER`. An added column fails nothing, since the INSERT names its columns. The crate classifies the server +error: a drift code (`TYPE_MISMATCH`, `NO_SUCH_COLUMN_IN_TABLE`, `CANNOT_PARSE_TEXT`, code 117, and friends) surfaces as `DynamicError::SchemaMismatch`, and the cache is invalidated so the next attempt re-fetches from `system.columns` and re-encodes. A non-drift error (network, auth) is returned as-is. See @@ -108,11 +88,7 @@ RowBinary is positional, so bytes encoded for a column's old type could parse as | `clickhouse.protocol` | `http`, `native` | `http` | | `clickhouse.insert_format` | `rowbinary` (aliases `row_binary`, `binary`), `jsoneachrow` (aliases `json_each_row`, `json`) | `rowbinary` | -The canonical spellings are the ones `docs/config-schema.json` carries, and the -aliases parse identically -- `row_binary` and `json_each_row` are the spellings -this page used before the key was wired, so both keep working. `native` is not -among them: it names a protocol this config rejects at boot, so accepting it as -a format spelling would have one word mean two opposite things on one page. +The canonical spellings are the ones `docs/config-schema.json` carries, and the aliases parse identically. `native` is refused as a format, because it names a protocol. Env form: `DFE_LOADER_CLICKHOUSE__INSERT_FORMAT=json_each_row`. The startup log line `Insert format configured format=...` reports the value that was applied. diff --git a/docs/clickhouse/schema-cache.md b/docs/clickhouse/schema-cache.md index 0fb2ac4b..abf77811 100644 --- a/docs/clickhouse/schema-cache.md +++ b/docs/clickhouse/schema-cache.md @@ -48,16 +48,18 @@ required one. See [types.md](types.md). `DynamicSchemaCache` is a TTL map keyed by `database.table`, shared across insert tasks behind an `Arc`. It is lazy, not background-refreshed: -- `get` returns the schema only if it was fetched within the TTL; an expired +- `get` returns the schema only if it was fetched within the TTL, so an expired entry reads as a miss. - A miss triggers a fetch, and the result is inserted under the key. -- `invalidate(table)` drops one entry; `invalidate_all` clears the map. +- `invalidate(table)` drops one entry, and `invalidate_all` clears the map. Lazy TTL keeps the steady state cheap (no timer threads, no work for idle tables) while bounding how long a stale schema can persist after a benign `ALTER`. The TTL is set once when the cache is constructed, from `schema.cache_ttl_secs` (default 300 s). +The loader keeps a second cache of the same read, `SchemaCache` in `src/clickhouse/schema.rs`, which decides the fields the header extractor promotes. Its entries live for `schema.cache_ttl_secs` too, and every 60 s a background task hands each entry within 30 s of expiry to the schema resolver, which re-reads its columns and comments together. + ## Mismatch recovery A TTL alone does not cover a schema change that happens mid-batch. So the insert @@ -66,14 +68,11 @@ signal -- a code name such as `TYPE_MISMATCH`, `NO_SUCH_COLUMN` or `INCORRECT_DATA`, or a drift phrase such as "no such column" or "cannot parse" -- the table's cache entry is invalidated. The retry re-fetches from `system.columns` and re-encodes against the current schema. A non-drift error (network, auth, server down) is -returned unchanged, so genuine outages are not mistaken for schema drift. +returned unchanged, so an outage is not mistaken for schema drift. -A row the server can never accept draws the same signals: an integer no ClickHouse integer type holds, in a JSON column, comes back as code 117 whatever the schema. So the server's code decides, never the words beside it, which echo the row. A code on the retry list in `src/clickhouse/error.rs` (`RETRY_CODES`: a busy or overloaded server, the network, replication and Keeper, the disk or object store under the table, and operator fixes such as a missing table or a refused login) holds the batch for another attempt. Any other code re-reads `system.columns`. A table unchanged since the rows were encoded makes the refusal the rows' own: the insert returns it as permanent, salvage isolates the bad row for the DLQ, and the rest land. A changed table is drift, recovered as above. `TOO_MANY_PARTS` for one INSERT over `max_partitions_per_insert_block` is a refusal, not a retry, because salvage's split is what lands it. A failure with no server code at all never reaches the DLQ. +A row the server can never accept draws the same signals: an integer no ClickHouse integer type holds, in a JSON column, comes back as code 117 whatever the schema. So the server's code decides, never the words beside it, which echo the row. A code on the retry list in `src/clickhouse/error.rs` (`RETRY_CODES`: a busy or overloaded server, the network, replication and Keeper, the disk or object store under the table, and operator fixes such as a missing table or a refused login) holds the batch for another attempt. Any other code re-reads `system.columns`. A table unchanged since the rows were encoded makes the refusal the rows' own: the insert returns it as permanent, salvage isolates the bad row for the DLQ, and the rest land. A changed table is drift, recovered as above. `TOO_MANY_PARTS` for one INSERT over `max_partitions_per_insert_block` is a refusal, not a retry, because salvage's split is what lands it. A server failure with no code never reaches the DLQ. Only the encoder's own refusal of a value does, since it holds the schema it read the value against. -This is the fix trail for the schema-cache-miss class of bug: a column added by -an out-of-band `ALTER` no longer fails every insert to that table until a restart -- the first -rejected insert clears the stale entry and the retry succeeds against the new -schema. +A column dropped or retyped by an out-of-band `ALTER` fails one attempt: the rejection clears the stale entry and the retry succeeds against the new schema. An added column fails nothing, since the INSERT names its columns, and it fills once the cached schema refreshes. `schema.refresh_on_error` (default `true`) switches this recovery. With `false` a drift error leaves both the encoder's entry and the loader's cached schema in diff --git a/docs/clickhouse/tls.md b/docs/clickhouse/tls.md index 1371a139..0a4495bd 100644 --- a/docs/clickhouse/tls.md +++ b/docs/clickhouse/tls.md @@ -28,7 +28,7 @@ flowchart LR | Setting | Effect | |---------|--------| -| `clickhouse.tls.enabled` | HTTPS for the connection | +| `clickhouse.tls.enabled` | TLS for the connection: HTTPS, or native TLS | | `clickhouse.tls` absent | TLS turns on for a host on 8443 or 9440, or under `clickhouse.cloud` | The SNI name and the name checked against the certificate are the host in `clickhouse.hosts`, so connect by a name (or IP) the certificate lists. @@ -39,9 +39,7 @@ A private CA is trusted the way the system trusts it: add it to the image's trus A trust store with no readable CA certificate fails the client at startup instead of connecting unverified. -## What is deliberately not offered - -`clickhouse.tls.ca_cert_file`, `cert_file`, `key_file` and `skip_verify` are refused at startup: no client reads them. There is no skip-verify mode, and client-certificate (mTLS) auth is out of scope today. +`clickhouse.tls.ca_cert_file`, `cert_file`, `key_file` and `skip_verify` are refused by `config-check` and at startup: no client reads them. There is no skip-verify mode and no client-certificate (mTLS) auth. ## Example: private-CA cluster @@ -58,4 +56,4 @@ clickhouse: SSL_CERT_FILE=/etc/dfe/ca-bundle.pem # the private CA plus any public roots still needed ``` -The CA bundle is non-secret and is distributed with the deployment. Keep the cluster password in a Secret, never in committed config. See [../configuration.md](../configuration.md) for the cascade and [../deployment/](../deployment/README.md) for how secrets are supplied at runtime. +The CA bundle is non-secret and is distributed with the deployment. Keep the cluster password in a Secret, never in committed config: the chart fills `DFE_LOADER__CLICKHOUSE__PASSWORD` from the `clickhouse-password` key the deployment contract declares (`Config::secrets` in `src/config/loader.rs`). See [../configuration.md](../configuration.md) for the cascade. diff --git a/docs/clickhouse/types.md b/docs/clickhouse/types.md index 8e044c04..29f48307 100644 --- a/docs/clickhouse/types.md +++ b/docs/clickhouse/types.md @@ -16,8 +16,7 @@ types (possibly wrapped in `Nullable` or `LowCardinality`), and the encoder in `dfe-clickhouse`'s `dynamic` module has to turn a `Value` into the right bytes for it. Use this page when you are designing a schema. -Nothing here is loader-invented: it is the ClickHouse type system as of the -24.8+ line, annotated with how each type lands on the native wire. For how those +Nothing here is loader-invented: it is the ClickHouse type system, annotated with how each type lands on the native wire. For how those bytes are actually shipped (HTTP RowBinary or native blocks), see [insert-formats.md](insert-formats.md). For where the encoder hands its bytes to the client's insert calls, see [clickhouse-ext.md](clickhouse-ext.md). @@ -29,7 +28,7 @@ Sources: - [clickhouse-cpp GitHub](https://github.com/ClickHouse/clickhouse-cpp) - [JSON Type Blog Post](https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouse) - [ClickHouse 24.8 Release](https://clickhouse.com/blog/clickhouse-release-24-08) -- [JSON/Dynamic/Variant Production Ready PR](https://github.com/ClickHouse/ClickHouse/pull/77785) +- [ClickHouse PR 77785: JSON, Dynamic and Variant out of experimental](https://github.com/ClickHouse/ClickHouse/pull/77785) --- @@ -209,7 +208,7 @@ types discovered at runtime. - No need to declare types upfront - `max_types` limits separate storage columns (default 32) -- Types beyond limit stored as String +- Types beyond the limit share one variant in binary form (`SharedVariant`) **Native protocol:** @@ -302,11 +301,11 @@ The loader writes in one of two formats, dispatched by `clickhouse.insert_format `DynamicInsert` reflects the target schema from `system.columns`, encodes a `Map` against that schema column-by-column, and sets `input_format_binary_read_json_as_string=1` for JSON-typed columns. The loader -never calls `clickhouse::Row` -- everything is dynamic. DDL and schema queries go +never calls `clickhouse::Row` -- everything is dynamic. Schema and comment queries go through the same client via `ClickHouseQueryClient`. The full transport x format matrix is in [insert-formats.md](insert-formats.md). -Type support comes from the loader's own encoder, not the client crate. A column +Type support comes from `dfe-clickhouse`'s encoder, not the `clickhouse` crate. A column type the encoder does not cover (a `Variant` column, for one) fails the write with `DynamicError::UnsupportedType`, and the rows are held rather than dead-lettered. @@ -389,7 +388,6 @@ A third coordinate is read and ignored. A geohash, a latitude outside -90 to 90, Elasticsearch lets any field carry one value or many, so a scalar bound for an `Array(T)` column is written as a one-element array, and T's own rules then decide whether it is valid. `"related_ip": "52.108.0.3"` lands in an `Array(IPv6)` column as `['::ffff:52.108.0.3']`. An object is wrapped only where the element is read from one (`JSON`, `Map`, `Point`). Bound for an array of scalars, it stays an encoding error. A null or absent array is the empty array. For an `Array(Point)` column, a bare `[lon, lat]` is one point, as Elasticsearch reads it. -The rationale for RowBinary as the default: schema-reflected RowBinary skips the -server-side JSON parse, cuts ClickHouse CPU, and gives full type support without -requiring compile-time schemas. Records stay a `Map` from +The rationale for RowBinary as the default: schema-reflected RowBinary leaves the server no row text to parse, cuts ClickHouse CPU, and gives full type support without +requiring compile-time schemas. A JSON column's value is still JSON text the server parses. Records stay a `Map` from intake to insert -- encoding happens against `system.columns` at runtime. diff --git a/docs/configuration.md b/docs/configuration.md index f4b29d27..36fd12de 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -15,7 +15,7 @@ dfe-loader reads its config in four layers. Higher layers win. Every setting has ```mermaid flowchart TB FLAT["1. Flat ENV (DFE_LOADER_CLICKHOUSE_HOSTS=a:8123,b:8123)"] - ENV["2. Nested ENV (DFE_LOADER_CLICKHOUSE__PROTOCOL=http), ./.env included"] + ENV["2. Nested ENV (DFE_LOADER_CLICKHOUSE__PROTOCOL=http)"] YAML["3. The YAML file --config or DFE_LOADER_CONFIG names, else ./config.yaml"] HARD["4. hard-coded defaults"] FLAT --> ENV --> YAML --> HARD @@ -25,13 +25,9 @@ Nested ENV names are derived from the key: `clickhouse.protocol` -> `DFE_LOADER_ A file that `--config` or `DFE_LOADER_CONFIG` names must exist. A missing one refuses startup rather than running on the defaults. -Only the `.env` in the working directory is read. A `.env` in a parent directory belongs to another project and is never loaded. +A `.env` is read only when `--config` names no file, and then only the one in the working directory, into the environment the two ENV layers read. A `.env` in a parent directory belongs to another project and is never loaded. -The scaling gate (`scaling.enabled`, `scaling.memory_gate_threshold`) is read by -scalo's own cascade, which discovers `defaults.yaml` / `settings.yaml` but never -the file named by `--config`, so set it with `DFE_LOADER_SCALING__ENABLED` and -`DFE_LOADER_SCALING__MEMORY_GATE_THRESHOLD`. A value for it in the `--config` -file drives the loader's own weights and is reported as a mismatch at startup. +scalo's own cascade, which builds the scaling engine KEDA reads, takes the `--config` file and the nested ENV form too (`DFE_LOADER_SCALING__ENABLED` or `DFE_LOADER__SCALING__ENABLED`), so `scaling.enabled` and `scaling.memory_gate_threshold` set either way reach it. > A config value is only useful if the ACTION matches it. The integration tests > assert the observable outcome (the row written, the transport used), not just @@ -44,9 +40,9 @@ file drives the loader's own weights and is reported as a mismatch at startup. | Setting | Values | Effect | |---------|--------|--------| | `clickhouse.protocol` | `http` (default), `native` | HTTP (8123), HTTPS (8443), or the native protocol (9000, 9440 with TLS) through `dfe_clickhouse::TcpClient` | -| `clickhouse.insert_format` | `row_binary` (default), `json_each_row` | binary (server skips JSON parsing) vs self-describing JSON | -| `clickhouse.hosts` | list | `native` fails over across every host, round-robin. `http` uses the first host only. With TLS, `native` verifies every host against the first one's name | -| `clickhouse.database` | string, `dfe` (default) | database the connection targets -- DDL, schema reflection, and any table name that arrives without a `db.` prefix | +| `clickhouse.insert_format` | `row_binary` (default), `json_each_row` | binary, so the server parses no row text, vs self-describing JSON | +| `clickhouse.hosts` | list | `native` fails over across every host, round-robin. `http` uses the first host only. With TLS, `native` verifies each host against its own name | +| `clickhouse.database` | string, `dfe` (default) | database the connection opens on, and the one the readiness probe reads `system.tables` and `system.columns` for | `clickhouse.database` is the CONNECTION database and is separate from `routing.default_db` (also `dfe`), which is the database a routed row lands in. @@ -54,12 +50,12 @@ Both are cascade-overridable like any other setting -- `DFE_LOADER_CLICKHOUSE__D or `clickhouse.database` in yaml. The protocol and format compose. The default `http + row_binary` is the fast -path; `json_each_row` is the diagnostic fallback: +path, and `json_each_row` is the diagnostic fallback: | `insert_format` | `protocol = http` | `protocol = native` | |-----------------|-------------------|---------------------| -| `row_binary` (default) | `insert_formatted_with` FORMAT RowBinaryWithDefaults | `FORMAT Native` blocks over the native protocol | -| `json_each_row` | `insert_formatted_with` FORMAT JSONEachRow | refused at startup | +| `row_binary` (default) | `insert_formatted_with` FORMAT RowBinaryWithNamesAndTypes | `FORMAT Native` blocks over the native protocol | +| `json_each_row` | `insert_formatted_with` FORMAT JSONEachRow | refused when the pipeline starts, not by `config-check` | See [clickhouse/insert-formats.md](clickhouse/insert-formats.md). @@ -69,7 +65,7 @@ HTTPS verifies the server's certificate and host name against the container's sy | Setting | Values | Effect | |---------|--------|--------| -| `clickhouse.tls.enabled` | bool | HTTPS for the connection. When `clickhouse.tls` is absent, a host on 8443/9440 or under `clickhouse.cloud` turns it on | +| `clickhouse.tls.enabled` | bool | TLS for the connection: HTTPS, or native TLS. When `clickhouse.tls` is absent, a host on 8443/9440 or under `clickhouse.cloud` turns it on | | `clickhouse.tls.ca_cert_file`, `cert_file`, `key_file`, `skip_verify` | -- | refused at startup: no client reads them | See [clickhouse/tls.md](clickhouse/tls.md). @@ -79,12 +75,12 @@ See [clickhouse/tls.md](clickhouse/tls.md). | Setting | Default | Bounds | |---------|---------|--------| | `clickhouse.connect_timeout_ms` | `5000` | opening a TCP connection | -| `clickhouse.request_timeout_ms` | `60000` | the first try of every query (schema reads, health), and an insert waiting to hand over its next chunk. A query that times out doubles the bound for the next one, up to 300 s, so a slow but working ClickHouse still answers; a quick answer narrows it again | +| `clickhouse.request_timeout_ms` | `60000` | the first try of every query (schema reads, health), and an insert waiting to hand over its next chunk. A query that times out doubles the bound for the next one, up to 300 s, so a slow but working ClickHouse still answers. A quick answer narrows it again | | `clickhouse.insert_timeout_ms` | `60000` | an insert waiting for the server's answer once its rows are sent | -A timed-out insert is held and sent again on a later flush, and its Kafka offset stays uncommitted until it lands. 60 s is about 270x the ClickHouse CPU cost of the largest default flush (20,000 rows at a measured 91,000 rows per CPU-second), and two rounds of timed-out concurrent inserts still fit Kafka's 300 s `max.poll.interval.ms`, which a blocked flush cycle counts against. Through a full outage the query bound widens, so an insert that first has to read its table's schema can hold one flush cycle past that limit; the consumer then rejoins with the batch's offsets still uncommitted. +A timed-out insert is held and sent again on a later flush, and its Kafka offset stays uncommitted until it lands. Two rounds of timed-out concurrent inserts still fit the 300 s `max.poll.interval.ms` scalo's Kafka transport sets by default, which a blocked flush cycle counts against. Through a full outage the query bound widens, so an insert that first has to read its table's schema can hold one flush cycle past that limit, and the consumer then rejoins with the batch's offsets still uncommitted. -Idle connections close on the client side after 2 s. +Idle HTTP connections close on the client side after 2 s. ### Capture (what gets stored) @@ -93,6 +89,8 @@ Idle connections close on the client side after 2 s. | `metadata.capture_mode` | `full` (default), `json_only`, `raw_only`, `extracted_only` | `_json` + `_raw` population -- see [architecture.md](architecture.md#capture-modes) | | `metadata.table_capture_modes` | map | per-table override of `capture_mode` | | DDL `@capture_mode: ` | comment tag | highest-priority override, per table | +| `metadata.capture_raw` | `true` (default), `false` | whether `metadata.raw_source_fields` may fill `_raw` | +| `metadata.capture_json` | -- | read by nothing: `capture_mode` decides `_json`, unless a directive on the column fills it ([pipeline/capture-modes.md](pipeline/capture-modes.md)) | Precedence, highest wins: **DDL tag > per-table > global**. @@ -101,7 +99,7 @@ Precedence, highest wins: **DDL tag > per-table > global**. | Setting | Effect | |---------|--------| | `routing.table_fields` | fields whose value sets the table (dot-notation for nested). A null or empty value counts as absent | -| `routing.default_db` / `default_table` | fallbacks when no field matches | +| `routing.default_db` / `default_table` | where a record lands when neither a field nor its topic names a table, or the table it names does not exist | | `routing.org_id_field` / `default_org_id` | where the `_org_id` tenant is read from, and the tenant a record with none lands in (`default`) | | `routing.org_routes` | per-org database routing, keyed on the tenant | | `routing.external_topics` | topics an external pipeline writes directly. A record there keeps its `_org_id` and `_source` only when they are the source's own or on the topic's allow-list ([routing](pipeline/routing.md#external-pipeline-topics)) | @@ -132,19 +130,17 @@ The gRPC transport reads no Kafka topic, so a gRPC loader loads no dead letters | `geoip`, `enrichment.reputation` / `risk_scoring` | inject enriched columns on the promoted row -- see [pipeline/enrichment.md](pipeline/enrichment.md) | | `buffer.flush_rows` / `flush_bytes` / `flush_age_secs` | per-table flush triggers: 20000 rows, the rows' input bytes reaching 64 MiB, or 5 s since the last flush, whichever comes first | -Insert retry, batch salvage and insert concurrency are not settings. Every build retries a transient insert failure up to 5 times with jittered backoff, binary-splits a batch with a data error down to the bad rows, and runs at most 8 inserts at once. A batch that still fails is held and retried as described in [operations.md](operations.md). +Insert retry, batch salvage and insert concurrency are not settings. Every build binary-splits a batch with a data error down to the bad rows and runs at most 8 inserts at once. Inside one flush a RowBinary insert retries up to 5 times with jittered backoff only to re-read a stale, missing or unreadable schema, and a JSONEachRow insert retries a transient failure the same way. A timeout, a dropped connection or a retryable server code returns the batch at once, and the held batch is retried as described in [operations.md](operations.md). `schema.refresh_on_error` (default `true`) decides whether a RowBinary insert that fails on schema drift drops the table's cached schema. With `false` the stale schema is kept until it expires -- see [clickhouse/schema-cache.md](clickhouse/schema-cache.md#mismatch-recovery). ## Hot-reload -Config hot-reloads on change: `hot_reload.enabled` is on unless set false, and the loader watches the file `--config` names. Some fields are safe to apply live. The rest require a restart and are logged as a warning rather than silently applied: +Config hot-reloads on change: `hot_reload.enabled` is on unless set false, and the loader watches the file `--config` or `DFE_LOADER_CONFIG` names. Some fields are safe to apply live. A change to most others is logged as a restart warning and the running value kept: -- **Restart-required:** `clickhouse.protocol`, ClickHouse URL/hosts, - `clickhouse.insert_format`, `clickhouse.tls.enabled`, the - `clickhouse.*_timeout_ms` bounds, Kafka connection -- all of these are baked - into the `Client` at build time. Also `routing.dlq`, `dlq_load`, `field_mapping` and `enrichment`, which are built once at startup. -- **Safe to reload:** buffer thresholds, `metadata` (capture modes included), `timestamp_dq`, `field_sanitization`, and `routing` other than `routing.dlq`. +- Safe to reload: buffer thresholds, `metadata` (capture modes included), `timestamp_dq`, `field_sanitization`, and `routing` other than `routing.dlq`. +- Restart-required, with a warning: `transport`, `kafka`, `grpc`, `clickhouse` (protocol, hosts, `insert_format`, TLS and the timeouts are built into the client), `payload`, `metrics`, `logging`, `scaling`, `hot_reload`, `schema`, `geoip`, `computed_columns`, `column_directives`, `routing.dlq`, `dlq_load`, `field_mapping` and `enrichment`. +- Restart-required, with no warning: `memory` and `batch_processing`. scalo's runtime builds the memory guard and the batch engine from them once, at startup (`ServiceRuntime` in scalo-rs `src/cli/runtime.rs`). On Kubernetes the deployment contract lists `routing` in `hot_paths`, so the chart leaves it out of the pod's config checksum. A `routing` edit reaches the running pods through the refreshed ConfigMap, and an edit anywhere else still rolls them. diff --git a/docs/deployment/README.md b/docs/deployment/README.md index 24104eec..500d9927 100644 --- a/docs/deployment/README.md +++ b/docs/deployment/README.md @@ -10,10 +10,7 @@ # Deployment -How dfe-loader gets built into artefacts and what it depends on at deploy time. -This covers publishing the container image and Helm chart that downstream -projects (dfe-docker, dfe-operator) pull, and the shared `dfe-schemas` -definitions the loader resolves at runtime to drive its table DDL. +How dfe-loader gets built into artefacts, and where the tables it writes come from. dfe-docker and dfe-infra pull the published image and chart, and the loader reads its tables' schema from ClickHouse, never from dfe-schemas. -- [publishing.md](publishing.md) -- container image + Helm chart publishing via CI: Dockerfile, `.hyperi-ci.yaml`, generated tags, and consumer usage. -- [schemas.md](schemas.md) -- loader-side reference for the shared schema definitions: submodule setup, resolution order, per-file versioning, and DFE `expr` directives. +- [publishing.md](publishing.md) -- container image and Helm chart publishing via CI: the generated Dockerfile, `.hyperi-ci.yaml`, the tags and who pins them. +- [schemas.md](schemas.md) -- where a table's schema comes from, and the column directives the loader reads off it. diff --git a/docs/deployment/publishing.md b/docs/deployment/publishing.md index 25531fc4..6fabbd28 100644 --- a/docs/deployment/publishing.md +++ b/docs/deployment/publishing.md @@ -11,21 +11,19 @@ # Container and chart publishing dfe-loader ships as a multi-arch container image and a Helm chart, built and -pushed by CI on every release. Downstream projects (dfe-docker, dfe-operator) -reference an immutable OCI tag instead of building from source or downloading -loose binaries. +pushed by CI on every release. dfe-infra's `versions.yaml` pins the image tag, its digest and the chart, and dfe-docker pins the image through `DFE_LOADER_VERSION`, so neither builds from source. Both artefacts go to GHCR. The image is `ghcr.io/hyperi-io/dfe-loader` and the chart is pushed to `oci://ghcr.io/hyperi-io/charts`. CI authenticates with `GITHUB_TOKEN`, so no registry secret is needed. ```mermaid flowchart TB REL["Release tag (vX.Y.Z)"] - BIN["CI cross-build
linux/amd64 + linux/arm64 binaries"] + BIN["CI build
linux/amd64 + linux/arm64 binaries"] DOCK["Build Dockerfile
wraps pre-built binary"] PUSH["Push image
version tags"] EMIT["dfe-loader generate-artefacts
emits the deployment contract"] HELM["Assemble thin chart on scalo-service
package + push OCI artifact"] - REG[("GHCR")] + REG[(GHCR)] REL --> BIN --> DOCK --> PUSH --> REG BIN --> EMIT --> HELM --> REG @@ -52,7 +50,7 @@ release: `library` moves with the scalo version in `Cargo.toml`: a scalo-service release ships the schema for, and renders, only the contract version its scalo release writes. To see the chart a release would ship, build the binary and run `hyperi-ci chart assemble --binary target/debug/dfe-loader --image ghcr.io/hyperi-io/dfe-loader:@sha256: --version `. It prints the chart directory it wrote. -The `Dockerfile` is generated from the app's deployment contract by scalo and copies in the binary CI has already cross-compiled. Do not edit it by hand. Regenerate it with `dfe-loader --emit-dockerfile`. `tests/integration/helm_contract.rs` fails when the committed `Dockerfile` drifts from the contract. +The `Dockerfile` is generated from the app's deployment contract by scalo and copies in the binary CI has already built. Do not edit it by hand. Regenerate it with `dfe-loader --emit-dockerfile`. `tests/integration/helm_contract.rs` fails when the committed `Dockerfile` drifts from the contract. ## Tags generated @@ -63,11 +61,9 @@ For a stable release `v`: - `ghcr.io/hyperi-io/dfe-loader:sha-` A prerelease (e.g. `v-rc.1`) gets its version tag and `sha-` -only. It never moves `latest`. +only. It never moves `latest`, and nor does a stable release on an older line while a newer one holds it. -## Verification - -After a release: +After a release, check the tags: ```bash docker pull ghcr.io/hyperi-io/dfe-loader:v @@ -75,26 +71,9 @@ docker run --rm ghcr.io/hyperi-io/dfe-loader:v --help docker manifest inspect ghcr.io/hyperi-io/dfe-loader:v ``` -## Consumer usage - -### Docker Compose (dfe-docker) +## Consumers -```yaml -services: - dfe-loader: - image: ghcr.io/hyperi-io/dfe-loader:${DFE_LOADER_VERSION:-latest} - ports: - - "9090:9090" - volumes: - - ./config/loader.yaml:/etc/dfe/loader.yaml:ro -``` +- dfe-infra deploys the published chart through Argo CD at the tag and digest `versions.yaml` names. +- dfe-docker's compose service takes `${IMAGE_REGISTRY:-ghcr.io/hyperi-io}/dfe-loader:${DFE_LOADER_VERSION}`, which `make stack VERSION=X.Y.Z` pins, and publishes the metrics port on `${DFE_LOADER_PROMETHEUS_PORT:-9091}`. -### Kubernetes (dfe-operator) - -```yaml -containers: - - name: dfe-loader - image: ghcr.io/hyperi-io/dfe-loader:v - ports: - - containerPort: 9090 -``` +The container's entrypoint is `dfe-loader --config /etc/dfe/loader.yaml`, so a consumer mounts the config there. diff --git a/docs/deployment/schemas.md b/docs/deployment/schemas.md index 559b60ed..71c4d6ad 100644 --- a/docs/deployment/schemas.md +++ b/docs/deployment/schemas.md @@ -21,7 +21,7 @@ loader by two steps, and nothing here needs a checkout of it. ```mermaid flowchart LR YAML["dfe-schemas
YAML definitions"] - APPLY["dfe-schema
(dfe-engine image)"] + APPLY["dfe-engine
applies the manifest at boot"] CH[(ClickHouse system.columns)] LOADER["dfe-loader
SchemaCache"] @@ -39,9 +39,11 @@ A schema YAML's `expr` becomes part of the column COMMENT in the deployed DDL, alongside any human-readable description: ```sql -COMMENT '@renamed: client_ip -- Source address of the connection' +COMMENT '@renamed: client_ip - Source address of the connection' ``` +The parser ends a source path at the first ` - `, so the description after it is never read as part of the path. + The loader reads the directives it acts on and ignores the rest. Config always wins over a COMMENT -- the exact resolution order is in [`src/column_meta/mod.rs`](../../src/column_meta/mod.rs). @@ -50,7 +52,7 @@ wins over a COMMENT -- the exact resolution order is in |-----------|--------------|--------| | `@skip` | `skip: true` | Omit the column from the insert | | `@default:value` | `default:` | Substitute when the column is null or absent | -| `@source:path` | `renamed:` | Source field path(s) for this column; a `\|` fallback also fills `default:` | +| `@source:path` | `renamed:` | Source field path(s) for this column, and a `\|` fallback also fills `default:` | | `@renamed:path` | `renamed:` | Source field path(s) for this column | | `@computed:expr` | `computed:` | CEL expression producing the value | | `@coerce:category` | `coerce:` | None: the encoder's coercion rules are fixed. The DDL directive logs a warning, the config key is refused | @@ -60,24 +62,20 @@ unknown directives (`DIRECTIVE_NAMES` in [`src/column_meta/mod.rs`](../../src/column_meta/mod.rs) lists only the six directives above). What makes the loader omit a value for a `@generated` column is ClickHouse's own DEFAULT/MATERIALIZED/ALIAS clause on it, reported -as `default_kind` in `system.columns` -- the applier writes that clause from +as `default_kind` in `system.columns` -- dfe-schemas' renderer writes that clause from the schema's `default:` and `attribute:` fields, not by parsing the `@generated:` text. `@config: path` documents that a mapping is driven by -loader config (for example `routing.org_id_field`) rather than hardcoded; +loader config (for example `routing.org_id_field`) rather than hardcoded, and nothing reads the comment text itself. See [ddl-directives.md](../clickhouse/ddl-directives.md) for the whole expression language. ## Getting the tables there in the first place -The `dfe-schema` entry point on the dfe-engine image creates or reconciles every -DFE table from the dfe-schemas YAML. dfe-infra runs it as a Job before the data -plane starts and dfe-docker as a compose init service, so a profile with no -engine still gets its tables. A loader pointed at a database that has not been -applied finds no columns and says so. +dfe-engine applies the dfe-schemas manifest at boot: it takes the schema lock, renders the manifest for the platform it reaches, and creates what is absent or additively changed. Nothing else creates a DFE table, no Job and no init container. A loader pointed at a database the engine has not applied finds no columns: records wait in the pending-schema buffer, and a missing default table is logged at `warn`. ## Related docs -- [common-header.md](../transport/common-header.md) -- the common header columns and their DDL +- [common-header.md](../transport/common-header.md) -- the header columns and what the loader writes into each - [ddl-directives.md](../clickhouse/ddl-directives.md) -- field mapping expression language - [dfe-schemas](https://github.com/hyperi-io/dfe-schemas) -- canonical schema documentation diff --git a/docs/feature-flags.md b/docs/feature-flags.md index b7c622d0..143a8ea6 100644 --- a/docs/feature-flags.md +++ b/docs/feature-flags.md @@ -10,10 +10,7 @@ # Feature flags -dfe-loader is a binary, not a library, so it carries far fewer features than -scalo. The defaults are empty -- everything the running loader needs is -always compiled; the flags exist for the allocator, test scaffolding, and the -PGO build. +dfe-loader ships as a binary, its library target serving that binary, the tests and the benches, so it carries far fewer features than scalo. The defaults are empty: everything the running loader needs is always compiled, and the flags exist for the allocator, test scaffolding and the PGO build. ## Loader features @@ -22,18 +19,15 @@ PGO build. | `jemalloc` | off | `tikv-jemallocator`, `tikv-jemalloc-ctl` | jemalloc allocator (DFE policy: jemalloc at every channel, no mimalloc) | | `transport-memory` | off | `scalo/transport-memory` | in-process Memory transport for unit tests | | `testcontainers` | off | -- | enable the integration tests that run against real ClickHouse and Kafka containers, see `tests/TESTING.md` | -| `pgo-driver` | off | `rdkafka` | build the PGO workload producer binary; the main loader binary is unaffected | +| `pgo-driver` | off | `rdkafka` | build the PGO workload producer binary, leaving the main loader binary unaffected | -`default = []`. Production builds opt in to `jemalloc` explicitly; CI opts in to -`testcontainers` for the docker test lane. See -[deployment/](deployment/README.md) for the release build. +`default = []`. hyperi-ci adds `jemalloc` to every release-track build, and `.hyperi-ci.yaml` lists `testcontainers`, `jemalloc` and `transport-memory` among the feature sets its tests compile. See [performance/overview.md](performance/overview.md) for what each channel builds. ## The ClickHouse client is not a feature The loader does not feature-gate its ClickHouse client. It always builds the official `clickhouse` crate from crates.io, with the `lz4`, `time`, `inserter` -and rustls TLS features, plus `dfe-clickhouse` with only its `ext` feature for -the server-exception parser. Neither is patched or taken from git. See +and rustls TLS features, plus `dfe-clickhouse` with `ext` (the server-exception parser), `tls` (the native-protocol client) and `dynamic` (the dynamic RowBinary insert). Neither is patched or taken from git. See [clickhouse/clickhouse-ext.md](clickhouse/clickhouse-ext.md). ## scalo features @@ -42,5 +36,5 @@ The bulk of the loader's capability surface comes from scalo (config cascade, logging, metrics, health, shutdown, transports, memory guard, CLI, top). Those are selected in the `scalo` dependency entry, pinned to a crates.io release (never a path or patch override). For the scalo feature tree -see the scalo docs; for how the loader wires them at startup see +see the scalo docs, and for how the loader wires them at startup see [architecture.md](architecture.md). diff --git a/docs/operations.md b/docs/operations.md index 41274133..488f3f86 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -25,6 +25,8 @@ scalo). The default subcommand is `run`. | `version` | print version and build metadata | | `top` | live TUI dashboard polling the running loader's `/metrics` | | `metrics-manifest` | print the metric catalogue as JSON without starting the loader | +| `config-schema` | write `config-schema.{json,yaml}` and `capability-catalog.{json,yaml}` (`--dir`, default `docs`) | +| `generate-artefacts` | write the metrics manifest, the deployment contract and the container spec (`--output-dir`, default `docs`) | Loader settings take `DFE_LOADER_*` environment variables: flat (`DFE_LOADER_CLICKHOUSE_HOSTS`) or nested on a double underscore @@ -34,72 +36,38 @@ env names (`LOG_LEVEL`, `METRICS_ADDR`). See ## Check before you boot -`config-check` runs the same validation the orchestrator runs at startup, so a -bad config fails fast and visibly rather than at first insert. The guards -include the transport/format matrix: +`config-check` loads the cascade and runs `Config::validate`, so an empty broker or host list, an unknown protocol, an unread TLS key, a zero flush trigger or a default table that is `dlq_load.table` fails before boot. Two checks run only when the pipeline starts, before the readiness probe, so `config-check` passes them: - `clickhouse.insert_format: jsoneachrow` with `clickhouse.protocol: native` is refused -- JSONEachRow needs HTTP. Use `protocol: http`, or the `rowbinary` default (which works on both). -- Port/transport mismatches (native on 8123, HTTP on 9000/9440) are flagged. +- HTTP on a native port (9000, 9440) is refused, and native on 8123 is logged as a warning. See [clickhouse/insert-formats.md](clickhouse/insert-formats.md). ## Observe -- **Metrics** -- Prometheus text on `/metrics`. The loader emits the DFE metric - groups (dual-emit: the older names such as `buffer_rows` and - `batches_flushed_total` alongside the standard group names), plus rdkafka consumer stats and config-reload counters. On the direct - gRPC transport there are no broker offsets, so `kafka_offsets_committed_total` - and the consumer offset counters stay flat while the transport counters move. -- **Computed columns** -- a CEL expression that fails leaves its column unset and the record goes on. `dfe_loader_computed_eval_failures_total{table, column, stage}` counts each failure, and the first runtime failure per table and column logs a warning with the error. See [clickhouse/ddl-directives.md](clickhouse/ddl-directives.md). -- **Catalogue** -- `dfe-loader metrics-manifest` prints every metric the loader - registers, so a dashboard or scaling rule can be written against the - catalogue rather than a running pod. -- **Counted in records** -- a message carrying several records is split into one - record per element before anything counts it, so `messages_received_total` and - `events_per_second` read in records, matching `rows_inserted_total`. The two - wire formats are counted apart, so the pair that moves says which producer is - batching: `dfe_loader_batched_array_messages_total` / - `..._batched_array_records_total` for a JSON array, and - `dfe_loader_batched_ndjson_messages_total` / `..._batched_ndjson_records_total` - for newline-separated records. -- **`top`** -- `dfe-loader top` renders those metrics as a TUI. `--once`, - `--json`, and `--filter` make it scriptable; the JSON/TSV output pipes into - shell tooling. -- **Health** -- the probes (`/livez`, `/readyz`) from the scalo health pillar, for Kubernetes liveness/readiness/startup. `/readyz` stays 503 until ClickHouse accepts the configured user and answers reads of `system.tables` and `system.columns` for the configured database. Each refusal is logged with ClickHouse's error code (516 for a wrong password) and retried on the `sink_stack` backoff, a wait that doubles from `sink_stack.min_backoff_ms` (100 ms) to `sink_stack.max_backoff_ms` (30 s), plus up to as much again in jitter. `pipeline_ready` reads 1 once it passes. A ClickHouse config that can never work, such as `http` on port 9000 or 9440, is refused before the probe and the loader exits. Intake waits on the same check: until it passes the loader joins no consumer group and opens no gRPC listener, so a pod with a wrong password takes no partitions from the pods that can write, and a shutdown before then exits without starting the pipeline. Kafka and the gRPC peer play no part in readiness. -- **Broker not up yet** -- a transport that cannot start, such as Kafka auto-discovery with no broker reachable, is retried on the same backoff instead of ending the process. Each failure is logged at `warn` with the error and counted in `pipeline_retries_total{stage="transport_start"}`. A config the transport refuses still exits, since no retry fixes it, and a shutdown during the retries exits cleanly. One attempt can take the 10 s Kafka metadata timeout, so a shutdown may wait that long. - -## Hot-reload - -Config changes are picked up live. Fields that only affect the running pipeline -(buffer thresholds, capture modes, routing) apply on the next batch. Fields baked into a connection or client at build time -(`clickhouse.protocol`, ClickHouse URL/hosts, `insert_format`, TLS settings, -Kafka connection, `routing.dlq`, `field_mapping`, `enrichment`) are restart-required: the reloader logs a warning naming the -field and keeps the running value rather than applying a half-change. See -[configuration.md](configuration.md#hot-reload). +- Prometheus text on `/metrics`: scalo's DFE metric groups, the older names such as `buffer_rows` and `batches_flushed_total` emitted beside them, consumer stats and config-reload counters. On the gRPC transport there are no broker offsets, so `kafka_offsets_committed_total` and the consumer offset counters stay flat while the transport counters move. +- `dfe-loader metrics-manifest` prints every metric the loader registers, so a dashboard or scaling rule can be written against it rather than a running pod. The per-table series appear on first use and are not in it: `dfe_loader_unknown_table_fallback_total`, `dfe_loader_permanent_reject_total`, `loader_messages_by_table_total`, `loader_json_max_paths_exceeded_total`, `loader_buffer_rows_by_table` and `loader_buffer_bytes_by_table`. +- A message carrying several records is split before anything counts it, so `messages_received_total` and `events_per_second` read in records, like `rows_inserted_total`. `dfe_loader_batched_array_*` counts JSON arrays and `dfe_loader_batched_ndjson_*` newline-separated records, so the pair that moves names the producer that batches. +- A failing CEL expression leaves its column unset and the record goes on, counted in `dfe_loader_computed_eval_failures_total{table, column, stage}`. See [clickhouse/ddl-directives.md](clickhouse/ddl-directives.md). +- `dfe-loader top` renders the metrics as a TUI. `--once` prints one scrape to stdout, `--json` prints it as JSON, and `--filter` takes a name regex. +- `/livez` and `/readyz` come from scalo's health pillar. `/readyz` stays 503 until ClickHouse accepts the configured user and answers reads of `system.tables` and `system.columns` for the configured database, and `pipeline_ready` reads 1 once it passes. Each refusal is logged with ClickHouse's code (516 for a wrong password) and retried on the `sink_stack` backoff, doubling from `sink_stack.min_backoff_ms` (100 ms) to `sink_stack.max_backoff_ms` (30 s) plus up to as much again in jitter. Until it passes the loader joins no consumer group and opens no gRPC listener, so a pod with a wrong password takes no partitions from the pods that can write. Kafka and the gRPC peer play no part in readiness. +- A transport that cannot start, such as Kafka with no broker reachable, is retried on the same backoff, logged at `warn` and counted in `pipeline_retries_total{stage="transport_start"}`. A config the transport refuses exits, and one attempt can take the 10 s Kafka metadata timeout, so a shutdown may wait that long. -## When inserts fail - -The insert tail is built to degrade, not stall: +Hot-reload rules are in [configuration.md](configuration.md#hot-reload). -- **Batch salvage** -- a data error binary-splits the batch to isolate the bad - row(s); only those go to the DLQ, the good rows still land. -- **Sink-down gate** -- a flush cycle in which every insert failed opens the - circuit gate on the scaling signal (`scaling_circuit_open`), which - zeroes the pressure KEDA reads: more pods cannot relieve a dead ClickHouse. - The next successful insert closes it. A failed insert never sends its batch to - the DLQ. The batch is held and retried (below). -- **Schema-cache recovery** -- a drift error (a column added by an out-of-band - `ALTER`) invalidates the cached schema and the retry re-fetches and - re-encodes. Any server code outside the retry list (a busy server, the network, Keeper, the disk or object store under the table, or an operator fix such as a missing table or grant) is a refusal of the rows: against a table that has not changed it is the row's own data, never drift, so salvage dead-letters that row and the rest of the batch lands. See [clickhouse/schema-cache.md](clickhouse/schema-cache.md). -- **One commit per flush cycle** -- Kafka offsets commit once per cycle, after the inserts and the DLQ hand-over. On each partition the commit stops below the lowest offset not placed yet: a buffered row, a row waiting on its schema, a failed batch, or a dead letter the DLQ refused. The loader holds a failed batch or refused dead letter and retries it with jittered backoff until it lands. A dead letter no DLQ backend can ever hold -- one over a Kafka-only DLQ's `message.max.bytes` as the DLQ writes it -- is dropped and counted in `pipeline_dead_letters_dropped_total{reason="too_large"}` instead, so it never holds a commit for good. A held batch whose table ClickHouse has since reported absent is retried against the default table, where new records for that table already go, and counted in `dfe_loader_unknown_table_fallback_total`. When a routing reload removes the source a batch is held for, the loader asks ClickHouse about its table at once, so the batch moves on the next retry, within about `buffer.flush_age_secs`. A table the routing still names is expected back and is not asked about. For any other dropped table the loader only learns it is gone when it next re-reads that table's schema, so until then the affected partition's commit stays below the held batch -- up to about `schema.cache_ttl_secs` (default 300 s) plus the 60 s schema refresh interval. Rows above the floor re-deliver as duplicates after a restart (at-least-once). +## When inserts fail -## ClickHouse outages +The insert tail degrades rather than stalls: -Readiness ignores an outage, since flipping it would pull every loader out of its Service at once. `clickhouse_connected` reports it: 1 once ClickHouse answered the startup probe or the latest insert, 0 once an insert went unanswered. An idle loader keeps its last reading. +- A data error binary-splits the batch down to the bad rows, which go to the DLQ, and the rest land. +- A flush cycle in which every insert failed opens the circuit gate on the scaling signal (`scaling_circuit_open`), which zeroes the pressure KEDA reads, since more pods cannot relieve a dead ClickHouse. The next successful insert closes it. A failed insert never sends its batch to the DLQ. +- A column dropped or retyped by an out-of-band `ALTER` is drift: the cached schema is dropped and the retry re-encodes against a fresh read. A server code on the retry list (a busy server, the network, Keeper, the disk or object store under the table, or an operator fix such as a missing table or grant) holds the batch. Any other code against a table that has not changed is the row's own data, so salvage dead-letters that row. See [clickhouse/schema-cache.md](clickhouse/schema-cache.md). +- Kafka offsets commit once per cycle, after the inserts and the DLQ hand-over. On each partition the commit stops below the lowest offset not placed yet: a buffered row, a row waiting on its schema, a failed batch, or a dead letter the DLQ refused. The loader holds a failed batch or refused dead letter and retries it with jittered backoff until it lands. A dead letter over a Kafka-only DLQ's `message.max.bytes` is dropped and counted in `pipeline_dead_letters_dropped_total{reason="too_large"}`, so it never holds a commit for good. Rows above the floor re-deliver as duplicates after a restart. +- A held batch whose table ClickHouse has since reported absent is retried against the default table and counted in `dfe_loader_unknown_table_fallback_total`. A routing reload that removes the batch's source makes the loader ask about its table at once. Any other dropped table is found gone at its next schema read, up to about `schema.cache_ttl_secs` plus the 60 s refresh interval, and the partition's commit stays below the held batch until then. -Failed batches are held. On Kafka their reservations grow until the memory governor pauses the assigned partitions. On gRPC a held Push is answered `UNAVAILABLE` and its sender retries. +Readiness ignores an outage, since flipping it would pull every loader out of its Service at once. `clickhouse_connected` reports it: 1 once ClickHouse answered the startup probe or the latest insert, 0 once an insert went unanswered. On Kafka the held batches' reservations grow until the memory governor pauses the assigned partitions, and on gRPC a held Push is answered `UNAVAILABLE`. ## Rebalances @@ -109,10 +77,7 @@ With `kafka.acknowledgements.enabled: false` offsets commit at receipt, so the b ## DLQ -Dead-lettered messages go to the DLQ (Kafka primary, file fallback). Routing -misses, unparseable payloads, and salvaged bad rows all land there with enough -context to replay or inspect. DLQ and consumer errors are debounced in the logs -so a burst does not flood. +Dead-lettered messages go to the DLQ (Kafka first, file fallback, file only on gRPC). Payloads that are not JSON or fail processing, records whose schema never arrived within `schema.pending_max_age_secs`, and salvaged bad rows land there with their reason. A routing miss does not: it lands in the default table. DLQ and consumer errors are debounced in the logs so a burst does not flood. A JSON record nested deeper than 64 levels is dead-lettered as it arrives, before any stage parses it, with the reason `payload nesting exceeds the maximum parse depth of 64`, and counted on `validation_failures_total{reason="out_of_range"}`. A batch too deep to split goes to the DLQ whole. The bound is the one scalo's parse path uses. @@ -130,4 +95,4 @@ The figure it checks is what the kernel charges the loader. That is the cgroup's ## Scaling -The contract's ScaledObject scales on CPU and carries a disabled Kafka lag trigger, so raw consumer lag never scales the loader. In a DFE deployment, dfe-infra's values set the CPU target to 70%, add the scaling-pressure trigger (target 70, read through dfe-keda-shim) and a 120 s cooldown. The slim profile clears the pressure trigger, so slim scales on CPU alone. The engine's [KEDA scaling page](https://github.com/hyperi-io/dfe-engine/blob/main/docs/deployment/keda-scaling.md) covers the triggers for every app. +The contract's ScaledObject scales on CPU and carries a disabled Kafka lag trigger, so raw consumer lag never scales the loader. In a DFE deployment, dfe-infra's values set the CPU target to 70% and add the scaling-pressure trigger (target 70, read through dfe-keda-shim). The cooldown is the contract's 300 s. The slim profile clears the pressure trigger, so slim scales on CPU alone. The engine's [KEDA scaling page](https://github.com/hyperi-io/dfe-engine/blob/main/docs/deployment/keda-scaling.md) covers the triggers for every app. diff --git a/docs/performance/README.md b/docs/performance/README.md index 553b3ec1..cc395b81 100644 --- a/docs/performance/README.md +++ b/docs/performance/README.md @@ -10,9 +10,7 @@ # Performance -Where the time goes and how to get it back. dfe-loader splits performance work -into build-time optimisations (allocator, LTO, PGO, BOLT -- now CI-automated) -and runtime tuning (batch sizes, concurrent inserts), with profiling tools for -finding the hot spots in between. +Where the time goes and how to get it back. dfe-loader splits performance work into build-time optimisations (allocator, LTO, PGO, BOLT), which hyperi-ci applies, and runtime tuning (batch sizes), with profiling tools for finding the hot spots in between. -- [overview.md](overview.md) -- the optimisation guide: jemalloc, LTO, PGO and BOLT build optimisations, insert-throughput and batch tuning, and CPU/memory/flame-graph profiling. +- [overview.md](overview.md) -- what CI builds, the allocator, batch tuning, and CPU, heap and flame-graph profiling. +- [history.md](history.md) -- why the loader's parser and insert encoding are the ones it uses. diff --git a/docs/performance/history.md b/docs/performance/history.md index d90328ea..c20d473e 100644 --- a/docs/performance/history.md +++ b/docs/performance/history.md @@ -11,10 +11,7 @@ # Decision and parser-selection history This is the rationale trail behind the loader's hot-path choices -- which JSON -parser, which insert encoding, and why. Each entry is a permanent record of a -targeted bake-off against the specific DFE use-case, not a general benchmark. -Keep it when you re-evaluate: the bake-offs cost real time, and the conclusions -are still load-bearing. +parser, which insert encoding, and why. Each entry records a targeted bake-off against the DFE use-case, not a general benchmark. For the current insert path and the dynamic encoder, see [../clickhouse/insert-formats.md](../clickhouse/insert-formats.md) and @@ -23,15 +20,9 @@ the "why we landed here", not the "how it works now". ## Why RowBinary is the default -RowBinary ships pre-typed bytes. The server does no JSON parse and no type -inference -- it reads column values straight off the wire in `system.columns` -order. For a high-throughput loader that is the difference between ClickHouse -spending CPU on ingest formatting and spending it on merges. +RowBinary ships pre-typed bytes. The server parses no row text and infers no types, reading each value off the wire in the order the INSERT names the columns. Only a JSON column's value travels as text for the server to parse. For a high-throughput loader that is the difference between ClickHouse spending CPU on ingest formatting and spending it on merges. -The dynamic encoder turns one `Map` plus its reflected schema -into RowBinary bytes client-side. ClickHouse receives pre-typed binary and skips -parsing entirely. Missing columns take their server-side default; extra columns -are dropped. +The dynamic encoder turns one `Map` plus its reflected schema into RowBinary bytes client-side. A defaulted column the row leaves out takes its server-side default, and a field with no column is dropped. JSONEachRow is retained as a fallback, not a throughput choice. It is self-describing and forgiving -- the server coerces types and tolerates schema @@ -43,8 +34,8 @@ dynamic encoder does not yet cover. An early trial used `clickhouse-arrow` (Arrow IPC columnar inserts) as a candidate fast path. It was dropped: RowBinary matched Arrow throughput for this -workload because end-to-end insert time is network-dominated, not encode- -dominated, so the columnar advantage did not show up at the pipeline level. +workload because insert time from flush to answer is dominated by the network, not the encode, +so the columnar advantage did not show up at the pipeline level. RowBinary also kept the encoder a single row-wise/per-column path off the same reflected schema, with no extra Arrow dependency or schema-mapping layer. @@ -65,14 +56,7 @@ Bench: `benches/bakeoff.rs`. ### Mison -- evaluated, rejected (2025-Q4) -- **Result:** 4-7% throughput improvement over sonic-rs in targeted - benchmarks. -- **Rejection reason:** the improvement was insufficient to justify maintaining - a separate codebase (mison required a fork with custom Rust bindings). The - cost/benefit was negative. All mison code was deleted in commit - `2a7a635`. -- **Bench:** separate bake-off repo (not retained -- results documented here - only). +Mison gave a 4-7% throughput improvement over sonic-rs in targeted benchmarks, too little to justify maintaining a separate codebase: it needed a fork with custom Rust bindings. All mison code was deleted in commit `2a7a635`. The bench lived in a separate bake-off repo, not kept, and these results are its only record. ### simd-json (0.17) -- evaluated, rejected (2026-03-11) @@ -99,24 +83,23 @@ Key observations: - At 30 schema columns `sonic_dom` is **2.7x faster** than `sonic_selective`. - At 15 schema columns with a nested payload, **4.2x faster** (`get_from_slice` scans the whole doc per miss). -- At 15 schema columns with a flat payload, essentially tied (59.7 ms vs +- At 15 schema columns with a flat payload, tied (59.7 ms vs 58.1 ms). - Routing: simd-json 23% faster for flat30, ~2% for nested (within noise). **Rejection reasons:** -1. **Mandatory clone.** simd-json requires `&mut [u8]` (in-place string - unescaping). The DFE pipeline holds payloads as `Arc<[u8]>` for zero-copy - `_json` splice. Every parse would need `raw.to_vec()` -- a full memcpy per - message. This is architecturally incompatible. -2. **Net gain too small.** After implementing `sonic_dom` (single full parse, - same dependency), simd-json is only ~28-33% faster for flat extraction and - ~12% for nested. Extraction is <10% of total pipeline time (dominated by - 40-75 ms network I/O per batch). Net pipeline improvement: ~2-3% -- below - the 5% mison rejection threshold. -3. **No lazy field access.** sonic-rs `get_from_slice` navigates to a field - without building any DOM. simd-json has no equivalent -- it always builds a - full tape first. +1. A mandatory clone. simd-json requires `&mut [u8]` (in-place string + unescaping), and the DFE pipeline holds payloads as `Arc<[u8]>` for the + `_json` splice, so every parse would need `raw.to_vec()`, a full memcpy per + message. +2. Too small a net gain. Against `sonic_dom` (single full parse, + same dependency), simd-json was ~28-33% faster for flat extraction and + ~12% for nested. Extraction was <10% of total pipeline time (dominated by + 40-75 ms network I/O per batch), so the net pipeline improvement was ~2-3%, + below the 5% mison rejection threshold. +3. No lazy field access. sonic-rs `get_from_slice` navigates to a field + without building any DOM, and simd-json always builds a full tape first. **Action taken:** `HeaderExtractor` switched from `get_from_slice` x N to `sonic_rs::from_slice` + O(1) hash lookups -- a 2.7-4.2x extraction speedup @@ -127,55 +110,10 @@ Key observations: The choices below predate the current module layout but remain the reasons the hot path is built the way it is. -### Per-table buffering +Each destination `db.table` has its own row buffer. An INSERT goes to one table, so a batch holds one table's rows. High-volume tables flush more often, and low-volume tables wait for the age trigger. A table's failed batch holds back the commit of only the partitions its rows came from. Each row is a `Map`, so the buffer layer needs no fixed schema, and rows accumulate only until a flush threshold (rows, bytes or age). -Each destination `db.table` has its own row buffer, because: +`_json` is never inserted into the promoted `Map`. The payload bytes are copied once into a reference-counted `Arc<[u8]>` and spliced as `_json` at serialise time, never re-serialised. That is why the extractor promotes only the schema's columns instead of flattening the whole payload. -- **ClickHouse target.** Inserts are per-table; all rows in a batch go to one - table. -- **Independent flush.** High-volume tables flush more often; low-volume tables - wait for the age trigger. One table's failure does not block another's - offset commit. -- **Schema flexibility.** Each row is a `Map` -- no fixed schema - required at the buffer layer. -- **Bounded memory.** Rows accumulate until a flush threshold (rows, bytes, or - age). +A payload must open a JSON object or array (`{` or `[` after any leading whitespace). Anything else, MessagePack included, is dead-lettered as not JSON before a parser reads it. -### Zero-copy `_json` - -`_json` is never inserted into the promoted `Map`. The full payload is held once -as a reference-counted `Arc<[u8]>` and spliced as `_json` only at serialise -time -- no heap allocation until the final insert body is assembled. This is why -the extractor promotes only schema-matched columns (~10-30 entries) instead of -flattening the whole payload (200+ fields). - -### Single-byte format auto-detect - -JSON object/array starts (`{`, `[`) are always `< 0x80`; MsgPack map/array -starts are always `>= 0x80`. The ranges are mutually exclusive for top-level -structured event data, so format detection is one byte read and one branch -- -O(1), sub-nanosecond. JSON and MsgPack can be mixed on the same topic; after -detection the `Arc` bytes are always valid JSON. (Edge case: bare MsgPack -primitives at top level would misdetect as JSON, but event pipelines do not -emit those.) - -### Delta coercion only - -Coercion runs only on the promoted columns and only for the four cases the wire -format cannot fix server-side. Everything else (String<->numeric, 1/0->Bool, -null->DEFAULT) is left to ClickHouse: - -| Delta coercion | Why needed | -|---|---| -| Epoch ms/us/ns -> DateTime64 ISO string | CH takes the integer at face value at column precision | -| ISO 8601 `T` separator -> space | Default `basic` parser rejects `T` | -| UUID without hyphens -> RFC 4122 | CH UUID rejects bare hex | -| IPv4 integer -> dotted-decimal | CH IPv4 rejects integers | - -The encoder carries these cases, in `dfe-clickhouse`'s `dynamic` module. - -## Future optimisations - -- **SIMD GeoIP** -- vectorised IP lookups across a batch. -- **Workspace crate extraction** -- split `crates/buffer` and friends out of - the binary crate. +With RowBinary the server takes the bytes as typed, so the encoder in `dfe-clickhouse`'s `dynamic` module converts each value to its column's type itself: an epoch read by magnitude at the column's precision, zoned text to UTC, hyphen-less UUID hex, an integer IPv4, the geo forms and a single value bound for an `Array`. A value it cannot convert fails that row, which salvage dead-letters. diff --git a/docs/performance/overview.md b/docs/performance/overview.md index 4d262bcd..d7d666f1 100644 --- a/docs/performance/overview.md +++ b/docs/performance/overview.md @@ -10,282 +10,75 @@ # Performance optimisation -dfe-loader is a high-throughput hot path, so performance work splits cleanly in -two: build-time optimisations (allocator, LTO, PGO, BOLT) that CI now automates, -and runtime tuning (batch sizes, concurrent inserts) that lives in config. This -page covers both, plus the profiling tools to find out where the time goes. +Performance work splits in two: build-time optimisations (allocator, LTO, PGO, BOLT) that hyperi-ci applies, and runtime tuning (batch sizes) that lives in config. This page covers both, plus the tools for finding where the time goes. ```mermaid flowchart TB - subgraph Build["Build-time (CI-automated)"] + subgraph Build["Build-time (hyperi-ci)"] ALLOC["jemalloc allocator"] LTO["LTO (thin/fat)"] - PGO["PGO (+10-20%)"] - BOLT["BOLT (+5-15% on top of PGO)"] + PGO["PGO"] + BOLT["BOLT (after PGO)"] end - subgraph Runtime["Runtime (config-tuned)"] + subgraph Runtime["Runtime (config)"] BATCH["Batch sizes
flush_rows / flush_bytes / flush_age_secs"] - CONC["max_concurrent_inserts"] end BIN["Optimised binary"] ALLOC & LTO & PGO & BOLT --> BIN BIN --> RUN["Running loader"] - BATCH & CONC --> RUN + BATCH --> RUN ``` -> **CI integration notice (2026-04+):** -> hyperi-ci now applies **Tier 1** build optimisations (jemalloc allocator + fat -> LTO) automatically on `beta` and `release` channels. **Tier 2** (PGO + BOLT) -> is opt-in per project via `.hyperi-ci.yaml`. The manual `cargo pgo` / `cargo -> pgo bolt` commands documented below are now for **local development and -> one-off profiling only** -- CI handles them automatically when configured. -> -> See the project standards for the full contract and the channel x tier -> table that governs CI build optimisation behaviour. -> -> The rest of this document remains relevant for: -> -> - Understanding *why* each optimisation matters (the CI just automates them) -> - Local profiling / one-off performance investigation -> - Runtime tuning (batch sizes, concurrent inserts -- NOT build-time concerns) +## What CI builds -## Build optimisations +hyperi-ci picks the optimisations by release channel (`src/hyperi_ci/languages/rust/optimize.py` in hyperi-ci): -### Profile-Guided Optimisation (PGO) +| Channel | Allocator | LTO | PGO and BOLT | +|---|---|---|---| +| alpha | jemalloc | thin | no | +| beta | jemalloc | fat | no | +| release | jemalloc | fat | PGO when the repo has a workload, BOLT with it unless `build.rust.optimize.bolt.enabled: false` | -PGO can provide **10-20% performance improvement** by using runtime profiling -data to guide compiler optimisations. See -[cargo-pgo](https://github.com/Kobzol/cargo-pgo). +This repo's workload is `scripts/pgo-workload.sh`, and `.hyperi-ci.yaml` builds its `pgo-driver` before the workload clock starts. The workload must exercise the real hot path: parse, transform, serialise and insert. Port checks, health probes and startup paths mislead the compiler and can make PGO a net loss. Confirm what shipped from the published artefact, not from the build log. hyperi-ci's guide is `docs/runtime/pgo-bolt.md` in its repo. -> **CI:** hyperi-ci runs this automatically on the `release` channel when -> configured. The commands below are for **local profiling only**. +`Cargo.toml` sets `lto = "thin"` and `codegen-units = 1` for `release`, and CI overrides the LTO mode per channel through `CARGO_PROFILE_RELEASE_LTO`, leaving `Cargo.toml` untouched. -#### Local setup (manual) +## The allocator -```bash -# Install the tool -cargo install cargo-pgo -rustup component add llvm-tools-preview - -# Step 1: Build instrumented binary -cargo pgo build - -# Step 2: Run with representative workload -./target/x86_64-unknown-linux-gnu/release/dfe-loader \ - --config test-config.yaml \ - # Run for sufficient time to gather profiles - -# Step 3: Build optimised binary -cargo pgo optimize -``` - -#### Best practices for PGO workloads - -- Run the instrumented binary with a **representative workload** -- Include all common code paths (various message types, error handling) -- Profile for at least 5-10 minutes of sustained load -- Re-profile when making significant code changes -- **CRITICAL:** The workload must exercise REAL data-processing paths -- - parse, transform, serialise, insert. **Do NOT use port checks, health - probes, or "does the service start" tests.** Profile data from startup / - readiness paths misleads the compiler and produces NEGATIVE PGO gains. - -### BOLT post-link optimisation - -BOLT provides an **additional 5-15% improvement** on top of PGO by optimising -code layout in the final binary. See -[LLVM BOLT](https://github.com/llvm/llvm-project/tree/main/bolt). - -> **CI:** hyperi-ci runs BOLT automatically on the `release` channel when -> `optimize.bolt.enabled: true` is set in `.hyperi-ci.yaml` AND `pgo.enabled` -> is also true (BOLT requires PGO profile data). Linux-only. The commands -> below are for **local profiling only**. - -#### Local setup (manual) - -```bash -# Build with BOLT (requires BOLT installed) -cargo pgo bolt build - -# Run with perf profiling -./target/x86_64-unknown-linux-gnu/release/dfe-loader-bolt-instrumented \ - --config test-config.yaml -# BOLT uses perf data, so run: perf record -e cycles:u -j any,u -- ./binary - -# Optimise with BOLT -cargo pgo bolt optimize - -# Combined PGO + BOLT -cargo pgo optimize -cargo pgo bolt optimize --with-pgo -``` - -#### Requirements - -- Linux only (ELF binaries) -- x86_64 or AArch64 -- LLVM BOLT tools installed - -### Link-Time Optimisation (LTO) - -Already configured in `Cargo.toml`: - -```toml -[profile.release] -lto = "thin" # Use "fat" for max perf, slower compile -codegen-units = 1 # Better optimisation -``` - -For maximum performance: - -```toml -lto = "fat" # More aggressive, 2-3x longer compile -``` - -## Memory allocator - -DFE policy (2026-04-17): **jemalloc at every channel, no mimalloc**. One -allocator across the fleet means one profiling story (`jeprof`), one set of -perf-trace symbols, one debugging playbook. - -hyperi-ci adds `--features jemalloc` automatically on every channel -(spike/alpha/beta/release). For local builds, opt in with: +The `jemalloc` feature installs `tikv-jemallocator` as the `#[global_allocator]` in `src/main.rs`. `default = []`, so a local build opts in: ```bash cargo build --release --features jemalloc ``` -### Verification on stripped release binaries - -Release builds set `strip = true`, so `nm` will not see allocator symbols. -Use `strings`: +Release builds set `strip = true`, so `nm` sees no allocator symbols. `strings` does: ```bash strings target/release/dfe-loader | grep -ciE 'jemalloc|je_mallctl' -# Expect > 0 ``` -Expect a ~3.5% binary-size increase from the static jemalloc link -(per dfe-receiver canary 1, 2026-04-17). - -### Benchmark (jemalloc vs system glibc, OLAP-style workload) - -| Metric | System (glibc) | jemalloc | -|--------|----------------|----------| -| Throughput | Baseline | +15-25% | -| RSS | Baseline | +5-10% | -| Large-batch p99 latency | Baseline | -10-20% | - -## Insert throughput - -dfe-loader writes to ClickHouse through crates.io `clickhouse` over HTTP, or -`dfe-clickhouse`'s `TcpClient` over the native protocol, with the dynamic -RowBinary layer from `dfe-clickhouse`'s `dynamic` module. The default path is RowBinary via -`DynamicInsert` (schema-reflected typed encoding -- ClickHouse skips JSON -parsing), over HTTP through `Client::insert_formatted_with` -(`FORMAT RowBinaryWithDefaults`), or as `FORMAT Native` blocks with -`clickhouse.protocol: native`. No throughput comparison of the two has been -measured. Set -`insert_format = "json_each_row"` to fall back to `FORMAT JSONEachRow` over HTTP -via `Client::insert_formatted_with`. Tuning below applies to both formats. - -### Batch size - -Larger batches amortise per-insert overhead. Default flush thresholds: - -- `flush_rows = 20000` rows -- `flush_bytes = 67108864` (64 MiB) -- `flush_age_secs = 5` - -Increase `flush_rows` and `flush_bytes` for higher throughput at the cost of -latency. - -### Allocator feature - -The `jemalloc` feature is declared in `Cargo.toml` and applies globally via -`#[global_allocator]` in `src/main.rs`: - -```toml -[features] -jemalloc = ["dep:tikv-jemallocator", "dep:tikv-jemalloc-ctl"] -``` - -mimalloc was removed on 2026-04-17 per the DFE allocator policy. +`tikv-jemallocator` prefixes jemalloc's symbols, so its options are read from `_RJEM_MALLOC_CONF`, not `MALLOC_CONF`. Heap profiling needs the crate's `profiling` feature, which this build does not enable, and symbols to read the profile with: the `profiling` cargo profile keeps them. ## Runtime tuning -### Batch sizes - | Setting | Default | Tuning guidance | |---------|---------|-----------------| -| `flush_rows` | 20,000 | Increase for throughput, decrease for latency | -| `flush_bytes` | 64 MiB | A ceiling for oversized rows: at 1-2 KB rows the 20,000-row trigger fires first | -| `flush_age_secs` | 5s | Lower for real-time, higher for batch | - -### Concurrent inserts - -```yaml -clickhouse: - max_concurrent_inserts: 8 # Default, good for most workloads -``` +| `buffer.flush_rows` | 20,000 | Increase for throughput, decrease for latency | +| `buffer.flush_bytes` | 64 MiB | A ceiling for oversized rows: at 1-2 KB rows the 20,000-row trigger fires first | +| `buffer.flush_age_secs` | 5 | Lower for real-time, higher for batch | -- Increase for high-latency ClickHouse connections -- Decrease if ClickHouse is CPU-bound +Insert concurrency is not a setting: at most 8 inserts run at once. RowBinary over HTTP (`FORMAT RowBinaryWithNamesAndTypes`) is the default, and `clickhouse.protocol: native` sends `FORMAT Native` blocks instead. dfe-clickhouse's [insert-formats.md](https://github.com/hyperi-io/dfe-clickhouse-rs/blob/main/docs/insert-formats.md) measures the two against each other. `insert_format: json_each_row` falls back to `FORMAT JSONEachRow` over HTTP. ## Profiling -### CPU profiling with perf - ```bash +# CPU, with perf perf record -g ./target/release/dfe-loader --config config.yaml perf report -``` -### Memory profiling with jemalloc - -```bash -# Enable jemalloc profiling -export MALLOC_CONF="prof:true,prof_prefix:jeprof.out" -./target/release/dfe-loader --config config.yaml - -# Analyse -jeprof --svg ./target/release/dfe-loader jeprof.out.*.heap > heap.svg -``` - -### Flame graphs - -```bash -# Install +# Flame graph cargo install flamegraph - -# Generate cargo flamegraph --bin dfe-loader -- --config config.yaml ``` -## Recommended production build - -**For production releases: let hyperi-ci do this.** Push to a `release`-channel -project and CI applies the full optimisation pipeline automatically. - -**For local development / one-off profiling:** - -```bash -# Full optimisation build -cargo build --release --features jemalloc - -# With PGO (if you have profile data) -cargo pgo optimize --features jemalloc - -# With PGO + BOLT (maximum optimisation) -cargo pgo optimize --features jemalloc -cargo pgo bolt optimize --with-pgo -``` - -## References - -- [cargo-pgo documentation](https://github.com/Kobzol/cargo-pgo) -- [The Rust Performance Book](https://nnethercote.github.io/perf-book/) -- [LLVM BOLT](https://github.com/llvm/llvm-project/tree/main/bolt) -- [jemalloc tuning](https://github.com/jemalloc/jemalloc/wiki/Getting-Started) -- hyperi-ci's PGO and BOLT runtime guide (`runtime/pgo-bolt.md` in the hyperi-ci - repo's docs) -- tiers, channels and workload design +For a local PGO or BOLT run, [cargo-pgo](https://github.com/Kobzol/cargo-pgo) builds the instrumented binary (`cargo pgo build`), takes the profile from a workload run, and rebuilds with it (`cargo pgo optimize`, then `cargo pgo bolt optimize --with-pgo` for BOLT, Linux only). Further reading: [The Rust Performance Book](https://nnethercote.github.io/perf-book/), [LLVM BOLT](https://github.com/llvm/llvm-project/tree/main/bolt) and [jemalloc tuning](https://github.com/jemalloc/jemalloc/wiki/Getting-Started). diff --git a/docs/pipeline/README.md b/docs/pipeline/README.md index c1f682d2..fc911faa 100644 --- a/docs/pipeline/README.md +++ b/docs/pipeline/README.md @@ -21,12 +21,12 @@ flowchart LR ## Docs -- [overview.md](overview.md) -- the six stages, in order. +- [overview.md](overview.md) -- the stages, in order. - [routing.md](routing.md) -- choosing `db.table` from message fields. - [capture-modes.md](capture-modes.md) -- what is stored as `_json` / `_raw`, and the three-level precedence. - [enrichment.md](enrichment.md) -- GeoIP, reputation, risk scoring. -- [parallelism.md](parallelism.md) -- the parallelisation playbook. +- [parallelism.md](parallelism.md) -- the parallel and sequential phases of a batch. For the buffer-and-insert tail see [../clickhouse/insert-formats.md](../clickhouse/insert-formats.md) and the diff --git a/docs/pipeline/capture-modes.md b/docs/pipeline/capture-modes.md index e3be850b..2ceb117f 100644 --- a/docs/pipeline/capture-modes.md +++ b/docs/pipeline/capture-modes.md @@ -19,15 +19,19 @@ governs the two payload-retention columns. |------|---------|--------|-----| | `full` (default) | full payload (JSON) | the record's own raw field, else NULL | path queries plus text search | | `json_only` | full payload (JSON) | NULL | path queries, never a `_raw` copy | -| `raw_only` | NULL | full source payload (String) | save ClickHouse CPU, no JSON-type overhead | -| `extracted_only` | NULL | NULL | promoted columns only | +| `raw_only` | NULL, unless a directive fills it | full source payload (String) | save ClickHouse CPU, no JSON-type overhead | +| `extracted_only` | NULL, unless a directive fills it | NULL | promoted columns only | `_raw` and `_json` are not copies of each other. `_json` is the parsed payload as a native ClickHouse JSON value (path-based queries). `_raw` is the original bytes as received (the tailed log line, DB row, or raw syslog). A table can keep one, both, or neither. -Under `full`, a JSON record fills `_raw` only from a `_raw` field of its own, kept as sent, such as a line the receiver captured. Any other JSON record is kept once, in `_json`, and `_raw` stays NULL. A text field such as `logoriginal` is not copied into `_raw` by default, because `_json` already stores its text. To promote one for a source that needs it, name it in `metadata.raw_source_fields` (first match wins) or in an `@renamed` directive on the `_raw` column. +Under `full`, a JSON record fills `_raw` only from a raw field of its own (`raw` or `_raw`), kept as sent, such as a line the receiver captured. Any other JSON record is kept once, in `_json`, and `_raw` stays NULL. A text field such as `logoriginal` is not copied into `_raw` by default, because `_json` already stores its text. To promote one for a source that needs it, name it in `metadata.raw_source_fields` (first match wins) or in an `@renamed` directive on the `_raw` column. + +`metadata.capture_raw` (default `true`) is what lets `raw_source_fields` fill `_raw`. `metadata.capture_json` is read by nothing: `capture_mode` decides `_json`. + +A directive on the `_json` column itself, such as `@source: record`, makes `_json` a promoted column. It then holds that field's value, or an empty object when the record has none, under every mode, `raw_only` and `extracted_only` included, and the payload is not spliced in. The DLQ table dfe-schemas renders uses `@source: _json` to keep a dead letter's original payload. ## Precedence @@ -43,23 +47,20 @@ flowchart TB GLOB -->|else| WIN ``` -- **DDL directive** -- a `@capture_mode: ` comment on the table DDL. The - table owns its capture policy; this wins over any config. See - [../clickhouse/ddl-directives.md](../clickhouse/ddl-directives.md). -- **Per-table config** -- `metadata.table_capture_modes` maps a table to a mode. -- **Global default** -- `metadata.capture_mode`, applied to any table without a - more specific setting. +- A `@capture_mode: ` tag in the table's `COMMENT` wins over any config, since the table owns its capture policy. See [../clickhouse/ddl-directives.md](../clickhouse/ddl-directives.md#table-level-tags). +- `metadata.table_capture_modes` maps a `db.table` to a mode. +- `metadata.capture_mode` applies to any table without a more specific setting. ## The setting drives the action A capture mode is only worth anything if the row that lands matches it. The processor tests assert the observable action, not just that the config parsed -(`src/pipeline/processor.rs`): +(`src/pipeline/processor.rs`), on a table with no directive on `_json`: - `full` -> the written row has `_json` populated, and `_raw` only when the record carried a raw field of its own. - `json_only` -> the written row has `_json` populated and `_raw` NULL, always. - `raw_only` -> the written row has `_raw` = the full payload and `_json` NULL. -- `extracted_only` -> the written row has neither `_json` nor `_raw`; only the +- `extracted_only` -> the written row has neither `_json` nor `_raw`, only the promoted columns. That is the rule for every config-cascade switch in the loader: the test proves diff --git a/docs/pipeline/enrichment.md b/docs/pipeline/enrichment.md index f729649f..18e41748 100644 --- a/docs/pipeline/enrichment.md +++ b/docs/pipeline/enrichment.md @@ -10,7 +10,7 @@ # Enrichment -Enrichment adds derived columns to a row from its IP fields: geographic data, +Enrichment adds derived columns to a row from its IP fields (`enrichment.ip_fields`, first match wins): geographic data, reputation flags, and a composite risk score. Each provider is an independent toggle, off by default. Enriched values are written as flat columns on the promoted row only -- never folded into `_json`, so the payload stays the payload @@ -72,15 +72,13 @@ The credentials are redacted in logs and config dumps. Supply them through the s Enrichment runs after extraction and before buffering, on the promoted row. The lookups are cached, so a hot IP is resolved once and reused. Because the columns -are flat and additive, a table opts in simply by declaring the enrichment +are flat and additive, a table opts in by declaring the enrichment columns in its schema -- an enriched value is written only where the target table has a column for it, the same promotion rule the rest of the pipeline follows. -## The setting drives the action - Like every config-cascade switch in the loader, an enrichment toggle is only meaningful if the row that lands reflects it: enable GeoIP and the geo columns -on the written row are populated; leave it off and they are not. See +on the written row are populated, leave it off and they are not. See [../configuration.md](../configuration.md) and the hot-path context in [overview.md](overview.md). diff --git a/docs/pipeline/overview.md b/docs/pipeline/overview.md index a0e7d9d8..852fc53c 100644 --- a/docs/pipeline/overview.md +++ b/docs/pipeline/overview.md @@ -18,7 +18,7 @@ time. The tail batches per table and inserts. ```mermaid flowchart LR - M["bytes (Kafka/gRPC/Memory)"] + M["bytes (Kafka/gRPC)"] S["split batched messages"] P["JSON parse"] R["route -> db.table"] @@ -50,41 +50,31 @@ untouched, so the format check and the DLQ see exactly what arrived: a scalar array, an empty array, a single object however it is formatted, a truncated tail, and any body whose elements are not all objects. -The split belongs at the consumer rather than at each producer in turn. #128 -taught dfe-receiver not to forward an array; dfe-transform-elastic then lost -five records a message to the same defect in the other wire format (#184). A -batch that reaches the next stage unsplit is refused as an unsplit batch and named as -such, rather than coming back as a ClickHouse encode error on an empty column -or as a parse error about trailing characters. +The split belongs at the consumer, so a producer's choice of wire format cannot lose records. An array that reaches the processor unsplit is refused as `payload is a JSON array, not a record object` rather than coming back as a ClickHouse encode error on an empty column. ## 1. Parse A payload must open a JSON object or array (`{` or `[` after any leading -whitespace) and is parsed with sonic-rs (SIMD). The parse is zero-copy where it -can be -- the original bytes are retained as `Arc<[u8]>` so the full payload can -be written as `_raw` or `_json` later without re-serialising. Anything else is -dead-lettered as `payload is not JSON` with its first 8 bytes in hex, and a -payload that opens like JSON but does not parse is dead-lettered with the parse -error. +whitespace) and is parsed with sonic-rs (SIMD). The original bytes are kept once as `Arc<[u8]>`, so the full payload can be written as `_raw` or `_json` later without re-serialising. Anything else is dead-lettered as `payload is not JSON` with its first 8 bytes in hex, and a payload that opens like JSON but does not parse is dead-lettered with the parse error. -JSON is the only payload format. MessagePack, supported in DFE/XDR 2.0 and 2.1, is deprecated in DFE 2.2 and no longer accepted: the JSON path (SIMD parsing with sonic-rs, zstd on the wire) is fast enough that MessagePack gave no CPU saving. +JSON is the only payload format. MessagePack, which DFE/XDR 2.0 and 2.1 accepted, was removed in DFE 2.2, and a MessagePack record is dead-lettered as not JSON: the JSON path (SIMD parsing with sonic-rs, zstd on the wire) is fast enough that MessagePack gave no CPU saving. ## 2. Route -Routing picks `db.table` from the message before any flattening, using the configured `routing.table_fields` (dot-notation for nested access). No field carrying a value falls back to the topic's source, then `routing.default_table`. The database is `routing.default_db` unless `routing.org_routes` lists the record's `_org_id` tenant. See [routing.md](routing.md). +Routing picks `db.table` from the message before any flattening, using the configured `routing.table_fields` (dot-notation for nested access). No field carrying a value falls back to the topic's source, then `routing.default_table`, and a table ClickHouse says does not exist falls back to `routing.default_table` too. The database is `routing.default_db` unless `routing.org_routes` lists the record's `_org_id` tenant. See [routing.md](routing.md). ## 3. Extract The target table's schema (reflected and cached, see [../clickhouse/schema-cache.md](../clickhouse/schema-cache.md)) decides which fields are promoted to typed columns. The header extractor pulls the common -header fields; the rest of the declared columns are pulled from the payload by +header fields, and the rest of the declared columns are pulled from the payload by name. Anything not promoted stays in the payload, available as `_json`. ## 4. Enrich Optional enrichment (`geoip`, `reputation`, `risk`) injects flat columns onto -the promoted row only -- never into `_json`. Each is a toggle; the lookups are +the promoted row only -- never into `_json`. Each is a toggle, and the lookups are cached. See [enrichment.md](enrichment.md). ## 5. Buffer and capture @@ -100,6 +90,4 @@ The encoder reshapes only the values ClickHouse cannot take as sent: epoch vs IS ## Parallelism -The hot path scales across a worker pool with per-table batching downstream. The -remediation playbook for parallel correctness is in -[parallelism.md](parallelism.md). +The hot path scales across a worker pool with per-table batching downstream. [parallelism.md](parallelism.md) describes the parallel and sequential phases. diff --git a/docs/pipeline/routing.md b/docs/pipeline/routing.md index 745f2f70..f0f3486c 100644 --- a/docs/pipeline/routing.md +++ b/docs/pipeline/routing.md @@ -19,7 +19,7 @@ flowchart TB PIN{"read from a dlq_load topic?"} DLQTBL["dlq_load.table"] TBF{"a table_fields value?"} - TOPIC{"topic is {source}_land / {source}_load?"} + TOPIC{"topic is {source}_land / {source}_load?"} DEFDB["default_db"] ORGDB["the org's database"] DEFTBL["default_table"] @@ -44,7 +44,7 @@ flowchart TB |---------|--------| | `routing.table_fields` | fields whose value selects the table, first one carrying a value wins. Dot-notation reaches nested fields (`event.category`). Default `["_source"]` | | `routing.default_db` | database for every org `org_routes` does not list -- default `dfe` | -| `routing.default_table` | table used when neither a `table_fields` value nor the topic names one -- default `main`, so an unrouted message lands in `dfe.main` | +| `routing.default_table` | table used when neither a `table_fields` value nor the topic names one, or the table named does not exist -- default `main`, so an unrouted message lands in `dfe.main` | | `routing.source_to_table` | maps a source value to the table its records land in, `{cisco-ios: network}` say -- default `{}`, so a source lands in the table of its own name | | `routing.topic_suffixes` | suffixes that mark a topic as one source's own -- default `_land` / `_load`, so `cisco-ios_load` names the source `cisco-ios`. With `kafka.topics` and `kafka.topic_regex` unset, auto-discovery subscribes to the topics ending in one, so an empty list or an empty suffix is refused | | `routing.org_id_field` | field the tenant is read from -- default `_org_id`, the common-header field the first hop stamps | @@ -89,7 +89,7 @@ The tenant is replaced at `org_id_field`. The source is replaced in `_source` an ## A routing edit applies without a restart -The loader watches its config file (`hot_reload.enabled`, on unless set false). On a change it rebuilds the router, the header extractor and the default table from the new `routing` and `metadata`, and pre-warms the schemas of any table the new routing names. Rows already buffered keep the table they were routed to, and no offset is committed for them until they land. `routing.dlq` is the exception: the DLQ is opened at startup, so a change there logs a restart warning. The deployment contract lists `routing` in `hot_paths`, so on Kubernetes an edit there reaches the running pods without rolling them. +The loader watches its config file (`hot_reload.enabled`, on unless set false). On a change it rebuilds the router, the header extractor and the default table from the new `routing` and `metadata`. Startup pre-warms the schemas of the default table and every `source_to_table` target. A reload pre-warms the ones it holds no fresh schema for, and leaves a table already held absent to the recheck described below. A source table the pre-warm finds absent is not recorded absent: the engine writes a source's routing and creates its table together, so the source's first batch resolves the table from the pending-schema buffer instead. An absent default table is reported at once. Rows already buffered keep the table they were routed to, and no offset is committed for them until they land. `routing.dlq` is the exception: the DLQ is opened at startup, so a change there logs a restart warning. The deployment contract lists `routing` in `hot_paths`, so on Kubernetes an edit there reaches the running pods without rolling them. ## The topic names the source when the record does not @@ -101,7 +101,7 @@ a record arriving there with no `table_fields` value routes to `dfe.cisco-ios`. A topic names a source only when one of `routing.topic_suffixes` strips off it AND what is left does not resolve to `default_table`. Both halves matter: `main_land` strips to the landing table, so its records have no -source of their own and keep the default fallback; a topic with no suffix to +source of their own and keep the default fallback. A topic with no suffix to strip is not `{source}_land` or `{source}_load` and names no source at all. `source_to_table` is applied before the comparison, so a source mapped onto the landing table is landing traffic too. @@ -111,11 +111,7 @@ named no table, so `dfe_loader_routing_field_absent_total` increments and a warning names the topic, the table and how many records it stands for, once a minute per topic. -A table ClickHouse confirms does not exist is held for 60 seconds, then -re-resolved, so a table created later starts receiving its own data without a -pod restart. The warning names the move into absence and not the state: the -re-resolved answer is the one already held, so a dead source name is announced -once rather than once a minute for the life of the deployment. +A record whose table ClickHouse confirms does not exist lands in `default_table` instead, counted in `dfe_loader_unknown_table_fallback_total{table}`. The answer is held, then the table is resolved again, so a table created later starts receiving its own data without a pod restart: after 60 seconds, or for a `source_to_table` target the routing names, after 1 second doubling up to 30. The warning names the move into absence and not the state, so a dead source name is announced once rather than once a minute for the life of the deployment. The default table is never held absent, because nothing falls back from it. A record routed to it while it does not exist waits in the pending-schema buffer and is resolved again while it waits, until the table exists or the record reaches `schema.pending_max_age_secs` and is retired like any record whose schema never arrived. diff --git a/docs/transport/README.md b/docs/transport/README.md index 804b7a9e..93adf55b 100644 --- a/docs/transport/README.md +++ b/docs/transport/README.md @@ -1,7 +1,7 @@ P G --> P - M --> P P --> REST ``` ## In this section -- [grpc.md](grpc.md) -- the gRPC transport design: proto schema, config surface, - client/server/bidirectional modes, K8s health and scaling, and the v1 -> v2 - mesh evolution. -- [common-header.md](common-header.md) -- the common header schema injected into - every event table regardless of transport: full field reference, DDL, ORDER BY - / PARTITION BY / codec / index design, and profiles. +- [grpc.md](grpc.md) -- the gRPC listener: how it answers a Push, the proto, its config and the deployment contract's port. +- [common-header.md](common-header.md) -- the header columns every table starts with, and what the loader writes into each. -See [../architecture.md](../architecture.md) for where the transport layer sits in -the overall system. +See [../architecture.md](../architecture.md) for where the transport layer sits in the overall system. diff --git a/docs/transport/common-header.md b/docs/transport/common-header.md index 5bd26ad0..d2d08bb9 100644 --- a/docs/transport/common-header.md +++ b/docs/transport/common-header.md @@ -8,419 +8,46 @@ Copyright: (c) 2026 HYPERI PTY LIMITED --> -# Common header schema v2 +# Common header -Every record in every DFE table starts with the common header: events, context and metrics alike. The loader injects it before insert, whichever transport the record arrived on (a Kafka topic or a gRPC push). The rest of the stack keys on it: row-level security on `_org_id`, uniqueness on `_uuid`, time windows and partitions on its timestamps, and the console's time and organisation filters on the same columns. A change to a header field reaches every table the engine's schema phase creates and every query that filters on it, which is why the header moves only with its schema version. This page documents the full field set, the DDL, and the ORDER BY / PARTITION BY / codec / index decisions behind it. +Every record in every DFE table starts with the common header: events, context and metrics alike. The loader fills it before insert, whichever transport the record arrived on (a Kafka topic or a gRPC push). The rest of the stack keys on it: row-level security on `_org_id`, time windows and partitions on its timestamps, and the console's time and organisation filters on the same columns. A change to a header field reaches every table the engine's schema phase creates and every query that filters on it, which is why the header moves only with its schema version. -## Overview +The header's DDL belongs to dfe-schemas: its column types, ORDER BY, partitioning, codecs, indexes and TTL come from the `timeseries`, `minimal` and `passthrough` profiles under [`common-header/`](https://github.com/hyperi-io/dfe-schemas/tree/main/common-header), rendered by dfe-schemas and applied by dfe-engine's schema phase. dfe-schemas' [profile reference](https://github.com/hyperi-io/dfe-schemas/blob/main/docs/meta-schema.md#common-header-profiles) describes them, and `make render` in a dfe-schemas clone prints the DDL a deployment gets. This page covers what the loader writes into each column. -Every table shares a common header that provides: +## The header is the base of every table -1. **Temporal ordering** -- multiple timestamp fields for different use cases. -2. **Multi-tenancy** -- organisation ID for row-level security (RLS). -3. **Deduplication** -- time-ordered UUIDs for uniqueness. -4. **Auditability** -- original payload preservation. -5. **Extensibility** -- dynamic metadata via JSON fields. +- The landing table (`dfe.main` by default, another with `routing.default_table`) has the header and nothing else. It is the catch-all for unrouted records. +- Every other table (`dfe.auth`, say) is the header plus its own data columns. -## Common header vs complete table schema +The loader creates no table, the landing table included. It reads the schema the deployment created and holds records for a table that does not exist yet in its pending-schema buffer. -The common header is the **base** of every table's schema -- not the complete -schema itself. The loader injects these system fields into every record. +## What the loader writes -- **Landing table** (`dfe.main` by default, another with `routing.default_table`): - schema IS just the common header. This is the catch-all for unrouted records. -- **Other tables** (e.g., `dfe.auth`, `dfe.metrics`): schema = common header - plus data-specific columns, alongside the common header. +Types are the `timeseries` profile's as dfe-schemas renders them. The loader reads the live types from `system.columns` and encodes against those. -The loader creates no table, the landing table included. It reads the schema the -deployment created (in DFE, the engine's schema phase) and holds records for a -table that does not exist yet in its pending-schema buffer. +| Column | Type | What the loader writes | +|---|---|---| +| `_timestamp_load` | `DateTime64(3,'UTC') DEFAULT now64(3)` | Nothing: the INSERT leaves it out and ClickHouse fills the load time | +| `_timestamp` | `DateTime64(3,'UTC')` | The record's `timestamp`, `@timestamp` or `_timestamp`, the first present, held to `timestamp_dq`, else the ingest time. A directive cannot move it | +| `_timestamp_received` | `DateTime64(3,'UTC')` | The first hop's `_timestamp_received` stamp, then the fields the column's directive names, then the ingest time | +| `_uuid` | `Nullable(UUID) DEFAULT generateUUIDv7()` | Nothing: ClickHouse generates a time-ordered UUID | +| `_org_id` | `LowCardinality(String)` | The `routing.org_id_field` value (default `_org_id`), else `routing.default_org_id`. No directive or sender field can change where it is read | +| `_source` | `LowCardinality(Nullable(String))` | The column's directive, `first(_source) \| topic_name`: the record's `_source`, else the source its topic names | +| `_raw` | `Nullable(String)` | A raw field the record carries (`raw` or `_raw`) under `full`, the whole payload under `raw_only`, else NULL | +| `_json` | `JSON(max_dynamic_paths=2048)` | The received payload under `full` and `json_only`, spliced from the bytes as they arrived. A directive on the column, as on the DLQ table's `@source: _json`, fills it from the field it names under every mode | +| `_tags` | `JSON` | The column's directive, `first(_tags/tags/meta/metadata.tags)`, wrapped in an object (below) | -```text -+----------------------------------+ -| landing table (dfe.main) | <- schema = common header ONLY -| +----------------------------+ | -| | common header | | -| | (_timestamp, _org_id, ...) | | -| +----------------------------+ | -+----------------------------------+ +[capture-modes.md](../pipeline/capture-modes.md) covers `_json` and `_raw`, and [routing.md](../pipeline/routing.md) covers `_org_id` and the topic's source. -+----------------------------------+ -| non-default table (dfe.auth) | <- schema = common header + data columns -| +----------------------------+ | -| | common header | | -| | (_timestamp, _org_id, ...) | | -| +----------------------------+ | -| | data columns | | -| | (user_id, action, ip, ...) | | -| +----------------------------+ | -+----------------------------------+ -``` - -The profile system (dfe-schemas `common-header/*.yaml`) controls which common -header fields are injected. The profile does NOT define data-specific columns -- -those come from the source data or are defined in the table's own DDL. - -## Common header DDL (default table) - -This is the common header as the default table uses it. -Non-default tables would include these columns alongside their own data-specific -columns. - -```sql -CREATE TABLE IF NOT EXISTS {db}.{table} -( - -- Primary timestamps - `_timestamp_load` DateTime64(3) DEFAULT now64(3) CODEC(Delta, ZSTD(1)), - `_timestamp` DateTime64(3) CODEC(Delta, ZSTD(1)), - `_timestamp_received` Nullable(DateTime64(3)) CODEC(Delta, ZSTD(1)), - - -- Identity - `_uuid` UUID DEFAULT generateUUIDv7(), - `_org_id` LowCardinality(String) CODEC(ZSTD(1)), - - -- Source identifier - `_source` LowCardinality(String) CODEC(ZSTD(1)), - - -- Payload preservation - `_raw` Nullable(String) CODEC(ZSTD(3)), - `_json` Nullable(JSON(max_dynamic_paths = 2048)) CODEC(ZSTD(3)), - - -- Metadata - `_tags` Nullable(JSON) CODEC(ZSTD(3)), - - -- Indexes - INDEX idx_timestamp _timestamp TYPE minmax GRANULARITY 1, - INDEX idx_raw_text _raw TYPE text(tokenizer = 'default') GRANULARITY 64 -) -ENGINE = {engine} -ORDER BY (_org_id, _timestamp_load, _uuid) -PARTITION BY (toYYYYMM(_timestamp_load), _org_id) -SETTINGS index_granularity = 8192 -``` - -## Column reference - -### `_timestamp_load` - -| Property | Value | -|----------|-------| -| Type | `DateTime64(3)` | -| Nullable | No | -| Default | `now64(3)` | -| Codec | `Delta, ZSTD(1)` | -| Source | `@generated: now64(3)` | - -**Purpose:** When the data was loaded into ClickHouse. - -**Rationale:** - -- Primary query filter for operational dashboards ("show me events from the last hour"). -- More reliable than event timestamp for time-windowed queries. -- ClickHouse generates via DEFAULT -- loader omits this field. -- All rows in a batch get the same load timestamp (acceptable trade-off). - -**Why first in ORDER BY after `_org_id`:** - -- Every dashboard query filters by recent time window. -- Combined with `_org_id`, provides excellent query locality. -- Enables efficient "tail -f" style queries. - -### `_timestamp` - -| Property | Value | -|----------|-------| -| Type | `DateTime64(3)` | -| Nullable | No | -| Default | None | -| Codec | `Delta, ZSTD(1)` | -| Source | `@source: timestamp \| now()` | - -**Purpose:** When the event actually occurred (event time). - -**Rationale:** - -- Event time semantics for analytics and time-series analysis. -- May differ significantly from load time (buffered, delayed, replayed events). -- Fallback to `now()` ensures no NULL values in non-nullable column. -- minmax index enables efficient range scans. - -**Field resolution order:** - -1. `timestamp` field in source data. -2. `@timestamp` field (common in log formats). -3. `_timestamp` field. -4. Fallback to `now()` if all missing/invalid. - -**Validation:** - -- Must be within reasonable bounds (not before 1970, not far in future). -- Millisecond precision preserved. -- Auto-detection of epoch seconds/millis/micros/nanos. +A numeric epoch is read by magnitude as seconds, milliseconds, microseconds or nanoseconds. A zoned text value, `2026-09-07T15:53:53.385+10:00` say, is rewritten as the UTC instant it names (`2026-09-07T05:53:53.385Z`), every fraction digit kept, so the offset is applied rather than dropped. The same rewrite applies to any `DateTime` or `DateTime64` column the header pass fills. `timestamp_dq` holds `_timestamp` to `max_future_seconds` (default 600) and `max_past_seconds` (default 0, no lower bound). A value outside them, or one that does not parse once `correct_known_bad` has tried the known bad forms, becomes the ingest time. -**Wire format:** +## JSON columns take an object -An RFC3339 value -- `2026-09-07T05:53:53.385Z`, which is what the receiver and -both transforms emit -- is normalised to ClickHouse's own text form -(`2026-09-07 05:53:53.385`, UTC) before the insert, and an offset is applied -rather than truncated. Carried verbatim it stops the `JSONEachRow` `DateTime64` -reader at the zone suffix, which rejects the whole batch with code 27 and loses -every row in it. The same normalisation applies to any `DateTime`/`DateTime64` -column the header pass fills, not only this one. +A ClickHouse `JSON` column accepts only an object at the top level, so the loader turns every value it writes into one, `_tags` included, whichever rule filled it and on both insert formats: -### `_timestamp_received` - -| Property | Value | -|----------|-------| -| Type | `Nullable(DateTime64(3))` | -| Nullable | Yes | -| Default | None | -| Codec | `Delta, ZSTD(1)` | -| Source | `@source: timestamp_received` | - -**Purpose:** When the receiver/loader first received the event. - -**Rationale:** - -- Useful for latency analysis (time from generation to receipt). -- The loader reads the first hop's `_timestamp_received` stamp ahead of the column's directive, then the directive's fields, then the ingest time. No other key such as `received_at` is read unless the directive names it. -- Nullable because most sources do not provide this. -- Distinct from `_timestamp_load` which is insert time. - -**Use cases:** - -- Pipeline latency monitoring: `_timestamp_load - _timestamp_received`. -- Event age at receipt: `_timestamp_received - _timestamp`. -- SLA compliance tracking. - -### `_uuid` - -| Property | Value | -|----------|-------| -| Type | `UUID` | -| Nullable | No | -| Default | `generateUUIDv7()` | -| Codec | None (UUID is already compact) | -| Source | `@generated: generateUUIDv7()` | - -**Purpose:** Unique identifier for each record. - -**Rationale:** - -- UUIDv7 is time-ordered (millisecond precision + random suffix). -- Sortable within timestamp -- useful for deterministic ordering. -- Generated by ClickHouse DEFAULT -- no client-side generation needed. -- Last in ORDER BY to ensure uniqueness within (org_id, timestamp_load). - -**Why UUIDv7 over other options:** - -| Option | Pros | Cons | -|--------|------|------| -| UUIDv4 | Widely supported | Not sortable, poor locality | -| UUIDv7 | **Time-ordered, sortable** | Requires ClickHouse 24.8+ | -| ULID | Time-ordered | Not native ClickHouse type | -| Snowflake | Compact, sortable | Requires coordination | -| xxhash64 | Fast | Not guaranteed unique | - -**ClickHouse 24.8+ functions:** - -```sql --- Thread-monotonic (guarantees ordering within thread) -generateUUIDv7() -generateUUIDv7ThreadMonotonic() - --- Slightly faster, no ordering guarantee -generateUUIDv7NonMonotonic() -``` - -### `_org_id` - -| Property | Value | -|----------|-------| -| Type | `LowCardinality(String)` | -| Nullable | No | -| Default | None | -| Codec | `ZSTD(1)` | -| Source | `routing.org_id_field` (default `_org_id`), never a column directive | - -**Purpose:** Organisation/tenant identifier for multi-tenancy and RLS. - -**Rationale:** - -- First in ORDER BY -- every query has an org_id filter (RLS). -- LowCardinality for dictionary encoding (orgs are highly repeated). -- Required field -- cannot be NULL. A record whose field is absent, null or - empty is written under `routing.default_org_id`. -- Read from the field `routing.org_id_field` names and nothing else, so no - column comment can point the tenant at a field a sender writes. - -**Row-Level Security (RLS):** - -```sql --- Create row policy for tenant isolation -CREATE ROW POLICY org_isolation ON dfe.main -FOR SELECT -USING _org_id = currentUser() -TO ALL; -``` - -In a DFE deployment the engine's reconciler writes the row policies that scope -each org's console users, so a deployment does not hand-write them. - -**Why LowCardinality:** - -- Typical deployment has <1000 unique orgs. -- Dictionary encoding reduces storage 10-100x for this column. -- Faster equality comparisons (dictionary index lookup). - -### `_raw` - -| Property | Value | -|----------|-------| -| Type | `Nullable(String)` | -| Nullable | Yes | -| Default | None | -| Codec | `ZSTD(3)` | -| Source | the record's own raw field: a receiver's raw capture, or an `@renamed` source | - -**Purpose:** Original raw data as received -- the data as it would appear in a -tailed log file or a database row, BEFORE any RFC/format parsing. - -Under `full` it is filled only when the source provides raw data, such as a syslog line the receiver captured. A JSON record is kept once, in `_json`, and `_raw` stays NULL. - -**NOT the same as `_json`:** `_raw` is the original wire format (e.g., raw syslog -RFC 3164/5424 line), while `_json` is the parsed/structured result. - -**Rationale:** - -- Captured BEFORE any transformation or parsing. -- Enables "grep-like" full-text searches across the original payload. -- Nullable -- can be disabled globally or per-table to save storage. -- Higher ZSTD level (3) for better compression of text. - -**Text search index (default, included in Common Header v2.1):** - -```sql -INDEX idx_raw_text _raw TYPE text(tokenizer = 'default') GRANULARITY 64 -``` - -The `text` index (GA in ClickHouse 26.2+) provides deterministic full-text search -with no false positives. Supported query functions: `hasToken()`, -`hasAnyTokens()`, `hasAllTokens()`, `LIKE`, `ILIKE`. Insert overhead ~50%. - -`_raw` without the index has no reason to exist. To avoid the overhead, disable -`_raw` entirely via `capture_mode` -- do not keep the column without the index. - -**Capture modes** control `_raw` and `_json` population: - -| `capture_mode` | `_json` | `_raw` | Use case | -|---|---|---|---| -| `full` (default) | Full payload (JSON type) | The record's own raw field, else NULL | Full observability | -| `json_only` | Full payload (JSON type) | NULL | Full observability without the `_raw` storage/index overhead | -| `raw_only` | NULL | Full Kafka payload (String) | CH CPU saving -- no JSON type overhead | -| `extracted_only` | NULL | NULL | Minimal -- only promoted schema fields | - -All four modes extract promoted fields to schema columns. Configurable at global, -per-table, and DDL levels. See `metadata.capture_mode` in config. - -### `_json` - -| Property | Value | -|----------|-------| -| Type | `Nullable(JSON(max_dynamic_paths = 2048))` | -| Nullable | Yes | -| Default | None | -| Codec | `ZSTD(3)` | -| Source | not a directive -- see Capture modes above | - -**Purpose:** Complete Kafka message as native ClickHouse JSON type for structured -path-based queries. - -**`max_dynamic_paths`:** Default raised to 2048 (from ClickHouse default 1024) to -cover multi-Beats deployments (ECS + Winlogbeat + Sysmon generates 500-1500 unique -paths). Per-column DDL only -- cannot be changed after data is inserted. Hard max -10,000. Paths exceeding the limit are stored in shared data (slower queries, data -not lost). - -**NOT the same as `_raw`:** `_json` is the parsed/structured result stored as -native columnar JSON. `_raw` is the original wire format before parsing. - -**Rationale:** - -- Captured BEFORE any transformation (preserves original structure). -- ClickHouse JSON type -- each JSON path stored as a native subcolumn. -- Enables ad-hoc queries on any field without schema changes. -- Nullable -- can be disabled to save storage. - -**ClickHouse JSON type benefits:** - -- Columnar storage within JSON (not stored as a string blob). -- Type inference per JSON path. -- Efficient queries: `SELECT _json.user.id, _json.action FROM events`. -- Supports nested objects and arrays. -- ClickHouse stores each path as a dense subcolumn with compression. - -**Requirements:** - -- ClickHouse 26.2+ for GA JSON type + `text` skip index (hard deck). - -**Configuration:** - -```yaml -metadata: - capture_mode: full # full (default) | json_only | raw_only | extracted_only -``` - -`capture_mode` controls `_json` and `_raw` population -- see the Capture Modes -section below. - -### `_tags` - -| Property | Value | -|----------|-------| -| Type | `Nullable(JSON)` | -| Nullable | Yes | -| Default | None | -| Codec | `ZSTD(3)` | -| Source | `@source: first(tags/_tags/meta/metadata.tags)` | - -**Purpose:** Metadata and collector/agent information. - -**Rationale:** - -- Config-driven source field list (first match wins). -- Stored as JSON for flexible querying. -- Contains collector metadata, enrichment tags, routing info. -- Nullable -- not all records have tags. - -**Field resolution order:** - -1. `tags` field. -2. `_tags` field. -3. `meta` field. -4. `metadata.tags` nested path. - -**Value format:** the column is a ClickHouse `JSON`, which only accepts an object at -the top level, and the loader converts every value it writes into any JSON column -to match -- whichever rule filled it (the `tags_fields` hoist or an -`@source`/`@renamed` column comment) and on both insert formats, `RowBinary` -and `JSONEachRow`. An object is stored as-is. An array -- ECS ships `tags` as an -array of keywords -- is stored whole under a `list` key, so `["forwarded"]` -lands as `{"list": ["forwarded"]}` and queries read it as `_tags.list`. A -scalar, such as a bare string, is stored under a `value` key. A string carrying -JSON text is stored as what that text holds, and text that does not parse is -stored as the string it is, under `value`. - -**Configuration:** - -```yaml -metadata: - tags_fields: [tags, _tags, meta, metadata.tags] - tags_output: _tags - drop_tags: false # Remove source tags after extraction -``` - -#### Reading the wrapped list back +- An object is stored as it is. +- An array is stored under a `list` key: ECS ships `tags` as an array, so `["forwarded"]` lands as `{"list": ["forwarded"]}` and queries read it as `_tags.list`. +- A scalar, and text that does not parse as JSON, is stored under a `value` key. Text that holds JSON is stored as the document it holds. ```sql SELECT _tags.list.:`Array(Nullable(String))` AS tags @@ -428,378 +55,27 @@ FROM dfe.main WHERE arrayExists(x -> x = 'filebeat', _tags.list.:`Array(Nullable(String))`) ``` -An object source passes through untouched, so `_tags` holding `{"list": [...]}` -cannot be told apart from a source that sent that key itself. - -## Underscore prefix convention - -All common header fields use an underscore prefix (`_timestamp`, `_org_id`, -`_uuid`, etc.). - -**Rationale:** - -- Avoids collision with source data fields. -- Source data often contains `timestamp`, `id`, `tags` etc. -- Clear visual distinction between system and data fields. -- Consistent with ClickHouse system columns (`_part`, `_partition_id`). - -**Example collision prevention:** - -```json -// Source event -{ - "timestamp": "2024-01-15T10:30:00Z", - "id": "user-123", - "tags": ["important", "urgent"] -} - -// Stored as: -// _timestamp = 2024-01-15T10:30:00Z (from source timestamp) -// _uuid = 01234567-... (generated) -// _tags = {"list": ["important", "urgent"]} (from source tags, wrapped for the JSON column) -// + all original fields preserved in _json -``` - -## ORDER BY design - -```sql -ORDER BY (_org_id, _timestamp_load, _uuid) -``` - -### Column order rationale +An object source passes through untouched, so `_tags` holding `{"list": [...]}` cannot be told apart from a source that sent that key itself. -1. **`_org_id` first:** - - Every query has an org_id filter (RLS enforcement). - - Data physically grouped by organisation. - - Partition pruning works with org_id in ORDER BY. +## When the header pass promotes nothing -2. **`_timestamp_load` second:** - - Primary time filter for operational queries. - - Recent data queries are most common. - - Good data locality for time-windowed dashboards. - -3. **`_uuid` last:** - - Ensures uniqueness within (org_id, timestamp_load). - - Deterministic ordering for reproducible queries. - - Enables efficient point lookups by UUID. - -### Query patterns optimised - -```sql --- Pattern 1: Recent events for an org (most common) -SELECT * FROM events -WHERE _org_id = 'acme' AND _timestamp_load > now() - INTERVAL 1 HOUR --- Uses: Full ORDER BY prefix - --- Pattern 2: Event by UUID -SELECT * FROM events -WHERE _org_id = 'acme' AND _uuid = '...' --- Uses: org_id prefix, UUID in last position - --- Pattern 3: Time range across orgs (admin dashboard) -SELECT _org_id, count() FROM events -WHERE _timestamp_load > now() - INTERVAL 1 DAY -GROUP BY _org_id --- Uses: Scans all orgs but time-filtered - --- Pattern 4: Event time analysis -SELECT * FROM events -WHERE _org_id = 'acme' AND _timestamp BETWEEN '...' AND '...' --- Uses: org_id prefix, minmax index on _timestamp -``` - -## PARTITION BY design - -```sql -PARTITION BY (toYYYYMM(_timestamp_load), _org_id) -``` - -### Rationale - -1. **Monthly partitions by load time:** - - Natural TTL boundary (drop old months). - - Reasonable partition count (~12-24 active). - - Aligns with typical retention policies. - -2. **Includes `_org_id`:** - - Enables per-org partition pruning. - - Efficient org-level data management. - - Only acceptable for <100 orgs (partition cardinality limit). - -### Partition management - -```sql --- View partitions -SELECT partition, rows, bytes_on_disk -FROM system.parts -WHERE database = 'common' AND table = 'events' -ORDER BY partition; - --- Drop old data (by month) -ALTER TABLE common.events DROP PARTITION ('202401', 'acme'); - --- Drop all data for an org -ALTER TABLE common.events DROP PARTITION ('*', 'acme'); - --- TTL-based retention (alternative) -ALTER TABLE common.events MODIFY TTL _timestamp_load + INTERVAL 90 DAY; -``` - -### Cardinality considerations - -| Orgs | Months | Partitions | Status | -|------|--------|------------|--------| -| 10 | 12 | 120 | Excellent | -| 50 | 12 | 600 | Good | -| 100 | 12 | 1,200 | Acceptable | -| 500 | 12 | 6,000 | Too many -- use shared schema | - -**For >100 orgs:** use shared schema without org in partition: - -```sql -PARTITION BY toYYYYMM(_timestamp_load) --- RLS handles isolation, not partitioning -``` - -## Codec selection - -### Delta + ZSTD(1) for timestamps - -```sql -`_timestamp_load` DateTime64(3) CODEC(Delta, ZSTD(1)) -``` - -**Rationale:** - -- Delta encoding exploits the monotonic timestamp nature. -- Adjacent timestamps differ by small amounts. -- ZSTD(1) compresses the delta-encoded values. -- Level 1 balances speed and compression. - -**Compression ratio:** typically 10-20x for timestamp columns. - -### ZSTD(3) for text/JSON - -```sql -`_raw` Nullable(String) CODEC(ZSTD(3)) -`_json` Nullable(JSON) CODEC(ZSTD(3)) -``` - -**Rationale:** - -- Higher level (3) for better compression of text. -- Text/JSON has high redundancy (field names, common values). -- Slightly slower compression but faster decompression. -- Worth the trade-off for storage-heavy columns. - -**Compression ratio:** typically 5-15x for JSON, 3-8x for raw text. - -### No codec for UUID - -```sql -`_uuid` UUID DEFAULT generateUUIDv7() -``` - -**Rationale:** - -- UUID is already 16 bytes (compact binary format). -- UUIDv7 has some entropy that does not compress well. -- Codec overhead not worth the minimal compression gain. - -## Index strategy - -### Primary index (sparse) - -The ORDER BY clause creates a sparse primary index: - -```sql -ORDER BY (_org_id, _timestamp_load, _uuid) -``` - -- Index entry every `index_granularity` rows (default: 8192). -- ~1 index entry per 8KB-64KB of data. -- Extremely memory-efficient for large tables. - -### minmax index on `_timestamp` - -```sql -INDEX idx_timestamp _timestamp TYPE minmax GRANULARITY 1 -``` - -**Purpose:** efficient time range queries on event time (not load time). - -**How it works:** - -- Stores min/max `_timestamp` per granule. -- Query `WHERE _timestamp BETWEEN x AND y` skips granules outside range. -- GRANULARITY 1 = one min/max per index granularity (8192 rows). - -**When it helps:** - -- Queries filtering on event time, not load time. -- Analytics queries: "events that occurred in Q1". -- Audit queries: "what happened at timestamp X". - -### Text index on `_raw` (default) - -```sql --- Standard: text index (GA in ClickHouse 26.2+, required by hard deck) -INDEX idx_raw_text _raw TYPE text(tokenizer = 'default') GRANULARITY 64 -``` - -The common header DDL carries this index on every `_raw` column. Under -`capture_mode: extracted_only` `_raw` is NULL and the index has nothing to hold. - -**Trade-offs:** - -| Index type | Pros | Cons | -|------------|------|------| -| `text` (default) | GA in 26.2+, accurate, no false positives | ~50% insert overhead on `_raw` | -| `ngrambf_v1` (legacy) | Lower storage | False positives, and no reason to prefer it on 26.2+ | -| None | Zero overhead | Full scan for text search -- use `capture_mode = "extracted_only"` instead | - -If you do not need full-text search on the raw payload, set -`capture_mode = "extracted_only"` -- `_raw` is NULL and the text index is skipped -entirely. - -## Engine selection - -The schema supports three MergeTree variants: - -### SharedMergeTree (recommended) - -```sql -ENGINE = SharedMergeTree() -``` - -**Requirements:** ClickHouse Cloud or 24.1+ with shared storage. - -**Benefits:** - -- Automatic replication without ZooKeeper. -- Adding a replica moves no data: every replica reads the same shared storage. -- No replica configuration needed. - -### ReplicatedMergeTree (clustered) - -```sql -ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/{db}/{table}', '{replica}') -``` - -**Requirements:** ZooKeeper or ClickHouse Keeper. - -**Benefits:** - -- Traditional replication with explicit control. -- Well-tested, stable. -- Works with any ClickHouse version. - -### MergeTree (single-node) - -```sql -ENGINE = MergeTree() -``` - -**Use case:** development, testing, single-node deployments. - -**Benefits:** - -- Simplest configuration. -- No external dependencies. -- Fastest for local development. - -The loader does not pick the engine: it writes to whatever table the deployment -created. - -## Source field mapping - -Using the DDL Expression Language (see [../clickhouse/ddl-directives.md](../clickhouse/ddl-directives.md)): - -| Column | Expression | Description | -|--------|------------|-------------| -| `_timestamp_load` | `@generated: now64(3)` | DB generates on insert | -| `_timestamp` | `@source: timestamp \| now()` | From source with fallback | -| `_timestamp_received` | `@source: timestamp_received` | Optional, from source | -| `_uuid` | `@generated: generateUUIDv7()` | DB generates on insert | -| `_org_id` | not a directive | `routing.org_id_field` (default `_org_id`), else `routing.default_org_id` | -| `_raw` | `@source: first(raw/_raw)` | The source's own raw data, never a JSON payload `_json` holds | -| `_json` | not a directive | Populated by capture_mode, not by a column expression | -| `_tags` | `@source: first(tags/_tags/meta/metadata.tags)` | First match wins | - -### When the header pass promotes nothing - -A payload the header pass cannot read -- not JSON, not a JSON object, or a valid -object matching no column in the table -- once returned an empty field map and -the row landed anyway, every column at its type default: no `_source`, no -`_tags`, `_timestamp` at epoch zero. A row like that reads as data while -carrying none, and cannot say where it came from. - -The message is now rejected to the DLQ instead. Every one of those rejections -logs an ERROR naming the table and the reason, at most once a minute, and counts -`dfe_loader_header_pass_skipped_total`. The table name stays out of the metric's -labels: it comes from a payload field with no allowlist, so labelling it would -let untrusted input grow the label set without bound. +A payload the header pass cannot read -- not JSON, not a JSON object, or a valid object matching no column in the table -- goes to the DLQ rather than landing as a row of type defaults with no `_source` and no `_tags`. Each rejection is counted in `dfe_loader_header_pass_skipped_total`, and an ERROR names the table and the reason at most once a minute. The table name stays out of the metric's labels: it comes from a payload field with no allowlist, so labelling it would let untrusted input grow the label set without bound. A table with a `_json` column keeps the whole payload, so a field no column holds is still stored. A table without one, such as dfe-hyperdx's own tables, keeps only its columns. Each top-level field that lands nowhere counts `dfe_loader_fields_unstored_total`, and a WARN names the table and the fields at most once a minute. DFE's own `_` fields are not counted. -## Configuration reference +## The profiles -### Loader config +| Profile | Columns | Use case | +|---|---|---| +| `timeseries` (default) | 9 | Full event ingestion | +| `minimal` | 5: no `_timestamp_received`, `_source`, `_raw` or `_tags` | High-volume structured data | +| `passthrough` | 4: `_timestamp_load`, `_uuid`, `_org_id`, `_json` | A transparent bridge, no `_timestamp` | -```yaml -routing: - org_id_field: _org_id # Field to extract for _org_id - default_org_id: default # Tenant for a record with none - org_routes: [] # Empty = shared schema - table_fields: [event_category] # Field for table routing - default_db: dfe - default_table: main - -metadata: - capture_mode: full # full (default) | json_only | raw_only | extracted_only - tags_fields: [tags, _tags, meta, metadata.tags] - tags_output: _tags - drop_tags: false # Keep tags in _json after extraction -``` - -## Implementation files +The profile YAML reaches the loader only as deployed ClickHouse columns, never as a file. See [schemas.md](../deployment/schemas.md). | File | Purpose | -|------|---------| -| `src/clickhouse/schema.rs` | Reads the deployed columns from `system.columns` on a TTL | +|---|---| +| `src/transform/extractor.rs` | Fills the header columns on the default `json_primary` path | +| `src/clickhouse/schema.rs` | Caches the deployed columns from `system.columns` | | `src/column_meta/mod.rs` | Parses the `@directive` annotations out of column comments | -| `src/schema/mod.rs` | Schema parsing, table tags, capability detection | -| `src/transform/transformer.rs` | Profile-driven field injection | - -The profile YAML itself lives in -[dfe-schemas](https://github.com/hyperi-io/dfe-schemas) under `common-header/` -and reaches the loader only as deployed ClickHouse columns -- see -[schemas.md](../deployment/schemas.md). - -## Profiles - -The common header is controlled by **profiles** -- named, versioned YAML -definitions that specify which system fields to inject and how to populate them. - -| Profile | Fields | Use case | -|---------|--------|----------| -| `timeseries` (default) | 9 | Full common header for record ingestion | -| `minimal` | 5 | High-volume structured data (no _raw,_tags, _source) | -| `passthrough` | 4 | Transparent bridge (no _timestamp injection) | - -Profiles define: - -- **Field set**: which common header columns to inject. -- **Field behaviour**: source expression for each field (how it is populated). -- **DDL structure**: ORDER BY, PARTITION BY, indexes, for the table DDL the - deployment creates. - -Profiles do NOT define data-specific columns. Those come from the source data or -are defined in the table's own DDL. - -## References - -- [../clickhouse/ddl-directives.md](../clickhouse/ddl-directives.md) -- field mapping expression language -- [ClickHouse MergeTree](https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree) -- [ClickHouse JSON Type](https://clickhouse.com/docs/en/sql-reference/data-types/json) -- [Row-Level Security](https://clickhouse.com/docs/en/guides/sre/user-management/row-level-security) +| `src/schema/mod.rs` | Parses table `COMMENT` tags | diff --git a/docs/transport/grpc.md b/docs/transport/grpc.md index 8fc1d9f8..4404c838 100644 --- a/docs/transport/grpc.md +++ b/docs/transport/grpc.md @@ -21,13 +21,13 @@ flowchart LR R["dfe-receiver
(gRPC client)"] T["transform
(client and server)"] L["dfe-loader
(gRPC server)"] - CH[("ClickHouse")] + CH[(ClickHouse)] R -->|Push| L R -->|Push| T T -->|Push| L L --> CH - L -.PushResponse = answer.-> R + L -.->|PushResponse is the answer| R ``` ## How the loader answers a Push diff --git a/reference/clickhouse_rls.md b/reference/clickhouse_rls.md deleted file mode 100644 index 6e9a9a6e..00000000 --- a/reference/clickhouse_rls.md +++ /dev/null @@ -1,459 +0,0 @@ -# ClickHouse Row-Level Security (RLS) for Multi-Tenancy - -**Version:** 1.0 -**Date:** 2026-01-12 -**Status:** Required Dependency - ---- - -## Overview - -The DFE loader uses a shared-schema multi-tenancy model by default, where all organisations' data is stored in common tables (e.g., `common.auth`, `common.api`). To enforce data isolation between organisations, ClickHouse Row Policies must be configured to restrict access based on the `_org_id` field. - -**Key Principle:** Each ClickHouse role sees only rows where `_org_id` matches their organisation, with no way to bypass this filter. - ---- - -## Architecture - -### Default Routing Behaviour - -```text -Event → Extract event_category → Route to common.{event_category} - → Extract org_id → Store in _org_id field - → No per-org database routing by default - -Examples: - {"org_id": "acme", "event_category": "auth"} → common.auth (_org_id='acme') - {"org_id": "bigcorp", "event_category": "auth"} → common.auth (_org_id='bigcorp') - {"event_category": "api"} → common.api (_org_id=NULL or default) -``` - -### Per-Org Database Routing (Optional) - -When specific organisations require database-level isolation (compliance, volume, etc.): - -```yaml -# config.yaml -routing: - default_db: "common" - - # Option 1: Allowlist specific orgs - routed_orgs: - - "acme" # → acme.auth, acme.api, etc. - - "bigcorp" # → bigcorp.auth, bigcorp.api, etc. - - # Option 2: Route all orgs to own databases - route_all_by_org: false # Default: shared schema -``` - -**When routed to own database:** RLS is optional (database-level grants provide isolation). - ---- - -## Row Policies (Required for Shared Schema) - -### Prerequisites - -- ClickHouse 20.5+ (row policies introduced) -- `_org_id String` column in all tables -- `_org_id` included in ORDER BY for index efficiency - -### Basic Policy - -```sql --- Create role for organisation -CREATE ROLE IF NOT EXISTS acme_reader; - --- Grant read access to common database -GRANT SELECT ON common.* TO acme_reader; - --- Enforce row-level filtering by _org_id -CREATE ROW POLICY acme_filter -ON common.auth -FOR SELECT -TO acme_reader -USING _org_id = 'acme' -AS RESTRICTIVE; - --- Repeat for all tables in common database -CREATE ROW POLICY acme_filter_api -ON common.api -FOR SELECT -TO acme_reader -USING _org_id = 'acme' -AS RESTRICTIVE; - --- Create user with role -CREATE USER acme_user -IDENTIFIED WITH sha256_password BY 'secure_password' -DEFAULT ROLE acme_reader; -``` - -### How It Works - -```sql --- User query (as acme_user) -SELECT * FROM common.auth WHERE timestamp > now() - INTERVAL 7 DAY; - --- Automatically becomes (policy enforced) -SELECT * FROM common.auth -WHERE _org_id = 'acme' -- Added by row policy - AND timestamp > now() - INTERVAL 7 DAY; - --- Attempting to bypass fails -SELECT * FROM common.auth WHERE _org_id = 'bigcorp'; --- Returns 0 rows - policy filter is ANDed with user's WHERE clause -``` - -### Policy Types - -```sql --- RESTRICTIVE (mandatory filter, ANDed with query) -CREATE ROW POLICY org_filter -ON common.auth -USING _org_id = 'acme' -AS RESTRICTIVE; -- Default behaviour - --- PERMISSIVE (optional filter, ORed with other PERMISSIVE policies) -CREATE ROW POLICY allow_own_org -ON common.auth -USING _org_id = currentUser() -AS PERMISSIVE; - -CREATE ROW POLICY allow_public -ON common.auth -USING is_public = 1 -AS PERMISSIVE; --- Result: WHERE (_org_id = currentUser()) OR (is_public = 1) -``` - ---- - -## Admin Access (Bypass Row Policies) - -Admins need unrestricted access across all organisations. - -### Option 1: Exclude Admin Role - -```sql --- Create admin role -CREATE ROLE IF NOT EXISTS admin_reader; -GRANT SELECT ON common.* TO admin_reader; - --- Apply row policies to all EXCEPT admin -CREATE ROW POLICY enforce_org_isolation -ON common.auth -FOR SELECT -TO ALL EXCEPT admin_reader -- Admin bypasses policy -USING _org_id = 'acme' -AS RESTRICTIVE; -``` - -### Option 2: Permissive Policy for Admin - -```sql --- Admin role with unrestricted access -CREATE ROLE admin_reader; - --- Permissive policy allows all -CREATE ROW POLICY admin_sees_all -ON common.auth -FOR SELECT -TO admin_reader -USING 1 = 1 -- Always true -AS PERMISSIVE; -``` - -**Recommendation:** Use Option 1 (exclude from policy) for clarity. - ---- - -## Applying Policies to All Tables - -### Manual Approach - -```sql --- Get list of tables -SELECT name FROM system.tables -WHERE database = 'common' - AND engine LIKE '%MergeTree%'; - --- Create policy for each table -CREATE ROW POLICY acme_filter_auth ON common.auth FOR SELECT TO acme_reader USING _org_id = 'acme' AS RESTRICTIVE; -CREATE ROW POLICY acme_filter_api ON common.api FOR SELECT TO acme_reader USING _org_id = 'acme' AS RESTRICTIVE; -CREATE ROW POLICY acme_filter_network ON common.network FOR SELECT TO acme_reader USING _org_id = 'acme' AS RESTRICTIVE; --- ... repeat for all tables -``` - -### Automated Script (Bash) - -```bash -#!/bin/bash -ORG_ID="acme" -ROLE_NAME="${ORG_ID}_reader" -DATABASE="common" - -# Get list of tables -TABLES=$(clickhouse-client --query " - SELECT name FROM system.tables - WHERE database = '${DATABASE}' - AND engine LIKE '%MergeTree%' - FORMAT TSV -") - -# Create role -clickhouse-client --query "CREATE ROLE IF NOT EXISTS ${ROLE_NAME};" -clickhouse-client --query "GRANT SELECT ON ${DATABASE}.* TO ${ROLE_NAME};" - -# Create row policy for each table -for TABLE in $TABLES; do - POLICY_NAME="${ORG_ID}_filter_${TABLE}" - clickhouse-client --query " - CREATE ROW POLICY IF NOT EXISTS ${POLICY_NAME} - ON ${DATABASE}.${TABLE} - FOR SELECT - TO ${ROLE_NAME} - USING _org_id = '${ORG_ID}' - AS RESTRICTIVE; - " - echo "Created policy: ${POLICY_NAME} on ${DATABASE}.${TABLE}" -done - -echo "Created role: ${ROLE_NAME} with row policies for all tables in ${DATABASE}" -``` - -### Automated Script (SQL Function) - -```sql --- Not directly supported - ClickHouse doesn't have CREATE FUNCTION for DDL --- Use external script or manual creation -``` - ---- - -## Performance Optimization - -### Schema Design - -Include `_org_id` in ORDER BY for efficient filtering: - -```sql -CREATE TABLE common.auth ( - timestamp DateTime64(3), - _org_id String, - _uuid UUID DEFAULT generateUUIDv7(), - -- ... other fields -) ENGINE = MergeTree() -ORDER BY (timestamp, _org_id, _uuid) -- _org_id in primary key -PARTITION BY (toYYYYMM(timestamp), _org_id) -- Partition by month + org -SETTINGS index_granularity = 8192; -``` - -**Benefits:** - -- **Index usage:** Primary key includes `_org_id` for fast filtering -- **Partition pruning:** Queries only read relevant org's partitions -- **Granule skipping:** ClickHouse skips granules (8192 rows) not matching org - -### Query Performance - -```sql --- User query -SELECT COUNT(*) FROM common.auth -WHERE timestamp > now() - INTERVAL 7 DAY; - --- With row policy (added automatically) -SELECT COUNT(*) FROM common.auth -WHERE _org_id = 'acme' -- Policy filter - AND timestamp > now() - INTERVAL 7 DAY; - --- ClickHouse execution: --- 1. Partition pruning: only (202601, 'acme'), (202602, 'acme') --- 2. Primary key index: timestamp range --- 3. Granule-level _org_id filtering (marks file) --- 4. Final WHERE filter -``` - -**Expected Overhead:** <5% for simple policies (ClickHouse documentation). - ---- - -## Security Considerations - -### Cannot Bypass - -Row policies are enforced **before** query execution at the ClickHouse server level: - -- ✅ Cannot bypass with `SETTINGS` changes -- ✅ Cannot bypass with query syntax tricks -- ✅ Cannot bypass with JOINs to other tables (policy applies recursively) -- ✅ Survives server restart (stored in system tables) -- ✅ Applies to views, subqueries, CTEs - -### Audit - -```sql --- View active row policies -SELECT * FROM system.row_policies -WHERE database = 'common'; - --- View policy details -SHOW CREATE ROW POLICY acme_filter ON common.auth; - --- Test as user --- (Requires logging in as that user) -``` - ---- - -## Migration Path - -### Phase 1: Add _org_id Column - -```sql --- Add column to existing tables -ALTER TABLE common.auth ADD COLUMN IF NOT EXISTS _org_id String DEFAULT ''; - --- Backfill from routing logic (application-specific) --- NOTE: Loader will populate _org_id for new inserts automatically -``` - -### Phase 2: Modify Schema (Optional but Recommended) - -```sql --- Rebuild table with _org_id in ORDER BY for better performance --- WARNING: This is a heavy operation on large tables - -CREATE TABLE common.auth_new ( - timestamp DateTime64(3), - _org_id String, - _uuid UUID DEFAULT generateUUIDv7(), - -- ... copy all columns -) ENGINE = MergeTree() -ORDER BY (timestamp, _org_id, _uuid) -- Include _org_id -PARTITION BY (toYYYYMM(timestamp), _org_id); -- Partition by org - --- Copy data -INSERT INTO common.auth_new SELECT * FROM common.auth; - --- Atomic swap -RENAME TABLE common.auth TO common.auth_old, - common.auth_new TO common.auth; - --- Verify, then drop old table -DROP TABLE common.auth_old; -``` - -### Phase 3: Create Row Policies - -```sql --- Per organisation (see scripts above) --- Apply to all tables in common database -``` - -### Phase 4: Create Users - -```sql --- Per organisation -CREATE USER acme_user IDENTIFIED BY 'secure_password' DEFAULT ROLE acme_reader; -CREATE USER bigcorp_user IDENTIFIED BY 'secure_password' DEFAULT ROLE bigcorp_reader; -``` - ---- - -## Verification - -### Test Isolation - -```sql --- As acme_user -SELECT DISTINCT _org_id FROM common.auth; --- Should return only: acme - -SELECT COUNT(*) FROM common.auth WHERE _org_id = 'bigcorp'; --- Should return: 0 - --- As admin_reader -SELECT _org_id, COUNT(*) FROM common.auth GROUP BY _org_id; --- Should return all orgs: --- acme | 150000 --- bigcorp | 250000 --- customer3 | 50000 -``` - -### Test Query Performance - -```sql --- Enable query profiling -SET send_logs_level = 'trace'; - --- Run query as org user -SELECT COUNT(*) FROM common.auth WHERE timestamp > now() - INTERVAL 1 DAY; - --- Check execution plan -EXPLAIN SELECT COUNT(*) FROM common.auth WHERE timestamp > now() - INTERVAL 1 DAY; - --- Look for: --- - "Condition: _org_id = 'acme'" (policy applied) --- - "Parts: X/Y" (partition pruning working) --- - "Granules: X/Y" (granule skipping working) -``` - ---- - -## Troubleshooting - -### Policy Not Applied - -```sql --- Check policy exists -SELECT * FROM system.row_policies WHERE name LIKE '%acme%'; - --- Check user's roles -SELECT * FROM system.role_grants WHERE user_name = 'acme_user'; - --- Verify policy targets correct role -SHOW CREATE ROW POLICY acme_filter ON common.auth; -``` - -### User Sees No Data - -```sql --- Check _org_id values in table -SELECT DISTINCT _org_id FROM common.auth LIMIT 10; - --- Check if _org_id matches policy --- (May be case sensitivity, whitespace, etc.) - --- As admin, check data exists -SELECT COUNT(*) FROM common.auth WHERE _org_id = 'acme'; -``` - -### Performance Degradation - -```sql --- Check if _org_id in ORDER BY -SHOW CREATE TABLE common.auth; - --- Check partition strategy -SELECT partition, count() FROM system.parts -WHERE database = 'common' AND table = 'auth' -GROUP BY partition -ORDER BY partition; - --- Consider adding _org_id to ORDER BY or PARTITION BY -``` - ---- - -## References - -- [ClickHouse Row Policies Documentation](https://clickhouse.com/docs/en/operations/access-rights#row-policy-management) -- [ClickHouse Security Best Practices](https://clickhouse.com/docs/en/operations/security) -- DFE Loader: `reference/common_header.sql` - Common header schema - ---- - -**Last Updated:** 2026-01-12 -**Maintainer:** HyperI Platform Team diff --git a/tests/PERFORMANCE_TESTING.md b/tests/PERFORMANCE_TESTING.md index 880ad282..4ccfdb95 100644 --- a/tests/PERFORMANCE_TESTING.md +++ b/tests/PERFORMANCE_TESTING.md @@ -1,240 +1,36 @@ -# Performance Testing with Metrics Snapshots +# Comparing metrics snapshots -This document describes how to use metrics snapshots to validate that code changes improve performance. - -## Overview - -The loader already exposes comprehensive Prometheus metrics. The metrics snapshot system captures these metrics before/after changes and generates comparison reports. - -## Quick Start - -### 1. Capture Baseline Metrics - -Before making performance changes, capture a baseline: +`tests/common/metrics.rs` turns Prometheus exposition text into a snapshot, compares two snapshots and reports what moved. `tests/performance_example.rs` shows the workflow on two fixed texts, so it checks the comparison, not the loader's speed. ```rust use common::metrics::MetricsSnapshot; -use dfe_loader::metrics::Metrics; - -// Run your workload -let metrics = Metrics::new(); -// ... process messages ... - -// Capture snapshot -let baseline = MetricsSnapshot::capture(&metrics.registry(), "baseline_v1"); -baseline.save("benchmarks/baseline.json")?; -``` - -### 2. Make Your Changes - -Implement your performance optimization. - -### 3. Capture Current Metrics -Run the same workload with your changes: +// Text from a running loader's /metrics, or MetricsManager::render(). +let baseline = MetricsSnapshot::from_text(&baseline_text, "baseline"); +baseline.save(".tmp/metrics_baseline.json")?; -```rust -let current = MetricsSnapshot::capture(&metrics.registry(), "optimized_v2"); -current.save("benchmarks/current.json")?; -``` - -### 4. Compare & Validate - -```rust +let current = MetricsSnapshot::from_text(¤t_text, "candidate"); let comparison = current.compare(&baseline); comparison.print_report(); -comparison.save_markdown("benchmarks/comparison.md")?; - -// Assert no regressions -assert!(comparison.regressions.is_empty(), - "Performance regressions detected"); -``` - -## Example Output - -```text -=== Metrics Comparison Report === -Baseline: baseline_v1 (abc123) -Current: optimized_v2 (def456) - -✅ Improvements (3): - loader_insert_latency_seconds_sum : 0.72 → 0.43 (-40.0%) - loader_messages_processed_total : 10000.00 → 12000.00 (+20.0%) - loader_messages_received_total : 10000.00 → 12000.00 (+20.0%) - -❌ Regressions (0): - -📊 Summary: - Improvements: 3 - Regressions: 0 - Unchanged: 10 - -✅ PASS: Performance improved with no regressions -``` - -## Available Metrics - -The loader tracks: - -- **Throughput**: `loader_messages_processed_total`, `loader_rows_inserted_total` -- **Latency**: `loader_insert_latency_seconds_sum`, `loader_insert_latency_by_table_seconds` -- **Errors**: `loader_insert_errors_total`, `loader_messages_dlq_total` -- **Buffer**: `loader_buffer_rows`, `loader_buffer_bytes`, `loader_buffer_tables` -- **Kafka**: `loader_kafka_offsets_committed_total`, `loader_kafka_lag` - -## Regression Detection - -The system automatically detects regressions: - -- **Latency/Errors**: Lower is better → increase is regression -- **Throughput**: Higher is better → decrease is regression -- **Memory/Buffer**: Lower is better → increase is regression - -## Best Practices - -### 1. Consistent Workloads - -Ensure baseline and current tests use identical workloads: - -- Same number of messages -- Same message sizes -- Same table distribution -- Same batch sizes - -### 2. Multiple Runs - -Run tests 3-5 times and average results to account for variance. - -### 3. Save Baselines - -Commit baseline snapshots to git for historical tracking: - -```bash -git add benchmarks/baseline_*.json -git commit -m "perf: baseline before optimization X" -``` - -### 4. CI/CD Integration - -Add performance regression tests to CI: - -```bash -cargo test --test performance_example -- --nocapture -``` - -## Integration with Existing Benchmarks - -The metrics snapshot system works with existing criterion benchmarks: - -```rust -// benches/pipeline.rs -use criterion::{black_box, criterion_group, criterion_main, Criterion}; -use dfe_loader::metrics::Metrics; -use common::metrics::MetricsSnapshot; - -fn bench_pipeline(c: &mut Criterion) { - let metrics = Metrics::new(); - - c.bench_function("pipeline_throughput", |b| { - b.iter(|| { - // Your benchmark code - metrics.messages_processed.inc(); - }); - }); - - // After benchmark, capture metrics - let snapshot = MetricsSnapshot::capture(&metrics.registry(), "pipeline_bench"); - snapshot.save("target/bench_snapshot.json").ok(); -} - -criterion_group!(benches, bench_pipeline); -criterion_main!(benches); -``` - -## Workflow Example - -### Before Optimization - -```bash -# Run baseline -cargo test --test performance_example baseline -- --nocapture - -# Save snapshot -cp target/metrics_current.json benchmarks/baseline_pre_optimization.json -git add benchmarks/baseline_pre_optimization.json -git commit -m "perf: baseline before JSON parser optimization" -``` - -### After Optimization - -```bash -# Make your changes -vim src/payload/parse.rs - -# Run same test -cargo test --test performance_example optimized -- --nocapture - -# Compare -cargo test --test performance_example compare -- --nocapture - -# Review report -cat target/metrics_comparison.md -``` - -### If Regressions Detected - -```bash -# Investigate which metrics regressed -cat target/metrics_comparison.md - -# Adjust implementation -vim src/payload/parse.rs - -# Re-test -cargo test --test performance_example -- --nocapture -``` - -## See Also - -- [tests/performance_example.rs](performance_example.rs) - Full working example -- [tests/common/metrics.rs](common/metrics.rs) - Metrics snapshot implementation -- [src/metrics/mod.rs](../src/metrics/mod.rs) - Available metrics (`dfe-loader metrics-manifest` prints the full catalogue) - -## Troubleshooting - -### Metrics Not Captured - -Ensure the registry is properly registered: - -```rust -let registry = Registry::new(); -let metrics = Metrics::with_registry(registry.clone()); - -// Use metrics.registry() for snapshot -let snapshot = MetricsSnapshot::capture(®istry, "test"); +comparison.save_markdown(".tmp/metrics_comparison.md")?; +assert!(comparison.regressions.is_empty()); ``` -### False Regressions +A snapshot records the time, the git commit (`git rev-parse --short HEAD`) and each metric's value, summed across its label sets. `MetricsSnapshot::load` reads a saved one back. -Histogram buckets may show "regressions" that aren't real - focus on `_sum` metrics: +## How a change is judged -```rust -// Good: actual latency sum -loader_insert_latency_seconds_sum +`compare` puts each metric that moved under improvements or regressions by its name: -// Misleading: just bucket counts -loader_insert_latency_seconds_bucket -``` +- `latency`, `error`, `dlq` or `lag` in the name: lower is better. +- `processed`, `inserted` or `flushed`: higher is better. +- `buffer` or `memory`: lower is better. +- Anything else: higher is better. -### Git Commit Not Captured - -Ensure you're in a git repository, from the dfe-loader checkout: +Histogram `_bucket` and `_count` series move with the request count, so judge latency on `_sum`. The example test leaves those two out of its regression check for that reason. ```bash -git status +cargo nextest run --test performance_example --no-capture ``` ---- - -**Last Updated:** 2026-01-12 -**Author:** HyperI Platform Team +`dfe-loader metrics-manifest` lists the metric names a snapshot of a running loader holds (see [../docs/operations.md](../docs/operations.md#observe)). diff --git a/tests/TESTING.md b/tests/TESTING.md index 57a3ebf5..27a20b23 100644 --- a/tests/TESTING.md +++ b/tests/TESTING.md @@ -6,8 +6,10 @@ The general rules are hyperi-ai's testing standard (`standards/universal/testing | Level | Runs | Command | |---|---|---| -| core | locally, every push and PR | `cargo nextest run --all-features` | -| full | nightly (`schedule` in `.github/workflows/ci.yml`) and before every release (`test.full.required_for_release: true`) | `cargo nextest run --all-features --run-ignored all --ignore-default-filter` | +| core | locally, every push and PR | `cargo nextest run --features testcontainers` | +| full | nightly (`schedule` in `.github/workflows/ci.yml`) and before every release (`test.full.required_for_release: true`) | `cargo nextest run --features testcontainers --run-ignored all --ignore-default-filter` | + +CI runs each tier once per feature set `.hyperi-ci.yaml` lists under `quality.rust.features`, so `jemalloc` and `transport-memory` are covered as well as `testcontainers`. | stack | dfe-infra's post-deploy acceptance, not this repo | - | Core is nextest's default selection. The `default-filter` in `.config/nextest.toml` holds every test that needs a real ClickHouse or Kafka out of it, bar the two in `tests/integration/shared_services.rs`. The `#[ignore]` tests in `tests/e2e/` run only under full, against the shared ClickHouse and Kafka (`require_e2e_services`). A missing service fails them, it never skips them. hyperi-ci runs core on push and PR, and full on the schedule, on a release, or on a dispatch with `test-tier: full`. @@ -32,6 +34,8 @@ The ClickHouse pin lives in `tests/common/containers.rs` and follows dfe-infra ` Each ClickHouse container gets a random password for `default`, never logged, which tests read with `clickhouse_password()`. `default` may create users and grant to them. An embedded single-node Keeper backs the `dfe_replicated` database (`Replicated` engine) that the `ReplicatedMergeTree()` tests in `inserter.rs` and `rls.rs` write to. Those tests fail, not skip, when it is missing (`require_replicated!`). Its certificate names `localhost` and the Docker host, signed by a CA made for the run, which tests read back with `shared_clickhouse_ca()`. Kafka is one KRaft broker advertising the host port Docker published. +## How a run shares its containers + scalo's `deployment::test_support::shared_service` gives every test process in a run the same container per service: - The first process to ask creates it, named `-integration--r`, where `` is the start of `NEXTEST_RUN_ID` and `` is `HYPERI_TEST_CONTAINER_PREFIX`, else `dfe-loader-test`. Every other process finds it by that name and waits for it to be ready. @@ -59,7 +63,7 @@ Each `HYPERI_TEST_*` setting is read from the process environment, then the proj - ClickHouse: `HYPERI_TEST_CLICKHOUSE_HOST`, `_HTTP_PORT`, `_HTTPS_PORT`, `_NATIVE_PORT`, `_NATIVE_TLS_PORT`, `_USER`, `_PASSWORD`, `_CA_FILE`, `_CLUSTER`, `_DB_PREFIX` - Kafka: `HYPERI_TEST_KAFKA_BOOTSTRAP`, `_SECURITY_PROTOCOL`, `_SASL_MECHANISM`, `_USER`, `_PASSWORD`, `_CA_FILE`, `_TOPIC_PREFIX` -On HyperI dev machines `hyperi-ai secrets devex-env` (hyperi-ai 4.1.30 or later) writes them to `~/.config/hyperi-ai/devex-test.env`, mode 0600, and prints only the path and the key count. Point `HYPERI_TEST_ENV_FILE` at that file. No host, port or credential belongs in this repo. +On HyperI dev machines `hyperi-ai secrets devex-env` writes them to `~/.config/hyperi-ai/devex-test.env`, mode 0600, and prints only the path and the key count. Point `HYPERI_TEST_ENV_FILE` at that file. No host, port or credential belongs in this repo. A run on a pet creates what it needs and drops it: