Repository navigation
docs(loader): correct docs against the code - #350
Merged
Merged
Conversation
README.md, CONTRIBUTING.md, BENCHMARKS.md, every page under docs/ and the two test guides now match the code and config schema. They also match scalo, dfe-clickhouse and dfe-schemas. - HTTP inserts use `RowBinaryWithNamesAndTypes`. ClickHouse still parses `_json` and every other JSON column from text. - `config-check` runs `Config::validate` only. ClickHouse protocol, port and insert format checks run when the pipeline starts. - The default `full` capture stores a JSON payload in `_json` with `_raw` NULL. Nothing reads `capture_json`. - Pre-warm never marks a source table as absent. This keeps the source's first rows out of the default table while the engine is still creating its table. - common-header.md points at the DDL dfe-schemas renders. Its separate copy no longer matched and is removed. - A merge publishes only with a `Release: true` trailer. - A reload ignores `memory` and `batch_processing` without a warning. - reference/clickhouse_rls.md is removed. Nothing links it and no claim in it holds. - Seven mermaid diagrams that failed to parse now render.
docs/performance/overview.md named dfe-clickhouse-rs's `docs/insert-formats.md` as a bare path, which hyperi-ci's doc-paths check reads as a file in this repo. It is now a link to that repo.
The README now names DFE_LOADER_CONFIG beside --config and the ./config.yaml fallback. Only a named file is hot-reloaded (src/main.rs:92). BENCHMARKS.md names the payloads the bakeoff runs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Correct README.md, CONTRIBUTING.md, BENCHMARKS.md, every page under docs/ and the two test guides against the code and config schema. Align them with scalo, dfe-clickhouse and dfe-schemas at the pinned versions.
All 195 FAIL rows are fixed. The ledger has 496 rows: PASS 281, FAIL 195, UNCHECKABLE 12, Accepted 7, Rejected 1. UNCHECKABLE covers claims the repo cannot show, such as a past bake-off. Accepted and Rejected cover prose warnings on reference pages and checker flags on ClickHouse names.
RowBinaryWithNamesAndTypes. ClickHouse still parses every JSON column, including_json, from text.config-checkrunsConfig::validateonly. The ClickHouse protocol, port and format checks run at pipeline start.fullcapture keeps a JSON payload in_jsonwith_rawNULL. Nothing readscapture_json. A directive on_jsonfills it under every mode.Release: truetrailer.memoryandbatch_processing.This PR changes docs only. The RowBinary and
config-checkerrors in code comments are unchanged. docker-compose.dev.yaml and reference/common_header.sql are unchanged.195 corrected rows
docker compose -f docker-compose.dev.yaml up -d+cargo run -- --config config.dev.yamlmoves real data{/[is NotJson (processor.rs)config.example.yamlis the annotated reference"revert"Varies"Release: truetrailer gate and dispatch##sectionsrole.otel_writer,role.hunt_runner). dfe-engine keepsdata.repository(dfe-engine CLAUDE.md "Repository service")git ls-files src--test integration_tests"some cases need ClickHouse".-- --ignoredfor e2etestcontainers. hyperi-ci src/hyperi_ci/languages/rust/test.py:53 runs the full tier as--run-ignored all --ignore-default-filterskip_if_no_clickhouse!()andskip_if_no_kafka!()return early and PASS"rg "skip_if_no_(clickhouse|kafka)!" tests: 0 uses. tests/common/mod.rs:223-260acquire_clickhouseskips without the feature or with an unreachable named server outside CI. It panics in CIDFE_LOADER_", "A single-underscore nested name ... is dropped, which config_reachability.rs checks"__spellings. It does not check a dropped single onevalidate()rejects" native + json_each_rowconfig-checkpasses native-json.yaml.ClickHouseConfig::validateruns at pipeline startcel"Keep it on>=0.14.5, <0.15", "(Cargo.toml:46-51)"celentry in Cargo.tomlgrpc.listencheck this side lacks"Cargo.toml:37takes scaloplugins_builtin/loader.py"git cat-file -e origin/main:src/dfe_engine/services/plugins_builtin/loader.py: missing. dfe-engine appmgmt/contract.py:54, :292-315 readsconfig-schema.jsondfe-schemaentry point creates every destination table"[project.scripts]has nodfe-schema. Table creation is described in dfe-schemas README.md:16topic_suffix: _land. dfe-transform-vrl src/kafka/mod.rs:474syslog_load. The slim/mesh profiles show the producers (R3)insert_format"RowBinary over the HTTPClient"RowBinaryWithNamesAndTypes,RowBinaryWithDefaultsonly for a row that supplies no columnClickHouseQueryClientdoes "DDL"FORMAT RowBinaryWithDefaults, one INSERT per batchRowBinaryWithNamesAndTypes, one INSERT per set of defaulted columns, up to 4 open.RowBinaryWithDefaultsonly for a row with no column_jsonunder every mode. Links to capture-modes.mdSchemaMismatch(e.g.ALTER ... ADD COLUMN)".envfeeds both ENV layers only without--config--config", set it with the env, a file value is reported as a mismatchinsert_format"server skips JSON parsing"nativeverifies every host against the first one's name"clickhouse.databaseused for "DDL" and any table without adb.prefixclickhouse.databasesets the session and probe databaserow_binary+ http =RowBinaryWithDefaults.json_each_row+ native refused at startupRowBinaryWithNamesAndTypes. The pipeline refusesjson_each_row+ native at start.config-checkaccepts ittls.enabled"HTTPS for the connection". Auto-on for 8443/9440/clickhouse.cloudcapture_raw,capture_jsoncapture_modedecides_jsonunless a directive on the column fills it, as CM2--confignames"DFE_LOADER_CONFIGtoomemoryandbatch_processingreload behaviour--helpconfig-checkruns the same validation the orchestrator runs", refusing jsoneachrow+native and flagging port mismatchesconfig-checkrunsConfig::validateonly. ClickHouse protocol, port and format checks run at pipeline startmetrics-manifestprints "every metric the loader registers"metrics-manifesttop--once,--json,--filter. "JSON/TSV output"ALTER)"##sections, two sections over 400 words[lib] dfe_loader, used by main.rs:24-29, tests and benchesjemallocexplicitly; CI opts in totestcontainersfor the docker test lane"dfe-clickhousewith only itsextfeature"["ext", "tls", "dynamic"]INSERT ... FORMAT RowBinaryWithDefaults", one INSERT for rows with differing columnsRowBinaryWithDefaultsover HTTPRowBinaryWithNamesAndTypesjson_each_row+ native "rejected atconfig-check"dfe-clickhouse0.1.5 that includes a parameterisedJSON(max_dynamic_paths = N)column"system.columnsorder"RowBinaryWithDefaultsonly for a row with no column)ALTER ... ADD COLUMN"native"names a protocol this config rejects at boot"SchemaCacheand its background refreshALTERno longer fails every insert"max_types(SharedVariant)"ClickHouseQueryClienttls.enabled"HTTPS for the connection"-- @...above the columnCOMMENTin every example@source: field | now()"if missing/invalid"@source: field | default_valuewith a bare word_tagsfrom the first field inmetadata.tags_fields"tags_fields.metadata.tags_fieldsis a dotted config key@renamed"No-op if destination exists: ... prevents overwriting upstream-populated fields"@config: routing.user_id_field@renamed"no-op if the destination is already present". Four bold-lead itemsjson_primarypath uses the header extractor (processor.rs:295-316)@schema_source,@schema_version,@created_by, "core= pre-supplied by dfe-loader"default_tableused when nothing names a table_jsonNULL underraw_onlyandextracted_onlyfull_rawfills "only from a_rawfield of its own"rawor_rawcapture_jsonandcapture_rawsimplyip_fields, first match wins_timestamp_load DateTime64(3) ... CODEC(Delta, ZSTD(1)), same for_timestamp_timestamp_received Nullable(DateTime64(3))DateTime64(3,'UTC'), not Nullable_uuid UUIDnot NullableNullable(UUID) DEFAULT generateUUIDv7()_source LowCardinality(String)LowCardinality(Nullable(String))_json Nullable(JSON(...)),_tags Nullable(JSON)JSON(max_dynamic_paths=2048),JSON. dfe-schemas common-header/timeseries.yaml 1.0.1: "ClickHouse rejects JSON inside Nullable"text(tokenizer = 'default') GRANULARITY 64ORDER BY (_org_id, _timestamp_load, _uuid),PARTITION BY (toYYYYMM(_timestamp_load), _org_id),index_granularity = 8192, and the ORDER BY / PARTITION BY rationale and cardinality tables_timestampwire format "normalised to ClickHouse's own text form (2026-09-07 05:53:53.385)"YYYY-MM-DDTHH:MM:SS.fffZ_timestamp_received@source: timestamp_received@source: first(_timestamp_received/timestamp_received/received_at). extractor.rs:516-527: stamp, directive, ingest time_tags@source: first(tags/_tags/meta/metadata.tags), resolution order tags, _tags, metafirst(_tags/tags/meta/metadata.tags). extractor.rs:284-290: the directive wins overtags_fields_raw@source: first(raw/_raw), no_sourcerow_jsonis the received payload underfullandjson_only_jsonunder every modemax_dynamic_pathsraised to 2048 for multi-Beats, "500-1500 unique paths", hard max 10,000gh api repos/hyperi-io/dfe-operator: 404. dfe-infra versions.yaml and dfe-docker docker-compose.yml:212 are the consumerssystem.columnsand runs no DDL (dfe-schemas README.md:16)DFE_LOADER_VERSION:${DFE_LOADER_VERSION:-latest},9090:9090,./config/loader.yaml:/etc/dfe/loader.yaml:ro${DFE_LOADER_PROMETHEUS_PORT:-9091}:9090, config through compose/dfe-app.ymldfe-schema (dfe-engine image)[project.scripts]: dfe-engine, dfe, dfe-hunt-runner, dfe-keda-shim, nodfe-schema. dfe-schemas README.md:16: "dfe-engine reads this package at boot ... applies"COMMENT '@renamed: client_ip -- Source address ...'-dfe-schemacreates the tables. dfe-infra runs it as a Job, dfe-docker as an init servicedfe-schemain dfe-infra argocd/helm or dfe-docker compose (git grep)dfe-loader --config test-config.yamlwith a comment after a line continuationoptimize.bolt.enabled: trueis set andpgo.enabledspiketo alphaFORMAT RowBinaryWithDefaults, "ClickHouse skips JSON parsing", "No throughput comparison of the two has been measured"clickhouse.max_concurrent_inserts: 8 # DefaultMALLOC_CONF="prof:true,..."thenjeprofon the release binary_rjem_, so options come from_RJEM_MALLOC_CONF. :38: heap profiling needs theprofilingfeature, which Cargo.toml:63-64 does not enable. The release binary is strippeddocs/insert-formats.mdas a bare path, which CI's doc-paths check read as this repo's filesystem.columnsorder". "skips parsing entirely". "Missing columns take their server-side default"_json, "no heap allocation until the final insert body"Arcper message{/[payloadcrates/buffercargo nextest run --all-features" on every push and PR. Full with--all-features--features testcontainers, and the feature sets CI runshyperi-ai secrets devex-env(hyperi-ai 4.1.30 or later)"Metrics::new(),metrics.registry(),MetricsSnapshot::capture(®istry, ...),Metrics::with_registry(...)Metrics::new(&MetricsManager). tests/common/metrics.rs hasfrom_text,save,load,compare, nocaptureand no registrybenchmarks/*.json,target/metrics_*.tmp/metrics_*->, "PASS:"loader_messages_processed_total,loader_insert_latency_by_table_seconds,loader_buffer_rows,loader_kafka_lagmetrics-manifestand corrects eight metric-name flagscargo test --test performance_example baseline|optimized|compareexample_metrics_snapshot_workflowjson_primarypath does not flatten (extractor.rs:17-21).logjsonis the old name of_json(loader.rs:1141-1150)sonic_rs::get_from_slice()for routing"from_slice(7d17a8d)routing.routed_orgs,routing.route_all_by_orgcommon.{event_category}common.apiand clickhouse.com security URL references