Skip to content

docs(loader): correct docs against the code - #350

Merged
catinspace-au merged 3 commits into
mainfrom
fix/docs-reality
Nov 15, 2026
Merged

catinspace-au merged 3 commits into
mainfrom
fix/docs-reality

Conversation

@catinspace-au

@catinspace-au catinspace-au commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Correct README.md, CONTRIBUTING.md, BENCHMARKS.md, every page under docs/ and the two test guides against the code and config schema. Align them with scalo, dfe-clickhouse and dfe-schemas at the pinned versions.

All 195 FAIL rows are fixed. The ledger has 496 rows: PASS 281, FAIL 195, UNCHECKABLE 12, Accepted 7, Rejected 1. UNCHECKABLE covers claims the repo cannot show, such as a past bake-off. Accepted and Rejected cover prose warnings on reference pages and checker flags on ClickHouse names.

  • HTTP inserts use RowBinaryWithNamesAndTypes. ClickHouse still parses every JSON column, including _json, from text.
  • config-check runs Config::validate only. The ClickHouse protocol, port and format checks run at pipeline start.
  • The default full capture keeps a JSON payload in _json with _raw NULL. Nothing reads capture_json. A directive on _json fills it under every mode.
  • Pre-warm never marks a source table absent. While the engine creates a source table, that source's first rows stay out of the default table.
  • common-header.md points at the DDL dfe-schemas renders. The page's DDL copy disagreed with dfe-schemas.
  • A merge publishes only with a Release: true trailer.
  • A reload neither applies nor warns about memory and batch_processing.
  • The README points to dfe-docker for a stack that moves data because docker-compose.dev.yaml is broken.
  • reference/clickhouse_rls.md is removed. Nothing links it, and its contents are wrong.
  • Seven mermaid diagrams that failed to parse now render.

This PR changes docs only. The RowBinary and config-check errors in code comments are unchanged. docker-compose.dev.yaml and reference/common_header.sql are unchanged.

195 corrected rows
Page Row Was Now
README.md R2 One-liner "Loads Kafka and gRPC streams into ClickHouse, batching small writes into large inserts" Uses the manifest's description
README.md R3 Last hop, taking what dfe-receiver sends over Kafka or direct gRPC The fetcher and transforms send too
README.md R7 "RowBinary inserts by default, so ClickHouse does no JSON parsing on the way in" No row text is parsed. JSON column values are still parsed
README.md R16 docker compose -f docker-compose.dev.yaml up -d + cargo run -- --config config.dev.yaml moves real data Points to dfe-docker for a stack that moves data
README.md R20 Insert "RowBinary by default, so ClickHouse skips JSON parsing" As R7
README.md R22 MessagePack "deprecated in DFE 2.2 and no longer accepted" No rmp crate in Cargo.toml. A payload not opening {/[ is NotJson (processor.rs)
README.md R26 "config.example.yaml is the annotated reference" docs/config-schema.yaml is the full list
README.md R32 default_db/default_table "where a record lands when no routing field matches" Also used when neither field nor topic names a table, or the named table does not exist
README.md R34 Documentation table: "Kafka, gRPC, the common header" under docs/transport/ docs/transport/ holds grpc.md and common-header.md, no Kafka page
CONTRIBUTING.md C3 revert "Varies" revert -> patch
CONTRIBUTING.md C8 "When your pull request is merged ... This happens automatically - no manual intervention required" Documents the Release: true trailer gate and dispatch
CONTRIBUTING.md C9 Five bold-lead steps. Nine ## sections Plain steps. Questions folded into the other sections
CONTRIBUTING.md C10 DCO intro "to ensure that" No "to ensure that" (DOC-REG-1)
CONTRIBUTING.md C11 "the only component that writes there" The dfe-schemas manifest renders write roles beside the loader's (role.otel_writer, role.hunt_runner). dfe-engine keeps data.repository (dfe-engine CLAUDE.md "Repository service")
CONTRIBUTING.md C14 Layout: routing/transform/enrich "Routing, coercion and capture, enrichment". No row for payload, kafka, column_meta, schema, metrics Capture is src/pipeline/capture.rs. Coercion is dfe-clickhouse's. Layout follows git ls-files src
CONTRIBUTING.md C16 Commands: --test integration_tests "some cases need ClickHouse". -- --ignored for e2e tests/integration/mod.rs gates most modules on testcontainers. hyperi-ci src/hyperi_ci/languages/rust/test.py:53 runs the full tier as --run-ignored all --ignore-default-filter
CONTRIBUTING.md C17 "skip_if_no_clickhouse!() and skip_if_no_kafka!() return early and PASS" rg "skip_if_no_(clickhouse|kafka)!" tests: 0 uses. tests/common/mod.rs:223-260 acquire_clickhouse skips without the feature or with an unreachable named server outside CI. It panics in CI
CONTRIBUTING.md C21 "figment strips exactly DFE_LOADER_", "A single-underscore nested name ... is dropped, which config_reachability.rs checks" loader.rs:412-416 also trims the next underscore. config_reachability.rs:340-355 checks the two __ spellings. It does not check a dropped single one
CONTRIBUTING.md C22 "validate() rejects" native + json_each_row config-check passes native-json.yaml. ClickHouseConfig::validate runs at pipeline start
CONTRIBUTING.md C24 cel "Keep it on >=0.14.5, <0.15", "(Cargo.toml:46-51)" Points to the cel entry in Cargo.toml
CONTRIBUTING.md C26 dfe-engine "scopes the broker check ... adds a grpc.listen check this side lacks" Points to the config-schema row
CONTRIBUTING.md C27 Cargo.toml:37 takes scalo Points to scalo in Cargo.toml without a line number
CONTRIBUTING.md C28 Dockerfile "at schema version 4" Points to the Dockerfile header for the schema version
CONTRIBUTING.md C30 dfe-engine "hand-reimplements this repo's config validation in plugins_builtin/loader.py" git cat-file -e origin/main:src/dfe_engine/services/plugins_builtin/loader.py: missing. dfe-engine appmgmt/contract.py:54, :292-315 reads config-schema.json
CONTRIBUTING.md C31 "dfe-engine's dfe-schema entry point creates every destination table" dfe-engine pyproject.toml [project.scripts] has no dfe-schema. Table creation is described in dfe-schemas README.md:16
CONTRIBUTING.md C32 Producers: "in the suite that producer is dfe-receiver" dfe-fetcher docs/config-schema.json:201 topic_suffix: _land. dfe-transform-vrl src/kafka/mod.rs:474 syslog_load. The slim/mesh profiles show the producers (R3)
docs/README.md D1 "High-performance loader from message transports (Kafka, gRPC, Memory)" Kafka or a gRPC push
docs/README.md D2 "RowBinary by default so ClickHouse skips JSON parsing" No row text is parsed
docs/README.md D7 insert_format "RowBinary over the HTTP Client" Supports native blocks too
docs/README.md D11 Diagram: Memory transport Removed
docs/README.md D12 Diagram: "HTTP: insert_formatted_with FORMAT RowBinaryWithDefaults" dfe-clickhouse insert.rs:441-446: RowBinaryWithNamesAndTypes, RowBinaryWithDefaults only for a row that supplies no column
docs/README.md D14 Where-things-live: "Kafka / gRPC / Memory" under transport/ No Kafka page. Memory is test-only
docs/architecture.md A2 "Parse once (SIMD, zero-copy)" Removed
docs/architecture.md A5 ClickHouseQueryClient does "DDL" Handles schema, comments and health
docs/architecture.md A9 Diagram and text: FORMAT RowBinaryWithDefaults, one INSERT per batch dfe-clickhouse v0.1.11 src/dynamic/insert.rs:441-446, :32-37: RowBinaryWithNamesAndTypes, one INSERT per set of defaulted columns, up to 4 open. RowBinaryWithDefaults only for a row with no column
docs/architecture.md A14 Capture-mode table and three-level precedence A directive can fill _json under every mode. Links to capture-modes.md
docs/architecture.md A17 "Schema-cache recovery on SchemaMismatch (e.g. ALTER ... ADD COLUMN)" Dropped or retyped columns
docs/architecture.md A20 Five bold-lead bullets Plain dot points
docs/configuration.md CF2 Diagram: "./.env included" in the nested layer .env feeds both ENV layers only without --config
docs/configuration.md CF8 scaling gate read by scalo's cascade, "never the file named by --config", set it with the env, a file value is reported as a mismatch The scaling gate reads the file and both nested spellings
docs/configuration.md CF10 insert_format "server skips JSON parsing" As R7
docs/configuration.md CF12 "With TLS, native verifies every host against the first one's name" client_http.rs:308-316 passes each endpoint's own host. dfe-clickhouse tcp/client.rs:89-98: "every endpoint is verified against its own host"
docs/configuration.md CF13 clickhouse.database used for "DDL" and any table without a db. prefix clickhouse.database sets the session and probe database
docs/configuration.md CF14 Matrix: row_binary + http = RowBinaryWithDefaults. json_each_row + native refused at startup RowBinaryWithNamesAndTypes. The pipeline refuses json_each_row + native at start. config-check accepts it
docs/configuration.md CF16 tls.enabled "HTTPS for the connection". Auto-on for 8443/9440/clickhouse.cloud HTTPS or native TLS
docs/configuration.md CF20 "60 s is about 270x ... at a measured 91,000 rows per CPU-second" Removed
docs/configuration.md CF24 Missing capture_raw, capture_json Both rows added. capture_mode decides _json unless a directive on the column fills it, as CM2
docs/configuration.md CF26 default_db/default_table "fallbacks when no field matches" As R32
docs/configuration.md CF35 "Every build retries a transient insert failure up to 5 times" inserter.rs:590-658: RowBinary returns a timeout, a dropped connection or a retryable code at once (:606-617). It retries only drift (:757-779). JSONEachRow retries Transient/Unknown (:930-950)
docs/configuration.md CF38 Hot-reload watches "the file --config names" main.rs:90-92 resolves DFE_LOADER_CONFIG too
docs/configuration.md CF39 Restart-required and safe lists Lists all restart and reload settings
docs/configuration.md CF43 Missing memory and batch_processing reload behaviour Listed as restart-required with no warning
docs/configuration.md CF42 Two semicolons Removed
docs/operations.md O2 Subcommand table: run, config-check, version, top, metrics-manifest Two subcommands added, with flags from their --help
docs/operations.md O5 "config-check runs the same validation the orchestrator runs", refusing jsoneachrow+native and flagging port mismatches config-check runs Config::validate only. ClickHouse protocol, port and format checks run at pipeline start
docs/operations.md O9 metrics-manifest prints "every metric the loader registers" Lists the metrics printed by metrics-manifest
docs/operations.md O11 top --once, --json, --filter. "JSON/TSV output" Describes each flag's behaviour
docs/operations.md O17 Hot-reload paragraph: the reloader names "the field" Links to configuration.md
docs/operations.md O19 "a drift error (a column added by an out-of-band ALTER)" Dropped or retyped columns
docs/operations.md O20 "Any server code outside the retry list (a busy server, the network, ...)" The retry list is unchanged. Any other code against an unchanged table is a row error
docs/operations.md O25 DLQ "Kafka primary, file fallback". "Routing misses ... land there" Documents which failures go to the DLQ. gRPC uses files only
docs/operations.md O30 dfe-infra: CPU 70%, pressure trigger 70 via dfe-keda-shim, "a 120 s cooldown" The contract's 300 s
docs/operations.md O33 Seven and four bold-lead bullets. File 1643 words, 10 ## sections, two sections over 400 words Plain bullets. Hot-reload and ClickHouse outages folded into the other sections
docs/feature-flags.md FF1 "dfe-loader is a binary, not a library" Cargo.toml:12-14 [lib] dfe_loader, used by main.rs:24-29, tests and benches
docs/feature-flags.md FF3 "Production builds opt in to jemalloc explicitly; CI opts in to testcontainers for the docker test lane" hyperi-ci optimize.py:31-33, :51-55 adds the allocator feature. .hyperi-ci.yaml:24-29 sets the features
docs/feature-flags.md FF5 "dfe-clickhouse with only its ext feature" Cargo.toml:74 ["ext", "tls", "dynamic"]
docs/feature-flags.md FF8 A semicolon Removed
docs/clickhouse/clickhouse-ext.md CE4 "one INSERT ... FORMAT RowBinaryWithDefaults", one INSERT for rows with differing columns As IF9
docs/clickhouse/insert-formats.md IF2 Diagram and matrix: RowBinaryWithDefaults over HTTP RowBinaryWithNamesAndTypes
docs/clickhouse/insert-formats.md IF3 json_each_row + native "rejected at config-check" Refused when the pipeline starts
docs/clickhouse/insert-formats.md IF5 "A native block declares each column's type, so the server casts a value to a changed column type instead of refusing" Both protocols refuse the changed type. The refusal is treated as drift
docs/clickhouse/insert-formats.md IF6 "As of dfe-clickhouse 0.1.5 that includes a parameterised JSON(max_dynamic_paths = N) column" Removed
docs/clickhouse/insert-formats.md IF7 "The server does no JSON parse ... in system.columns order" insert.rs:62-66, :454-458: JSON as text. The header names the INSERT's columns (:436-446)
docs/clickhouse/insert-formats.md IF9 1 MiB chunks, "one INSERT ... per batch", rows with differing columns "share one INSERT" insert.rs:32-37, :97-103 (one INSERT per set of defaulted columns, up to four open on HTTP), :400-410 (RowBinaryWithDefaults only for a row with no column)
docs/clickhouse/insert-formats.md IF11 Drift example "ALTER ... ADD COLUMN" As A17
docs/clickhouse/insert-formats.md IF16 "the spellings this page used before the key was wired". native "names a protocol this config rejects at boot" Removed. The refusal is stated
docs/clickhouse/insert-formats.md IF19 A semicolon Removed
docs/clickhouse/schema-cache.md SC4 Missing the loader's own SchemaCache and its background refresh Documents both
docs/clickhouse/schema-cache.md SC8 "A failure with no server code at all never reaches the DLQ" Server failures with no code
docs/clickhouse/schema-cache.md SC9 "This is the fix trail ... a column added by an out-of-band ALTER no longer fails every insert" Dropped or retyped columns, as A17
docs/clickhouse/types.md T1 "the ClickHouse type system as of the 24.8+ line" Removed
docs/clickhouse/types.md T4 Dynamic: "Types beyond limit stored as String" dfe-clickhouse v0.1.11 docs/type-support.md: a value "stored past max_types (SharedVariant)"
docs/clickhouse/types.md T8 "DDL and schema queries" through ClickHouseQueryClient Schema and comment queries
docs/clickhouse/types.md T9 "Type support comes from the loader's own encoder, not the client crate" The encoder is dfe-clickhouse's (Cargo.toml:74). clickhouse-ext.md: "The loader holds no encoder of its own"
docs/clickhouse/types.md T15 "RowBinary skips the server-side JSON parse" As R7
docs/clickhouse/tls.md TL2 tls.enabled "HTTPS for the connection" Native TLS too (client_http.rs:309-313)
docs/clickhouse/tls.md TL8 "out of scope today" "out of scope"
docs/clickhouse/tls.md TL10 "[deployment/] for how secrets are supplied at runtime" Names the contract's secret env var
docs/clickhouse/ddl-directives.md DD1 "You write them as comments above (or beside, in CSV) each column", and every SQL example puts -- @... above the column Uses the column COMMENT in every example
docs/clickhouse/ddl-directives.md DD6 @source: field | now() "if missing/invalid" Missing or null (three places)
docs/clickhouse/ddl-directives.md DD7 @source: field | default_value with a bare word A quoted literal
docs/clickhouse/ddl-directives.md DD9 "The applier (dfe-engine's schema module) writes that DEFAULT clause" dfe-schemas' renderer, applied by dfe-engine
docs/clickhouse/ddl-directives.md DD11 "_tags from the first field in metadata.tags_fields" Its directive, else tags_fields. metadata.tags_fields is a dotted config key
docs/clickhouse/ddl-directives.md DD17 @renamed "No-op if destination exists: ... prevents overwriting upstream-populated fields" The paths are read in place of the column's name
docs/clickhouse/ddl-directives.md DD22 CSV schema format section Removed
docs/clickhouse/ddl-directives.md DD23 SQL DDL example with @config: routing.user_id_field Example from the dfe-schemas render
docs/clickhouse/ddl-directives.md DD24 Data-prep list: @renamed "no-op if the destination is already present". Four bold-lead items Table and plain prose
docs/clickhouse/ddl-directives.md DD25 "The data prep stage (transformer) implements these expressions" The default json_primary path uses the header extractor (processor.rs:295-316)
docs/clickhouse/ddl-directives.md DD26 Table tags: @schema_source, @schema_version, @created_by, "core = pre-supplied by dfe-loader" Documents which table tags are written and read
docs/pipeline/README.md PR1 overview.md "the six stages" No stage count
docs/pipeline/README.md PR2 parallelism.md "the parallelisation playbook" parallelism.md describes the two phases
docs/pipeline/overview.md PO1 Diagram "bytes (Kafka/gRPC/Memory)" As D1
docs/pipeline/overview.md PO4 "#128 taught dfe-receiver ... dfe-transform-elastic then lost five records ... (#184)" Removed
docs/pipeline/overview.md PO5 "refused as an unsplit batch and named as such ... or as a parse error about trailing characters" Describes the array case only
docs/pipeline/overview.md PO7 "The parse is zero-copy where it can be" The bytes are kept once
docs/pipeline/overview.md PO8 MessagePack deprecated and no longer accepted As R22
docs/pipeline/overview.md PO10 Missing absent-table fallback orchestrator.rs:788-841
docs/pipeline/overview.md PO14 "The remediation playbook for parallel correctness is in parallelism.md" parallelism.md describes the phases
docs/pipeline/overview.md PO15 A semicolon Removed
docs/pipeline/routing.md RT4 default_table used when nothing names a table orchestrator.rs:788-841 also uses it for an absent table
docs/pipeline/routing.md RT9 Reload rebuilds router, extractor, default table. "pre-warms the schemas of any table the new routing names" Names the routing tables and separates startup from reload. Pre-warm never marks a source table absent
docs/pipeline/routing.md RT13 "A table ClickHouse confirms does not exist is held for 60 seconds, then re-resolved" Documents fallback routing, its counter and the short recheck
docs/pipeline/capture-modes.md CM2 Mode table: _json NULL under raw_only and extracted_only "NULL, unless a directive fills it", with a paragraph on the directive
docs/pipeline/capture-modes.md CM4 Under full _raw fills "only from a _raw field of its own" raw or _raw
docs/pipeline/capture-modes.md CM6 Missing capture_json and capture_raw Documents both
docs/pipeline/enrichment.md EN13 "a table opts in simply by declaring" No simply
docs/pipeline/enrichment.md EN14 Missing which fields are the IP fields Schema EnrichmentConfig ip_fields, first match wins
docs/transport/README.md TR1 "Kafka, gRPC and Memory -- are provided by scalo" Kafka and gRPC. Memory is a test adapter
docs/transport/README.md TR3 grpc.md covers "client/server/bidirectional modes, K8s health and scaling, and the v1 -> v2 mesh evolution" grpc.md covers the server only: answering a Push, proto, config, contract
docs/transport/README.md TR4 common-header.md: "injected into every event table ... DDL, ORDER BY / PARTITION BY / codec / index design" common-header.md points to dfe-schemas' DDL
docs/transport/README.md TR5 A semicolon Removed
docs/transport/common-header.md CH1 _timestamp_load DateTime64(3) ... CODEC(Delta, ZSTD(1)), same for _timestamp Types from the render. Codecs left to dfe-schemas
docs/transport/common-header.md CH2 _timestamp_received Nullable(DateTime64(3)) Render: DateTime64(3,'UTC'), not Nullable
docs/transport/common-header.md CH3 _uuid UUID not Nullable Render: Nullable(UUID) DEFAULT generateUUIDv7()
docs/transport/common-header.md CH4 _source LowCardinality(String) Render: LowCardinality(Nullable(String))
docs/transport/common-header.md CH5 _json Nullable(JSON(...)), _tags Nullable(JSON) Render: JSON(max_dynamic_paths=2048), JSON. dfe-schemas common-header/timeseries.yaml 1.0.1: "ClickHouse rejects JSON inside Nullable"
docs/transport/common-header.md CH6 Indexes minmax GRANULARITY 1 and text(tokenizer = 'default') GRANULARITY 64 Removed. dfe-schemas owns the indexes
docs/transport/common-header.md CH7 ORDER BY (_org_id, _timestamp_load, _uuid), PARTITION BY (toYYYYMM(_timestamp_load), _org_id), index_granularity = 8192, and the ORDER BY / PARTITION BY rationale and cardinality tables Removed
docs/transport/common-header.md CH8 Engine section recommends SharedMergeTree, "Requires ClickHouse Cloud or 24.1+" Removed
docs/transport/common-header.md CH9 UUIDv7 "Requires ClickHouse 24.8+", codec ratios "10-20x", "5-15x", "<1000 orgs", "10-100x" Removed
docs/transport/common-header.md CH11 _timestamp wire format "normalised to ClickHouse's own text form (2026-09-07 05:53:53.385)" datetime.rs:133-146, :157-173: rewritten as YYYY-MM-DDTHH:MM:SS.fffZ
docs/transport/common-header.md CH12 _timestamp_received @source: timestamp_received dfe-schemas 1.0.3 @source: first(_timestamp_received/timestamp_received/received_at). extractor.rs:516-527: stamp, directive, ingest time
docs/transport/common-header.md CH15 _tags @source: first(tags/_tags/meta/metadata.tags), resolution order tags, _tags, meta dfe-schemas 1.0.3 first(_tags/tags/meta/metadata.tags). extractor.rs:284-290: the directive wins over tags_fields
docs/transport/common-header.md CH16 Source mapping table: _raw @source: first(raw/_raw), no _source row One table from the render
docs/transport/common-header.md CH18 "once returned an empty field map ... The message is now rejected" The message is rejected
docs/transport/common-header.md CH21 Implementation files: transformer "profile-driven field injection", schema/mod.rs "capability detection" The default path fills the header in extractor.rs. schema/mod.rs holds TableTags only
docs/transport/common-header.md CH24 _json is the received payload under full and json_only States that a directive can fill _json under every mode
docs/transport/common-header.md CH22 max_dynamic_paths raised to 2048 for multi-Beats, "500-1500 unique paths", hard max 10,000 Removed
docs/deployment/README.md DE1 Downstream "dfe-docker, dfe-operator" pull the artefacts gh api repos/hyperi-io/dfe-operator: 404. dfe-infra versions.yaml and dfe-docker docker-compose.yml:212 are the consumers
docs/deployment/README.md DE2 "the shared dfe-schemas definitions the loader resolves at runtime to drive its table DDL" The loader reads system.columns and runs no DDL (dfe-schemas README.md:16)
docs/deployment/README.md DE3 schemas.md covers "submodule setup, resolution order, per-file versioning" schemas.md: no submodule, no resolution order
docs/deployment/publishing.md PB1 Downstream "dfe-docker, dfe-operator" reference an immutable tag dfe-infra versions.yaml (tag, digest, chart) and dfe-docker DFE_LOADER_VERSION
docs/deployment/publishing.md PB3 Diagram "CI cross-build" CI build
docs/deployment/publishing.md PB8 "copies in the binary CI has already cross-compiled" As PB3
docs/deployment/publishing.md PB11 dfe-docker example :${DFE_LOADER_VERSION:-latest}, 9090:9090, ./config/loader.yaml:/etc/dfe/loader.yaml:ro dfe-docker e6563d8 docker-compose.yml:212-216: version needed, ${DFE_LOADER_PROMETHEUS_PORT:-9091}:9090, config through compose/dfe-app.yml
docs/deployment/publishing.md PB12 "Kubernetes (dfe-operator)" example Removed
docs/deployment/schemas.md SM2 Diagram dfe-schema (dfe-engine image) dfe-engine origin/main pyproject.toml [project.scripts]: dfe-engine, dfe, dfe-hunt-runner, dfe-keda-shim, no dfe-schema. dfe-schemas README.md:16: "dfe-engine reads this package at boot ... applies"
docs/deployment/schemas.md SM3 Example COMMENT '@renamed: client_ip -- Source address ...' -
docs/deployment/schemas.md SM7 dfe-schema creates the tables. dfe-infra runs it as a Job, dfe-docker as an init service dfe-schemas README.md:16: "Nothing else creates a ClickHouse object ...: not an Argo CD Job, not a compose init container". No dfe-schema in dfe-infra argocd/helm or dfe-docker compose (git grep)
docs/performance/README.md PF1 "now CI-automated". Runtime tuning of "concurrent inserts" Concurrency is fixed at 8 (inserter.rs:98-108). No config key in the schema
docs/performance/README.md PF2 history.md unlinked docs/performance/ holds history.md
docs/performance/overview.md PV1 Notice "(2026-04+): hyperi-ci now applies Tier 1 (jemalloc + fat LTO) on beta and release; Tier 2 opt-in" Per-channel table
docs/performance/overview.md PV2 PGO "10-20%", BOLT "5-15%" Removed
docs/performance/overview.md PV3 Manual PGO snippet runs dfe-loader --config test-config.yaml with a comment after a line continuation Documents the cargo-pgo steps and names the workload
docs/performance/overview.md PV4 BOLT runs only when optimize.bolt.enabled: true is set and pgo.enabled optimize.py:208-211: BOLT defaults on with PGO
docs/performance/overview.md PV7 "DFE policy (2026-04-17)", "every channel (spike/alpha/beta/release)" No dated policy stamp. optimize.py:31-33 has alpha, beta, release, and maps a legacy spike to alpha
docs/performance/overview.md PV9 "~3.5% binary-size increase (per dfe-receiver canary 1, 2026-04-17)". jemalloc vs glibc table Removed
docs/performance/overview.md PV10 Insert throughput: FORMAT RowBinaryWithDefaults, "ClickHouse skips JSON parsing", "No throughput comparison of the two has been measured" As A9, R7. dfe-clickhouse v0.1.11 docs/insert-formats.md "Native against HTTP RowBinary" measures them
docs/performance/overview.md PV12 clickhouse.max_concurrent_inserts: 8 # Default No such key in docs/config-schema.yaml. configuration.md: "insert concurrency [is] not a setting"
docs/performance/overview.md PV13 MALLOC_CONF="prof:true,..." then jeprof on the release binary tikv-jemalloc-sys 0.7.1 README.md:81-83 prefixes symbols with _rjem_, so options come from _RJEM_MALLOC_CONF. :38: heap profiling needs the profiling feature, which Cargo.toml:63-64 does not enable. The release binary is stripped
docs/performance/overview.md PV16 dfe-clickhouse's docs/insert-formats.md as a bare path, which CI's doc-paths check read as this repo's file A link to the dfe-clickhouse-rs page
docs/performance/history.md PH1 "Keep it when you re-evaluate ... the conclusions are still load-bearing" Removed
docs/performance/history.md PH2 "The server does no JSON parse ... in system.columns order". "skips parsing entirely". "Missing columns take their server-side default" As IF7, IF9
docs/performance/history.md PH7 "One table's failure does not block another's offset commit" The commit floor is per partition (orchestrator.rs:2861-2877), so tables sharing a partition share it
docs/performance/history.md PH8 Zero-copy _json, "no heap allocation until the final insert body" processor.rs:373 allocates the Arc per message
docs/performance/history.md PH9 Single-byte JSON/MsgPack auto-detect, "JSON and MsgPack can be mixed on the same topic" MessagePack removed. processor.rs refuses a non-{/[ payload
docs/performance/history.md PH10 "Delta coercion only ... Everything else is left to ClickHouse", the four-row table The encoder converts every value, leaving the server no RowBinary coercion (dfe-clickhouse encode.rs:23 and the type arms)
docs/performance/history.md PH11 Future optimisations incl. crates/buffer Removed
docs/performance/history.md PH12 Four bold-lead bullets Plain prose
tests/TESTING.md TT2 Core "cargo nextest run --all-features" on every push and PR. Full with --all-features --features testcontainers, and the feature sets CI runs
tests/TESTING.md TT13 "hyperi-ai secrets devex-env (hyperi-ai 4.1.30 or later)" Removed
tests/PERFORMANCE_TESTING.md PT1 Metrics::new(), metrics.registry(), MetricsSnapshot::capture(&registry, ...), Metrics::with_registry(...) src/metrics/mod.rs Metrics::new(&MetricsManager). tests/common/metrics.rs has from_text, save, load, compare, no capture and no registry
tests/PERFORMANCE_TESTING.md PT2 Paths benchmarks/*.json, target/metrics_* tests/performance_example.rs:36-76 writes .tmp/metrics_*
tests/PERFORMANCE_TESTING.md PT3 Example output with emoji and arrows tests/common/metrics.rs:183-227 prints "Improvements (n):", ->, "PASS:"
tests/PERFORMANCE_TESTING.md PT4 Metric names loader_messages_processed_total, loader_insert_latency_by_table_seconds, loader_buffer_rows, loader_kafka_lag Points to metrics-manifest and corrects eight metric-name flags
tests/PERFORMANCE_TESTING.md PT5 cargo test --test performance_example baseline|optimized|compare The target has one test, example_metrics_snapshot_workflow
tests/PERFORMANCE_TESTING.md PT6 "Add performance regression tests to CI" with this example The example compares two fixed texts. It does not compare loader performance
tests/PERFORMANCE_TESTING.md PT8 "Last Updated: 2026-01-12", "Author: HyperI Platform Team" Removed
BENCHMARKS.md BM1 "Last updated: 2025-12-28", Rust 1.83.0, and every timing table Removed. history.md keeps the dated bake-offs
BENCHMARKS.md BM2 Transform "with logjson capture", flatten as a hot-path stage The default json_primary path does not flatten (extractor.rs:17-21). logjson is the old name of _json (loader.rs:1141-1150)
BENCHMARKS.md BM3 "Mison Structural Index (Experimental)" with "Current implementation" and next steps Removed
BENCHMARKS.md BM4 "Use sonic_rs::get_from_slice() for routing" history.md: the extractor moved to one from_slice (7d17a8d)
BENCHMARKS.md BM6 Version history 0.9.0-0.11.0 Removed
reference/clickhouse_rls.md RL1 routing.routed_orgs, routing.route_all_by_org Removed with the page
reference/clickhouse_rls.md RL2 Default routing to common.{event_category} Removed with the page
reference/clickhouse_rls.md RL3 Row policies "must be configured", by hand or script Removed with the page
reference/clickhouse_rls.md RL4 Migration path: ADD COLUMN, a new table and INSERT SELECT Removed with the page
reference/clickhouse_rls.md RL5 "Version: 1.0, Date: 2026-01-12", "Maintainer: HyperI Platform Team". Nothing links the page Page removed with the common.api and clickhouse.com security URL references
diagrams MD1 docs/architecture.md diagram renders Valid
diagrams MD2 docs/clickhouse/ddl-directives.md diagram renders Valid
diagrams MD3 docs/clickhouse/insert-formats.md diagram renders Valid
diagrams MD4 docs/deployment/publishing.md diagram renders Valid
diagrams MD5 docs/pipeline/routing.md diagram renders Valid
diagrams MD6 docs/README.md diagram renders Valid
diagrams MD7 docs/transport/grpc.md diagram renders Valid

README.md, CONTRIBUTING.md, BENCHMARKS.md, every page under docs/ and the two test guides now match the code and config schema. They also match scalo, dfe-clickhouse and dfe-schemas.

- HTTP inserts use `RowBinaryWithNamesAndTypes`. ClickHouse still parses `_json` and every other JSON column from text.
- `config-check` runs `Config::validate` only. ClickHouse protocol, port and insert format checks run when the pipeline starts.
- The default `full` capture stores a JSON payload in `_json` with `_raw` NULL. Nothing reads `capture_json`.
- Pre-warm never marks a source table as absent. This keeps the source's first rows out of the default table while the engine is still creating its table.
- common-header.md points at the DDL dfe-schemas renders. Its separate copy no longer matched and is removed.
- A merge publishes only with a `Release: true` trailer.
- A reload ignores `memory` and `batch_processing` without a warning.
- reference/clickhouse_rls.md is removed. Nothing links it and no claim in it holds.
- Seven mermaid diagrams that failed to parse now render.
docs/performance/overview.md named dfe-clickhouse-rs's `docs/insert-formats.md` as a bare path, which hyperi-ci's doc-paths check reads as a file in this repo. It is now a link to that repo.
The README now names DFE_LOADER_CONFIG beside --config and the ./config.yaml fallback. Only a named file is hot-reloaded (src/main.rs:92). BENCHMARKS.md names the payloads the bakeoff runs.
@catinspace-au
catinspace-au merged commit 4116e02 into main Nov 15, 2026
14 checks passed
@catinspace-au
catinspace-au deleted the fix/docs-reality branch November 15, 2026 11:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant