Skip to content

docs(engine): correct docs against the code - #1172

Merged
catinspace-au merged 2 commits into
mainfrom
fix/docs-reality
Nov 15, 2026
Merged

catinspace-au merged 2 commits into
mainfrom
fix/docs-reality

Conversation

@catinspace-au

Copy link
Copy Markdown
Contributor

Correct README.md and docs/ against the engine code, the dfe-schemas wheel, openapi.json and the named sibling repos at origin/main. The generated route map is unchanged.

558 ledger rows: PASS 378, FAIL 104 (fixed), register or structure FAIL 60 (fixed), UNCHECKABLE 13, accepted 2, rejected 1. hyperi-ai docs claims FAIL results drop from 709 to 689. The 689 results cover:

  • The generated route map's planned ids (193).
  • Names present in code that the tool misses.
  • Sibling-repo or ClickHouse names.

Rewrite four pages about designs that were never built or have since moved: the edge auth page, the CLI design, the commit standard and the engine-loader schema contract.

Delete the hyperdx fork maintenance page's superseded rebase model. The integrations index links the fork's own runbook. Fold the one-link reference index into docs/README.md.

Every touched page passes hyperi-ai docs prose check with no deny. hyperi-ci lint-docs passes:

  • Every doc path resolves.
  • 60 mermaid blocks parse.
  • markdownlint is clean.

On dfe-hyperdx origin/main, jwt-verify.ts reads hyperdx_role from /auth/me and runs refuseReadOnlyWrites (role-claim.ts:12, :67, :92).

Page Old claim Change
README.md "The image bakes the dfe-schemas trees in at /app/schemas" Correct the seed path and copy
README.md docs/reference/ is "the API and CLI reference" Link the route map
architecture.md "Five tiers select defaults ... mesh" Four selectable tiers. mesh is in the charts and refused
data-evolution.md Common header "_json, _timestamp, timestamp, _tags", "_timestamp and timestamp drive partitioning, retention" _timestamp_load partitions and expires. _timestamp reads the record's timestamp
data-evolution.md Source name is "both the topic and the table name" Name the table and <name>_land / <name>_load
data-evolution.md Loader MetadataConfig carries capture_json/capture_raw with disable_json_tables/disable_raw_tables metadata.capture_mode and table_capture_modes
auth-envoy-topology.md "one Envoy front layer that behaves identically in Kubernetes and Docker", jwt_authn verifies the engine to... Describe each mode
auth-envoy-topology.md Single origin, /observe/* -> hyperdx Own host or listener
auth-envoy-topology.md /api/* -> engine /api/v1
auth-envoy-topology.md Each app re-verifies the JWT against JWKS engine and dfe-hyperdx verify
auth-envoy-topology.md Envoy strips client X-Oidc-* / Authorization State that stripping is opt-in. Document the engine's trust_proxy_auth_headers (settings.py:1512-1521, default false)
auth-envoy-topology.md Transport "TLS 1.3, AES-256-GCM, P-384, hybrid ML-KEM" 1.2 default, 1.3 overlay
auth-envoy-topology.md "At rest: the signing private key is AES-256 wrapped" PEM in the secrets backend. Protection is the backend's
auth-envoy-topology.md Reachability guard: docker apps not published to the host State that the engine is on loopback
local-dev.md "mints an ephemeral ES384 JWT authority" Loads or mints its key
local-dev.md "retries for a minute" Up to 180 s
local-dev.md dfe-docker's clickhouse/server-custom.xml declares the SQL_ prefix, a stock image does not Remove the prefix claim. Keep SQL_current_tenant_id (governance/ch/render.py:181)
local-dev.md dfe-ui talks to http://localhost:8003/api and ${DFE_UI_BASE_URL}/rules/{id} Correct the variable names
README.md docs/reference/README.md is one line and one link Delete the index. docs/README.md and README.md link the route map directly. docs/README.md was its only inbound link (rg reference/README)
archive/* archive/README: helm research superseded by "HelmValuesCompiler + a copy of deep_merge" Describe what exists
archive/* archive/README: "plugin entry points live in code" State that neither was pursued
control-plane/README.md dfe-ui "verifies the engine-minted JWT" dfe-ui carries it and reads its expiry. The engine verifies it
control-plane/README.md @dfe/dfe-engine-types Correct the package name
control-plane/README.md Prism contract mock "(dev + CI)" Link the compose file. Dev only
control-plane/governed-ops-design.md Default registry: helmvars, infravars, sources, library, governance (accounts, groups, roles, actions, poli... Name every class
control-plane/governed-ops-design.md governance/ch_rbac.py, build_group_sql, ddl_artifact writing ddl/ch-rbac/.sql, GroupChProvisioner Describe governance/ch
control-plane/governed-ops-design.md Hunts run as a per-group CH user with quota and profile Name the service role and hunt tiers for groups
control-plane/governed-ops-design.md Claim settles on a winner by claim time then owner id. A doubled window is idempotent Correct the settle claim. State that exclusion must hold
control-plane/governed-ops-design.md {window} substitutes timestamp_load >= start AND < end, field "fixed to timestamp_load" _timestamp_load default, overridable
control-plane/gitops-commit-standard.md Directory-per-environment layout clusters//, envs//values/, a shared base/, promotion PRs Describe the layout the engine writes
control-plane/gitops-commit-standard.md author = acting human, committer = bot, "[already split]" Actor is the DFE-Actor trailer
control-plane/gitops-commit-standard.md Every machine commit signed. Argo verifies with sourceIntegrity State that commits are unsigned
control-plane/gitops-commit-standard.md "auto-sync + self-heal + prune are ON (prune added gradually). Webhooks configured" Correct the policy
control-plane/gitops-commit-standard.md A CI lint dfe local gitops lint Remove
control-plane/oidc-rbac-architecture.md Egress admin.googleapis.com List both hosts
control-plane/oidc-rbac-architecture.md Envoy SecurityPolicy fallback injects X-Oidc-Subject/X-Oidc-Groups State that it is opt-in and groups do not arrive
control-plane/oidc-rbac-architecture.md Role claim "member for any other account" List three values
control-plane/rbac.md Headers "injected by Envoy Gateway SecurityPolicy" incl. X-Oidc-Groups Opt-in jwtAuthn. Groups do not arrive
control-plane/rbac.md Example key's long part Use a 32-char example
control-plane/rbac.md Section numbering jumps 12 -> 14 -> 15 -> 17 Renumber
control-plane/rbac-vocabulary.md dfe-deploy does "deployment-time group seeding" dfe-deploy ships none. The engine seeds
control-plane/idp-setup/README.md dfe local governed helm ... and action invoke Correct the subcommands
control-plane/dfe-cli-design.md dfe-engine carries a break-glass dfe local governed ... in cli/governed_ops.py Correct the page
control-plane/dfe-cli-design.md PyInstaller binary, install.sh, Homebrew tap, ghcr.io/hyperi-io/dfe, .pkg/.msi, dfe upgrade Mark as not built
control-plane/dfe-cli-design.md ~/.dfe/config + ~/.dfe/credentials, DFE_CONFIG_DIR override Correct the layout
control-plane/dfe-cli-design.md Credential chain flag > env > cached session > profile > in-cluster token Correct the order
control-plane/dfe-cli-design.md dfe configure family, dfe configure oidc, browser dfe login Correct the commands
control-plane/dfe-cli-design.md "openapi/openapi.json (74 operations, 23 routers)" and the tree read from openapi.json Remove the count and name the factory
control-plane/dfe-cli-design.md Noun lists dfe rules validate/publish/rollback, dfe deployments, dfe queries, dfe system health Use one example
control-plane/dfe-cli-design.md --region analogue --org, --page-size, --max-items, --no-verify-ssl, --cli-auto-prompt, completion, --messag... Remove
control-plane/type-safety-sync.md Clone with DFE_UI_TOKEN, "PAT with permission ..." Use the App credential
control-plane/ui-api-guide.md GET /queries/health Remove the row
control-plane/ui-api-guide.md GET /operations/{id}/stream SSE and its EventSource example Remove. Poll the operation
control-plane/ui-api-guide.md "API Surface (74 Endpoints)" Remove the count. The spec is the full list
control-plane/ui-api-guide.md Login posts application/x-www-form-urlencoded Use a JSON example
control-plane/ui-api-guide.md OIDC row "X-Oidc-Subject (set by Envoy), Production SSO" Split into two rows
control-plane/ui-api-guide.md CLI dfe usage list for /usage dfe usage get
control-plane/ui-api-guide.md Prism compose snippet stoplight/prism:4, -h 0.0.0.0 Link the compose file
control-plane/ui-api-guide.md "CI validates the committed spec" with a manual diff Name the test
integrations/dfe-ui-guide.md "Next.js 16 + React 19", "Yarn 4" Link package.json
integrations/dfe-ui-guide.md packages are "an icon set, a logger, and dfe-engine-types" Mark the package list as partial
integrations/dfe-ui-guide.md dfe-hyperdx "not built into it" Embedded in an iframe
integrations/dfe-ui-guide.md "In production the intent is OIDC at the edge via Envoy + the X-Oidc-* headers" Describe the engine's OIDC relying party
integrations/dfe-ui-guide.md Roadmap: plan doc docs/superpowers/plans/2026-06-30-governed-ops-api.md, Track A/B, @dfe-engine/client, Orv... Replace with a short list of what is not built
integrations/hyperdx-fork-maintenance.md "Status: PROPOSED", track upstream by release tag and rebase the DFE commit set Delete the page. integrations/README.md links the fork's own runbook
data-plane/source.md rules is a source component, source.rules runs on the hunt runner Remove the rows. Describe how rules bind
data-plane/source-definition.md SchemaBuilder / ClickHouseSchema pipeline, SchemaModifier ALTERs, SchemaBuilder validation SchemaBuilderV2 and the schema applier
data-plane/source-flow.md Profile table lists mesh as a profile Mark mesh as refused
data-plane/sigma.md "About 3% of the SigmaHQ set" Remove
data-plane/expressions-cel.md API compile(expr), "The Rust crate is the single source of truth" Correct the names. Remove the unsupported claim
data-plane/expressions-cel.md Diagram: dfe-loader transformer.rs Name computed.rs in the mermaid diagram
data-plane/schema-sync.md Engine side type_maps.csv + schema CSVs. Loader schemas/profiles/*.yaml, COMMON-HEADER.md, DDL-EXPRESSION.m... Correct the files
data-plane/schema-sync.md "dfe-loader still consumes it as a schemas/ git submodule", "auto-init tables" State that the loader creates none
data-plane/schema-sync.md Directives @source, @generated, @renamed, @captured, @computed List the loader's directives
data-plane/schema-sync.md CI step .github/workflows/schema-compat.yml running dfe_engine.schema.generate_profiles and a loader schema... State that no cross-repo job exists. Name the directive contract test (tests/unit/test_services/test_loader_directive_contract.py:11)
data-plane/schema-sync.md Loader ProfileDiff migrates on a version change The engine's apply adds columns
data-plane/schema-sync.md "Profile definitions (YAML) owner dfe-engine", SchemaBuilder / ClickHouseSchema Correct the description
data-plane/schema-sync.md Monorepo path section Remove
data-plane/schema-classes.md "13 primitives" in the file map, "one of 15 values" in the rules Remove the restated count
data-plane/schema-classes.md expr and comment joined by an em-dash Correct the separator
data-plane/schema-classes.md registries/types.yaml loads as YAML 1.1 through PyYAML YAML 1.2 via ruamel
data-plane/schema-bootstrap.md "the widest Elastic product table is 688 KB" Remove the number
data-plane/schema.md Index table: dimension set(0) Add the entry
data-plane/schema.md similarity_search granularity 1 Corrected
data-plane/schema.md Banked measurements: "229M rows/s -> 98M rows/s", "45x faster", "10-100x faster" Remove
data-plane/schema.md "slave from the deployed ClickHouse schema ONLY ... never an in-flight mismatch ... no cache invalidation r... Describe the cache
data-plane/schema.md Lifecycle "Schema field modified -> ALTER TABLE MODIFY COLUMN (type/codec/comment)" Corrected
data-plane/schema.md Directives live in comment. The format example puts @source in comment Use expr in the example and field table
data-plane/schema.md Header columns in the example (_timestamp datetime, _org_id @source: org_id, _raw/_json @captured) Header columns come from the profile. The example holds data columns only
data-plane/schema.md Directive table @generated, @captured as loader directives List the six directives. Mark @generated as unparsed
data-plane/schema.md "column comments have highest precedence -- above built-in presets and external remap files" Corrected
data-plane/schema.md Table COMMENT @schema_source: core Corrected
data-plane/schema.md Elastic mapping table Use the table from the code
data-plane/query-api-python.md HTTP mode "sends no credentials, so it is answered only by an engine with auth disabled" Name the credential and header
data-plane/query-api-python.md Constructor signature base_url, direct, timeout_seconds Completed
data-plane/hunt-runner-scaling.md "Three tables already exist ... hunt_schedule is the fourth", "the four CH tables" Name all tables without a count
deployment/README.md "Five tiers", mesh as a tier Four selectable tiers. mesh is in the charts only
deployment/README.md "DocumentDB operator running on CloudNativePG PostgreSQL 18" Link the values file
deployment/README.md "Canal IP-masquerades pod egress to the node IP" Remove
deployment/keda-scaling.md Shim "folded into the dfe-engine chart" dfe-extras
deployment/transports.md mesh listed as a profile to pick Mark mesh as refused
deployment/observability-standard.md scalo-py docs/core-pillars/LOGGING.md Corrected
deployment/supply-chain-pinning.md "Dockerfile base images and the Helm chart image ref still use floating tags" Both pinned
deployment/supply-chain-pinning.md dfe-docker DFE images "float to :latest and must be brought onto tag@sha256" Correct the mechanism

Fix or cut every claim in README.md and docs/ that contradicts the code,
the dfe-schemas wheel or the named sibling repos.

Rewrite four pages about unbuilt or outdated designs to match the code:
the edge auth page, the CLI design, the commit standard and the
engine-loader schema contract.

Remove the hyperdx fork maintenance page because its rebase model is
outdated. Point the integrations index at the fork's own runbook.
Move the reference index's one link into docs/README.md.

Every touched page passes hyperi-ai docs prose check with no deny.
hyperi-ci lint-docs passes.
The edge auth table says which routes take an OIDC SecurityPolicy with gateway OIDC off and on. It names Kafbat's own login. It cites the network-policies chart for what lets traffic into the DFE namespaces.

Reword bold-lead bullets and two sentences starting with a capitalised repo name.
@catinspace-au
catinspace-au merged commit 324ee36 into main Nov 15, 2026
19 checks passed
@catinspace-au
catinspace-au deleted the fix/docs-reality branch November 15, 2026 19:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant