From e1827ac0666879bf7797a85da5fbf4da2f57ebb7 Mon Sep 17 00:00:00 2001 From: Derek Date: Sat, 19 Sep 2026 13:33:12 +1000 Subject: [PATCH] fix(docs): add a README Context section and the architecture doc Two gaps. The README had no working guide, and there was no architecture doc at all, so the startup ordering rules and the PKI traps lived only as comments in the modules that implement them. docs/architecture.md now carries what is in the image and how each binary is verified, the entrypoint's ordered sequence, the two iptables chains and the four ordering rules that hold them together, and the three PKI invariants that are not guessable from the code's shape. All of it read out of the Dockerfile, scripts/entrypoint.py, scripts/lib/network.py and scripts/lib/pki.py this turn. docs/README.md gains the index entry, because tests/unit/test_docs_links.py fails the suite on a doc the landing page does not link. Verified: tests/unit/test_docs_links.py passes, 5 of 5. The wider unit tier is 467 passed, 3 failed, and the three are issue #38 firing on an unrelated change -- Chart.yaml is at appVersion v2.1.14 on main while VERSION says 2.1.15. Nothing here touches Chart.yaml, VERSION or CHANGELOG.md. Not verified: the integration, e2e and k8s tiers were not run. --- README.md | 57 ++++++++++++++ docs/README.md | 3 + docs/architecture.md | 180 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 240 insertions(+) create mode 100644 docs/architecture.md diff --git a/README.md b/README.md index d0781e3..0165484 100644 --- a/README.md +++ b/README.md @@ -313,3 +313,60 @@ Pull requests welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for the workflow, commit-message convention, and config conventions. Security issues: see [SECURITY.md](SECURITY.md). Community expectations: [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md). + +## Context + +### What this is + +A VPN server shipped as one container image. Python inside, but not a Python +package -- the PyPI publish is opted out and a wheel is built only for pipeline +parity. Anyone deploys it, so defaults are generic and anything site-shaped +belongs in a profile or the deployment values. It replaced dfe-vpn, now +retired, so older references to that repo are stale. + +### Where things live + +| Path | What it holds | +|---|---| +| `scripts/entrypoint.py` | Thin orchestrator. Owns the startup ORDER | +| `scripts/lib/` | One module per concern -- config, network, pki, openvpn, wireguard, stunnel, wstunnel, oauth2, health, metrics, process, download | +| `config/*.template` | `${VAR}` templates rendered at runtime | +| `profiles/` | Opt-in site presets loaded by `CULVERT_PROFILE` | +| `deploy/` | Helm chart and compose fragment, GENERATED by `scripts/generate-deploy-artefacts.py` | +| `docs/architecture.md` | Startup order, the two iptables chains, the PKI invariants | + +### Commands that prove a change + +`hyperi-ci check` locally, `tests/run-all.sh` for the tiers. + +Green in CI is the UNIT tier alone. `.hyperi-ci.yaml` disables integration, e2e +and k8s, turns coverage off, disables `vulture` and leaves `ty` non-blocking. +Run the rest yourself: + +```bash +pytest tests/e2e/ -m e2e -v # needs Docker and TUN +pytest tests/k8s/ -m k8s -v # needs a kubeconfig and CULVERT_K8S_* +``` + +`addopts` carries no `-m "not e2e"`, so a bare `pytest` tries the e2e tier and +fails without Docker rather than skipping. + +### What tends to bite + +| Don't | Do | Why | +|---|---|---| +| Write the skip-CI marker literally in a commit message | Describe it | GitHub scans the whole message, body included. PRs #35 and #36 got zero CI runs that way and were abandoned for #37 | +| Hand-edit `deploy/helm/culvert` | Re-run `scripts/generate-deploy-artefacts.py` | It renders from the scalo deployment contract, and a unit test compares the committed output against a fresh render | +| Call `easyrsa init-pki` | Create the directories directly, as `lib/pki.py` does | On 3.2.x it removes the PKI directory unconditionally. That is a volume mount, so it fails outright, and where it succeeds it takes the CA ([issue #11](https://github.com/hyperi-io/culvert/issues/11)) | +| Check a renamed doc's links on macOS | Let `tests/unit/test_docs_links.py` do it | It compares against `git ls-files`, so case matters. A new doc also has to be linked from `docs/README.md` | +| Read a green PR as proof the image runs | Confirm the Container job built | The Ubuntu 26.04 bump and a broken `--proxy` config each passed every check and died at startup | + +### Where this sits + +Inbound: **hyperi-io/scalo-py**, the only suite edge. `pyproject.toml` declares +`scalo[metrics,secrets-vault,secrets-aws]` bounded below 3 at runtime, plus a +dev-only `scalo[deployment]` range the artefact generators need and the runtime +never imports. Granular extras on purpose. + +Outbound: nothing in the suite graph consumes culvert, and it is an optional +member. It reaches consumers as a public container image on GHCR. diff --git a/docs/README.md b/docs/README.md index e0d2a4e..4a48bfb 100644 --- a/docs/README.md +++ b/docs/README.md @@ -30,6 +30,9 @@ for one of the opinionated `CULVERT_PROFILE=` presets. ## Understand a decision +- [architecture.md](architecture.md) - what is in the image, the startup order, + the two iptables chains and why their order is what it is, and the PKI + invariants. Start here before changing any of them. - [addressing.md](addressing.md) - why the tunnels default to `10.8.0.0/22`, and when to move to the CGNAT range instead. - [performance.md](performance.md) - the cipher stance (AES-256-GCM the diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..ac3c840 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,180 @@ +# Architecture + +What culvert is made of, why it is one container instead of a stack, and the +invariants that no single file makes obvious. Look a fact up here before you +change the startup order, the firewall chains or the PKI paths. + +## The short version + +A VPN server is easy to install and hard to keep correct. The usual answers +either mutate a host with a shell script, or ship a management plane, a +database and a web UI to configure one daemon. Culvert takes the other route: +one image, configuration by environment variable, and every capability past +plain OpenVPN over UDP an explicit opt-in. Upgrading is a `docker pull`. + +That choice is what shapes everything below. There is no state store, so a +replica is defined entirely by its environment and its mounted volumes. There +is no control plane, so the container itself has to refuse to start when a +control it was asked for did not install. + +## What is in the image + +The base is `ubuntu:26.04`, pinned by digest. Every third-party binary comes +from upstream directly, and each arrives with its own verification: + +| Component | Source | How it is verified | +|---|---|---| +| OpenVPN 2.7+ | The project's signed apt repo | Downloaded keyring must hold exactly one primary key, whose fingerprint is pinned. Installed version then checked against a floor | +| easy-rsa, iptables, wireguard-tools, stunnel4 | Ubuntu archive | apt, over the base image's trust | +| openvpn-auth-oauth2 | Pinned GitHub release `.deb` | SHA256 per architecture, checked before `dpkg` | +| wstunnel | Pinned GitHub release tarball | SHA256 per architecture, checked before extract | +| Python dependencies | `requirements-docker.txt`, exported from `uv.lock` | `pip --require-hashes`, so a drifted release fails the build | + +Counting `pub` records rather than `fpr` records on the OpenVPN keyring is +deliberate. A key with signing subkeys emits one `fpr` line per subkey, all +legitimately the same primary, so counting fingerprints would reject a valid +key and counting keys catches a second one smuggled in alongside the real one. + +The apt repo serves one version per suite and cannot be pinned, which is why +the version floor is asserted after install rather than requested before it. +The server config depends on 2.7 behaviour, so building on 2.6 would change it +silently. `openvpn-auth-oauth2` gets the artefact treatment instead because its +apt repo serves `releases/latest` and cannot be pinned at all. + +DCO is supported but its DKMS package is deliberately absent. Inside a +container DKMS would compile against kernel headers that are not there, and the +module has to be loaded on the host regardless -- `vm-setup/setup-host.sh` does +that. Leaving it out drops a compiler toolchain from the image at no cost to +the capability. + +## One entrypoint, one ordered sequence + +`scripts/entrypoint.py` is a thin orchestrator. Every piece of real work lives +in a focused module under `scripts/lib/`, and the entrypoint's job is the +order. + +``` +config -> directories -> log rotation -> network -> routing control + -> forward guards -> [OpenVPN branch] -> [WireGuard branch] + -> observability -> CRL refresh -> client download -> block +``` + +The OpenVPN branch initialises PKI, renders the UDP, HTTPS and TCP server +configs from `config/*.template`, sets up stunnel, generates any auto-clients +and starts the OIDC helper. The WireGuard branch brings up `wg0` through +`wg-quick`, starts wstunnel and starts a connection monitor. `CULVERT_PROTOCOL` +selects one, the other, or both, and `both` runs both branches in full. + +In WireGuard-only mode there is no OpenVPN process to block on, so the +entrypoint marks itself started and ready and waits on signals. The kernel +interface is already up at that point, which is why readiness flips there +rather than in the server starter. + +## PKI: two modes, and local disk is the source of truth + +Local mode has Easy-RSA generate a CA and a server certificate. External mode +fetches them through scalo secrets from a file mount, OpenBao or AWS Secrets +Manager. In both modes the files under the PKI directory are what OpenVPN +reads, and a remote provider is the update mechanism rather than the authority. +That is what lets an external deployment keep serving when the provider is +unreachable, on cached certificates, with a warning. + +Three things here are not guessable from the code's shape: + +- **`easyrsa init-pki` is never called.** All it creates is three directories, + and it insists on removing the PKI directory first, unconditionally on 3.2.x + with no opt-out. The directory is a volume mount, so the removal fails + outright, and where it could succeed it would take the CA and every + certificate issued from it. 3.1.x's `init-pki soft` is not a portable + substitute either, because 3.2.x reads that argument as a curve name. + Culvert creates the directories itself, and `build-ca` still refuses to + overwrite an existing CA, so a half-built PKI fails loudly instead of being + replaced. +- **The `tls-crypt-v2` server key is minted locally only when no provider + supplies one.** A client's key derives from it, so two servers with + different keys reject each other's clients. Once + `CULVERT_SECRETS_TC_KEY_PATH` is set, a failed fetch is fatal rather than a + fall back to minting, because an install that comes up healthy and rejects + clients at random is worse than one that refuses to start. +- **CRL refresh differs by mode and cannot be shared.** Local PKI regenerates + from its own CA. External PKI has no CA key present, so the only way to pick + up a revocation is to re-fetch. OpenVPN reads the CRL at startup, so a new + file on disk changes nothing until the server is told to re-read it. + +A CRL that reaches its `nextUpdate` makes OpenVPN refuse every client, +including valid ones. The refresher therefore reports remaining life on every +cycle, not just whether the last refresh worked. + +## Enforcement is two iptables chains, and the order matters + +NAT and MSS clamping go in the base `FORWARD` and `POSTROUTING` paths. +Everything policy-shaped lives in two dedicated chains, both rebuilt on each +start so restarts cannot stack duplicates: + +- `CULVERT_FWD` -- opt-in routing control. Client isolation, the egress + allow-list, and the downstream-admin gate that lets an operator initiate back + into the tunnels. +- `CULVERT_GUARD` -- always on by default. Drops forwarded client traffic to + link-local, which on a cloud instance is the metadata service and this host's + credentials. + +Four ordering rules hold this together, and each exists because the obvious +arrangement is wrong: + +1. `CULVERT_GUARD` is installed AFTER `CULVERT_FWD`. Both insert at `FORWARD` + position 1, so the last one installed is evaluated first, and the guard has + to be. It is separate from routing control precisely because routing control + is opt-in and the guard must hold on a default configuration. +2. `CULVERT_FWD` is detached from `FORWARD` before it is flushed, built + completely, and jumped at last. A flushed chain that is still jumped falls + through to the `FORWARD` policy, so there must be no window where it is + jumped and empty. +3. Client-to-client rules sit above the conntrack accept, so cross-tunnel + traffic is decided without reference to state and a RELATED packet cannot + slip between clients. +4. Reverse-admin replies get a NAT `RETURN`, scoped to established flows only. + Without it the general masquerade rewrites the client's reply to the + server's own address and the exchange never completes. Scoping it keeps a + client that initiates towards the admin range masqueraded. + +Rules of this kind are not advisory, so a failed install raises and the +container exits. Serving clients while reporting a control that is not there is +the outcome being avoided. Chain creation and rule deletion stay tolerant, +because both are expected to fail on a normal restart. + +The guard is scoped to `FORWARD` deliberately, which leaves the server's own +access to the metadata service intact -- external PKI on AWS authenticates with +exactly those instance credentials. The tunnels carry IPv4 only and only +`net.ipv4.ip_forward` is enabled, so there is no ip6tables setup at all. + +## Observability is one listener + +`CULVERT_METRICS_ADDR` (default `0.0.0.0:9090`) carries `/livez` and `/readyz` +always, and `/metrics` when metrics are enabled. OTLP is a push to a collector, +so it adds no listener. Health and metrics deliberately share the port so that +one published port covers the operator surface and nothing about it rides on +the VPN listeners. + +## Deployment artefacts are generated, not authored + +`deploy/helm/culvert` and the compose fragment are rendered from culvert's +scalo deployment contract by `scripts/generate-deploy-artefacts.py`. Editing +the chart by hand puts it out of step with the contract, and a unit test +compares the committed output against a fresh render. The chart's `appVersion` +is a known weak point: the release commit stamps `VERSION` and the changelog +without re-rendering, so the chart can ship pointing at the previous image +([issue #38](https://github.com/hyperi-io/culvert/issues/38)). + +## Invariants, collected + +- Configuration is `CULVERT_*` environment variables through scalo's cascade, + with profile YAML as an extra source. Explicit env always beats a profile. +- `/etc/vpn` and `/var/log/vpn` are the canonical paths and nothing aliases + them. Compatibility symlinks to the legacy OpenVPN directories were removed + rather than forced, because the package already owns those as real + directories and the link landed inside one, which silently defeated PKI + persistence. +- A replica holds no state beyond its volumes, which is what makes horizontal + scaling possible and what makes a shared `tls-crypt-v2` key mandatory for it. +- Defaults are generic, never site-specific. Anything site-shaped belongs in a + profile or the deployment values.