hyperi-io · catinspace-au · Sep 19, 2026 · Sep 19, 2026
@@ -313,3 +313,60 @@ Pull requests welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for the
 workflow, commit-message convention, and config conventions. Security
 issues: see [SECURITY.md](SECURITY.md). Community expectations:
 [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md).
+
+## Context
+
+### What this is
+
+A VPN server shipped as one container image. Python inside, but not a Python
+package -- the PyPI publish is opted out and a wheel is built only for pipeline
+parity. Anyone deploys it, so defaults are generic and anything site-shaped
+belongs in a profile or the deployment values. It replaced dfe-vpn, now
+retired, so older references to that repo are stale.
+
+### Where things live
+
+| Path | What it holds |
+|---|---|
+| `scripts/entrypoint.py` | Thin orchestrator. Owns the startup ORDER |
+| `scripts/lib/` | One module per concern -- config, network, pki, openvpn, wireguard, stunnel, wstunnel, oauth2, health, metrics, process, download |
+| `config/*.template` | `${VAR}` templates rendered at runtime |
+| `profiles/` | Opt-in site presets loaded by `CULVERT_PROFILE` |
+| `deploy/` | Helm chart and compose fragment, GENERATED by `scripts/generate-deploy-artefacts.py` |
+| `docs/architecture.md` | Startup order, the two iptables chains, the PKI invariants |
+
+### Commands that prove a change
+
+`hyperi-ci check` locally, `tests/run-all.sh` for the tiers.
+
+Green in CI is the UNIT tier alone. `.hyperi-ci.yaml` disables integration, e2e
+and k8s, turns coverage off, disables `vulture` and leaves `ty` non-blocking.
+Run the rest yourself:
+
+```bash
+pytest tests/e2e/ -m e2e -v     # needs Docker and TUN
+pytest tests/k8s/ -m k8s -v     # needs a kubeconfig and CULVERT_K8S_*
+```
+
+`addopts` carries no `-m "not e2e"`, so a bare `pytest` tries the e2e tier and
+fails without Docker rather than skipping.
+
+### What tends to bite
+
+| Don't | Do | Why |
+|---|---|---|
+| Write the skip-CI marker literally in a commit message | Describe it | GitHub scans the whole message, body included. PRs #35 and #36 got zero CI runs that way and were abandoned for #37 |
+| Hand-edit `deploy/helm/culvert` | Re-run `scripts/generate-deploy-artefacts.py` | It renders from the scalo deployment contract, and a unit test compares the committed output against a fresh render |
+| Call `easyrsa init-pki` | Create the directories directly, as `lib/pki.py` does | On 3.2.x it removes the PKI directory unconditionally. That is a volume mount, so it fails outright, and where it succeeds it takes the CA ([issue #11](https://github.com/hyperi-io/culvert/issues/11)) |
+| Check a renamed doc's links on macOS | Let `tests/unit/test_docs_links.py` do it | It compares against `git ls-files`, so case matters. A new doc also has to be linked from `docs/README.md` |
+| Read a green PR as proof the image runs | Confirm the Container job built | The Ubuntu 26.04 bump and a broken `--proxy` config each passed every check and died at startup |
+
+### Where this sits
+
+Inbound: **hyperi-io/scalo-py**, the only suite edge. `pyproject.toml` declares
+`scalo[metrics,secrets-vault,secrets-aws]` bounded below 3 at runtime, plus a
+dev-only `scalo[deployment]` range the artefact generators need and the runtime
+never imports. Granular extras on purpose.
+
+Outbound: nothing in the suite graph consumes culvert, and it is an optional
+member. It reaches consumers as a public container image on GHCR.
@@ -30,6 +30,9 @@ for one of the opinionated `CULVERT_PROFILE=` presets.
 
 ## Understand a decision
 
+- [architecture.md](architecture.md) - what is in the image, the startup order,
+  the two iptables chains and why their order is what it is, and the PKI
+  invariants. Start here before changing any of them.
 - [addressing.md](addressing.md) - why the tunnels default to `10.8.0.0/22`,
   and when to move to the CGNAT range instead.
 - [performance.md](performance.md) - the cipher stance (AES-256-GCM the

@@ -0,0 +1,180 @@
+# Architecture
+
+What culvert is made of, why it is one container instead of a stack, and the
+invariants that no single file makes obvious. Look a fact up here before you
+change the startup order, the firewall chains or the PKI paths.
+
+## The short version
+
+A VPN server is easy to install and hard to keep correct. The usual answers
+either mutate a host with a shell script, or ship a management plane, a
+database and a web UI to configure one daemon. Culvert takes the other route:
+one image, configuration by environment variable, and every capability past
+plain OpenVPN over UDP an explicit opt-in. Upgrading is a `docker pull`.
+
+That choice is what shapes everything below. There is no state store, so a
+replica is defined entirely by its environment and its mounted volumes. There
+is no control plane, so the container itself has to refuse to start when a
+control it was asked for did not install.
+
+## What is in the image
+
+The base is `ubuntu:26.04`, pinned by digest. Every third-party binary comes
+from upstream directly, and each arrives with its own verification:
+
+| Component | Source | How it is verified |
+|---|---|---|
+| OpenVPN 2.7+ | The project's signed apt repo | Downloaded keyring must hold exactly one primary key, whose fingerprint is pinned. Installed version then checked against a floor |
+| easy-rsa, iptables, wireguard-tools, stunnel4 | Ubuntu archive | apt, over the base image's trust |
+| openvpn-auth-oauth2 | Pinned GitHub release `.deb` | SHA256 per architecture, checked before `dpkg` |
+| wstunnel | Pinned GitHub release tarball | SHA256 per architecture, checked before extract |
+| Python dependencies | `requirements-docker.txt`, exported from `uv.lock` | `pip --require-hashes`, so a drifted release fails the build |
+
+Counting `pub` records rather than `fpr` records on the OpenVPN keyring is
+deliberate. A key with signing subkeys emits one `fpr` line per subkey, all
+legitimately the same primary, so counting fingerprints would reject a valid
+key and counting keys catches a second one smuggled in alongside the real one.
+
+The apt repo serves one version per suite and cannot be pinned, which is why
+the version floor is asserted after install rather than requested before it.
+The server config depends on 2.7 behaviour, so building on 2.6 would change it
+silently. `openvpn-auth-oauth2` gets the artefact treatment instead because its
+apt repo serves `releases/latest` and cannot be pinned at all.
+
+DCO is supported but its DKMS package is deliberately absent. Inside a
+container DKMS would compile against kernel headers that are not there, and the
+module has to be loaded on the host regardless -- `vm-setup/setup-host.sh` does
+that. Leaving it out drops a compiler toolchain from the image at no cost to
+the capability.
+
+## One entrypoint, one ordered sequence
+
+`scripts/entrypoint.py` is a thin orchestrator. Every piece of real work lives
+in a focused module under `scripts/lib/`, and the entrypoint's job is the
+order.
+
+```
+config -> directories -> log rotation -> network -> routing control
+       -> forward guards -> [OpenVPN branch] -> [WireGuard branch]
+       -> observability -> CRL refresh -> client download -> block
+```
+
+The OpenVPN branch initialises PKI, renders the UDP, HTTPS and TCP server
+configs from `config/*.template`, sets up stunnel, generates any auto-clients
+and starts the OIDC helper. The WireGuard branch brings up `wg0` through
+`wg-quick`, starts wstunnel and starts a connection monitor. `CULVERT_PROTOCOL`
+selects one, the other, or both, and `both` runs both branches in full.
+
+In WireGuard-only mode there is no OpenVPN process to block on, so the
+entrypoint marks itself started and ready and waits on signals. The kernel
+interface is already up at that point, which is why readiness flips there
+rather than in the server starter.
+
+## PKI: two modes, and local disk is the source of truth
+
+Local mode has Easy-RSA generate a CA and a server certificate. External mode
+fetches them through scalo secrets from a file mount, OpenBao or AWS Secrets
+Manager. In both modes the files under the PKI directory are what OpenVPN
+reads, and a remote provider is the update mechanism rather than the authority.
+That is what lets an external deployment keep serving when the provider is
+unreachable, on cached certificates, with a warning.
+
+Three things here are not guessable from the code's shape:
+
+- **`easyrsa init-pki` is never called.** All it creates is three directories,
+  and it insists on removing the PKI directory first, unconditionally on 3.2.x
+  with no opt-out. The directory is a volume mount, so the removal fails
+  outright, and where it could succeed it would take the CA and every
+  certificate issued from it. 3.1.x's `init-pki soft` is not a portable
+  substitute either, because 3.2.x reads that argument as a curve name.
+  Culvert creates the directories itself, and `build-ca` still refuses to
+  overwrite an existing CA, so a half-built PKI fails loudly instead of being
+  replaced.
+- **The `tls-crypt-v2` server key is minted locally only when no provider
+  supplies one.** A client's key derives from it, so two servers with
+  different keys reject each other's clients. Once
+  `CULVERT_SECRETS_TC_KEY_PATH` is set, a failed fetch is fatal rather than a
+  fall back to minting, because an install that comes up healthy and rejects
+  clients at random is worse than one that refuses to start.
+- **CRL refresh differs by mode and cannot be shared.** Local PKI regenerates
+  from its own CA. External PKI has no CA key present, so the only way to pick
+  up a revocation is to re-fetch. OpenVPN reads the CRL at startup, so a new
+  file on disk changes nothing until the server is told to re-read it.
+
+A CRL that reaches its `nextUpdate` makes OpenVPN refuse every client,
+including valid ones. The refresher therefore reports remaining life on every
+cycle, not just whether the last refresh worked.
+
+## Enforcement is two iptables chains, and the order matters
+
+NAT and MSS clamping go in the base `FORWARD` and `POSTROUTING` paths.
+Everything policy-shaped lives in two dedicated chains, both rebuilt on each
+start so restarts cannot stack duplicates:
+
+- `CULVERT_FWD` -- opt-in routing control. Client isolation, the egress
+  allow-list, and the downstream-admin gate that lets an operator initiate back
+  into the tunnels.
+- `CULVERT_GUARD` -- always on by default. Drops forwarded client traffic to
+  link-local, which on a cloud instance is the metadata service and this host's
+  credentials.
+
+Four ordering rules hold this together, and each exists because the obvious
+arrangement is wrong:
+
+1. `CULVERT_GUARD` is installed AFTER `CULVERT_FWD`. Both insert at `FORWARD`
+   position 1, so the last one installed is evaluated first, and the guard has
+   to be. It is separate from routing control precisely because routing control
+   is opt-in and the guard must hold on a default configuration.
+2. `CULVERT_FWD` is detached from `FORWARD` before it is flushed, built
+   completely, and jumped at last. A flushed chain that is still jumped falls
+   through to the `FORWARD` policy, so there must be no window where it is
+   jumped and empty.
+3. Client-to-client rules sit above the conntrack accept, so cross-tunnel
+   traffic is decided without reference to state and a RELATED packet cannot
+   slip between clients.
+4. Reverse-admin replies get a NAT `RETURN`, scoped to established flows only.
+   Without it the general masquerade rewrites the client's reply to the
+   server's own address and the exchange never completes. Scoping it keeps a
+   client that initiates towards the admin range masqueraded.
+
+Rules of this kind are not advisory, so a failed install raises and the
+container exits. Serving clients while reporting a control that is not there is
+the outcome being avoided. Chain creation and rule deletion stay tolerant,
+because both are expected to fail on a normal restart.
+
+The guard is scoped to `FORWARD` deliberately, which leaves the server's own
+access to the metadata service intact -- external PKI on AWS authenticates with
+exactly those instance credentials. The tunnels carry IPv4 only and only
+`net.ipv4.ip_forward` is enabled, so there is no ip6tables setup at all.
+
+## Observability is one listener
+
+`CULVERT_METRICS_ADDR` (default `0.0.0.0:9090`) carries `/livez` and `/readyz`
+always, and `/metrics` when metrics are enabled. OTLP is a push to a collector,
+so it adds no listener. Health and metrics deliberately share the port so that
+one published port covers the operator surface and nothing about it rides on
+the VPN listeners.
+
+## Deployment artefacts are generated, not authored
+
+`deploy/helm/culvert` and the compose fragment are rendered from culvert's
+scalo deployment contract by `scripts/generate-deploy-artefacts.py`. Editing
+the chart by hand puts it out of step with the contract, and a unit test
+compares the committed output against a fresh render. The chart's `appVersion`
+is a known weak point: the release commit stamps `VERSION` and the changelog
+without re-rendering, so the chart can ship pointing at the previous image
+([issue #38](https://github.com/hyperi-io/culvert/issues/38)).
+
+## Invariants, collected
+
+- Configuration is `CULVERT_*` environment variables through scalo's cascade,
+  with profile YAML as an extra source. Explicit env always beats a profile.
+- `/etc/vpn` and `/var/log/vpn` are the canonical paths and nothing aliases
+  them. Compatibility symlinks to the legacy OpenVPN directories were removed
+  rather than forced, because the package already owns those as real
+  directories and the link landed inside one, which silently defeated PKI
+  persistence.
+- A replica holds no state beyond its volumes, which is what makes horizontal
+  scaling possible and what makes a shared `tls-crypt-v2` key mandatory for it.
+- Defaults are generic, never site-specific. Anything site-shaped belongs in a
+  profile or the deployment values.