rfc: regional security principles and key management deployment proposal - #21
rfc: regional security principles and key management deployment proposal#21hannahhoward wants to merge 2 commits into
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The regional security RFC moves the region wrap from A256KW to AES-256-GCM inside the region's secrets manager. Swap the algorithm in the keys table and point to the new RFC; tenant and Hilt wraps unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
bajtos
left a comment
bajtos
left a comment
There was a problem hiding this comment.
Great proposal, I agree with it!
I have a few comments to consider, see below.
|
|
||
| ## Principles | ||
|
|
||
| **1. Two properties are winnable at the region, and we build for both.** A powered-off disk, imaged after theft, RMA, or decommissioning, yields no object plaintext, no key material, and no credential that still works. And a region we have decided to cut off does not come back up: revoking it centrally prevents the appliance from starting. Every design decision below serves these two properties. |
There was a problem hiding this comment.
A powered-off disk, imaged after theft, RMA, or decommissioning, yields no object plaintext, no key material, and no credential that still works.
Isn't this easily achieved using full-disk encryption?
|
|
||
| **2. Discovery is the caveat on both.** The revocation lever helps once we know to pull it. A malicious operator or a stolen machine that we discover is contained from the next boot. Discovered too late, there is very little left to do: the system has been serving that adversary plaintext all along. This is a limit of the problem itself, and no design spend moves it. Remote kill of a *running* appliance (a central-triggered seal) would narrow the window between discovery and containment; it is worth looking at and is lower priority than getting the at-rest and startup properties right. | ||
|
|
||
| **3. The running system has no defense against its own operator.** The appliance holds plaintext CEKs and object bytes in memory whenever it serves a read, and the operator controls the hypervisor, the kernel, and the network path. PR #19's threat model states this plainly: "The region operator has hypervisor access and can read the SSH host key and any authorized-keys material." Neither of the obvious conclusions follows. Reaching for exotic protection against this adversary (trusted execution environments as a requirement) spends complexity on a property we cannot have. Concluding that at-rest custody is pointless because memory is readable anyway gives away the two properties we can have. |
There was a problem hiding this comment.
Reaching for exotic protection against this adversary (trusted execution environments as a requirement) spends complexity on a property we cannot have. Concluding that at-rest custody is pointless because memory is readable anyway gives away the two properties we can have.
In my limited understanding of confidential computing, the host cannot read the memory of a VM running inside a TEE.
I believe that running our appliance inside a TEE would protect us from this class of attacks.
At the same time, I think we are better off trusting our region operators not to exfiltrate data, at least for the first few regions.
|
|
||
| ## Principles | ||
|
|
||
| **1. Two properties are winnable at the region, and we build for both.** A powered-off disk, imaged after theft, RMA, or decommissioning, yields no object plaintext, no key material, and no credential that still works. And a region we have decided to cut off does not come back up: revoking it centrally prevents the appliance from starting. Every design decision below serves these two properties. |
There was a problem hiding this comment.
I want to point out another property that should be easy to achieve and make our appliance meaningfully more secure:
- Do not keep any plaintext/key material/credentials anywhere on the filesystem.
Even if a malicious node operator sshs into the running VM and looks around, they cannot access any sensitive content by reading files. The only way to access sensitive stuff is by reading the VM's memory, and extracting secrets from raw memory bytes requires much more sophisticated effort.
|
|
||
| ## Design | ||
|
|
||
| ### One home for secrets: a local OpenBao, rooted centrally |
|
|
||
| ### One home for secrets: a local OpenBao, rooted centrally | ||
|
|
||
| Each appliance runs OpenBao, listening on a unix socket, with raft storage on the appliance disk. It is the single home for regional secrets: the Region KEK (a non-exportable transit key), the provider wallet key, TLS leaf keys, Postgres credentials, service identity PEMs, and the S3 credentials the installer currently writes into config files. Nothing secret sits in a plain file. |
There was a problem hiding this comment.
TLS leaf keys
I would like to provision TLS certs using Caddy's built-in features.
It seems that Caddy can use Vault as the storage backend via 3rd-party plugin, so I guess we are good here.
|
|
||
| The switch also adds a capability A256KW cannot offer. Context binding means a wrapped CEK authenticates only against its own (space, blob digest): transplanting wrap material between rows fails outright (verified: wrong context yields an authentication failure). And because `derived=true` gives each context its own subkey, no single GCM key accrues invocations toward NIST's per-key bound no matter how many billions of objects a region holds. Binding rides `context` rather than `associated_data` deliberately: OpenBao's rewrap endpoint honors context and rejects AAD-bound ciphertexts, and rewrap is how rotation stays clean — rotate the key, then batched `rewrap` moves stored ciphertexts from `vault:v1:` to `vault:v2:` entirely inside OpenBao, with no CEK ever appearing in any process of ours (verified end to end). Old versions keep decrypting until `min_decryption_version` advances, so rewrap campaigns can be lazy. | ||
|
|
||
| GCM fails badly under nonce reuse. The realistic trigger is a cloned VM image or an entropy-starved first boot rather than request volume. Context derivation already reduces any one subkey to a handful of invocations; the remaining discipline — seeded entropy at first boot, no snapshot reuse of a running appliance — goes on the host checklist below. |
There was a problem hiding this comment.
seeded entropy at first boot, no snapshot reuse of a running appliance
I think this is a new requirement we need to consider when decided how to run appliance (#19), can you please confirm?
There was a problem hiding this comment.
According to Claude, when using Vault Transit, the nonces are managed by Vault.
With Transit, nonce generation is inside Vault's barrier. Omit the nonce parameter entirely on POST /v1/transit/encrypt/:key and Vault draws a fresh 96-bit nonce from its own CSPRNG per operation, then embeds it in the returned blob:
vault:v1:<base64(nonce || ciphertext || tag)>Because the nonce ships inside the ciphertext string, there's also nothing extra to persist alongside your wrapped key — you store one opaque string.
Failure modes mapped to Transit - I think we should be good?
| Concern | Under Transit |
|---|---|
| Key rotation / re-wrap | Each rotate mints a new key version. v2 is a different GCM key than v1, so nonce space doesn't overlap. Non-issue. |
| Migration script run twice | Produces two ciphertexts of the same plaintext under two different random nonces. Correct behavior, no reuse. |
| Two hosts, independent RNGs | Randomness comes from the Vault cluster, not your hosts. Non-issue. |
| Snapshot-restore rewinds a counter | Random nonces are stateless — there's no counter to rewind. But a rollback loses key versions created after the snapshot, making ciphertexts under those versions permanently undecryptable. That's a durability risk, not a nonce risk, and it's the one you should actually design around. |
One more comment from Claude:
What you do still own: the birthday bound
Random 96-bit nonces collide at the birthday bound. NIST SP 800-38D caps invocations per key at 2³² when IVs are random, to hold collision probability at ~2⁻³². Vault doesn't enforce this for you — key versions are the mechanism.
For KEK operations (wrapping CEKs, not bulk data) you're realistically 4–5 orders of magnitude below 2³² for the service's lifetime. Set auto_rotate_period anyway — it bounds blast radius and forces you to exercise the rewrap path before you need it in anger.
|
|
||
| ### Host hardening replaces process-level memory protection | ||
|
|
||
| Following OpenBao's own post-mlock guidance: swap disabled (`memory.swap.max=0` in the unit's cgroup) or encrypted, core dumps off, seeded entropy at first boot, no snapshot/clone reuse. One host checklist protects OpenBao's process and Ingot's alike. The irreducible key material in Ingot is one object's CEK for the duration of its own request, which is the blast radius we accept for being the process that serves plaintext. The dead-disk property carries one wiring dependency alongside this checklist: the PUT pipeline must encrypt bytes before they rest in the spool (the [FIL-481](https://linear.app/filecoin-foundation/issue/FIL-481)/[FIL-482](https://linear.app/filecoin-foundation/issue/FIL-482) ordering), or the spool must live on a bound volume. |
There was a problem hiding this comment.
Following OpenBao's own post-mlock guidance: swap disabled (
memory.swap.max=0in the unit's cgroup) or encrypted, core dumps off, seeded entropy at first boot, no snapshot/clone reuse. One host checklist protects OpenBao's process and Ingot's alike.
Seems like another thing we need to incorporate in our deployment setup.
|
|
||
| Single-operation latency is a serial round trip and does not vary with cores at all; a slower core moves it from ~0.1 ms toward ~0.2 ms, which stays invisible inside a GET that already walks the MST and fetches from Piri. The per-request cost decomposes into roughly 100 µs of connection, JSON, and auth handling against 6–9 µs of actual key wrap per item. The bottleneck is request mechanics rather than cryptography, which is exactly what `batch_input` amortizes: a 2-core configuration still wraps over 300,000 keys per second because a batch pays the request overhead once. | ||
|
|
||
| The Region-KEK comparison appended to PR #19 (in its analysis notes) puts Vault transit at "0.3–1 ms" per operation, a "3k–10k PUT/s" ceiling, and "~1 hour" to rotate 10M parts, and concludes "this is putting a ceil on the maximum throughput we can achieve." Measured locally, the per-operation cost is three to ten times lower; the two-core floor roughly doubles the top of that estimate, and roughly meets it after halving for slower server cores. A 10M-part rotation at constrained batch rates is under a minute of transit time, with the Postgres row updates dominating the campaign. The intermediate-KEK layer that table motivates (held unwrapped in Ingot's memory, keeping the manager off the per-part path) addresses a throughput limit the local measurements do not reproduce, and it would carry more long-lived key material in application memory while giving up per-part custody, audit, and context binding. The same table's YubiHSM column is right, though: ~10 ms per operation makes an HSM on the per-part path infeasible. An HSM belongs at the root of trust: the central unseal today, a PKCS#11 provider later. |
| - Audit device on the appliance: per-wrap audit costs about 30% of sustained throughput, which we do not currently need. On by default, or off? | ||
| - Spool and encryption ordering: the dead-disk property requires the PUT pipeline to encrypt before bytes rest in the spool, or the spool to live on a bound volume. This belongs to the [FIL-481](https://linear.app/filecoin-foundation/issue/FIL-481)/[FIL-482](https://linear.app/filecoin-foundation/issue/FIL-482) wiring. | ||
| - A separated Piri on its own machine would change what an intruder can reach: Piri holds only FEE ciphertext, so access to a running Piri host could yield the provider wallet but not decrypted customer data. The Principle 3 concession attaches to the machine that decrypts, so a split deployment could keep bulk storage on less-trusted hardware. Relevant to future deployments. | ||
| - Who operates the central OpenBao, and its availability target; ties into PR #19's open question of who operates which layer. |
There was a problem hiding this comment.
Note #19 is not covering the Forge Central services (Hilt, Sprue, piri-signing-service, OpenBao).
I think that does not matter in the context of this RFC. We already know that we need to run a central OpenBao for Hilt, adding appliance-local OpenBao as another client should be a trivial improvement.
bajtos
left a comment
bajtos
left a comment
There was a problem hiding this comment.
After I learned more about what it means to operate an OpenBao instance (see Upgrade OpenBao version in my FilOne Appliance Deployment RFC), I prefer not to add OpenBao to every Ingot instance.
Instead, I propose adopting https://developers.google.com/tink as an embedded alternative to Vault/OpenBao - similarly to how SQLite is an embedded alternative to Postgres.
We can still use the central OpenBao to "unseal" the secrets (unwrap the local KEK used to decrypt secrets).
The key issue is embedded KMS means single process. Tink is fundamentally not designed to read from multiple processes at once. I could see this working for Piri, in the future, as I see SQLite and Tink being a reasonable path for a future where scaling Piri is done through one Piri per machine connected to a storage array. Fundamentally Ingot isn't designed this way -- the target is a stateless process that talks to an out-of-process database, and therefore also, an out of process secrets manager. the scaling model is horizontal scaling a la traditional web applications. Zero downtime blue-green ingot deploys is a first order priority (maybe not v1 but the first zero downtime improvement we ship) Ultimately, I just don't see deploying ingot an embeeded secrets manager working, and since we're stuck with Postgres and OpenBao, I think the first version just has Piri talking to those. |
|
|
||
| ### One home for secrets: a local OpenBao, rooted centrally | ||
|
|
||
| Each appliance runs OpenBao, listening on a unix socket, with raft storage on the appliance disk. It is the single home for regional secrets: the Region KEK (a non-exportable transit key), the provider wallet key, TLS leaf keys, Postgres credentials, service identity PEMs, and the S3 credentials the installer currently writes into config files. Nothing secret sits in a plain file. |
There was a problem hiding this comment.
One gap worth specifying: how the non-transit secrets (service PEMs, DSN, S3 creds) travel from Bao into Piri/Ingot processes.
Proposal: systemd LoadCredential= — pid1 materializes each secret into the unit's private ramfs (can't swap, per-unit mount namespace, 0400, gone on stop), apps keep reading file paths, quadlet bind-mounts %d/ into the container. Source is pluggable (plain file for dev, TPM blob for standalone, socket→Bao for enrolled) so the unit files are identical across modes. This is a documented host-OS feature, per Principle 4 and the "stock machinery" criterion — no secrets client in either app.
There was a problem hiding this comment.
I'm finding this hard to read and understand. I assume this means that there's essentially no change to Piri/Ingot for these credentials? We keep reading from file paths (or in the case of postgres DSN we add load from file support).
|
|
||
| Each appliance runs OpenBao, listening on a unix socket, with raft storage on the appliance disk. It is the single home for regional secrets: the Region KEK (a non-exportable transit key), the provider wallet key, TLS leaf keys, Postgres credentials, service identity PEMs, and the S3 credentials the installer currently writes into config files. Nothing secret sits in a plain file. | ||
|
|
||
| The local OpenBao's storage is sealed by a transit key held at a central OpenBao (`seal "transit"`). At boot the appliance authenticates to central, unwraps its barrier key, and unseals. With central unreachable, or the seal credential revoked, OpenBao 2.6.1 refuses to start at all (verified behavior). This is the startup-kill lever from Principle 1: revoking one credential at central makes everything on the disk permanently unreadable. The boot credential is CIDR-bound to the region's egress and single-use where the deployment permits, so an imaged disk replayed elsewhere fails, and any use of a stolen credential is visible at central. Hardware binding of the boot identity (TPM sealing at enrollment) is the eventual strengthening; it needs host-image work and is out of scope here. The boot-time dependency on central has the same failure envelope the appliance already accepts for auth, where Hilt-derived credentials expire daily; steady-state reads never call central, preserving the encryption RFC's Independence criterion. |
There was a problem hiding this comment.
Above we say:
Each appliance runs OpenBao [...] It is the single home for regional secrets: the Region KEK (a non-exportable transit key), the provider wallet key, TLS leaf keys, Postgres credentials, service identity PEMs, and the S3 credentials the installer currently writes into config files. Nothing secret sits in a plain file.
So the appliance operator runs OpenBao and puts all their sensitve data in it, yes?
Then we go onto say:
The local OpenBao's storage is sealed by a transit key held at a central OpenBao (
seal "transit"). At boot the appliance authenticates to central, unwraps its barrier key, and unseals. With central unreachable, or the seal credential revoked, OpenBao 2.6.1 refuses to start at all (verified behavior). This is the startup-kill lever from Principle 1: revoking one credential at central makes everything on the disk permanently unreadable.
Am I reading/understanding this right(?): If the central OpenBao is down, or revoked, a provider cannot read their provider wallet key, Postgres credentials, service identity PEMs, and S3 credentials?
Replace smelt's single-VM Docker Compose deployment for the five central services with Terraform-managed infrastructure: ECS/Fargate behind an ALB, one RDS Postgres instance with a database per service, and OpenBao as the secrets manager. Supports multiple stages through a `stage` variable. The substantial change from smelt is where secrets come from. smelt mints keys on an operator's laptop and stores them in 1Password; here a Go Lambda runs inside the VPC, writes private material straight to SSM Parameter Store, and returns only DIDs, wallet addresses and database names. No private key reaches Terraform state or a local disk. - Port smelt's keygen: Ed25519 identities, secp256k1 wallets with the three serializations each consumer expects, and the UCAN startup proofs, now issued in-process through ucantool's new ucandelegate API rather than by shelling out to its CLI. - Never regenerate an existing secret. Writes use Overwrite=false, so re-applying cannot abandon a funded wallet or change a registered DID. - Namespace parameters per service, so a task execution role reads only /forge/<stage>/<service>/* and a compromised task cannot reach another service's credentials. - Store OpenBao's data in Postgres and seal it with KMS. Fargate has no durable local disk, and KMS removes the unseal key, the 1Password item holding it, and the sidecar that applied it. - Give hilt an AppRole scoped to forge/hilt/data/tenant/* instead of the Vault root token, which smelt tracks as debt. - Expose OpenBao publicly, since fil-one/RFC#21 makes it the transit seal regional appliances authenticate against to unseal at boot. - Add a fund phase that signs the three FilecoinPay transactions inside AWS, so the payer key never leaves the account. Terraform never invokes it; `make fund-payer` shows a plan and requires confirmation. - Pin images by manifest digest rather than git SHA, which names the last commit instead of the code just built and collides against a dirty tree. Use ucantool's identity helpers rather than libforge's: libforge formats the signer into its error text with %s, and an ed25519 signer is a []byte with no String method, so a marshalling failure renders the raw private key into a logged error. Reported separately; sprue, hilt and swarf still depend on it. Signed-off-by: Miroslav Bajtoš <oss@bajtos.net> Assisted-by: Claude:claude-opus-5
|
@bajtos can I get a removal on the requested change, unless you still feel strongly about embedded secrets? |
|
|
||
| ## TL;DR | ||
|
|
||
| A region can defend exactly two security properties, and we should commit to both: **nothing on the appliance's disks is readable at rest, and a revoked appliance never comes back up.** Both are gated on discovery. Against an operator with sustained, undiscovered access to the running system, no design protects the data: the appliance decrypts objects to serve them, and the operator owns the hardware it runs on. |
There was a problem hiding this comment.
Both are gated on discovery
What does this mean?
undiscovered access to the running system
Huh?
|
|
||
| ### One home for secrets: a local OpenBao, rooted centrally | ||
|
|
||
| Each appliance runs OpenBao, listening on a unix socket, with raft storage on the appliance disk. It is the single home for regional secrets: the Region KEK (a non-exportable transit key), the provider wallet key, TLS leaf keys, Postgres credentials, service identity PEMs, and the S3 credentials the installer currently writes into config files. Nothing secret sits in a plain file. |
There was a problem hiding this comment.
I'm finding this hard to read and understand. I assume this means that there's essentially no change to Piri/Ingot for these credentials? We keep reading from file paths (or in the case of postgres DSN we add load from file support).
4 participants
📖 Preview
Summary
Regional security principles — the two properties a region can actually defend (nothing readable from a powered-off disk, and a revoked appliance never starts again) — and the key-management deployment that delivers them: a local OpenBao on each appliance, auto-unsealed from a central OpenBao, with the region wrap moving from A256KW to context-bound transit AES-256-GCM. Includes measured throughput down to a two-core floor, and amends the encryption RFC's keys table for the algorithm swap (tenant and Hilt wraps unchanged). Picks up the Region-KEK open question from #19.
Read the docs rendered:
🤖 Generated with Claude Code