deploy: make the OTP-signup broker + route survive redeploys#78
deploy: make the OTP-signup broker + route survive redeploys#78Alexgodoroja wants to merge 7 commits into
Conversation
Add the broker-signup pattern: obtain a per-user provider API key that a
provider gates behind an email OTP, with NO email input from the user and no
key held on our side after handoff.
- internal/otpmail: a receive-only mail server (catch-all SMTP for one domain
+ token-authed control API for provision/read-OTP/teardown). Keeps mail only
for provisioned addresses, tmpfs maildir, never logs the code or body.
- internal/otpsignup: a signed HTTP broker that provisions a mailbox, drives the
provider register→OTP→verify handshake, mints the key, and returns it. Reuses
the shared broker's ed25519 caller-identity verify; idempotent per caller with
an AES-GCM encrypted-at-rest ledger; per-IP Sybil cap.
- scaffold: a new signup `step: broker` (the adapter signs one keyless call to
the broker and caches {email,api_key} to secrets.json; ops stay byo) plus a
`step: account` local reader to retrieve the cached account. Emits the signer +
a broker_signup.go runtime; grants key.sign + net.dial-broker.
All provider- and host-specifics are configuration; nothing is compiled in.
Tests cover SMTP delivery, OTP parse, control API, the full signed mint,
idempotency, encryption at rest, the per-IP cap, and generated-project compile.
didit.signup {} now mints the per-user Didit key via the Pilot broker in one
call (no email, no code); adds didit.account to retrieve the cached account;
drops the two-step register/verify. Help + long description updated.
…deploy The signup broker (cmd/otpsignup-broker) and its public nginx route were set up by hand, so a broker redeploy (which regenerates broker.conf) wiped the route. Fold both into startup.sh + setup-broker-tls.sh so redeploys and fresh VMs recreate them: - Build cmd/otpsignup-broker; add a systemd unit that runs ONLY when broker-env metadata carries an OTPSIGNUP_* block (a broker VM without signup skips it). All app/provider/host specifics + secrets come from broker-env metadata; only a non-secret listen/DB default is in the unit. - setup-broker-tls.sh injects operator-provided nginx location blocks (from broker-extra-locations metadata) before the default `location /`, so the signup route survives every config regeneration. No app-specific route is baked into the repo. Validated on the live VM: regenerating broker.conf from scratch re-injects the route and the endpoint answers from the signup broker.
TeoSlayer
left a comment
There was a problem hiding this comment.
Security + quality review. Solid engineering with good test discipline, but one HIGH issue must be fixed before merge, plus a few MED items — and this branch is currently CONFLICTING (needs a rebase regardless).
HIGH — the anti-Sybil cap is trivially bypassable (X-Forwarded-For spoofing)
clientIP() in internal/otpsignup/broker.go trusts the first, client-supplied element of X-Forwarded-For, while the nginx vhost in deploy/setup-broker-tls.sh only appends the real address via $proxy_add_x_forwarded_for (never strips/replaces a forged header) and never sets X-Real-IP. Since a caller's "identity" is a free, self-generated ed25519 keypair (internal/broker/identity.go has no allowlist), anyone can spoof a fresh X-Forwarded-For per request and mint unlimited provider accounts — defeating MaxIdentitiesPerIP / MintCooldown, the only anti-farming control, and contradicting the guarantee stated in docs/BROKER-SIGNUP.md. TestPerIPCapBlocksSybil calls Broker.Signup() directly with a trusted ip, so the real HTTP-layer extraction has zero coverage.
Fix: derive the client IP from a trusted hop (set X-Real-IP in nginx and read that, or take the last XFF entry nginx appended), and add an HTTP-layer test that exercises clientIP().
MED
- Signup race (orphaned account): check-then-act with no per-caller lock — two concurrent
/signupfor the same new caller both run the full register→OTP→verify.INSERT OR IGNOREdrops the loser's row, but its HTTP response still returns an{email,api_key}thatstore.getwill never return again → a permanently orphaned real provider account. - Partial signups aren't resumable: context-cancel after
register()succeeds but before persist leaves an orphaned provider account; a retry mints a new one instead of resuming. - SMTP DoS: the public
:25listener ininternal/otpmail/server.gohas no concurrent-connection cap / per-IP throttle;readData()reads up to the 2-min deadline regardless ofMaxMsgBytes(slow-loris). - Cleartext control-plane: broker→otpmail is plain HTTP with only a static bearer token; docs describe cross-host deploy, so the token + OTP can traverse the network in cleartext. Consider TLS/mTLS + token rotation.
LOW
OTPSIGNUP_ENC_KEYsilently falls back to an ephemeral key — a restart of a misconfigured deploy irrecoverably orphans all sealed creds; fail hard (or warn loudly) instead.otpsignup-datadir created without an explicit0700.- nginx patch writes a fixed
/tmp/broker.conf.new(usemktemp). MintCooldownexists inConfigbut is never wired to an env var incmd/otpsignup-broker/main.go, so the knob is unreachable.
What's good
No hardcoded secrets anywhere; parameterized queries (no SQLi); no shell injection in the deploy scripts (quoted heredocs/awk, no eval/curl|sh); TLS correctly enforced (https-only validation, no skip-verify); constant-time token compares; tightly-scoped manifest grants (key.sign target self, explicit deduped hosts, no wildcards, fs.write scoped to $APP/secrets.json); and strong test coverage otherwise (auth, idempotency, at-rest encryption vs raw ciphertext, signed-vs-unsigned, full SMTP round-trip, real go build of scaffolded code).
Please fix the XFF-derived client IP (+ test) and the signup race at minimum, and rebase — then this is close.
…er-durable # Conflicts: # deploy/setup-broker-tls.sh # deploy/startup.sh # docs/BROKER-SIGNUP.md # internal/publish/submission.go # internal/scaffold/config.go # internal/scaffold/signup_test.go # internal/scaffold/templates/broker_signup.go.tmpl # internal/scaffold/templates/main.go.tmpl # internal/scaffold/templates/signup.go.tmpl
Fixes from the completed security review:
- HIGH: X-Forwarded-For spoofing bypassed the per-IP anti-Sybil cap.
clientIP() now trusts ONLY X-Real-IP (set by our own nginx from
$remote_addr, never client-controllable) or the raw RemoteAddr — it
never reads a client-supplied X-Forwarded-For. deploy/setup-broker-tls.sh
already sets X-Real-IP on the shared vhost location (from prior main
hardening); documented that any extra injected location must set it
too, since nginx doesn't inherit proxy_set_header across locations.
- MED: signup race. Broker.Signup now serializes the check-then-act
per caller id (a keyed sync.Map of per-caller mutexes), so concurrent
signups for the same caller can't both pass the "no existing account"
check and mint (and get billed for) a duplicate provider account.
- MED: SMTP DoS. The :25 listener now enforces a global concurrent-
connection cap plus a per-remote-IP cap before handing a connection
to the protocol handler, so a single source can't exhaust goroutines/
fds. Configurable via OTPMAIL_MAX_CONNS[_PER_IP], sane defaults.
- LOW hardening:
- OTPSIGNUP_ENC_KEY now fails hard for any persistent ledger instead
of silently falling back to an ephemeral key (which used to make
every existing row undecryptable — and silently re-minted — after
every restart). A :memory:-only store still auto-generates one.
- otpsignup-data dir created with explicit 0700 in deploy/startup.sh.
- setup-broker-tls.sh's extra-locations injection now rewrites
broker.conf in place (adopted from main's newer mechanism during
the rebase) instead of a fixed-path /tmp scratch file.
- MintCooldown wired to OTPSIGNUP_MINT_COOLDOWN (a Go duration) in
cmd/otpsignup-broker/main.go.
- Control-plane mTLS (broker<->mail server) is a larger deployment/
cert-management change and is intentionally deferred — documented in
docs/BROKER-SIGNUP.md#security-notes. As a quick interim mitigation,
both otpsignup.New and otpmail.New now enforce a minimum bearer-token
length so a weak shared secret isn't the actual boundary.
Every fix has a dedicated test (HTTP-layer forged-XFF test, a
concurrency test proving exactly one provider registration, SMTP
admission-control unit + real-listener tests, enc-key fail-hard tests,
weak-token rejection tests). go build ./... and go test ./... (incl.
-race on the touched packages) are clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The signup broker (
cmd/otpsignup-broker, added in #77) and its public nginx route were configured by hand on the broker VM. A broker redeploy regeneratesbroker.conf, which wiped the route — signup requests then fell through to the managed-key broker ({"error":"unknown app: didit"}).This folds both into the managed deploy so redeploys and fresh VMs recreate them:
startup.sh— buildscmd/otpsignup-brokerand installs a systemd unit that runs only whenbroker-envmetadata carries anOTPSIGNUP_*block (a broker VM that doesn't offer signup skips it). Every app/provider/host specific + secret comes frombroker-envmetadata; only a non-secret listen/DB default is in the unit.setup-broker-tls.sh— injects operator-provided nginx location blocks (frombroker-extra-locationsmetadata) beforelocation /, so the signup route survives every regeneration. No app-specific route is baked into the repo.Secrets and app/host specifics live in instance metadata, not the repo. Validated on the live VM: regenerating
broker.conffrom scratch re-injects the route and the endpoint answers from the signup broker. Merging this triggers a redeploy that stands the whole thing up cleanly.