Skip to content

deploy: make the OTP-signup broker + route survive redeploys#78

Open
Alexgodoroja wants to merge 7 commits into
mainfrom
deploy/otpsignup-broker-durable
Open

deploy: make the OTP-signup broker + route survive redeploys#78
Alexgodoroja wants to merge 7 commits into
mainfrom
deploy/otpsignup-broker-durable

Conversation

@Alexgodoroja

Copy link
Copy Markdown
Collaborator

The signup broker (cmd/otpsignup-broker, added in #77) and its public nginx route were configured by hand on the broker VM. A broker redeploy regenerates broker.conf, which wiped the route — signup requests then fell through to the managed-key broker ({"error":"unknown app: didit"}).

This folds both into the managed deploy so redeploys and fresh VMs recreate them:

  • startup.sh — builds cmd/otpsignup-broker and installs a systemd unit that runs only when broker-env metadata carries an OTPSIGNUP_* block (a broker VM that doesn't offer signup skips it). Every app/provider/host specific + secret comes from broker-env metadata; only a non-secret listen/DB default is in the unit.
  • setup-broker-tls.sh — injects operator-provided nginx location blocks (from broker-extra-locations metadata) before location /, so the signup route survives every regeneration. No app-specific route is baked into the repo.

Secrets and app/host specifics live in instance metadata, not the repo. Validated on the live VM: regenerating broker.conf from scratch re-injects the route and the endpoint answers from the signup broker. Merging this triggers a redeploy that stands the whole thing up cleanly.

Alex Godoroja added 5 commits July 7, 2026 16:39
Add the broker-signup pattern: obtain a per-user provider API key that a
provider gates behind an email OTP, with NO email input from the user and no
key held on our side after handoff.

- internal/otpmail: a receive-only mail server (catch-all SMTP for one domain
  + token-authed control API for provision/read-OTP/teardown). Keeps mail only
  for provisioned addresses, tmpfs maildir, never logs the code or body.
- internal/otpsignup: a signed HTTP broker that provisions a mailbox, drives the
  provider register→OTP→verify handshake, mints the key, and returns it. Reuses
  the shared broker's ed25519 caller-identity verify; idempotent per caller with
  an AES-GCM encrypted-at-rest ledger; per-IP Sybil cap.
- scaffold: a new signup `step: broker` (the adapter signs one keyless call to
  the broker and caches {email,api_key} to secrets.json; ops stay byo) plus a
  `step: account` local reader to retrieve the cached account. Emits the signer +
  a broker_signup.go runtime; grants key.sign + net.dial-broker.

All provider- and host-specifics are configuration; nothing is compiled in.
Tests cover SMTP delivery, OTP parse, control API, the full signed mint,
idempotency, encryption at rest, the per-IP cap, and generated-project compile.
didit.signup {} now mints the per-user Didit key via the Pilot broker in one
call (no email, no code); adds didit.account to retrieve the cached account;
drops the two-step register/verify. Help + long description updated.
…deploy

The signup broker (cmd/otpsignup-broker) and its public nginx route were set up
by hand, so a broker redeploy (which regenerates broker.conf) wiped the route.
Fold both into startup.sh + setup-broker-tls.sh so redeploys and fresh VMs
recreate them:

- Build cmd/otpsignup-broker; add a systemd unit that runs ONLY when broker-env
  metadata carries an OTPSIGNUP_* block (a broker VM without signup skips it).
  All app/provider/host specifics + secrets come from broker-env metadata; only
  a non-secret listen/DB default is in the unit.
- setup-broker-tls.sh injects operator-provided nginx location blocks (from
  broker-extra-locations metadata) before the default `location /`, so the
  signup route survives every config regeneration. No app-specific route is
  baked into the repo.

Validated on the live VM: regenerating broker.conf from scratch re-injects the
route and the endpoint answers from the signup broker.

@TeoSlayer TeoSlayer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security + quality review. Solid engineering with good test discipline, but one HIGH issue must be fixed before merge, plus a few MED items — and this branch is currently CONFLICTING (needs a rebase regardless).

HIGH — the anti-Sybil cap is trivially bypassable (X-Forwarded-For spoofing)

clientIP() in internal/otpsignup/broker.go trusts the first, client-supplied element of X-Forwarded-For, while the nginx vhost in deploy/setup-broker-tls.sh only appends the real address via $proxy_add_x_forwarded_for (never strips/replaces a forged header) and never sets X-Real-IP. Since a caller's "identity" is a free, self-generated ed25519 keypair (internal/broker/identity.go has no allowlist), anyone can spoof a fresh X-Forwarded-For per request and mint unlimited provider accounts — defeating MaxIdentitiesPerIP / MintCooldown, the only anti-farming control, and contradicting the guarantee stated in docs/BROKER-SIGNUP.md. TestPerIPCapBlocksSybil calls Broker.Signup() directly with a trusted ip, so the real HTTP-layer extraction has zero coverage.
Fix: derive the client IP from a trusted hop (set X-Real-IP in nginx and read that, or take the last XFF entry nginx appended), and add an HTTP-layer test that exercises clientIP().

MED

  • Signup race (orphaned account): check-then-act with no per-caller lock — two concurrent /signup for the same new caller both run the full register→OTP→verify. INSERT OR IGNORE drops the loser's row, but its HTTP response still returns an {email,api_key} that store.get will never return again → a permanently orphaned real provider account.
  • Partial signups aren't resumable: context-cancel after register() succeeds but before persist leaves an orphaned provider account; a retry mints a new one instead of resuming.
  • SMTP DoS: the public :25 listener in internal/otpmail/server.go has no concurrent-connection cap / per-IP throttle; readData() reads up to the 2-min deadline regardless of MaxMsgBytes (slow-loris).
  • Cleartext control-plane: broker→otpmail is plain HTTP with only a static bearer token; docs describe cross-host deploy, so the token + OTP can traverse the network in cleartext. Consider TLS/mTLS + token rotation.

LOW

  • OTPSIGNUP_ENC_KEY silently falls back to an ephemeral key — a restart of a misconfigured deploy irrecoverably orphans all sealed creds; fail hard (or warn loudly) instead.
  • otpsignup-data dir created without an explicit 0700.
  • nginx patch writes a fixed /tmp/broker.conf.new (use mktemp).
  • MintCooldown exists in Config but is never wired to an env var in cmd/otpsignup-broker/main.go, so the knob is unreachable.

What's good

No hardcoded secrets anywhere; parameterized queries (no SQLi); no shell injection in the deploy scripts (quoted heredocs/awk, no eval/curl|sh); TLS correctly enforced (https-only validation, no skip-verify); constant-time token compares; tightly-scoped manifest grants (key.sign target self, explicit deduped hosts, no wildcards, fs.write scoped to $APP/secrets.json); and strong test coverage otherwise (auth, idempotency, at-rest encryption vs raw ciphertext, signed-vs-unsigned, full SMTP round-trip, real go build of scaffolded code).

Please fix the XFF-derived client IP (+ test) and the signup race at minimum, and rebase — then this is close.

teovl and others added 2 commits July 24, 2026 15:20
…er-durable

# Conflicts:
#	deploy/setup-broker-tls.sh
#	deploy/startup.sh
#	docs/BROKER-SIGNUP.md
#	internal/publish/submission.go
#	internal/scaffold/config.go
#	internal/scaffold/signup_test.go
#	internal/scaffold/templates/broker_signup.go.tmpl
#	internal/scaffold/templates/main.go.tmpl
#	internal/scaffold/templates/signup.go.tmpl
Fixes from the completed security review:

- HIGH: X-Forwarded-For spoofing bypassed the per-IP anti-Sybil cap.
  clientIP() now trusts ONLY X-Real-IP (set by our own nginx from
  $remote_addr, never client-controllable) or the raw RemoteAddr — it
  never reads a client-supplied X-Forwarded-For. deploy/setup-broker-tls.sh
  already sets X-Real-IP on the shared vhost location (from prior main
  hardening); documented that any extra injected location must set it
  too, since nginx doesn't inherit proxy_set_header across locations.

- MED: signup race. Broker.Signup now serializes the check-then-act
  per caller id (a keyed sync.Map of per-caller mutexes), so concurrent
  signups for the same caller can't both pass the "no existing account"
  check and mint (and get billed for) a duplicate provider account.

- MED: SMTP DoS. The :25 listener now enforces a global concurrent-
  connection cap plus a per-remote-IP cap before handing a connection
  to the protocol handler, so a single source can't exhaust goroutines/
  fds. Configurable via OTPMAIL_MAX_CONNS[_PER_IP], sane defaults.

- LOW hardening:
  - OTPSIGNUP_ENC_KEY now fails hard for any persistent ledger instead
    of silently falling back to an ephemeral key (which used to make
    every existing row undecryptable — and silently re-minted — after
    every restart). A :memory:-only store still auto-generates one.
  - otpsignup-data dir created with explicit 0700 in deploy/startup.sh.
  - setup-broker-tls.sh's extra-locations injection now rewrites
    broker.conf in place (adopted from main's newer mechanism during
    the rebase) instead of a fixed-path /tmp scratch file.
  - MintCooldown wired to OTPSIGNUP_MINT_COOLDOWN (a Go duration) in
    cmd/otpsignup-broker/main.go.

- Control-plane mTLS (broker<->mail server) is a larger deployment/
  cert-management change and is intentionally deferred — documented in
  docs/BROKER-SIGNUP.md#security-notes. As a quick interim mitigation,
  both otpsignup.New and otpmail.New now enforce a minimum bearer-token
  length so a weak shared secret isn't the actual boundary.

Every fix has a dedicated test (HTTP-layer forged-XFF test, a
concurrency test proving exactly one provider registration, SMTP
admission-control unit + real-listener tests, enc-key fail-hard tests,
weak-token rejection tests). go build ./... and go test ./... (incl.
-race on the touched packages) are clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants