Skip to content

omnigraph-server: runtime reload of the bearer-token source (SIGHUP or admin endpoint) #434

Description

@blarghmatey

Summary

omnigraph-server reads its bearer-token source exactly once, at process boot. There is no way to add or revoke an actor's token without restarting the server. We'd like a runtime reload — a SIGHUP handler, or an admin endpoint — for whichever token source is configured.

Current behaviour (0.8.1)

From docs/user/operations/server.md, "Auth model":

Tokens are SHA-256 hashed on startup; plaintext is never persisted in memory.

docs/user/deployment.md documents all three sources — AWS Secrets Manager (OMNIGRAPH_SERVER_BEARER_TOKENS_AWS_SECRET), a JSON file or inline env (OMNIGRAPH_SERVER_BEARER_TOKENS_FILE / _JSON), and the single legacy OMNIGRAPH_SERVER_BEARER_TOKEN — as read at startup, with no documented poll interval, cache TTL, or reload mechanism for any of them.

Why it matters to us

We run omnigraph-server as a shared, multi-tenant graph service on Kubernetes, with a {actor_id: token} map synced from Vault into a mounted Secret. Per-user tokens are provisioned from an identity-provider group, so the map changes whenever someone is onboarded or offboarded — this isn't a rare config edit, it's routine.

The awkward part is that our own MCP server reads the same physical file and re-stats it on a cache miss, so a newly-added user is live on their very next request on our side — while omnigraph-server keeps returning 401 for that same token until its process is bounced. One file, two readers, one of which needs a restart.

We work around it today by having the secrets operator rolling-restart the Deployment whenever the rendered token map changes, which does make "write the token" and "restart the server" a single operation. But the cost is real: our data tier is deliberately single-replica with a Recreate strategy — the storage format is strict-single-version, so two binaries must never hold the same object store concurrently — which turns every onboarding into a brief hard outage of the graph rather than a rolling one.

What we'd like

A way to tell a running server to re-read its configured token source, ideally mirroring the mtime/size-based reload we already do on our side:

  • SIGHUP — simplest to adopt; no new API surface, no auth story, works with a plain kill -HUP sidecar or an operator's post-sync hook.
  • An authenticated admin endpoint (e.g. POST /admin/reload-tokens) — more discoverable and scriptable, and would fit alongside the existing flat /healthz.

Either is sufficient for us; SIGHUP alone would fully solve it. Re-hashing the whole map on reload is fine — we're not asking for incremental registration, just "read the source again."

One smaller question

Is the AWS Secrets Manager source (--features aws) refreshed on any interval at runtime, or is it also strictly boot-time? The docs don't say either way. If that path does refresh, switching to it would sidestep this entirely for us without any new upstream API — and it'd be worth documenting explicitly, since the three sources currently read as equivalent on this point.

Happy to test a patch against our deployment if that's useful.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions