[mzueri]

Problem

Nodes that share one home directory, e.g. the nodes of a multi-node Slurm job on a shared filesystem (Clariden), all load the same ~/.config/opentela/keys/id and announce the same peer ID. --seed doesn't help: a seed is only used when no key file exists yet. The gateway then answers 503 "No provider found" for the model.

They also all write their peer databases to ~/.ocfcore/ocfcore.<id>.db. A node only removes its own database on a clean shutdown, so nodes that get killed leave theirs behind. On Clariden, 52 of them (19 GB) filled the 50 GB home quota, and OpenTela then failed to start with disk quota exceeded.

The current workaround in model-launch (swiss-ai/model-launch#263) is to run OpenTela with HOME and --config-dir pointed at a per-job /tmp folder, and copy the user's cfg.yaml into it. It also still breaks when two OpenTela processes run on one node (sglang router + replica 0 head). The Euler deployment overrides HOME to /tmp for the same database reason.

Change

Two new otela start flags:

• --ephemeral-identity: the node uses an identity generated in memory, once per process. No key file is read or written. It overrides --seed. The instance link (deploy key) signs with the same in-memory key, so the linked peer is the one that serves. Using it without --datadir logs a warning.
• --datadir (or OF_DATADIR): the existing datadir setting (usage store) now also moves the peer database to <datadir>/ocfcore. Unset keeps ~/.ocfcore.

A multi-node job then only needs:

otela start --ephemeral-identity --datadir "$(mktemp -d)" ...

No HOME override and no copying of cfg.yaml; the user's config is read from its normal place.

Neither flag is bound with viper.BindPFlag: bound keys are written into the cfg.yaml seeded on first run, so a one-off --ephemeral-identity or temporary --datadir would stick to every later start. initConfig applies them per run, like --seed.

Identity selection moves out of newHost into loadOrCreateIdentity, so it can be unit-tested without starting a libp2p host. A short note in the spin-up tutorial explains when to use the flags.

For the maintainers to weigh

• An ephemeral node links a new peer to the account on every run, if a deploy key is used.
• Peer-ID allowlists can't cover ephemeral nodes. This is opt-in, and the default behaviour is unchanged.
• Anyone who already sets datadir in cfg.yaml gets their peer database moved there too. No config in this repo sets it, and the database is wiped at start by default (cleanslate).

Testing

• 9 new unit tests:
  • an existing key file is neither used nor overwritten, and no key file is written;
  • one identity per process, including concurrent callers, and a new identity per new process;
  • the default stays stable;
  • --datadir moves the database;
  • the instance link uses the served identity;
  • the flags are not saved into cfg.yaml. This test runs in a child test process, because other tests' viper.Reset would hide the bug.
• Mutation check: each fix was reverted in turn, and its test failed every time.
• End to end with the built binary:
  • three standalone nodes on one shared HOME get three distinct peer IDs;
  • nothing is written to the shared home, and nothing is saved into cfg.yaml;
  • databases are removed on SIGTERM;
  • a kill -9'd node leaves its database only in its own datadir;
  • a later plain otela start gets its persisted identity back.
• go test -race ./... passes (also with -shuffle=on -count=3 on the touched packages), make lint reports 0 issues, and the linux/amd64 and linux/arm64 builds succeed (Go 1.25).
• Not yet tested on Clariden itself; that will happen through model-launch once this is released.

[mzueri]

@xzyaoi @robmsmt @AryanAhadinia
(can not assign reviewer)