[mzueri]

Problem

OpenTela loads its identity key from ~/.config/opentela/keys/id and keeps its peer database at ~/.ocfcore/ocfcore.<peer id>.db. On Clariden $HOME is one shared filesystem, so every node of a multi-node replica announces the same peer ID and writes the same database. The nodes race to create the file: the loser gets file exists / stale file handle, and OpenTela crashes (a nil-pointer panic in the Badger datastore) or the gateway answers 503 "No provider found". It is intermittent, so some launches get through.

Change

Both $OPENTELA_BIN start invocations (_opentela_wrap, _opentela_wrap_metrics_only) now pass the flags eth-easl/OpenTela#68 added (release v0.2.7):

• --ephemeral-identity: an in-memory peer identity per process, no key file read or written.
• --datadir "$OPENTELA_DATADIR": the peer database (and usage store) go to a node-local directory made with mktemp -d /tmp/opentela-${SLURM_JOB_ID}.XXXXXX instead of the shared ~/.ocfcore. One directory per OpenTela process, so two processes on a node (an sglang router and the replica's head) do not share a database. The rank script removes it on exit.

This replaces the earlier version of this PR (a per-job HOME, --config-dir and a cfg.yaml copy): no HOME override and no config copy are needed, and the user's cfg.yaml is read from its normal place.

Requirement

An OpenTela binary with --ephemeral-identity and --datadir (v0.2.7). The prod binary in opentela-share on Clariden has both flags (checked with otela start --help). I did not check the dev channel.

Side effects

• A fresh peer ID per node per job instead of a stable one per user. Please flag if anything (dashboards, allowlists) relies on a stable ID.

Tested

• Unit tests: tests/unit passes (462 tests); ruff check and ruff format are clean. New tests check that every rendered head and follower script has the flags and no --config-dir or HOME override, and run the wrapper against a fake otela to check the datadir is per process, outside $HOME, and removed on exit.
• Live: a 2-node vLLM replica on Clariden (highprio) with this branch's rendering. No panic or file exists on either node; the two nodes announced different peer IDs; the head registered has_service: true, status: ready and the follower has_service: false (metrics-only, as before); a chat request through the serving API returned a reply; no new directories appeared in the shared ~/.ocfcore.
• Not checked on a live node: that the per-process directory is removed on exit (covered by the unit test only), and the two-OpenTela-processes-per-node (sglang router) case.

[AryanAhadinia]

I didn't got the idea of the PR. I think the peer ID you are pointing to is the bootstrap peer, which indeed is fixed across the system.

Could you please elaborate a bit more?

Maybe @xzyaoi can give us an input here as well.

[mzueri]

I didn't got the idea of the PR. I think the peer ID you are pointing to is the bootstrap peer, which indeed is fixed across the system.

Could you please elaborate a bit more?

Maybe @xzyaoi can give us an input here as well.

I wanted to evaluate chonk/megachonk which requires us to use multiple nodes for model launch now. This caused issues because every node ends up with the same OpenTela identity and this raises an error.

This PR attempts to give each node its own identity. Or is this feature already supported and I missed it?

Probably there is also a cleaner way of doing it upstream... I am currently using this PR branch for evals and it works now.

[xzyaoi]

I didn't got the idea of the PR. I think the peer ID you are pointing to is the bootstrap peer, which indeed is fixed across the system.
Could you please elaborate a bit more?
Maybe @xzyaoi can give us an input here as well.

I wanted to evaluate chonk/megachonk which requires us to use multiple nodes for model launch now. This caused issues because every node ends up with the same OpenTela identity and this raises an error.

This PR attempts to give each node its own identity. Or is this feature already supported and I missed it?

Probably there is also a cleaner way of doing it upstream... I am currently using this PR branch for evals and it works now.

I think if you have multiple slurm jobs, each one of them should get a different $HOME, if you do not mount the same? I recall I have a setup of multiple concurrent slurm job and each use a different $HOME.

But yes, if they conflict, it is an issue. IIRC you can also provide a --seed so they will generate different ids.

[mzueri]

I didn't got the idea of the PR. I think the peer ID you are pointing to is the bootstrap peer, which indeed is fixed across the system.
Could you please elaborate a bit more?
Maybe @xzyaoi can give us an input here as well.

I wanted to evaluate chonk/megachonk which requires us to use multiple nodes for model launch now. This caused issues because every node ends up with the same OpenTela identity and this raises an error.
This PR attempts to give each node its own identity. Or is this feature already supported and I missed it?
Probably there is also a cleaner way of doing it upstream... I am currently using this PR branch for evals and it works now.

I think if you have multiple slurm jobs, each one of them should get a different $HOME, if you do not mount the same? I recall I have a setup of multiple concurrent slurm job and each use a different $HOME.

But yes, if they conflict, it is an issue.

The issue for me arises already when submitting 1 slurm job using multiple nodes. Then $HOME is the same for all nodes. Opentela looks in the shared $HOME directory and the ID's for all nodes become the same.

So you suggest using a container which redirects $HOME ? This PR redirects to /tmp

[xzyaoi]

I didn't got the idea of the PR. I think the peer ID you are pointing to is the bootstrap peer, which indeed is fixed across the system.
Could you please elaborate a bit more?
Maybe @xzyaoi can give us an input here as well.

I wanted to evaluate chonk/megachonk which requires us to use multiple nodes for model launch now. This caused issues because every node ends up with the same OpenTela identity and this raises an error.
This PR attempts to give each node its own identity. Or is this feature already supported and I missed it?
Probably there is also a cleaner way of doing it upstream... I am currently using this PR branch for evals and it works now.

I think if you have multiple slurm jobs, each one of them should get a different $HOME, if you do not mount the same? I recall I have a setup of multiple concurrent slurm job and each use a different $HOME.
But yes, if they conflict, it is an issue.

The issue for me arises already when submitting 1 slurm job using multiple nodes. Then $HOME is the same for all nodes. Opentela looks in the shared $HOME directory and the ID's for all nodes become the same.

So you suggest using a container which redirects $HOME ? This PR redirects to /tmp

Yes, I think this is the right.

Withdrawing approval pending review feedback (router/head identity sharing, config-dir seeding).

[robmsmt]

Thanks for the thorough diagnosis. The root cause is right, and the fix works for the multi-node replica case. I've withdrawn my earlier approval until the points below are settled.

1. Router and replica-0 head may still share an identity
_render_router_launch runs router.sh with srun --nodelist=${nodes[0]} --overlap, and replica 0's head also runs on nodes[0]. Both go through _opentela_wrap*, so both resolve to /tmp/opentela-${SLURM_JOB_ID}. If those two srun steps share /tmp, the router (service llm) and head 0 (metrics-only) load the same key and announce the same peer ID, which is the failure this PR is meant to remove.

• Suggest putting a role in the dir name (-router / -head / -follower), or using ${SLURM_STEP_ID}.
• Please include a router-fronted sglang launch with replicas>1 in the live testing, and check for distinct peer IDs on node 0.
• Tests: the router script isn't covered by the flag-count test, and nothing asserts that router and head dirs differ.

2. Code shape
/tmp/opentela-${SLURM_JOB_ID} is hardcoded in the seed, the arg, and the test. Defining it once as a shell variable (e.g. otela_cfg_dir=...) and building the seed and --config-dir arg from one helper would avoid drift. A failed cp is also silent, so a log line saying which cfg was used would help debugging.

3. Upstream follow-up
--config-dir moves both the identity key and cfg.yaml, which is why this PR has to copy the user's config around. It would be cleaner if OpenTela let the key location be set separately (e.g. an --identity-dir or similar flag). Then model-launch wouldn't need the seeding snippet, and users' configs couldn't be silently dropped. Fine to merge this as a stopgap, but worth opening an issue on the OpenTela side.

Smaller notes

• SLURM_JOB_ID is stable across requeue, so a requeued job on the same node reuses its /tmp identity. That is probably fine, but it contradicts the "never reuses a dead peer's identity" wording.
• Nothing enforces the prod >= v0.0.7 requirement, so an old binary fails with an unknown-flag error.
• Someone should check whether anything relies on a stable peer ID (dashboards, allowlists).

[AryanAhadinia]

Automated review

This repository has no .github/review-criteria.yml on main, so there are no review criteria to check this PR against.

[mzueri]

@xzyaoi @robmsmt @AryanAhadinia I made a PR into upstream opentela repo to make a cleaner fix (per-node identity):
eth-easl/OpenTela#68
Could someone please review (can't add a reviewer)? Thanks!

I smoke-tested it on Clariden (2 nodes, shared $HOME, deployed dev build + the change): distinct peer IDs, routing between the nodes works, nothing written to $HOME, DBs removed on exit.

[robmsmt]

Mel we can close this now because we merged the changes into opentela I believe?

[mzueri]

Mel we can close this now because we merged the changes into opentela I believe?

Yes, resolved upstream:
eth-easl/OpenTela#68
Thank you @xzyaoi !

[mzueri]

@robmsmt sorry, only upstream fix is done, this is still relevant. Could you pls have a look?

[sonarqubecloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud