Repository navigation
Auto link opt in - #69
Merged
Merged
Conversation
Wallet-less nodes linked to OpenTela Cloud via a deploy key previously published no identity attestation, so the cloud's ownership checks and peers' trust computation saw an inconsistent record. Derive a stable Ed25519 key from the deploy key (domain-separated SHA-256 -> seed) and sign the identity attestation with it, so Owner == attested pubkey and the signature verifies without a Solana wallet. - wallet.DeriveIdentityKeyFromDeployKey: deterministic, path-independent. - wallet.SignIdentityWithKey: sign with a raw Ed25519 key. - protocol.deployKeyIdentity + InitializeMyself fallback. - tests for derivation stability and consistent record. Entire-Checkpoint: 01M4TC34XQTYJ5E62907C3BWWD
'otela init' pre-generates the libp2p identity key used by --config-dir and the mesh-bench runner, but had no way to control it: the key was always random, so a later 'start --seed N' just loaded that random key and the seed had no effect. Add --seed to 'otela init' and generate a reproducible, distinct PeerID from a non-zero seed. To make the seed actually reproducible, the seeded path now uses Ed25519: Go's RSA generation is deliberately non- deterministic from a seeded reader, so the old RSA-based --seed never produced a stable PeerID. RSA is retained for the seed=0 random path, and existing persisted keys are always loaded unchanged. - protocol.GenerateKeyFromSeed (Ed25519) and GenerateAndWriteKey(seed). - newHost reuses the shared keygen instead of duplicating it. - init_test: same seed => same key, different seed => different key. - docs: correct the false '--seed 0 is deterministic' claim. Entire-Checkpoint: 01M4TC3GYSW0FQSHZM0RJJE6S4
'otela start' used to re-link the node with any stored deploy key on every boot. Same-peer re-links are free, but each new peer ID spends a key use, so nodes that mint fresh identities (e.g. --ephemeral-identity on Slurm) drained the key on every restart and left phantom instances. Linking at boot now requires instance.auto_link (--auto-link / OF_INSTANCE_AUTO_LINK=true). An explicit 'otela start --deploy-key' still links that run. Without either, a stored key only logs a hint. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RijVobFYfzE6YvveLwKteU Entire-Checkpoint: 01M6E8V0Q98CKK4XRXFRJYYWCE
Contributor
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
… reservation (#67) * fix(protocol): worker connectivity watchdog retries bootstrap + relay reservation A worker is only routable from the head if it holds a bootstrap connection and a live relay reservation. Both were attempted once at startup (the reservation 10 s after the CRDT store starts); if that raced the network, the node stayed locally healthy but unreachable. Recovery relied on Reconnect() firing with 50% probability per minute, and the auto-reconnect loop only acts when there are zero connections. All failures logged at Debug, so nothing was visible. Add a watchdog (workers only) that, until both conditions hold, retries the bootstrap dial and relay reservation with 5s..60s jittered backoff, and logs state transitions at Info/Warn. Track reservation expiry so a lapsed reservation is re-made. Export IsBootstrapConnected/HasRelayReservation. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XjjN5e5xxxaFYKQcW51ytb * fix(protocol): make connectivity watchdog checks cheap and tie reservations to the advertised relay Review follow-ups for the worker connectivity watchdog: - Cache resolved bootstrap peers. Health checks are now in-memory reads; discovery (HTTP/DNS) runs only on refresh, reconnect, or when the cache is older than 10 minutes. An empty or failed resolution keeps the last known set, so a discovery outage no longer marks a connected node as disconnected. - Parse bootstrap multiaddrs one by one, skipping entries without a /p2p/<id> component instead of failing the whole batch. - Track reservations per relay peer and drop them when the last connection to that relay closes. "Relay reserved" now means the RelayPeer advertised in the CRDT is connected and holds an unexpired reservation, or AutoRelay advertises a circuit address through it. - Keep the advertised relay stable while it is still valid; otherwise prefer a freshly reserved bootstrap peer. - Guard MakeRelayReservations against overlapping runs from startup, Reconnect() and the watchdog. - Start the watchdog under the same condition as AutoRelay, and not in standalone mode. With no bootstrap peers resolved, log once and back off without making reservations. - Tests restore the package-global state they touch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RijVoeKQvu4rZ1PqgzEWKv Entire-Checkpoint: 01M6E5X1BDH1J2XKKHPEPX3D1G * fix(protocol): reserve on relay-capable peers in parallel; add docker NAT mesh test MakeRelayReservations tried every connected peer serially with a 10 s timeout each, so one watchdog step could block for minutes. Every node runs a relay service, so it also reserved on other NATed workers, which heads cannot route through (AutoRelay already covers worker-to-worker circuits). - Reserve only on bootstrap peers and peers the node table marks as relay/head or with a public address; fall back to all connected peers when none qualifies. - Run reservations in parallel (at most 8 in flight, 10 s timeout each). Successes are returned in candidate order, so pickAdvertisedRelay's preference is unchanged. Add TestRelayWatchdogMesh (integration tag): two isolated docker bridge networks simulate a public network and an HPC private network, so the head can reach workers only through the relay. It covers a baseline worker, a worker started while the relay is unreachable, and a relay restart while the workers stay connected to each other. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RijVoeKQvu4rZ1PqgzEWKv Entire-Checkpoint: 01M6E6QF8VP2TPTQ0EE3FE7J09 * test(integration): check the response body in the relay mesh routability probe The p2p forward error handler writes "ERROR: ..." with status 200, so a status check alone treated failed forwards as success. Require the worker's {"status":"ok"} health body. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RijVoeKQvu4rZ1PqgzEWKv Entire-Checkpoint: 01M6E7G9T32HJGVCJNDW5DEJPM * fix(server): report failed p2p forwards as 502/504 instead of 200 ErrorHandler wrote "ERROR: <reason>" without setting a status, so the response went out as 200. Clients and health checks took failed forwards for success. A relay that could not reach a worker also answered the head with 200, and the head passed that body to the client as a successful response. - ErrorHandler now responds 504 on deadline errors and 502 otherwise. The body keeps the "ERROR: <reason>" text. Responses carry X-Otela-Proxy-Error: 1 to mark them as generated by the proxy. - The service-forward retry path reuses ErrorHandler. - On a relay hop, the head treats a marked response from the relay as a transport failure and retries another worker. An unmarked 5xx from the worker's own service is still passed through. - The docker NAT mesh test asserts that a forward to an unreachable peer returns a marked 5xx. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RijVoeKQvu4rZ1PqgzEWKv Entire-Checkpoint: 01M6E7XGGN2KSSE0GY9YX8RBD7 * test(integration): exercise the relay v2 circuit path in the NAT mesh test The pub network used an RFC 1918 subnet. AutoRelay drops a relay's private addresses when building circuit addresses, so workers advertised no /p2p-circuit address. The head could never dial them directly, and the test only covered the HTTP relay hop, which does not depend on reservations. Code from before this PR passed the test. - Use 11.213.10.0/24, which libp2p treats as public, for the pub network. - Require both the relay hop and a direct circuit dial from the head in every scenario. With this change, code from before the PR fails the relay-restart scenario in 2 of 2 runs: the relay hop recovers, but the circuit dial fails for 90 s. This PR passes all scenarios in 3 of 3 runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RijVoeKQvu4rZ1PqgzEWKv Entire-Checkpoint: 01M6E99Z6NMQVKN8SQNBJWWMXN --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Co-authored-by: Xiaozhe Yao <askxzyao@gmail.com>
) * start: --ephemeral-identity and --datadir for nodes sharing a $HOME Nodes that share one home directory (e.g. the nodes of a multi-node Slurm job on a shared filesystem) all load the same ~/.config/opentela/keys/id and announce the same peer ID; --seed does not help, since a seed is only used when no key file exists yet. They also all write their peer databases to ~/.ocfcore, which fills the home quota when nodes are killed before they can clean up. - --ephemeral-identity: the node uses an in-memory identity generated once per process; no key file is read or written. Overrides --seed. The instance link (deploy key) signs with the same in-memory key, so the linked peer is the one that serves. Warns when used without --datadir. - --datadir (or OF_DATADIR): the existing datadir setting (usage store) now also moves the peer database to <datadir>/ocfcore. Unset keeps ~/.ocfcore. Neither flag is bound with viper.BindPFlag: bound keys are written into the cfg.yaml seeded on first run, so a one-off --ephemeral-identity or temporary --datadir would stick to every later start. initConfig applies them per run, like --seed. Identity selection moves out of newHost into loadOrCreateIdentity so it can be unit-tested without starting a libp2p host. * start: skip bootAccount for --component ingress An ingress-only process never brings up a libp2p host, so linking its identity (especially an ephemeral one) only spends deploy-key uses and leaves phantom instances in the console. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RijVobFYfzE6YvveLwKteU Entire-Checkpoint: 01M6E8BSW3HH9JZF5S51EM9B6V * ephemeral-identity: one identity resolver, env form, temp datadir Review follow-ups for --ephemeral-identity / --datadir: - protocol.NodeIdentity(seed) is now the single resolver for both the libp2p host and nodeLinkSigner, so linking and serving cannot drift (nodeLinkSigner previously re-implemented the selection and ignored the seed). ConfiguredSeed() shares the seed parsing. - One newKey helper for persisted, seeded and ephemeral keys. - OF_EPHEMERAL_IDENTITY enables the flag; the hyphenated viper key was unreachable via AutomaticEnv. Bound after the first-run cfg.yaml seed, so it never persists. - With --ephemeral-identity and no --datadir, node data goes to a per-process temp dir instead of piling up in the shared ~/.ocfcore. - datadir expands a leading ~ (common.DataDir), for both the peer database and the usage store. - `otela peer-id` warns on stderr when ephemeral identity is enabled. - Drop the dead connection-manager error branch in newHost. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RijVobFYfzE6YvveLwKteU Entire-Checkpoint: 01M6E8Q1ECER7DRR2Q2HTF3CZD * instance link: refuse seeded and ephemeral identities - nodeLinkSigner refuses a seeded identity: its key is derivable from a small integer, so linking it would let anyone who guesses the seed sign as an account-linked peer. Seeds stay for tests and benchmarks. - 'otela instance link' refuses when OF_EPHEMERAL_IDENTITY is set: the identity dies with the command, so the link would spend a deploy-key use on a peer that never comes online. Boot-time links by the serving node are unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RijVobFYfzE6YvveLwKteU Entire-Checkpoint: 01M6E8Z0MWVH1H60DHPYH383PN --------- Co-authored-by: Xiaozhe Yao <askxzyao@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Test Coverage Report 📊Click to view detailed coverageSummary: |
Resolve conflicts with #68 (ephemeral identity): nodeLinkSigner goes through protocol.NodeIdentity; seeded keys keep this branch's Ed25519 GenerateKeyFromSeed; GenerateAndWriteKey keeps its seed parameter. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RijVoa4Sf4BDt1BrxB1v4c Entire-Checkpoint: 01M6E9QKR0DMNQJH5VD77CQ6KR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.