[robmsmt]

Problem

A worker (no public-addr) is routable from the head only if it holds a bootstrap connection and a live relay reservation. Both are attempted once at startup (reservation: 10 s after the CRDT store starts, only over peers connected at that moment). If that races the network, the worker stays locally healthy (its framework /health is 200, /v1/self answers) but the gateway cannot route to it.

Recovery was unreliable:

• startAutoReconnect only acts when there are zero connections, and doesn't make reservations.
• Reconnect() (which re-reserves) fires with 50% probability per minute.
• Every failure on those paths logs at Debug, and the startup banner bootstrap_connected is a one-time snapshot, so nothing is visible.

Observed on Clariden (SwissAI): 5 of the last 10 SLURM jobs started with bootstrap_connected=false, across the same nodes that connected fine in other jobs. One 2-node job ran >1 h locally healthy but never became routable; the model disappeared from the serving API.

Change

• New connectivity.go: a watchdog (workers only) that, until bootstrap is connected and a relay reservation is held, retries the bootstrap dial and MakeRelayReservations with 5 s..60 s jittered backoff. Once healthy it re-checks every 30 s.
• Tracks reservation expiry, so a lapsed reservation is re-made.
• State transitions logged at Info/Warn (previously invisible).
• Exports IsBootstrapConnected() / HasRelayReservation() so callers (e.g. a health checker) can tell "locally up" from "routable". Not wired to any endpoint here; happy to add one if you'd like.

Testing

• connectivity_test.go: unit tests for the watchdog step logic (side-effect injection) and reservation expiry.
• go test ./internal/protocol/ passes (with -race).
• Not reproduced end-to-end: the original failure can't be forced, and the failing step isn't logged. The diagnosis is from reading the code paths above.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XjjN5e5xxxaFYKQcW51ytb

[xzyaoi]

Thanks for this, the diagnosis is spot on. I reviewed it and pushed a follow-up commit (07b8361) to this branch addressing the main issues. Summary:

Issues found in the original watchdog

1. bootstrapConnected re-resolved the bootstrap list on every check. With the default bootstrap.static, that is three HTTP fetches (5 s timeout each) every 30 s per worker, plus another resolution in tryReconnectToBootstraps.
2. If discovery was down or returned nothing, the check reported bootstrap_connected=false even with live connections to the heads.
3. peer.AddrInfosFromP2pAddrs fails the whole batch on one address without /p2p/<id>, so a single bad entry made the check fail forever.
4. Reservation state was a single global max-expiry that was never cleared on disconnect, so it could report "reserved" for up to an hour after the relay restarted.
5. A reservation on any peer counted, even one that is not the RelayPeer advertised in the CRDT. Heads could still fail to route, which is the case this PR targets.
6. Several MakeRelayReservations runs could overlap (startup goroutine, Reconnect()'s delayed call, the watchdog) and race on SetMyselfRelayPeer.
7. The watchdog also started in standalone mode and for reachability=public nodes, where it can never succeed.

What the follow-up commit changes

• Bootstrap peers are cached. Health checks are in-memory reads. Discovery runs only on startup, on reconnect, or when the cache is older than 10 min. An empty or failed resolution keeps the last known set.
• Bootstrap multiaddrs are parsed one by one, and bad entries are skipped.
• Reservations are tracked per relay and dropped when the last connection to that relay closes (DisconnectedF).
• "Relay reserved" now means: the advertised RelayPeer is connected and has an unexpired manual reservation, or AutoRelay advertises a circuit address through it (via EvtAutoRelayAddrsUpdated).
• MakeRelayReservations keeps the current RelayPeer while it is still valid. Otherwise it prefers a freshly reserved bootstrap peer. This avoids churn from peer iteration order.
• MakeRelayReservations has a single-flight guard.
• The watchdog starts under the same condition as AutoRelay and not in standalone mode. With no bootstrap peers resolved, it logs once and backs off without reserving.
• Tests restore the package-global state they touch. There are new tests for bad bootstrap entries, cache retention, per-relay validity, disconnect, the AutoRelay circuit path, and advertised-relay binding.

go test -race ./internal/protocol/ and go vet pass locally. I could not run golangci-lint locally because of a toolchain mismatch, so CI lint still needs to confirm. This has not been tested end to end.

Possible follow-ups, not in this PR

• MakeRelayReservations still tries every connected peer serially with a 10 s timeout each. Limiting it to bootstrap/relay-role peers and running in parallel would bound each step.
• IsBootstrapConnected / HasRelayReservation still have no callers. Exposing a routable field in /v1/self would give exactly the signal that was missing on Clariden.