Repository navigation
Charon can hang indefinitely at startup (does not recover when BN comes up) #4689
Description
Description
When Charon starts before its configured beacon node REST API is available (common with split hosts / LXC / Proxmox boot order), Charon can print one or two startup log lines (e.g. lock file loaded) and then go silent. The process remains running (systemctl shows active), but it never becomes operational.
Importantly: once the beacon node later becomes healthy, Charon does not recover. Killing/restarting Charon after the BN is up fixes it immediately.
This is worse than “BN unreachable while Charon keeps retrying,” because the process stays wedged and systemd Restart=on-failure never fires.
Environment
Charon installed via systemd (EthPillar), not Docker Compose
Topology: Charon + VC on one host/LXC; beacon node on a separate host/LXC
BN client: not Nimbus (so not the json_requests case)
After host reboot, BN CT and Charon CT can start in parallel / Charon can win the race
Workaround that works: restart Charon (or reboot the Charon CT) after BN REST is answering
Reproduction
Configure Charon with --beacon-node-endpoints=http://:5052
Ensure BN REST is down / not listening yet
Start Charon
Observe: few log lines, then silence; process stays up
Bring BN REST up and confirm it answers (/eth/v1/node/version)
Observe: Charon still stuck; VC still cannot usefully talk to Charon :3600
systemctl restart charon → Charon starts normally
Expected behavior
Charon should either:
Retry BN-dependent startup (genesis/spec/etc.) with backoff until the BN is reachable, then continue wiring and start the validator API; or
Fail fast and exit so the process supervisor can restart it once the BN is up
It should not remain permanently stuck in a half-started state after the BN recovers.
Suspected cause (from reading current main)
After "Lock file loaded", startup still synchronously depends on the BN before life.Run starts the validator API:
sse.StartListener → FetchGenesisTime / FetchSlotsConfig
wireCoreWorkflow → more BN fetches (NewDutyDeadlineFunc, NewDutyGater, fork/slot config)
Only then is the validator API (:3600) registered/started
Charon constructs the eth2 client with WithAllowDelayedStart(true), but then immediately requires successful BN calls during startup. If that path hangs or gets stuck without exiting, the process looks “active” while unhealthy, and delayed start does not help once the BN appears later.
Relevant areas:
app/app.go — order after lock load: newETH2Client → sse.StartListener → wireCoreWorkflow → life.Run
app/sse/listener.go — StartListener blocks on BN genesis/spec
app/eth2wrap/eth2wrap.go — WithAllowDelayedStart(true) on eth2http.New
Impact
Split BN / Charon deployments (separate VMs/LXCs) are unreliable across host reboot
Operators see VC logs complaining that Charon’s BN API is unavailable
Manual restart is required even after the BN is healthy
Systemd restart policy cannot heal a non-exiting hang
Activity
KaloyanTanev commentedon Sep 8, 2026
Can you provide more details on your setup and full (ideally debug) log from Charon?
charon startup workaround
b0a7 commentedon Sep 8, 2026
my systemd file was:
[Unit]
Description=Obol Charon DVT middleware for MAINNET
After=network-online.target
Wants=network-online.target
Documentation=https://docs.coincashew.com
[Service]
Type=simple
User=charon
Group=charon
Restart=on-failure
RestartSec=3
KillSignal=SIGINT
TimeoutStopSec=120
LimitNOFILE=65536
WorkingDirectory=/var/lib/charon
ExecStart=/usr/local/bin/charon run
--beacon-node-endpoints=http://myRemoteBN:5052
--builder-api
--execution-client-rpc-endpoint=http://myRemoteBN:8545
--feature-set-enable=fetch_att_on_block
--lock-file=/var/lib/charon/.charon/cluster-lock.json
--monitoring-address=127.0.0.1:3620
--p2p-external-hostname=myHost
--p2p-tcp-address=0.0.0.0:3810
--private-key-file=/var/lib/charon/.charon/charon-enr-private-key
--validator-api-address=127.0.0.1:3600
[Install]
WantedBy=multi-user.target
b0a7 commentedon Sep 9, 2026
log only shows:
Sep 04 12:50:06 charon[396]: 21:45:06.103 INFO app-start Charon starting {"version": "v1.11.0", "git_commit_hash": "e9f7121", "git_commit_time": "2026-08-25T13:11:20Z"}
Sep 07 18:08:15 systemd[1]: Stopping charon.service - Obol Charon DVT middleware for MAINNET...
Sep 07 18:10:15 systemd[1]: charon.service: State 'stop-sigterm' timed out. Killing.
b0a7 commentedon Sep 9, 2026
some AI analysis:
journal evidence points earlier than BN (EL / pre-relay)
After Charon starting, still before any relay log / Lock file loaded / BN wiring:
eth1wrap.RunAndWait if --execution-client-rpc-endpoint is set
loadClusterLock (may use EL for ERC-1271)
… then wireP2P → p2p.NewRelays → logs Resolved new relay
only then Lock file loaded
only then BN (sse.StartListener, scheduler cache refresh, etc.)
This wedged boot never reached relay resolve, so it cannot be explained by BN-not-ready alone.
EL is configured on this node
--execution-client-rpc-endpoint is set. That puts eth1wrap.RunAndWait on the hot path immediately after Charon starting.
WaitConnected is documented to use a 10s timeout and then return an error (process should exit → supervisor restart). A multi-day wedged PID with only Charon starting does not match a clean 10s fail-and-exit.
Possible angles to investigate in app/eth1wrap:
Dial / web3_clientVersion not failing cleanly when EL is down or half-open (TCP accept / blackhole) while Charon is racing EL at boot
ClientVersion holds a mutex across CallContext, and maybeReconnect() can block on a full buffered channel while holding that lock → potential deadlock if reconnect signals pile up
Separately, p2p.NewRelays should either log Resolved new relay or return timeout resolving bootnode ENR within ~1 minute — also inconsistent with a days-long non-exiting hang unless we never got that far.
BN path still relevant as a second class of failure
Synchronous BN fetches after Lock file loaded (sse.StartListener / workflow genesis-spec) remain a plausible hang/fail-closed mode for other boots.
KaloyanTanev commentedon Sep 9, 2026
Looking at the logs (or rather lack of such :)), I don't think it's BN-related, as Charon doesn't even reach the stage of reading its lock file. In
--execution-client-rpc-endpoint=http://myRemoteBN:8545/you've stated your remote BN, but this should be an URL for the EL client not the CL client. If you are not using ERC-1271 signatures for the lock file, but rather a default setup, you can omit overall the EL flag.b0a7 commentedon Sep 9, 2026
b0a7 commentedon Sep 15, 2026
should be fixed now in #4694