feat(health): detect stalled newHeads subscriptions (WS-only health signal)

[merklefruit]

Summary

vixy can report an EL upstream as healthy while its newHeads subscription has silently stalled, so WebSocket relay clients are served a frozen head for minutes. Observed in production on hoodi: an mk1 sequencer (WS) and taiko-driver (HTTP) both froze at the same L1 block while vixy's own monitor saw fresh blocks (via eth_blockNumber polling) and reported every node healthy with zero errors.

Root cause: health came only from HTTP polling, and the relay blindly forwarded whatever the upstream subscription delivered — there was no detection of a stalled subscription.

What this PR adds

A per-EL-node newHeads liveness probe and a separate, WebSocket-only subscription_healthy signal.

• health::subscription probe — vixy maintains its own newHeads subscription to each upstream and records the time of the last head. Detection is time-since-last-head (subscription_stall_timeout_ms), independent of max_el_lag and the HTTP poll.
• WS-only gating — subscription_healthy is consulted ONLY by WS relay selection (select_el_ws_node). It is never folded into is_healthy, so a stalled subscription never diverts HTTP traffic or flips the failover flag.
• WS selection prefers a fresh+healthy primary → any fresh+healthy node (backups included, since a sub-stale primary cannot serve WS) → falls back to plain health selection (so WS is never worse off than HTTP).
• Robust probe — the stall timer keys on the last head (not any frame), so WS keepalive pings cannot mask a stall; capped exponential reconnect backoff (2s→30s) + connect timeout; a rejected subscription is logged clearly instead of looping silently.
• Observability — /status now exposes sub_block_number + subscription_healthy; the el_node_subscription_healthy metric is emitted only when the probe is actually tracking a node (not falsely 1 when disabled / never-connected).
• Config-validated (subscription_stall_timeout_ms > 0 when enabled); defaults on (set subscription_health_enabled = false to disable).

Behavior on the incident

Polled head 2930983 while the relayed subscription was frozen at 2930965 → the probe sees no head within the stall timeout → the node is marked subscription-stale → the WS relay fails over to a fresh node, HTTP routing untouched. Previously this was a ~4-minute silent frozen head.

Findings addressed (from review of the initial approach)

The key fix is decoupling subscription health from is_healthy so it cannot cause HTTP over-failover or flip the global failover flag. Also: detection switched to time-since-last-head (so the subscription_stall_timeout_ms knob actually drives detection and no longer reuses max_el_lag); the stall timer keys on heads not frames; sub_block_number stores the latest head (not a monotonic max, so reorgs/resyncs don't permanently disable detection); capped backoff + connect timeout; subscribe-error logging; honest metric / /status semantics; config validation.

Deliberately not changed (low value / higher risk — happy to do if preferred)

• Per-head el_nodes write-lock + name lookup in record_head — runs at block cadence (~1 per 12s) over a handful of nodes; negligible. Per-node atomics were rejected because ElNodeState must stay Clone for /status snapshots.
• Sharing WS connect/subscribe code with the relay — the relay's frame handling is intertwined with subscription-ID translation/replay; extracting a shared helper adds coupling risk for little gain (duplicated surface is small: connect_async + a subscribe send + a newHeads parse).

Testing

• cargo fmt --check and cargo clippy -- -D warnings — clean
• cargo test — 109 passed (incl. new tests for freshness, backoff, WS selection, head parsing, subscribe-error detection)
• cargo test --test cucumber — 16 scenarios / 83 steps
• cargo build --release — clean

🤖 Generated with Claude Code