[serve] Env-gated observe layer4 mark-down on HAProxy servers

[harshit-anyscale]

Summary

When a node is drained (spot reclaim, compaction, scale-down), its displaced (soft-stopped) HAProxy processes keep serving established connections with a frozen config in which torn-down replicas remain nominally UP. Requests those processes accept are black-holed until hard-stop-after instead of failing over. This PR adds an env-gated observe layer4 error-limit <N> on-error mark-down to backend replica servers so live traffic itself marks dead servers DOWN and the existing backup fallback takes over. Default off; the rendered config is byte-identical when unset.

Chain of events

1. A node starts draining (2026-08-04 incident: spot reclaim; 2026-07-23: replica compaction).
2. The controller migrates the node's replicas; replacements are healthy elsewhere in ~2s.
3. Updated backend configs are pushed to every proxy.
4. Each HAProxy applies the new config by reloading: a new process starts, and the old one is soft-stopped (-sf) but keeps serving its established connections.
5. The old process's view is frozen: the stopped replicas stay UP in its map, and the replacements don't exist in it.
6. A request arrives on a connection held by the old process; the frontend accepts it.
7. Dispatch picks a stale server; the connect fails — the replica is gone.
8. Retries and redispatch only consider servers believed UP: the same dead ones. nbsrv > 0, so the backup fallback stays ineligible.
9. No error is ever returned; the client burns its full timeout (150s in our fleet).
10. Steps 6-9 repeat for every request landing on that process until hard-stop-after (400s) kills it: queued requests die as 502s (last client failures at drain+401s, to the second), and traffic then flows only to healthy processes.

Impact: 8 failed client requests over ~6.5 minutes, exactly on the apps whose replicas all sat on the drained node (echo 1/1 replica, highscale 2/2).

Fix

RAY_SERVE_HAPROXY_OBSERVE_MARK_DOWN_ENABLED (default off) renders observe layer4 error-limit <N> on-error mark-down (RAY_SERVE_HAPROXY_OBSERVE_ERROR_LIMIT, default 3) on HTTP and gRPC replica server lines. Connection-level failures observed by live traffic mark the server DOWN synchronously in the request path — verified in HAProxy source: __health_adjust() runs inline with no stopping guard, and soft-stop does not destroy check tasks, so this works inside displaced processes, the one place nothing else can. With all primaries DOWN, option redispatch + the existing backup fallback serve the request. Recovery is automatic: DOWN servers keep being health-probed and rise passes restore them in ~0.5s, so a false positive self-heals. Backup/fallback servers are never observed; router-path servers inherit state via track.

[gemini-code-assist]

Code Review

This pull request introduces configuration options to enable HAProxy's layer 4 error observation, allowing servers to be marked DOWN after consecutive connection failures without waiting for the health checker. The feedback suggests utilizing existing helper functions ('get_env_int_positive' and 'get_env_bool') to parse the new environment variables safely and robustly, which prevents potential startup crashes from invalid integer inputs and ensures consistent boolean parsing.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 168e001. Configure here.}