fix(470): auto-recovery probe + force T3 rebuild on RPC peer reconnection - #104
Draft
hydra-z[bot] wants to merge 5 commits into
Draft
fix(470): auto-recovery probe + force T3 rebuild on RPC peer reconnection#104hydra-z[bot] wants to merge 5 commits into
hydra-z[bot] wants to merge 5 commits into
Conversation
When an RPC peer (e.g. RTX 3060) crashes and restarts, the 5060 Ti's graph tensor data pointers become stale because the peer's new process has empty g_hydra_server_buffers. This causes graph_compute to fail with 'cannot reach peer' and the engine degrades to solo mode (~3 tok/s instead of ~42 tok/s COMBINED). This fix adds auto re-provision detection and T3 rebuild triggering: 1. ggml-rpc.cpp: Add peer_reconnected atomic flag to device context. Set when ggml_backend_rpc_graph_compute detects last_sock changed (peer restart). Return GGML_STATUS_FAILED to signal the engine. 2. ggml-rpc.cpp: Add ggml_backend_rpc_check_peer_reconnection() function that reads and clears the reconnection flag per device. 3. llama-context.cpp: After graph_compute failure, check all RPC devices for reconnection. If detected, set peer_reconnection_pending flag on the context. 4. llama-context.h: Add public peer_reconnection_pending flag. 5. server-context.cpp: In update_slots(), when all slots are idle and peer_reconnection_pending is set, call apply_t3_rebuild(force=true) to re-provision model layers on the fresh peer. 6. server-context.cpp: Add force parameter to apply_t3_rebuild() to bypass the unchanged-config optimization when needed. The engine continues serving solo during re-provision and returns to COMBINED mode after the T3 rebuild completes.
When an RPC peer (e.g. RTX 3060) crashes and restarts, buffer functions (get_tensor, set_tensor, init_tensor, free_buffer, clear, get_base) would GGML_ABORT via RPC_STATUS_ASSERT because the peer's new process has empty g_hydra_server_buffers and can't resolve stale remote_ptrs. This fix converts all RPC_STATUS_ASSERT calls in buffer functions to fail-soft: instead of crashing, they set the peer_reconnected flag on the device context and return gracefully. The engine will detect the flag on the next graph_compute call and trigger T3 rebuild to re-provision model layers on the fresh peer. Changes: - Add rpc_set_reconnect_flag() helper to set peer_reconnected atomically - get_tensor: return without data on RPC failure (fail-soft) - set_tensor: skip write on RPC failure (fail-soft) - init_tensor: skip padding init on RPC failure (fail-soft) - free_buffer: skip free on RPC failure (fail-soft) - clear: skip clear on RPC failure (fail-soft) - get_base: return nullptr on RPC failure (fail-soft) This prevents the engine from crashing when a peer restarts, allowing the auto re-provision (T3 rebuild) to restore COMBINED mode.
The reconnection check in llama_context::graph_compute was accidentally removed during the RPC_STATUS_ASSERT fail-soft conversion. This meant the peer_reconnection_pending flag was never set, so the T3 rebuild in update_slots() was never triggered after a peer restart. Restore the check: after graph_compute failure, scan all RPC devices for reconnection. If detected, set peer_reconnection_pending = true so update_slots() triggers apply_t3_rebuild(force=true).
- Add ggml_backend_rpc_check_any_peer_reconnection() — probes all RPC backends for stale peer_reconnected flag - PREFILL handler probes RPC peers before hydra_apply_config — sets peer_reconnection_pending if any peer reconnected - apply_t3_rebuild() force=true when peer_reconnection_pending is set (both sync PREFILL and apply_pending_hydra_config paths) - Removes offload_kqv=false (user: KV on CPU kills speed) WIP: race condition remains — peer can die between PREFILL probe and graph_compute. Need decode-path retry-on-fail for full auto-recovery.
…xisted graph_compute compared last_sock.lock() != sock and, on mismatch, set peer_reconnected and returned GGML_STATUS_FAILED. But last_sock is an empty weak_ptr on a fresh device context (engine start, and every T3 rebuild which recreates the context), so lock() returns nullptr and the mismatch fired on the FIRST compute every time. That forced a T3 rebuild on every engine start and looped: each rebuild recreates the context (empty last_sock) so the next compute failed again and the engine could never serve - the exact 'first request after restart always fails' symptom PR #104 targets. Guard the FAILED/flag path with prev_sock != nullptr so a genuine reconnection (we held a live socket that now differs) is what triggers re-provision, while a first connect proceeds normally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Adds auto-recovery for COMBINED mode when an RPC peer (RTX 3060) restarts. Previously, a peer restart caused:
Changes
ggml/src/ggml-rpc/ggml-rpc.cppggml_backend_rpc_check_any_peer_reconnection()— probes all RPC backends for stalepeer_reconnectedflagggml/include/ggml-rpc.hggml_backend_rpc_check_any_peer_reconnection()tools/server/server-context.cpphydra_apply_config— setspeer_reconnection_pendingif any peer reconnected. Bothapply_t3_rebuild()call sites (sync PREFILL +apply_pending_hydra_config) passforce=truewhen reconnection pendingHow it works
ggml_backend_rpc_check_any_peer_reconnection()peer_reconnectedflag set (from a prior failed RPC call), the PREFILL handler setsctx_tgt->peer_reconnection_pending = trueapply_t3_rebuild(force=true)skips the "model/params unchanged" early-exit and forces a full model reloadKnown limitation
Race condition: if the peer dies AFTER the PREFILL probe but BEFORE
graph_compute, the first request still fails. Theupdate_slotsreconnection check catches it on the next request. Full fix requires decode-path retry-on-fail.Test plan