Repository navigation
fix(fusion): keep local server health per server - #1149
Draft
sosidudku1 wants to merge 1 commit into
Draft
sosidudku1 wants to merge 1 commit into
sosidudku1 wants to merge 1 commit into
Conversation
With mixLocalModels two local servers can serve one fan-out, but the health mark set after a cut-off worker was shared, and the only probe asked the main server. A wedge on one server could fail a worker sent to the other, and an answer from the main server cleared the other's mark. Marks are now filed under the leg's provider id, and a leg is asked only through its own server's probe. The runtime has a probe for the main local server only, so a second server's leg is not checked rather than checked against the wrong one. (ATO-1318)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem (ATO-1318). With
mixLocalModelson, role legs can run on a second local server. The health mark from ATO-917 was one flag for all local legs, and the only probe (llama.probeHealth()) asks the main server (localModels.url). So a wedge on one server could fail a worker sent to the other, and the main server's answer could clear the other server's mark.Fix.
LocalLegHealthkeeps one mark per leg, filed under the worker's provider id.noteCut,causeandreadyForDispatchtake the leg;isSuspect()with no leg still means "any".probeLocalHealthFor(providerId)returns the probe for that leg's own server, orundefined. The runtime returns the main client's probe only for allama-serverentry with no URL or the main URL. A leg with no probe is not checked, as the issue asks.probeLocalHealthFor(tests, embedders) the oldprobeLocalHealthis used as before.Default config (one local server) behaves the same as before.
Tests: two new cases in
local-leg-health.test.ts(marks kept apart, a cut filed under the worker's own server); one existing case now names its leg.This PR and #1139 (ATO-1317) both edit the
noteCut(...)call at the end ofrunOneTask. Whichever merges second must keepoptions.providerIdas the second argument and #1139's cause text, or every cut is filed under"local"and the health check silently stops; the testfiles a cut under the worker's own servercatches it.Merge note. #1139 also appends tests to the end of
local-leg-health.test.ts; keep both sides.Not yet checked on the second Mac.