Repository navigation
fix(a2a_agent): bounded retry after a trajectory turn ends, and alertable read failures - #691
Draft
andrewpark-scaleai wants to merge 1 commit into
Draft
andrewpark-scaleai wants to merge 1 commit into
andrewpark-scaleai wants to merge 1 commit into
Conversation
…mes after the turn ends, and make read failures alertable
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
AgentEnv should have enough logs, metrics and alerts to debug every failure, and heal itself where code can. The live trajectory follower fails that bar in two ways.
stopset), a failed read gives up on the first failure. One transient 5xx or dropped connection loses the rest of the live trajectory. Before fix(a2a_agent): a trajectory follower ends with its step #618, the same branch retried every 0.2 s with no limit and no backoff, so a gone agent made the follower spin.consec=%dinside the message). A spinning follower cannot be alerted on, because nothing is searchable and the level never rises.What
stopis set, a failed read is retried up to 3 consecutive failures, sleeping 1 s then 2 s between them. A gone agent frees the step after about 3 s. A read that succeeds resets the count.event="live_trajectory_read_failed"withconsecandtask_idinextra. At 10 consecutive failures and above the level is ERROR instead of WARNING, so an alert can key on it.event="live_trajectory_follow_abandoned"withconsec.Design decision and tradeoff
The drain is bounded: 3 failures, about 3 s of waiting. A longer drain keeps more of the trajectory tail when the agent is briefly down, but holds the step's activity slot while it waits on an agent that is probably gone. A shorter drain, which is what main did (stop on the first failure), frees the slot at once but loses the tail on any blip.
Losing the tail is cheap here. The step marks the turn's end when its follower stops early, and the final trajectory still holds every event, so only the live view comes up short. Holding a slot is not cheap. The bound errs toward freeing the slot while giving one blip a chance to pass.
Test
25 passed. Against the old follower source, 3 of the new or changed tests fail:
test_an_agent_that_fails_after_the_turn_ends_is_retried_a_bounded_number_of_times(always-failing agent,stopset: ends within 4 reads)test_an_agent_that_recovers_after_the_turn_ends_still_gets_its_tail_readtest_a_failing_read_is_logged_with_its_count_and_escalates_to_error(checksevent,consec, level viacaplog)I ran these in a venv without
pytest-socket(no package index here), soconftest.pyimported a no-op stub of it. The full unit suite was not run.