Repository navigation
fix: retry ServiceFoundry GET calls after transient network failures - #1193
ramantehlan wants to merge 2 commits into
Conversation
A dropped connection or short gateway error on a read failed the request on the first attempt. Writes stay a single attempt. Signed-off-by: Raman Tehlan <ramantehlan@gmail.com>
🦋 Changeset detectedLatest commit: 16f6aac The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
The GET retry stays. The final request metric already records the outcome, so a separate retry counter and its tests are unnecessary. Signed-off-by: Raman Tehlan <ramantehlan@gmail.com>
| const AGENT_SKILLS_PAGE_SIZE = 100; | ||
| /** SFY resolve `@ArrayMaxSize(50)` — chunk larger AgentSpec skill lists. */ | ||
| const AGENT_SKILL_RESOLVE_CHUNK_SIZE = 50; | ||
| const RETRY_INITIAL_DELAY_MS = 1_000; |
There was a problem hiding this comment.
Is it possible to reuse the retry logic from packages/trueforge-core/src/core/util/ssrfGuard.ts
?
There was a problem hiding this comment.
The loop in withOutboundHttpRetries is private, and every attempt goes through guardedFetch, so this client cannot call it.
guardedFetchchecks the SSRF host list. In-cluster ServiceFoundry hosts end in.svc,.cluster, or.internal, and those suffixes are denied.- It replaces the request dispatcher with its own undici Agent, so the mutual-TLS dispatcher this client needs would be dropped.
- Retries there are recorded on
tfg_outbound_http_retries_totalwith amoduleofgeneric,model, ormcp.
The exported isRetryableOutboundTransportError only matches UND_ERR_CONNECT_TIMEOUT and, for an idempotent call, UND_ERR_HEADERS_TIMEOUT. This client fails with TimeoutError from AbortSignal.timeout, and with connection errors such as ECONNRESET. That helper would not retry those.
The status list, the 5s Retry-After cap, and the backoff could be shared if they were pulled out of guardedFetch. The fetch loop still has to stay here. The shared helper also does not retry 504, which these GETs do.
Summary
TRUEFOUNDRY_SERVICEFOUNDRY_HTTP_MAX_RETRIESdefaults to 2 extra attempts. Set it to 0 to keep a single attempt.Test plan
pnpm --filter @truefoundry/trueforge test -- tests/unit/truefoundry/peekUsers.test.ts tests/unit/truefoundry/listAgentSkillsPaging.test.ts tests/unit/truefoundry/getTenantControlPlaneUrl.test.ts tests/unit/truefoundry/TrueFoundryAgentStore.test.tsTRUEFOUNDRY_SERVICEFOUNDRY_HTTP_MAX_RETRIES=0restores a single attempt.Note
Medium Risk
Retries only apply to idempotent GETs, but failed reads can take longer (multiple timeouts/backoffs) before surfacing errors, and gateway statuses like 409 are replayed by design.
Overview
ServiceFoundry GET resilience:
TrueFoundryServiceFoundryServerClientnow replays GET requests after dropped connections,TimeoutError, common network error codes, and gateway-style statuses (408/409/429/502/503/504/520–524/530). POST, PUT, and DELETE still make a single attempt.Retries use exponential backoff with jitter;
Retry-Afteris honored when it is at most 5s, otherwise backoff applies. Each retry is logged at warn level via#waitForRetry.Configuration: new env
TRUEFOUNDRY_SERVICEFOUNDRY_HTTP_MAX_RETRIES(default 2 extra attempts after the first try; 0 disables retries). Wired throughconfig.ts,.env.example, and client construction inmain.ts. Existing unit tests passmaxRetries: 0so fetch call counts stay stable.Reviewed by Cursor Bugbot for commit 16f6aac. Bugbot is set up for automated code reviews on this repo. Configure here.