Skip to content

fix: retry ServiceFoundry GET calls after transient network failures - #1193

Open
ramantehlan wants to merge 2 commits into
mainfrom
sd/retry-failed-network-call
Open

ramantehlan wants to merge 2 commits into
mainfrom
sd/retry-failed-network-call

Conversation

@ramantehlan

@ramantehlan ramantehlan commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Retry ServiceFoundry GET calls after a dropped connection, a timeout, or a retryable gateway status. PUT, POST, and DELETE stay a single attempt.
  • TRUEFOUNDRY_SERVICEFOUNDRY_HTTP_MAX_RETRIES defaults to 2 extra attempts. Set it to 0 to keep a single attempt.

Test plan

  • pnpm --filter @truefoundry/trueforge test -- tests/unit/truefoundry/peekUsers.test.ts tests/unit/truefoundry/listAgentSkillsPaging.test.ts tests/unit/truefoundry/getTenantControlPlaneUrl.test.ts tests/unit/truefoundry/TrueFoundryAgentStore.test.ts
  • Confirm a short ServiceFoundry blip on session or model list succeeds after a retry warn, and a down control plane fails only after the configured attempts.
  • Confirm TRUEFOUNDRY_SERVICEFOUNDRY_HTTP_MAX_RETRIES=0 restores a single attempt.

Note

Medium Risk
Retries only apply to idempotent GETs, but failed reads can take longer (multiple timeouts/backoffs) before surfacing errors, and gateway statuses like 409 are replayed by design.

Overview
ServiceFoundry GET resilience: TrueFoundryServiceFoundryServerClient now replays GET requests after dropped connections, TimeoutError, common network error codes, and gateway-style statuses (408/409/429/502/503/504/520–524/530). POST, PUT, and DELETE still make a single attempt.

Retries use exponential backoff with jitter; Retry-After is honored when it is at most 5s, otherwise backoff applies. Each retry is logged at warn level via #waitForRetry.

Configuration: new env TRUEFOUNDRY_SERVICEFOUNDRY_HTTP_MAX_RETRIES (default 2 extra attempts after the first try; 0 disables retries). Wired through config.ts, .env.example, and client construction in main.ts. Existing unit tests pass maxRetries: 0 so fetch call counts stay stable.

Reviewed by Cursor Bugbot for commit 16f6aac. Bugbot is set up for automated code reviews on this repo. Configure here.

A dropped connection or short gateway error on a read failed the request on the first attempt. Writes stay a single attempt.

Signed-off-by: Raman Tehlan <ramantehlan@gmail.com>
@changeset-bot

changeset-bot Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 16f6aac

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@truefoundry/trueforge Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

The GET retry stays. The final request metric already records the outcome, so a separate retry counter and its tests are unnecessary.

Signed-off-by: Raman Tehlan <ramantehlan@gmail.com>
const AGENT_SKILLS_PAGE_SIZE = 100;
/** SFY resolve `@ArrayMaxSize(50)` — chunk larger AgentSpec skill lists. */
const AGENT_SKILL_RESOLVE_CHUNK_SIZE = 50;
const RETRY_INITIAL_DELAY_MS = 1_000;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to reuse the retry logic from packages/trueforge-core/src/core/util/ssrfGuard.ts
?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The loop in withOutboundHttpRetries is private, and every attempt goes through guardedFetch, so this client cannot call it.

  • guardedFetch checks the SSRF host list. In-cluster ServiceFoundry hosts end in .svc, .cluster, or .internal, and those suffixes are denied.
  • It replaces the request dispatcher with its own undici Agent, so the mutual-TLS dispatcher this client needs would be dropped.
  • Retries there are recorded on tfg_outbound_http_retries_total with a module of generic, model, or mcp.

The exported isRetryableOutboundTransportError only matches UND_ERR_CONNECT_TIMEOUT and, for an idempotent call, UND_ERR_HEADERS_TIMEOUT. This client fails with TimeoutError from AbortSignal.timeout, and with connection errors such as ECONNRESET. That helper would not retry those.

The status list, the 5s Retry-After cap, and the backoff could be shared if they were pulled out of guardedFetch. The fetch loop still has to stay here. The shared helper also does not retry 504, which these GETs do.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants