Skip to content

Surface dropped buildx session diagnostics and raise session healthcheck tolerance#117

Open
Trent-TSE wants to merge 1 commit into
mainfrom
devin/1784149660-session-drop-diagnostics
Open

Surface dropped buildx session diagnostics and raise session healthcheck tolerance#117
Trent-TSE wants to merge 1 commit into
mainfrom
devin/1784149660-session-drop-diagnostics

Conversation

@Trent-TSE

@Trent-TSE Trent-TSE commented Jul 15, 2026

Copy link
Copy Markdown

Summary

When a build maxes out runner memory (observed on sidestream-tech's cold-cache arm64 builds: ~0.04GB mem available, 93% iowait), the buildx client stalls past buildkitd's session healthcheck tolerance and the session carrying registry credentials is torn down mid-build. The eventual failure surfaces minutes later as a confusing push/auth error: no active session for <id>: context deadline exceeded (/src/session/auth/auth.go:104).

Two changes:

  • startBuildkitd now sets BUILDKIT_SESSION_HEALTHCHECK_MAX_FAILURES=10 by default (user driver-opts env.* values override it). This pairs with session: make healthcheck tolerance configurable via env vars buildkit#33, which makes the session healthcheck tolerance configurable via env vars — the var is inert until that lands in a fleet buildkit release.
  • New post-step diagnostic warnOnDroppedBuildkitSessions(): greps /tmp/buildkitd.log for healthcheck failed fatally / no active session and emits a core.warning explaining that memory pressure (not bad credentials) dropped the session, with the matching log lines.

Updated the two driver-opts tests that asserted no sudo env prefix when no opts are passed, since a default env var is now always present.

Verified with pnpm typecheck, pnpm lint, pnpm test (29 passed), and the lefthook pre-commit build (dist/ regenerated and committed).

Link to Devin session: https://app.devin.ai/sessions/47b6abfcecd948dcbae7d55badefa46c
Requested by: @Trent-TSE


View with Codesmith Autofix with Codesmith
Need help on this PR? Tag /codesmith with what you need. Autofix is disabled. (Staging)

…eck tolerance

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant