Skip to content

fix(ci): treat an unreadable cache size as unstable when settling - #7274

Merged
waleedlatif1 merged 1 commit into
stagingfrom
ci/settle-loop-treat-empty-du-as-unstable
Aug 29, 2026
Merged

fix(ci): treat an unreadable cache size as unstable when settling#7274
waleedlatif1 merged 1 commit into
stagingfrom
ci/settle-loop-treat-empty-du-as-unstable

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

Summary

cubic flagged this on #7273 and it's correct. Verified rather than taken on trust.

The settle loop initialises prev='', and since 4795d0426f made total() return empty on failure, two consecutive failed buildctl du reads compare equal and trip the stability counter:

du always failing -> loop exited after 2 iterations (of a possible 60)

So the loop gives up after ~2s instead of its 120s bound — precisely when du is failing and the prune is most likely still deleting, which is the exact case the wait exists to cover. Handing back early there risks the builder post-step SIGKILLing buildkitd and skipping the sticky disk commit.

Why guarding on non-empty is the right fix, with no downside

An empty reading can only mean du failed. buildctl prints its Total: line unconditionallycmd/buildctl/diskusage.go:161, outside any branch:

fmt.Fprintf(tw, "Reclaimable:\t%.2f\n", units.Bytes(reclaimable))
fmt.Fprintf(tw, "Total:\t%.2f\n", units.Bytes(total))

So a genuinely empty cache still reports Total: 0B and settles normally. There's no risk of adding 120s to builds with a cold cache — which was the obvious objection to this fix.

Confirmed across all three paths:

du behaviour iterations before exit
always fails 60 (waits the full bound)
steady value 3
empty cache (Total: 0B) 3

Severity

Lower than P1 in practice: since #7252 the step can't fail a build, so the consequence is a possibly-skipped disk commit — one run's cache lost, not a red build. But the logic is plainly wrong and the fix is one condition, so it's worth taking before promotion.

Type of Change

  • Bug fix

Testing

Reproduced the early exit and verified the fix under the exact shell flags GitHub uses (bash -e -o pipefail), across failing / steady / empty-cache readings. shellcheck clean, actionlint unchanged from baseline (13 pre-existing), bun run lint and all 39 check:audits pass.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

The settle loop initialised prev='' and, after 4795d04 made total() return
empty on failure, two consecutive failed `buildctl du` reads compared equal and
tripped the stability counter. The loop then exited after ~2s instead of its
120s bound — exactly when du is failing and the prune is most likely still
deleting, which is the case the wait exists to cover. Handing back early there
risks the builder post-step SIGKILLing buildkitd and skipping the sticky disk
commit.

An empty reading can only mean du failed. buildctl prints its `Total:` line
unconditionally (cmd/buildctl/diskusage.go), so an empty cache still reports
`Total: 0B` and settles normally. Guarding on a non-empty reading therefore
costs nothing in the healthy paths: verified a steady value and an empty cache
both still exit after 3 iterations, while a persistently failing du now waits
out all 60.

Reported by cubic on #7273.
@vercel

vercel Bot commented Aug 29, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
docs Building Building Preview Aug 29, 2026 7:14am

Request Review

@waleedlatif1
waleedlatif1 merged commit f516361 into staging Aug 29, 2026
20 of 21 checks passed
@waleedlatif1
waleedlatif1 deleted the ci/settle-loop-treat-empty-du-as-unstable branch August 29, 2026 07:14
@greptile-apps

greptile-apps Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR corrects the Docker build cache settle loop so failed, empty buildctl du readings cannot be mistaken for stable cache usage.

  • Requires a non-empty cache-size reading before incrementing the stability counter.
  • Preserves normal settling for successful readings, including an empty cache reported as Total: 0B.
  • Makes persistent unreadable states consume the existing bounded wait rather than exiting prematurely.

Confidence Score: 5/5

The PR appears safe to merge because the changed condition correctly distinguishes unreadable cache-size results from valid stable values.

The settle loop now counts only matching non-empty readings as stable, while failed reads remain bounded by the existing 60-iteration limit and valid zero-cache output continues to settle normally.

Important Files Changed

Filename Overview
.github/actions/docker-build/action.yml Adds a non-empty guard to the cache-size stability comparison, correctly preventing repeated failed reads from triggering early settlement.

Reviews (1): Last reviewed commit: "fix(ci): treat an unreadable cache size ..." | Re-trigger Greptile

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant