solver: resolve lazy blob cache opts per record during cache export - #7107
Draft
maxpetrusenkoagent wants to merge 1 commit into
Draft
solver: resolve lazy blob cache opts per record during cache export#7107maxpetrusenkoagent wants to merge 1 commit into
maxpetrusenkoagent wants to merge 1 commit into
Conversation
maxpetrusenkoagent
force-pushed
the
hermes/oss-pr-2026-09-02-buildkit-6893
branch
from
September 2, 2026 11:06
b9fc166 to
5ceae92
Compare
Fixes moby#6893 ExportTo only installed the record-specific cache opt getter (recordCtxOpts) when the context did not already carry one. The cache exporter callers always install a getter rooted at the exported result ref (withDescHandlerCacheOpts), so nested records inherited that getter and never re-rooted the lookup at their own state. For records that belong to other branches of the build graph, such as an earlier build stage consumed via COPY --from, the inherited getter cannot resolve the desc handlers for their lazy base blobs. Their LoadRef then failed with NeedsRemoteProviderError and the deps loop dropped the record silently, so mode=max cache export produced a degraded cache where steps re-ran on later builds. Re-root the lookup at each record's own state while keeping the inherited getter as a fallback for keys the record-specific getter cannot resolve. This preserves the caller-installed getter behavior from 2fcce87/051818cf3 while fixing the cross-stage drop. Adds solver unit tests covering both the cross-stage record drop and the caller-getter fallback, plus a stargz client integration test mirroring the issue reproducer. Signed-off-by: maxpetrusenkoagent <280464918+maxpetrusenkoagent@users.noreply.github.com>
maxpetrusenkoagent
force-pushed
the
hermes/oss-pr-2026-09-02-buildkit-6893
branch
from
September 2, 2026 11:06
5ceae92 to
d81baff
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #6893 —
mode=maxcache export silently drops earlier-stage cache records when a lazy/remote snapshotter (stargz/eStargz, overlaybd, ...) is used with a multi-stage build where a later stage consumes an earlier, different-base stage viaCOPY --from. On later builds, steps that should be restored from cache re-run instead, with no error reported.Root cause. The cache exporter callers install a cache-opt getter in the export context (
withDescHandlerCacheOpts) that resolves descriptor handlers for the exported final result ref chain only.ExportToused to re-root the getter at each record's own state (recordCtxOpts, an ancestor walk over the record's vertex graph) so every record could resolve the handlers for its own lazy blobs. Commit 051818c added aCacheOptGetterOf(ctx) == nilguard ("remotecache: only load desc handlers if not set") so that the caller-installed getter is kept and per-record re-rooting is skipped. Since the caller getter is rooted at a single ref chain, records from other branches of the graph (an earlier build stage) can no longer resolve their lazy base blobs:LoadReffails withNeedsRemoteProviderErrorand the deps loop drops the record silently, producing a degraded cache. Eager snapshotters are unaffected because their base blobs are materialized.Fix. Re-root the cache-opt lookup at each record's own state while keeping the inherited (caller-installed) getter as a fallback for keys the record-specific lookup cannot resolve. This restores the pre-regression behavior for nested/cross-stage records and preserves the behavior that 051818c/2fcce87cf intentionally added for the root record (handler resolution through the already-loaded final ref, e.g. when the record has no live solver state to walk).
Tests
solver/exporter_test.go(new, run locally):TestExportToLazyCrossStageRecordsUseTheirOwnCacheOpts— fails onmaster(intermediate record silently dropped from the export) and passes with the fix. Verified locally: red on master, green after the change.TestExportToKeepsCallerCacheOptsFallback— guards the caller-installed-getter contract; verified it fails against the naive "remove the guard only" variant and passes with the composed fix.go test ./solver/ -skip TestJobsIntegration— full solver unit suite passes (TestJobsIntegration requires worker spawn infra and fails identically on pristine master in this environment).client/client_cache_test.go(new) —testStargzCacheExportMaxCrossStageLazyBase, a containerd+stargz integration test mirroring the issue reproducer: two different eStargz bases, an earlier stage whose first RUN writes a random marker, a bust step forcing re-run,mode=maxregistry cache export, prune, then rebuild from cache import asserting the marker is unchanged (i.e., the RUN was restored from cache, not re-executed). Runs on thecontainerd-snapshotter-stargzCI worker (skips elsewhere, same as the existing stargz tests).Notes
go vetclean on changed files (pre-existingsolver/jobs.go:719vet warning untouched, present on master).mode=maxcache export silently drops build-stage layers with a lazy snapshotter + multi-stageCOPY --from#6893.