Skip to content

solver: resolve lazy blob cache opts per record during cache export - #7107

Draft
maxpetrusenkoagent wants to merge 1 commit into
moby:masterfrom
maxpetrusenkoagent:hermes/oss-pr-2026-09-02-buildkit-6893
Draft

solver: resolve lazy blob cache opts per record during cache export#7107
maxpetrusenkoagent wants to merge 1 commit into
moby:masterfrom
maxpetrusenkoagent:hermes/oss-pr-2026-09-02-buildkit-6893

Conversation

@maxpetrusenkoagent

Copy link
Copy Markdown

Description

Fixes #6893mode=max cache export silently drops earlier-stage cache records when a lazy/remote snapshotter (stargz/eStargz, overlaybd, ...) is used with a multi-stage build where a later stage consumes an earlier, different-base stage via COPY --from. On later builds, steps that should be restored from cache re-run instead, with no error reported.

Root cause. The cache exporter callers install a cache-opt getter in the export context (withDescHandlerCacheOpts) that resolves descriptor handlers for the exported final result ref chain only. ExportTo used to re-root the getter at each record's own state (recordCtxOpts, an ancestor walk over the record's vertex graph) so every record could resolve the handlers for its own lazy blobs. Commit 051818c added a CacheOptGetterOf(ctx) == nil guard ("remotecache: only load desc handlers if not set") so that the caller-installed getter is kept and per-record re-rooting is skipped. Since the caller getter is rooted at a single ref chain, records from other branches of the graph (an earlier build stage) can no longer resolve their lazy base blobs: LoadRef fails with NeedsRemoteProviderError and the deps loop drops the record silently, producing a degraded cache. Eager snapshotters are unaffected because their base blobs are materialized.

Fix. Re-root the cache-opt lookup at each record's own state while keeping the inherited (caller-installed) getter as a fallback for keys the record-specific lookup cannot resolve. This restores the pre-regression behavior for nested/cross-stage records and preserves the behavior that 051818c/2fcce87cf intentionally added for the root record (handler resolution through the already-loaded final ref, e.g. when the record has no live solver state to walk).

 mainCtx := ctx
-if CacheOptGetterOf(ctx) == nil && e.recordCtxOpts != nil {
-    ctx = e.recordCtxOpts(ctx)
-}
+if e.recordCtxOpts != nil {
+    if outer := CacheOptGetterOf(ctx); outer != nil {
+        // composed: record-specific getter first, inherited getter as fallback
+        ...

Tests

  • solver/exporter_test.go (new, run locally):
    • TestExportToLazyCrossStageRecordsUseTheirOwnCacheOpts — fails on master (intermediate record silently dropped from the export) and passes with the fix. Verified locally: red on master, green after the change.
    • TestExportToKeepsCallerCacheOptsFallback — guards the caller-installed-getter contract; verified it fails against the naive "remove the guard only" variant and passes with the composed fix.
    • go test ./solver/ -skip TestJobsIntegration — full solver unit suite passes (TestJobsIntegration requires worker spawn infra and fails identically on pristine master in this environment).
  • client/client_cache_test.go (new) — testStargzCacheExportMaxCrossStageLazyBase, a containerd+stargz integration test mirroring the issue reproducer: two different eStargz bases, an earlier stage whose first RUN writes a random marker, a bust step forcing re-run, mode=max registry cache export, prune, then rebuild from cache import asserting the marker is unchanged (i.e., the RUN was restored from cache, not re-executed). Runs on the containerd-snapshotter-stargz CI worker (skips elsewhere, same as the existing stargz tests).

Notes

Fixes moby#6893

ExportTo only installed the record-specific cache opt getter
(recordCtxOpts) when the context did not already carry one. The cache
exporter callers always install a getter rooted at the exported result
ref (withDescHandlerCacheOpts), so nested records inherited that getter
and never re-rooted the lookup at their own state. For records that
belong to other branches of the build graph, such as an earlier build
stage consumed via COPY --from, the inherited getter cannot resolve the
desc handlers for their lazy base blobs. Their LoadRef then failed with
NeedsRemoteProviderError and the deps loop dropped the record
silently, so mode=max cache export produced a degraded cache where
steps re-ran on later builds.

Re-root the lookup at each record's own state while keeping the
inherited getter as a fallback for keys the record-specific getter
cannot resolve. This preserves the caller-installed getter behavior
from 2fcce87/051818cf3 while fixing the cross-stage drop.

Adds solver unit tests covering both the cross-stage record drop and
the caller-getter fallback, plus a stargz client integration test
mirroring the issue reproducer.

Signed-off-by: maxpetrusenkoagent <280464918+maxpetrusenkoagent@users.noreply.github.com>
@maxpetrusenkoagent
maxpetrusenkoagent force-pushed the hermes/oss-pr-2026-09-02-buildkit-6893 branch from 5ceae92 to d81baff Compare September 2, 2026 11:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

mode=max cache export silently drops build-stage layers with a lazy snapshotter + multi-stage COPY --from

1 participant