Skip to content

ci(vcpkg): Fix recurring binary cache invalidation from runner image updates#2371

Open
bobtista wants to merge 3 commits intoTheSuperHackers:mainfrom
bobtista:bobtista/fix-vcpkg-ci-cache-stability
Open

ci(vcpkg): Fix recurring binary cache invalidation from runner image updates#2371
bobtista wants to merge 3 commits intoTheSuperHackers:mainfrom
bobtista:bobtista/fix-vcpkg-ci-cache-stability

Conversation

@bobtista
Copy link

@bobtista bobtista commented Mar 1, 2026

Summary

  • Add custom x86-windows triplet overlay with VCPKG_DISABLE_COMPILER_TRACKING so weekly MSVC version bumps on GitHub-hosted runners don't invalidate the vcpkg binary cache
  • Add --x-abi-tools-use-exact-versions to pin tool versions from vcpkg's own manifest instead of using system-installed versions
  • Bump cache key to v3 to flush the current stale cache and remove MSVC version from the key since compiler tracking is now disabled
  • Add VCPKG_OVERLAY_TRIPLETS to CMakePresets.json so the custom triplet is used in both CI and local builds

Test plan

  • First CI run will cache-miss (new v3 key prefix), build from source, and save cache
  • Subsequent CI runs should restore cached packages and skip ffmpeg rebuild (~17 min savings)

@greptile-apps
Copy link

greptile-apps bot commented Mar 1, 2026

Greptile Summary

Stabilized vcpkg binary cache by disabling compiler version tracking to prevent GitHub runner image updates from invalidating cached packages.

Key changes:

  • Created custom x86-windows triplet with VCPKG_DISABLE_COMPILER_TRACKING enabled
  • Removed MSVC version from cache key and bumped to v3 to flush stale cache
  • Added ${{ hashFiles('triplets/*.cmake') }} to cache key for proper invalidation when triplet configuration changes
  • Set VCPKG_OVERLAY_TRIPLETS in both workflow and CMakePresets.json for consistency between CI and local builds
  • Added --x-abi-tools-use-exact-versions to pin vcpkg tool versions from manifest

Impact:
This fixes the recurring issue where weekly MSVC compiler updates on GitHub-hosted runners invalidated the entire binary cache, forcing ~17-minute ffmpeg rebuilds. The changes are backward-compatible and well-scoped to vcpkg-based builds.

Confidence Score: 5/5

  • This PR is safe to merge with minimal risk
  • The changes are well-targeted CI/build configuration updates that address a specific caching issue. All modifications follow vcpkg best practices: custom triplet configuration is standard, cache key includes appropriate dependencies, and environment variables are correctly scoped. No runtime code changes, proper cache version bump prevents conflicts, and local builds remain consistent with CI through CMakePresets.json integration.
  • No files require special attention

Important Files Changed

Filename Overview
triplets/x86-windows.cmake Created custom x86-windows triplet with VCPKG_DISABLE_COMPILER_TRACKING to prevent weekly MSVC updates from invalidating the binary cache
.github/workflows/build-toolchain.yml Removed MSVC version tracking from cache key, bumped to v3, added triplet hash tracking, and configured VCPKG_OVERLAY_TRIPLETS and VCPKG_INSTALL_OPTIONS
CMakePresets.json Added VCPKG_OVERLAY_TRIPLETS to default-vcpkg preset to ensure custom triplet is used in both CI and local builds

Last reviewed commit: 7eb4dca

@xezon
Copy link

xezon commented Mar 1, 2026

What is triplets? The build time still shows with 23 minutes.

@bobtista
Copy link
Author

bobtista commented Mar 1, 2026

What is triplets? The build time still shows with 23 minutes.

A vcpkg triplet is a config file that tells vcpkg how to build packages (32 vs 64-bit, static vs dynamic linking, etc). We add a custom one that also tells vcpkg to ignore compiler version changes when caching. GitHub's runner images update MSVC weekly. vcpkg includes the exact compiler version in its cache hash by default. So every week the compiler version bumps, the hash changes, the cached binaries don't match, and ffmpeg gets rebuilt from source (~17 min) even though nothing in our code or dependencies changed. Disabling compiler tracking stops that cycle.

The build is like 6 minutes - I only re-ran one of the vcpkg jobs, I can run all of them again now though

# Exclude compiler version from ABI hash so that weekly GitHub runner image
# updates don't invalidate the binary cache. Minor MSVC version bumps do not
# cause ABI incompatibilities for this project.
set(VCPKG_DISABLE_COMPILER_TRACKING ON)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this possibly cause other issues in the future? If the binary cache is invalidated once a week, then why does it not keep the new binary cache after 1 build? Does that not need looking into?

Do we perhaps need a CI trigger to manually force purge and refresh the cache in case it is accidentially broken?

Do we know why exactly the cache is broken right now?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are two separate layers of caching.

  1. GitHub Actions cache — stores/restores a directory of files based on a key we define in the workflow. It just zips up the folder and gives it back when the key matches. It knows nothing about vcpkg.
  2. vcpkg binary cache — inside that directory, each package is a .zip file named by vcpkg's own ABI hash (which includes the exact compiler version like 14.44.35207). When vcpkg runs, it computes its hash and looks for a matching .zip in the directory.

So the problem was GitHub Actions says: "key v2-msvc14.44-... matches, here's your directory" and says cach-hit was true. Then vcpkg opens that directory, computes its hash with the full compiler version 14.44.35207, finds no matching .zip files (they were built with 14.43.xxxxx) and re-builds ffmpeg which takes forever (17 min). Because cache-hit was true as far as github is concerned, it doesn't save the new build. This fix removes the compiler version from vcpkg's hash, so both layers should stay in sync.

RE future issues, MSVC maintains ABI compatibility within the same major version (14.x), and the cache still invalidates when the vcpkg baseline or triplet config changes. We'd be very unlikely to have no other changes before the next major MSVC version - and yes we can always just bump the version in build-toolchain.yml to force purge and refresh the cache.

@xezon xezon added Major Severity: Minor < Major < Critical < Blocker Build Anything related to building, compiling labels Mar 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Build Anything related to building, compiling Major Severity: Minor < Major < Critical < Blocker

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants