Skip to content

fix(CDC): append chunks when downloading path-backed outputs#29545

Closed
tyler-french wants to merge 1 commit into
bazelbuild:release-9.1.0from
tyler-french:tfrench/fix-cdc-path-backed-download-9.1.0
Closed

fix(CDC): append chunks when downloading path-backed outputs#29545
tyler-french wants to merge 1 commit into
bazelbuild:release-9.1.0from
tyler-french:tfrench/fix-cdc-path-backed-download-9.1.0

Conversation

@tyler-french
Copy link
Copy Markdown
Contributor

@tyler-french tyler-french commented May 15, 2026

Fixes #29544

Using --remote_download_outputs=toplevel with --experimental_remote_cache_chunking and --disk_cache can cause Bazel to materialize a top-level output as a single CDC chunk instead of the full blob. This happens because the chunked downloader passes the final path-backed output stream into per-chunk downloads, letting DiskCacheClient's MaybePathBacked fast path copy each cached chunk directly over the final output path.

This PR fixes it by fetching each chunk through CombinedCache as bytes and then appending those bytes to the final output stream, preserving disk cache policy/write-through behavior for chunks while keeping the path-backed copy optimization for whole-blob downloads.

This is unintentionally fixed in #29341
But I think this might be worth a patch release in 9.1.1 and/or 8.7.1

Copilot AI review requested due to automatic review settings May 15, 2026 00:59
@tyler-french tyler-french requested a review from a team as a code owner May 15, 2026 00:59
@github-actions github-actions Bot added team-Remote-Exec Issues and PRs for the Execution (Remote) team awaiting-review PR is awaiting review from an assigned reviewer labels May 15, 2026
Comment thread src/main/java/com/google/devtools/build/lib/remote/ChunkedBlobDownloader.java Outdated
@tyler-french tyler-french force-pushed the tfrench/fix-cdc-path-backed-download-9.1.0 branch from eb4afad to 3bbc53b Compare May 15, 2026 15:36
@tyler-french tyler-french force-pushed the tfrench/fix-cdc-path-backed-download-9.1.0 branch from 3bbc53b to 0152704 Compare May 16, 2026 17:32
@tjgq
Copy link
Copy Markdown
Contributor

tjgq commented May 18, 2026

@bazel-io 9.1.1

@iancha1992
Copy link
Copy Markdown
Member

iancha1992 commented May 18, 2026

@tyler-french 9.1.0 is already released. Let's also wait until it is merged to the master branch first before we merge this to the release branch because we prefer the cherry-pick bot to do the work. I will reopen this if there's a merge conflict and help is needed. Thank you!

@iancha1992 iancha1992 closed this May 18, 2026
@github-actions github-actions Bot removed the awaiting-review PR is awaiting review from an assigned reviewer label May 18, 2026
@tyler-french
Copy link
Copy Markdown
Contributor Author

tyler-french commented May 18, 2026

@tyler-french 9.1.0 is already released. Let's also wait until it is merged to the master branch first before we merge this to the release branch because we prefer the cherry-pick bot to do the work. I will reopen this if there's a merge conflict and help is needed. Thank you!

EDIT: Thanks, just realized #29341 IS going into 9.1.1. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

team-Remote-Exec Issues and PRs for the Execution (Remote) team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants