add gcs.allow_multipart_upload and gcs.allow_multipart_download for parallel transfers of large files#1469
Merged
Conversation
Files bigger than gcs.parallel_upload_min_size (default 1GB) upload to GCS as multiple parallel parts via a dedicated gRPC client and compose into the final object, using the experimental EnableParallelUpload writer surface from cloud.google.com/go/storage v1.62+. Single-stream uploads of 100GB+ parts were the bottleneck for large-table backups (12h+), see #1028. - parallel_upload_part_size (0 = SDK default 16MiB, min 5MiB) and parallel_upload_max_concurrency (0 = SDK default min(4+CPU/2, 16)) - rejected together with endpoint, force_http and encryption_key: PCU works only via gRPC to storage.googleapis.com and Compose requires the same CSEK for source parts and destination - retry policy set client-level: part uploads create fresh ObjectHandles, per-object Retryer does not cover them - UploadPath now passes the real file size to PutFile so the min_size threshold works for uncompressed uploads - TestGCSParallelUpload runs against the real bucket with GCS_ENCRYPTION_KEY unset per command (run.sh exports it globally for CSEK coverage in TestGCS)
Signed-off-by: slach <bloodjazman@gmail.com>
Slach
changed the title
Slach
changed the title
Coverage Report for CI Build 29478480961Coverage increased (+0.2%) to 68.357%Details
Uncovered Changes
Coverage Regressions3 previously-covered lines in 2 files lost coverage.
Coverage Stats
💛 - Coveralls |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements parallel multipart transfers to/from GCS for large files, fix #1028.
Single-stream transfers of 100GB+ parts were the bottleneck for large-table backups (12h+ reported in the issue). Google shipped the feature requested in googleapis/google-cloud-go#3219 in two pieces, both used here:
EnableParallelUploadwriter surface incloud.google.com/go/storagev1.62+ (repo uses v1.63.1) — the stream splits into parts uploaded concurrently as temporary objects (gcs-go-sdk-pu-tmp/prefix in the bucket root) and composed into the final object. gRPC client only.transfermanager.Downloader(preview) — parallel range reads of a single object into a temporary file, works over the existing HTTP client.New config options (
gcssection)allow_multipart_uploadGCS_ALLOW_MULTIPART_UPLOADfalseupload_concurrencyGCS_UPLOAD_CONCURRENCYmultipart_upload_min_sizeGCS_MULTIPART_UPLOAD_MIN_SIZEallow_multipart_downloadGCS_ALLOW_MULTIPART_DOWNLOADfalses3.allow_multipart_downloaddownload_concurrencyGCS_DOWNLOAD_CONCURRENCYPart size for both directions is the existing
chunk_size(default 16MiB; the SDK bumps upload parts below 5MiB to 5MiB).Implementation notes
Connect()creates a dedicatedstorage.NewGRPCClientnext to the pooled HTTP clients; retry policy is set client-level because part uploads create freshObjectHandles that per-objectRetryerdoesn't cover.ValidateConfigrejectsallow_multipart_uploadcombined withendpoint,force_httporencryption_key(gRPC goes only to storage.googleapis.com; Compose requires the same CSEK for source parts and destination).DownloadCompressedStreamforces the streaming reader.UploadPathnow passes the real file size toPutFileso themultipart_upload_min_sizethreshold also works for uncompressed (upload_by_part) uploads.gcs-go-sdk-pu-tmp/(SDK cleanup of temporary upload parts is best-effort).Testing
make testpasses;ValidateConfigunit cases for the incompatible combinations and thedownload_concurrency > 1requirement.TestGCSMultipartintegration test against the realaltinity-qa-testbucket: ~30MB of Go-side-generated random data (LZ4-incompressible;randomString()is missing in old ClickHouse versions) with 1MiB threshold / 5MiBchunk_size, asserts the upload log went throughputFileMultipartand the download log through the multipart path, then restore verifies all rows. Databases are dropped via the version-safedropDatabasehelper (SYNCfor Atomic, otherwise the async drop leavesstore/<uuid>behind and restore fails on 21.3). Runs withGCS_ENCRYPTION_KEY=unset per command sincerun.shexports it globally for the CSEK coverage inTestGCS.🤖 Generated with Claude Code