From 0ac95e0a55f534441b7afecd346ead980be6d8b4 Mon Sep 17 00:00:00 2001 From: Steven Pall Date: Sun, 19 Jul 2026 23:30:22 -0700 Subject: [PATCH] [fix](cloud) set recycler S3 client requestTimeoutMs to avoid curl-28 on slow DeleteObjects (mirror #49315) (#64758) ## Proposed changes The cloud recycler builds its S3 client in `S3Accessor::init()` (`cloud/src/recycler/s3_accessor.cpp`) from an `Aws::Client::ClientConfiguration` that never sets `requestTimeoutMs`, so it keeps the SDK default of 3000ms. The vendored aws-sdk-cpp maps that to `CURLOPT_LOW_SPEED_TIME=3` / `CURLOPT_LOW_SPEED_LIMIT=1`, so any slow or large `DeleteObjects` request that can't sustain >1 byte/s for 3 seconds is aborted with curl error 28 ("Timeout was reached"). This is the same defect that #49315 fixed for the BE (`be/src/util/s3_util.cpp`, `requestTimeoutMs = 30000`), but the cloud recycler's client was missed by that change. On object stores with higher per-request latency (e.g. an OVH cold object-storage vault) the recycler wedges: every `delete_rowset_data` / `DeleteObjects` aborts at 3s with curlCode 28, the recycler burns its delete budget on timed-out ops, and the orphan backlog never drains. This change sets `requestTimeoutMs = 30000` and `connectTimeoutMs = 5000` on the recycler client, mirroring #49315. Symptom in MS recycler log before the fix: ``` s3_obj_client.cpp: failed to delete objects ... responseCode=-1 error="curlCode: 28, Timeout was reached" recycler.cpp: failed to delete rowset data, instance_id=... ``` ## Further comments Pure timeout-config change in the recycler S3 client path; no behavior change for fast object stores. `MaxDeleteBatch` is left unchanged. Signed-off-by: Steven Pall --- cloud/src/recycler/s3_accessor.cpp | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/cloud/src/recycler/s3_accessor.cpp b/cloud/src/recycler/s3_accessor.cpp index 8ff29694bd7c3e..0c90676d8e0aa7 100644 --- a/cloud/src/recycler/s3_accessor.cpp +++ b/cloud/src/recycler/s3_accessor.cpp @@ -449,6 +449,11 @@ int S3Accessor::init() { if (!_ca_cert_file_path.empty()) { aws_config.caFile = _ca_cert_file_path; } + // Mirror BE PR #49315: default ClientConfiguration leaves requestTimeoutMs=3000, + // which the vendored aws-sdk-cpp maps to CURLOPT_LOW_SPEED_TIME=3 and causes + // curl error 28 on slow/large S3 DeleteObjects (OVH cold vault). + aws_config.requestTimeoutMs = 30000; + aws_config.connectTimeoutMs = 5000; auto s3_client = std::make_shared( get_aws_credentials_provider(conf_), std::move(aws_config), Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never,