Skip to content

Commit c6b2a80

Browse files
committed
feat(rest): align HTTP retries and timeouts with Java
Configure REST transport through rest.client.* properties for connection and socket timeouts, retry limits, user-agent and idempotency-key lifetime. Apply initial and merged catalog properties through the client factory, preserving custom factories and the existing timeout constructor. Use Java's three-minute default connection timeout and zero-value timeout semantics. Enforce response inactivity deadlines in milliseconds with a synchronous libcurl transfer loop, without an additional executor. Authenticate once and replay the same request across retries. Decide HTTP status retries after headers and before reading the entity. Honor Idempotency-Key for retry-safe requests and bound keyed retries by the first-send time and maximum next interval, including Retry-After or jitter. Parse key lifetimes as ISO-8601 durations with Java-compatible handling of invalid and non-positive values. Retry eligible request-execution I/O failures without a delay. Propagate final response-body failures without replaying the request. Preserve redirect protection, transfer-error reporting and commit-state-unknown handling. Saturate oversized Retry-After conversions without overflow and avoid echoing malformed OAuth token responses. Document the transport policy and add regression coverage for retry rules, key lifetime budgets, stable authentication, timeout precision and duration boundaries.
1 parent 5045427 commit c6b2a80

9 files changed

Lines changed: 1427 additions & 112 deletions

File tree

‎mkdocs/docs/rest-catalog.md‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
<!--
2+
~ Licensed to the Apache Software Foundation (ASF) under one
3+
~ or more contributor license agreements. See the NOTICE file
4+
~ distributed with this work for additional information
5+
~ regarding copyright ownership. The ASF licenses this file
6+
~ to you under the Apache License, Version 2.0 (the
7+
~ "License"); you may not use this file except in compliance
8+
~ with the License. You may obtain a copy of the License at
9+
~
10+
~ http://www.apache.org/licenses/LICENSE-2.0
11+
~
12+
~ Unless required by applicable law or agreed to in writing,
13+
~ software distributed under the License is distributed on an
14+
~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
~ KIND, either express or implied. See the License for the
16+
~ specific language governing permissions and limitations
17+
~ under the License.
18+
-->
19+
20+
# REST catalog transport
21+
22+
The transport and retry properties follow the Java `HTTPClient` and
23+
`ExponentialHttpRequestRetryStrategy`. They apply to configuration discovery,
24+
OAuth token requests, catalog operations, and storage credential refresh requests.
25+
Initial requests use client properties; subsequent requests use merged catalog
26+
properties.
27+
28+
| Property | Default | Notes |
29+
| --- | --- | --- |
30+
| `rest.client.connection-timeout-ms` | 180000 | Non-negative integer; zero disables the configured deadline |
31+
| `rest.client.socket-timeout-ms` | Unbounded | Non-negative integer; zero disables the response inactivity timeout |
32+
| `rest.client.max-retries` | 5 | Positive integer; excludes the first attempt |
33+
| `rest.client.user-agent` | `iceberg-cpp/<version>` | Used when the request has no explicit `User-Agent` header |
34+
| `rest.client.idempotency-key-lifetime` | Unbounded | ISO-8601 duration, for example `PT30M`; zero or negative durations disable keyed retries; malformed or out-of-range values are ignored |
35+
36+
The socket timeout measures response inactivity rather than average transfer
37+
speed or total request duration. Connection setup and writing a request body
38+
use their own transport behavior. Transfers run synchronously in the calling
39+
thread, polling the remaining read deadline in milliseconds; no executor is
40+
required for HTTP retries.
41+
42+
HTTP response retries are decided after receiving the headers, before reading
43+
the response body:
44+
45+
- HTTP 429, and HTTP 503 carrying `Retry-After`, permit retries for any method.
46+
- GET, HEAD and DELETE, and requests carrying an `Idempotency-Key` header, also
47+
permit retries on HTTP 408, 500, 502, 503 and 504.
48+
- Positive numeric and HTTP-date `Retry-After` values are honored. Otherwise the
49+
wait doubles from 1 second up to 64 seconds, with up to 10% jitter. Invalid or
50+
non-positive values use this backoff; numeric durations beyond the millisecond
51+
range saturate at the maximum representable duration.
52+
53+
Recoverable I/O failures before response headers are received permit retries
54+
for idempotent methods or requests carrying `Idempotency-Key`. As in Java's
55+
I/O-exception retry path, these retries have no delay. Name resolution,
56+
connection, TLS, timeout, cancellation and premature-response-closure failures
57+
are not retried. Failures while reading a final response body are propagated,
58+
including when its HTTP status was successful.
59+
60+
For keyed requests, the lifetime starts at the first send. A retry is abandoned
61+
when elapsed time plus the maximum next retry interval reaches the configured
62+
lifetime. The bound includes `Retry-After` or the largest possible HTTP backoff
63+
jitter. As in Java, the I/O-error lifetime check uses the exponential interval
64+
even though the exception retry itself has no delay. Unkeyed requests are not
65+
constrained by this property.
66+
67+
Authentication runs once before the first send. Retries replay the same
68+
headers, body and idempotency key. A subsequent logical request authenticates
69+
again and can use an updated token.
70+
71+
Errors are classified from the `code` reported in the error body, falling back
72+
to the HTTP status when the body cannot be parsed. Service diagnostics remain
73+
available. Malformed OAuth token responses never echo response content.
74+
75+
Other Java transport options, including `rest.client.max-connections`,
76+
`rest.client.connections-per-route`, `rest.client.proxy.*` and
77+
`rest.client.tls.configurer-impl`, are not yet supported by this transport.
78+
79+
For temporary storage credentials and their lifetime, see [FileIO](file-io.md#refreshing-temporary-s3-credentials).

‎mkdocs/mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,7 @@ nav:
6161
- Verify a Release Candidate: verify-rc.md
6262
- API Documentation: api/index.html
6363
- SQL Catalog: sql-catalog.md
64+
- REST Catalog: rest-catalog.md
6465

6566
extra:
6667
social:

‎src/iceberg/catalog/rest/auth/oauth2_util.cc‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -49,9 +49,12 @@ constexpr std::string_view kAuthorizationHeader = "Authorization";
4949
constexpr std::string_view kBearerPrefix = "Bearer ";
5050

5151
Result<OAuthTokenResponse> ParseTokenResponse(const std::string& response_body) {
52-
ICEBERG_ASSIGN_OR_RAISE(auto json, FromJsonString(response_body));
53-
ICEBERG_ASSIGN_OR_RAISE(auto token_response, FromJson<OAuthTokenResponse>(json));
54-
ICEBERG_RETURN_UNEXPECTED(token_response.Validate());
52+
// Never propagate the underlying JSON errors: they echo the response body,
53+
// which may contain token material. FromJson already validates the response.
54+
auto json = FromJsonString(response_body);
55+
if (!json) return InvalidArgument("Invalid OAuth2 token response");
56+
auto token_response = FromJson<OAuthTokenResponse>(*json);
57+
if (!token_response) return InvalidArgument("Invalid OAuth2 token response");
5558
return token_response;
5659
}
5760

‎src/iceberg/catalog/rest/catalog_properties.h‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -74,7 +74,7 @@ class ICEBERG_REST_EXPORT RestCatalogProperties
7474
inline static constexpr std::string_view kHeaderPrefix = "header.";
7575
/// \brief HTTP connection timeout in milliseconds, matching Java REST clients.
7676
inline static Entry<int64_t> kClientConnectionTimeoutMs{
77-
"rest.client.connection-timeout-ms", 0};
77+
"rest.client.connection-timeout-ms", 180000};
7878
/// \brief HTTP socket/read timeout in milliseconds, matching Java REST clients.
7979
inline static Entry<int64_t> kClientSocketTimeoutMs{"rest.client.socket-timeout-ms", 0};
8080

0 commit comments

Comments
 (0)