Skip to content

Commit 780e128

Browse files
committed
Merge remote-tracking branch 'upstream/main' into fix/collect-nan-value-counts
2 parents a46dabe + c78e18e commit 780e128

44 files changed

Lines changed: 1758 additions & 1685 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/nightly-pypi-build.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,7 @@ jobs:
8888
- name: Publish to TestPyPI
8989
id: publish-testpypi
9090
continue-on-error: true
91-
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2
91+
uses: pypa/gh-action-pypi-publish@7245906fb61fa33a13554f9d5fb856fe61758086 # v1.14.3
9292
with:
9393
repository-url: https://test.pypi.org/legacy/
9494
skip-existing: true

‎.github/workflows/pypi-build-artifacts.yml‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,6 @@ jobs:
4747
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
4848
with:
4949
python-version: |
50-
3.10
5150
3.11
5251
3.12
5352
3.13
@@ -76,7 +75,7 @@ jobs:
7675
env:
7776
# Ignore 32 bit architectures
7877
CIBW_ARCHS: "auto64"
79-
CIBW_PROJECT_REQUIRES_PYTHON: ">=3.10,<3.15"
78+
CIBW_PROJECT_REQUIRES_PYTHON: ">=3.11,<3.15"
8079
# Keep these in sync with Python CI job `cibw-dev-env-smoke-test`
8180
# in .github/workflows/python-ci.yml to catch import-time regressions early.
8281
CIBW_BEFORE_TEST: "uv sync --directory {project} --only-group dev --no-install-project"

‎.github/workflows/python-ci.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ jobs:
3838
max-parallel: 15
3939
fail-fast: true
4040
matrix:
41-
python: ['3.10', '3.11', '3.12', '3.13', '3.14']
41+
python: ['3.11', '3.12', '3.13', '3.14']
4242

4343
steps:
4444
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
@@ -54,12 +54,12 @@ jobs:
5454
- name: Install system dependencies
5555
run: sudo apt-get update && sudo apt-get install -y libkrb5-dev # for kerberos
5656
- name: Check uv.lock is up to date
57-
if: matrix.python == '3.10'
57+
if: matrix.python == '3.11'
5858
run: uv lock --check
5959
- name: Install
6060
run: make install
6161
- name: Run linters
62-
if: matrix.python == '3.10'
62+
if: matrix.python == '3.11'
6363
run: make lint
6464
- name: Run unit tests with coverage
6565
run: COVERAGE=1 make test

‎.github/workflows/svn-build-artifacts.yml‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,6 @@ jobs:
4747
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
4848
with:
4949
python-version: |
50-
3.10
5150
3.11
5251
3.12
5352
3.13
@@ -70,7 +69,7 @@ jobs:
7069
env:
7170
# Ignore 32 bit architectures
7271
CIBW_ARCHS: "auto64"
73-
CIBW_PROJECT_REQUIRES_PYTHON: ">=3.10,<3.15"
72+
CIBW_PROJECT_REQUIRES_PYTHON: ">=3.11,<3.15"
7473
CIBW_BEFORE_TEST: "uv sync --directory {project} --only-group dev --no-install-project"
7574
CIBW_TEST_COMMAND: "uv run --directory {project} pytest tests/avro/test_decoder.py"
7675
# Skip free-threaded (PEP 703) builds until we evaluate decoder_fast support

‎dev/docker-compose-integration.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ services:
6262
- CATALOG_S3_ENDPOINT=http://object-store:9000
6363
- CATALOG_JDBC_STRICT__MODE=true
6464
object-store:
65-
image: rustfs/rustfs:1.0.3
65+
image: rustfs/rustfs:1.0.4
6666
container_name: pyiceberg-object-store
6767
networks:
6868
iceberg_net:

‎dev/docker-compose.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717

1818
services:
1919
object-store:
20-
image: rustfs/rustfs:1.0.3
20+
image: rustfs/rustfs:1.0.4
2121
container_name: pyiceberg-object-store
2222
ports:
2323
- 9000:9000

‎mkdocs/docs/configuration.md‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -784,6 +784,7 @@ catalog:
784784
| hive.hive2-compatible | true | Set to `true` when using a Hive 2.x metastore |
785785
| hive.kerberos-authentication | true | Using authentication via Kerberos |
786786
| hive.kerberos-service-name | hive | Kerberos service name (default hive) |
787+
| hive.kerberos-service-host | hive-host | Kerberos service host (default URI host) |
787788
| ugi | t-1234:secret | Hadoop UGI for Hive client. |
788789

789790
Hive 3 and newer need no extra configuration. When using a Hive 2.x metastore, set the compatibility flag:
@@ -844,6 +845,8 @@ catalog:
844845
| glue.access-key-id | admin | Configure the static access key id used to access the Glue Catalog |
845846
| glue.secret-access-key | password | Configure the static secret access key used to access the Glue Catalog |
846847
| glue.session-token | AQoDYXdzEJr... | Configure the static session token used to access the Glue Catalog |
848+
| glue.role-arn | arn:aws:... | AWS Role ARN. If provided, temporary credentials will be fetched by assuming this role before accessing the Glue Catalog |
849+
| glue.role-session-name | session | An optional identifier for the assumed role session used to access the Glue Catalog |
847850
| glue.max-retries | 10 | Configure the maximum number of retries for the Glue service calls |
848851
| glue.retry-mode | standard | Configure the retry mode for the Glue service. Default to standard. |
849852

@@ -902,6 +905,8 @@ catalog:
902905
| dynamodb.access-key-id | admin | Configure the static access key id used to access the DynamoDB Catalog |
903906
| dynamodb.secret-access-key | password | Configure the static secret access key used to access the DynamoDB Catalog |
904907
| dynamodb.session-token | AQoDYXdzEJr... | Configure the static session token used to access the DynamoDB Catalog |
908+
| dynamodb.role-arn | arn:aws:... | AWS Role ARN. If provided, temporary credentials will be fetched by assuming this role before accessing the DynamoDB Catalog |
909+
| dynamodb.role-session-name | session | An optional identifier for the assumed role session used to access the DynamoDB Catalog |
905910

906911
<!-- markdown-link-check-enable-->
907912

@@ -946,8 +951,10 @@ configures the AWS credentials for both Glue Catalog and S3 FileIO.
946951
| client.secret-access-key | password | Configure the static secret access key used to access both the Glue/DynamoDB Catalog and the S3 FileIO |
947952
| client.session-token | AQoDYXdzEJr... | Configure the static session token used to access both the Glue/DynamoDB Catalog and the S3 FileIO |
948953
| client.profile-name | default | Configure the AWS profile used to access both the Glue/DynamoDB Catalog and the S3 FileIO (only supported by `FsspecFileIO` currently) |
949-
| client.role-session-name | session | An optional identifier for the assumed role session. |
950-
| client.role-arn | arn:aws:... | AWS Role ARN. If provided instead of access_key and secret_key, temporary credentials will be fetched by assuming this role. |
954+
| client.assume-role.arn | arn:aws:... | AWS Role ARN. If provided instead of access_key and secret_key, temporary credentials will be fetched by assuming this role before accessing both the Glue/DynamoDB Catalog and the S3 FileIO. |
955+
| client.assume-role.session-name | session | An optional identifier for the assumed role session used to access both the Glue/DynamoDB Catalog and the S3 FileIO. |
956+
| client.role-arn | arn:aws:... | Same as `client.assume-role.arn`, used when that key is not set. |
957+
| client.role-session-name | session | Same as `client.assume-role.session-name`, used when that key is not set. |
951958

952959
<!-- prettier-ignore-start -->
953960

‎pyiceberg/catalog/__init__.py‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,7 @@
7474
from pyiceberg.view.metadata import ViewVersion
7575

7676
if TYPE_CHECKING:
77+
import boto3
7778
import pyarrow as pa
7879

7980
logger = logging.getLogger(__name__)
@@ -97,6 +98,54 @@
9798
EXTERNAL_TABLE = "EXTERNAL_TABLE"
9899
BOTOCORE_SESSION = "botocore_session"
99100

101+
DEFAULT_ROLE_SESSION_NAME = "pyiceberg"
102+
103+
104+
def _get_aws_session_with_assumed_role(
105+
session: boto3.Session,
106+
role_arn: str,
107+
role_session_name: str | None,
108+
region_name: str | None,
109+
) -> boto3.Session:
110+
"""Return a boto3 session whose credentials come from assuming an IAM role via STS.
111+
112+
This mirrors the assume-role behavior already available for the S3 FileIO (``client.role-arn`` /
113+
``client.role-session-name``) so that the AWS catalog clients (Glue, DynamoDB) can honor the same
114+
properties. The role is assumed on first use and again before the temporary credentials expire,
115+
so a long-lived catalog keeps working past the role session duration.
116+
117+
Args:
118+
session: The base boto3 session whose credentials are used to call STS.
119+
role_arn: The ARN of the role to assume.
120+
role_session_name: An identifier for the assumed-role session. Defaults to ``pyiceberg``.
121+
region_name: The region for the returned session.
122+
123+
Returns:
124+
A new boto3 session authenticated with refreshable assumed-role credentials.
125+
"""
126+
import boto3
127+
import botocore.session
128+
from botocore.credentials import AssumeRoleCredentialFetcher, DeferredRefreshableCredentials
129+
130+
source_credentials = session.get_credentials()
131+
if source_credentials is None:
132+
raise ValueError(f"No AWS credentials found to assume role {role_arn}")
133+
134+
fetcher = AssumeRoleCredentialFetcher(
135+
client_creator=session.client,
136+
source_credentials=source_credentials,
137+
role_arn=role_arn,
138+
extra_args={"RoleSessionName": role_session_name or DEFAULT_ROLE_SESSION_NAME},
139+
)
140+
botocore_session = botocore.session.Session()
141+
# botocore has no public setter for refreshable credentials on a session.
142+
botocore_session._credentials = DeferredRefreshableCredentials(
143+
refresh_using=fetcher.fetch_credentials,
144+
method="assume-role",
145+
)
146+
return boto3.Session(botocore_session=botocore_session, region_name=region_name)
147+
148+
100149
TABLE_METADATA_FILE_NAME_REGEX = re.compile(
101150
r"""
102151
(\d+) # version number

‎pyiceberg/catalog/dynamodb.py‎

Lines changed: 26 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@
3434
TABLE_TYPE,
3535
MetastoreCatalog,
3636
PropertiesUpdateSummary,
37+
_get_aws_session_with_assumed_role,
3738
)
3839
from pyiceberg.exceptions import (
3940
ConditionalCheckFailedException,
@@ -46,7 +47,17 @@
4647
NoSuchTableError,
4748
TableAlreadyExistsError,
4849
)
49-
from pyiceberg.io import AWS_ACCESS_KEY_ID, AWS_REGION, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, load_file_io
50+
from pyiceberg.io import (
51+
AWS_ACCESS_KEY_ID,
52+
AWS_ASSUME_ROLE_ARN,
53+
AWS_ASSUME_ROLE_SESSION_NAME,
54+
AWS_REGION,
55+
AWS_ROLE_ARN,
56+
AWS_ROLE_SESSION_NAME,
57+
AWS_SECRET_ACCESS_KEY,
58+
AWS_SESSION_TOKEN,
59+
load_file_io,
60+
)
5061
from pyiceberg.partitioning import UNPARTITIONED_PARTITION_SPEC, PartitionSpec
5162
from pyiceberg.schema import Schema
5263
from pyiceberg.serializers import FromInputFile
@@ -92,6 +103,8 @@
92103
DYNAMODB_ACCESS_KEY_ID = "dynamodb.access-key-id"
93104
DYNAMODB_SECRET_ACCESS_KEY = "dynamodb.secret-access-key"
94105
DYNAMODB_SESSION_TOKEN = "dynamodb.session-token"
106+
DYNAMODB_ROLE_ARN = "dynamodb.role-arn"
107+
DYNAMODB_ROLE_SESSION_NAME = "dynamodb.role-session-name"
95108

96109

97110
class DynamoDbCatalog(MetastoreCatalog):
@@ -107,14 +120,24 @@ def __init__(self, name: str, client: Optional["DynamoDBClient"] = None, **prope
107120
if client is not None:
108121
self.dynamodb = client
109122
else:
123+
region_name = get_first_property_value(properties, DYNAMODB_REGION, AWS_REGION)
110124
session = boto3.Session(
111125
profile_name=properties.get(DYNAMODB_PROFILE_NAME),
112-
region_name=get_first_property_value(properties, DYNAMODB_REGION, AWS_REGION),
126+
region_name=region_name,
113127
botocore_session=properties.get(BOTOCORE_SESSION),
114128
aws_access_key_id=get_first_property_value(properties, DYNAMODB_ACCESS_KEY_ID, AWS_ACCESS_KEY_ID),
115129
aws_secret_access_key=get_first_property_value(properties, DYNAMODB_SECRET_ACCESS_KEY, AWS_SECRET_ACCESS_KEY),
116130
aws_session_token=get_first_property_value(properties, DYNAMODB_SESSION_TOKEN, AWS_SESSION_TOKEN),
117131
)
132+
if role_arn := get_first_property_value(properties, DYNAMODB_ROLE_ARN, AWS_ASSUME_ROLE_ARN, AWS_ROLE_ARN):
133+
session = _get_aws_session_with_assumed_role(
134+
session=session,
135+
role_arn=role_arn,
136+
role_session_name=get_first_property_value(
137+
properties, DYNAMODB_ROLE_SESSION_NAME, AWS_ASSUME_ROLE_SESSION_NAME, AWS_ROLE_SESSION_NAME
138+
),
139+
region_name=region_name,
140+
)
118141
self.dynamodb = session.client(DYNAMODB_CLIENT)
119142

120143
self.dynamodb_table_name = self.properties.get(DYNAMODB_TABLE_NAME, DYNAMODB_TABLE_NAME_DEFAULT)
@@ -681,7 +704,7 @@ def _convert_dynamo_table_item_to_iceberg_table(self, dynamo_table_item: dict[st
681704
DYNAMODB_COL_NAMESPACE,
682705
DYNAMODB_COL_CREATED_AT,
683706
]:
684-
if prop not in table_dict.keys():
707+
if prop not in table_dict:
685708
raise NoSuchPropertyException(f"Iceberg required property {prop} is missing: {dynamo_table_item}")
686709

687710
table_type = table_dict[_add_property_prefix(TABLE_TYPE)]

‎pyiceberg/catalog/glue.py‎

Lines changed: 26 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@
3939
TABLE_TYPE,
4040
MetastoreCatalog,
4141
PropertiesUpdateSummary,
42+
_get_aws_session_with_assumed_role,
4243
)
4344
from pyiceberg.exceptions import (
4445
CommitFailedException,
@@ -50,7 +51,18 @@
5051
NoSuchTableError,
5152
TableAlreadyExistsError,
5253
)
53-
from pyiceberg.io import AWS_ACCESS_KEY_ID, AWS_PROFILE_NAME, AWS_REGION, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, FileIO
54+
from pyiceberg.io import (
55+
AWS_ACCESS_KEY_ID,
56+
AWS_ASSUME_ROLE_ARN,
57+
AWS_ASSUME_ROLE_SESSION_NAME,
58+
AWS_PROFILE_NAME,
59+
AWS_REGION,
60+
AWS_ROLE_ARN,
61+
AWS_ROLE_SESSION_NAME,
62+
AWS_SECRET_ACCESS_KEY,
63+
AWS_SESSION_TOKEN,
64+
FileIO,
65+
)
5466
from pyiceberg.partitioning import UNPARTITIONED_PARTITION_SPEC, PartitionSpec
5567
from pyiceberg.schema import Schema, SchemaVisitor, visit
5668
from pyiceberg.serializers import FromInputFile, ToOutputFile
@@ -131,6 +143,8 @@
131143
GLUE_ACCESS_KEY_ID = "glue.access-key-id"
132144
GLUE_SECRET_ACCESS_KEY = "glue.secret-access-key"
133145
GLUE_SESSION_TOKEN = "glue.session-token"
146+
GLUE_ROLE_ARN = "glue.role-arn"
147+
GLUE_ROLE_SESSION_NAME = "glue.role-session-name"
134148
GLUE_MAX_RETRIES = "glue.max-retries"
135149
GLUE_RETRY_MODE = "glue.retry-mode"
136150
GLUE_CONNECTION_S3_TABLES = "aws:s3tables"
@@ -349,14 +363,24 @@ def __init__(self, name: str, client: Optional["GlueClient"] = None, **propertie
349363
else:
350364
retry_mode_prop_value = get_first_property_value(properties, GLUE_RETRY_MODE)
351365

366+
region_name = get_first_property_value(properties, GLUE_REGION, AWS_REGION)
352367
session = boto3.Session(
353368
profile_name=get_first_property_value(properties, GLUE_PROFILE_NAME, AWS_PROFILE_NAME),
354-
region_name=get_first_property_value(properties, GLUE_REGION, AWS_REGION),
369+
region_name=region_name,
355370
botocore_session=properties.get(BOTOCORE_SESSION),
356371
aws_access_key_id=get_first_property_value(properties, GLUE_ACCESS_KEY_ID, AWS_ACCESS_KEY_ID),
357372
aws_secret_access_key=get_first_property_value(properties, GLUE_SECRET_ACCESS_KEY, AWS_SECRET_ACCESS_KEY),
358373
aws_session_token=get_first_property_value(properties, GLUE_SESSION_TOKEN, AWS_SESSION_TOKEN),
359374
)
375+
if role_arn := get_first_property_value(properties, GLUE_ROLE_ARN, AWS_ASSUME_ROLE_ARN, AWS_ROLE_ARN):
376+
session = _get_aws_session_with_assumed_role(
377+
session=session,
378+
role_arn=role_arn,
379+
role_session_name=get_first_property_value(
380+
properties, GLUE_ROLE_SESSION_NAME, AWS_ASSUME_ROLE_SESSION_NAME, AWS_ROLE_SESSION_NAME
381+
),
382+
region_name=region_name,
383+
)
360384
self.glue: GlueClient = session.client(
361385
"glue",
362386
endpoint_url=properties.get(GLUE_CATALOG_ENDPOINT),

0 commit comments

Comments
 (0)