Repository navigation
REST: Allow disabling remote signing with s3.remote-signing-enabled - #4134
Ankitraj-builds wants to merge 2 commits into
Conversation
| metadata_location = "s3://warehouse/database/table/metadata/00001.metadata.json" | ||
| rest_mock.get( | ||
| f"{TEST_URI}v1/namespaces/fokko/tables/table", | ||
| json={ | ||
| "metadata-location": metadata_location, | ||
| "metadata": example_table_metadata_with_snapshot_v1, | ||
| "config": {"s3.signer": "S3V4RestSigner", "s3.signer.uri": "http://signer"}, | ||
| }, | ||
| status_code=200, | ||
| request_headers=TEST_HEADERS, | ||
| ) | ||
| catalog = RestCatalog("rest", uri=TEST_URI, token=TEST_TOKEN, **{"s3.signer": ""}) | ||
| table = catalog.load_table(("fokko", "table")) |
There was a problem hiding this comment.
This seems akward to me. The catalog should pass valid information. @c-thiel what is Lakekeeper doing here?
There was a problem hiding this comment.
Thanks @Fokko. This was on the first version, which used an empty s3.signer. That is replaced now: users set s3.remote-signing-enabled=false instead (rambleraptor suggested this), so an empty value has no special meaning anymore.
The use case from #3888: some users already have their own S3 credentials, but the REST catalog sends s3.signer to every client. They then can’t turn signing off, and every S3 request makes an extra trip to the signer. Java uses the same s3.remote-signing-enabled property, and it’s also listed in #4140.
Happy to change the approach if you’d prefer something else.
rambleraptor
left a comment
There was a problem hiding this comment.
I've got a quick question.
Closes #3888
Rationale for this change
REST catalogs like Lakekeeper return remote signing settings (
s3.signer,s3.signer.uri) in the table response config. Since #3956 the FileIO property order is table metadata < catalog config < server config < vended credentials, so the server's signer always wins. A client that already has its own S3 credentials cannot turn remote signing off, and every S3 request pays an extra round trip to the signer.This adds a catalog property,
s3.remote-signing-enabled. When the catalog config sets it tofalse,RestCatalog._load_file_io()removess3.signerfrom the merged FileIO properties, so remote signing is off even if the server config sets a signer. When it is not set ortrue, nothing changes. This coversTable,StagedTableand scan planning FileIOs.Based on review feedback, this replaces the first version, which used an empty
s3.signeras the opt-out.Are these changes tested?
Yes, in
tests/catalog/test_rest.py:test_load_table_remote_signing_disabled_in_catalog_config: the server returnss3.signer, the catalog setss3.remote-signing-enabled=false, and the table FileIO has nos3.signer. This test fails onmain.test_load_table_server_signer_kept_unless_disabled: with the property not set, or set totrue, the server signer is kept.All 177 tests in
tests/catalog/test_rest.pypass locally, and lint passes.Are there any user-facing changes?
Yes. New REST catalog property
s3.remote-signing-enabled(defaulttrue), documented inconfiguration.md. Nothing changes for users who do not set it.