Skip to content

fix: honor write.object-storage.enabled in IcebergWriteExec - #121

Draft
smaheshwar-pltr wants to merge 3 commits into
apache:mainfrom
smaheshwar-pltr:design/core-table-generator
Draft

smaheshwar-pltr wants to merge 3 commits into
apache:mainfrom
smaheshwar-pltr:design/core-table-generator

Conversation

@smaheshwar-pltr

@smaheshwar-pltr smaheshwar-pltr commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Closes #122.

// Outline for now as requires apache/iceberg-rust#3730.

smaheshwar-pltr and others added 3 commits November 6, 2026 02:29
`IcebergWriteExec` always used `DefaultLocationGenerator`, so tables
with `write.object-storage.enabled=true` got the default
`{data}/{partition path}/{file}` layout instead of hashed paths, and
`write.object-storage.partitioned-paths=false` had no effect. Use
`TableLocationGenerator`, which picks the layout from the table
properties.

INSERT now fails when a `write.object-storage.*` property has an
invalid value, instead of ignoring it.
Move object-storage INSERT tests into the split test suite.

Generated-by: Codex
Compare each data file path with its expected layout, with the hash
directories and generated file name redacted, instead of rebuilding the
expected path with `ObjectStorageLocationGenerator`. Also check that
INSERT rejects an invalid `write.object-storage.enabled`.

Generated-by: Claude Code
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
.await?;

assert_eq!(
redacted_data_file_paths(&fixtures).await?,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With apache/iceberg-rust#3730, we could rely on Iceberg Rust behaviour and instead here just check that the file path matches what ObjectStorageLocationGenerator would yield by using it directly. I thought that being more explicit in the actual paths we expect might be nicer hence gone with this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

INSERT ignores write.object-storage.enabled

1 participant