Skip to content

feat(writer): add a location generator that follows table properties - #3730

Open
smaheshwar-pltr wants to merge 5 commits into
apache:mainfrom
smaheshwar-pltr:design/core-table-generator
Open

smaheshwar-pltr wants to merge 5 commits into
apache:mainfrom
smaheshwar-pltr:design/core-table-generator

Conversation

@smaheshwar-pltr

@smaheshwar-pltr smaheshwar-pltr commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

What changes are included in this PR?

See the issue description for details. This PR proposes adding TableLocationGenerator::new(&TableMetadata) to select the default or object-storage layout from table properties, and use it in the copy-on-write writer and writer examples.

We'd then use that also in Datafusion to respect the object storage property. apache/datafusion-iceberg#121 is how that might look.

Are these changes tested?

Yes, with table_location_follows_properties unit tests in crates/iceberg/src/writer/file_writer/location_generator.rs.

AI Disclosure

This PR was written by myself and Codex.

Writers that build a location generator from table metadata had to
choose between `DefaultLocationGenerator` and
`ObjectStorageLocationGenerator` themselves, or ignore
`write.object-storage.enabled`. Add `TableLocationGenerator`, which
makes that choice from the table properties, and use it in the
copy-on-write rewrite writer.

The layout is chosen internally, so more layouts can be added later
without breaking callers.
Generated-by: Codex
/// [`write.object-storage.enabled`](crate::spec::TableProperties::PROPERTY_WRITE_OBJECT_STORAGE_ENABLED)
/// is `true`, and [`DefaultLocationGenerator`] otherwise.
#[derive(Clone, Debug)]
pub struct TableLocationGenerator {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wrote up some context in #3729 on why I've gone with this approach but open to change / discussion if folks have thoughts. A new generator here lets us keep compatibility with current client behaviour, but the downside is that DefaultLocationGenerator reads naturally yet may not do what's desired.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IAlso, Java and Python support custom location providers so that is worth considering too w.r.t. the API design we introduce here. I think the proposed design should be able to support that by e.g. adding Custom(Arc<dyn <ErasedGenerator>) to the Layout enum)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think we should consider custom location provider here. enum seems like an acceptable workaround to me, but I don't want to block the custom location provider support for datafusion. Would appreciate more clarity here!

);
}

#[rstest]

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wary that I could be over-testing here (I did somehwat like how these tests read though)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a location generator that follows the table layout properties

2 participants