Repository navigation
feat(writer): add a location generator that follows table properties - #3730
smaheshwar-pltr wants to merge 5 commits into
Conversation
Writers that build a location generator from table metadata had to choose between `DefaultLocationGenerator` and `ObjectStorageLocationGenerator` themselves, or ignore `write.object-storage.enabled`. Add `TableLocationGenerator`, which makes that choice from the table properties, and use it in the copy-on-write rewrite writer. The layout is chosen internally, so more layouts can be added later without breaking callers.
Generated-by: Codex
Generated-by: Codex
Generated-by: Codex
| /// [`write.object-storage.enabled`](crate::spec::TableProperties::PROPERTY_WRITE_OBJECT_STORAGE_ENABLED) | ||
| /// is `true`, and [`DefaultLocationGenerator`] otherwise. | ||
| #[derive(Clone, Debug)] | ||
| pub struct TableLocationGenerator { |
There was a problem hiding this comment.
I wrote up some context in #3729 on why I've gone with this approach but open to change / discussion if folks have thoughts. A new generator here lets us keep compatibility with current client behaviour, but the downside is that DefaultLocationGenerator reads naturally yet may not do what's desired.
There was a problem hiding this comment.
IAlso, Java and Python support custom location providers so that is worth considering too w.r.t. the API design we introduce here. I think the proposed design should be able to support that by e.g. adding Custom(Arc<dyn <ErasedGenerator>) to the Layout enum)
There was a problem hiding this comment.
Yes, I think we should consider custom location provider here. enum seems like an acceptable workaround to me, but I don't want to block the custom location provider support for datafusion. Would appreciate more clarity here!
| ); | ||
| } | ||
|
|
||
| #[rstest] |
There was a problem hiding this comment.
I'm wary that I could be over-testing here (I did somehwat like how these tests read though)
Which issue does this PR close?
What changes are included in this PR?
See the issue description for details. This PR proposes adding
TableLocationGenerator::new(&TableMetadata)to select the default or object-storage layout from table properties, and use it in the copy-on-write writer and writer examples.We'd then use that also in Datafusion to respect the object storage property. apache/datafusion-iceberg#121 is how that might look.
Are these changes tested?
Yes, with
table_location_follows_propertiesunit tests incrates/iceberg/src/writer/file_writer/location_generator.rs.AI Disclosure
This PR was written by myself and Codex.