Repository navigation
Strip pandas metadata use schema types - #397
sergiopeixoto-seequent wants to merge 5 commits into
Conversation
…chema types Avoid pandas reconstruction metadata that can be incompatible across pandas versions. Load dataframes with nullable dtypes derived from the table schema.
|
I think this is closer to a reasonable solution, but not quite there yet. I would place higher importance on omitting pandas metadata from parquet files produced by the geoscience object helpers. I haven't thought much about whether we need to strip any such metadata when consuming parquet files. I suppose in the spirit of this change, it is also a reasonable thing to do. I think for converting pandas to pyarrow it could be cleaner to add to make add an format = ArrowTableFormat.from_pandas(df)
table = pa.Table.from_pandas(df, schema=format.pa_schema, preserve_index=False)The advantages here would be full control over dtype mappings in a common place that can be used for both import and export. |
|
Thanks, I’ve added ArrowTableFormat.from_pandas() and pa_schema, exposed ColumnFormat and BaseTableFormat.columns, and updated the conversion helpers to use them. One important detail: PyArrow still adds pandas metadata even with an explicit schema, so we still strip it when writing Parquet, preserving other metadata. Readers also ignore legacy pandas metadata. Category uploads retain dictionary encoding, and I added regression tests and clearer errors for malformed index metadata. |
wordsworthc
left a comment
There was a problem hiding this comment.
Another step in the right direction. There are, however, some recurring issues with reusing private members (make them public if they need to be shared), code organization (readability counts), and unnecessary/duplicated validation. At a glance there may also be issues with support for environments where pandas is not installed (which we explicitly support), although I have not had a chance to verify this.
Please see the attached comments for a non-exhaustive list of issues I can see
- Centralize schema inference and dtype mapping in ArrowTableFormat. - Use public conversion helpers and remove redundant validation and state. - Strip pandas metadata during DataFrame export and before reconstruction. - Preserve Arrow-only operation without pandas. - Add regression coverage for legacy metadata, categories, and schema validation.
|
Guarded the pandas-dependent downhole exports and verified that Arrow/NumPy operations remain available without pandas. Schema inference and dtype mapping now live on ArrowTableFormat, with public ColumnFormat and BaseTableFormat.columns. Shared conversion no longer requires private imports. |
wordsworthc
left a comment
There was a problem hiding this comment.
Thanks Sergio, looks a lot tidier now. One remaining comment regarding the readme, other than that I think we're ready to go 🎉
Description
Parquet files included pandas reconstruction metadata that could cause consumers with incompatible pandas versions to fail when loading extension arrays. This made reading SDK tables dependent on pandas-specific metadata rather than the table’s declared schema.
Solution
Removed pandas reconstruction metadata when writing Parquet tables and updated DataFrame loading to derive nullable dtypes from the table schema. Added regression tests covering metadata removal and schema-based dtype restoration.
Checklist