Skip to content

Strip pandas metadata use schema types - #397

Open
sergiopeixoto-seequent wants to merge 5 commits into
SeequentEvo:mainfrom
sergiopeixoto-seequent:strip_pandas_metadata_use_schema_types
Open

sergiopeixoto-seequent wants to merge 5 commits into
SeequentEvo:mainfrom
sergiopeixoto-seequent:strip_pandas_metadata_use_schema_types

Conversation

@sergiopeixoto-seequent

Copy link
Copy Markdown
Contributor

Description

Parquet files included pandas reconstruction metadata that could cause consumers with incompatible pandas versions to fail when loading extension arrays. This made reading SDK tables dependent on pandas-specific metadata rather than the table’s declared schema.

Solution

Removed pandas reconstruction metadata when writing Parquet tables and updated DataFrame loading to derive nullable dtypes from the table schema. Added regression tests covering metadata removal and schema-based dtype restoration.

Checklist

  • I have read the contributing guide and the code of conduct

…chema types

Avoid pandas reconstruction metadata that can be incompatible across pandas versions. Load dataframes with nullable dtypes derived from the table schema.
@sergiopeixoto-seequent
sergiopeixoto-seequent requested a review from a team as a code owner November 4, 2026 08:17
@wordsworthc

Copy link
Copy Markdown
Contributor

I think this is closer to a reasonable solution, but not quite there yet.

I would place higher importance on omitting pandas metadata from parquet files produced by the geoscience object helpers. I haven't thought much about whether we need to strip any such metadata when consuming parquet files. I suppose in the spirit of this change, it is also a reasonable thing to do.

I think for converting pandas to pyarrow it could be cleaner to add to make add an ArrowTableFormat.pa_schema property and an ArrowTableFormat factory method from dataframe. For extra value we could also add a BaseTableFormat.columns property and make _ColumnFormat public too. The df -> table conversion would then be:

format = ArrowTableFormat.from_pandas(df)
table = pa.Table.from_pandas(df, schema=format.pa_schema, preserve_index=False)

The advantages here would be full control over dtype mappings in a common place that can be used for both import and export.

@sergiopeixoto-seequent

Copy link
Copy Markdown
Contributor Author

@wordsworthc

Thanks, I’ve added ArrowTableFormat.from_pandas() and pa_schema, exposed ColumnFormat and BaseTableFormat.columns, and updated the conversion helpers to use them.

One important detail: PyArrow still adds pandas metadata even with an explicit schema, so we still strip it when writing Parquet, preserving other metadata. Readers also ignore legacy pandas metadata. Category uploads retain dictionary encoding, and I added regression tests and clearer errors for malformed index metadata.

@wordsworthc wordsworthc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another step in the right direction. There are, however, some recurring issues with reusing private members (make them public if they need to be shared), code organization (readability counts), and unnecessary/duplicated validation. At a glance there may also be issues with support for environments where pandas is not installed (which we explicitly support), although I have not had a chance to verify this.

Please see the attached comments for a non-exhaustive list of issues I can see

- Centralize schema inference and dtype mapping in ArrowTableFormat.
- Use public conversion helpers and remove redundant validation and state.
- Strip pandas metadata during DataFrame export and before reconstruction.
- Preserve Arrow-only operation without pandas.
- Add regression coverage for legacy metadata, categories, and schema validation.
@sergiopeixoto-seequent

Copy link
Copy Markdown
Contributor Author

Guarded the pandas-dependent downhole exports and verified that Arrow/NumPy operations remain available without pandas.

Schema inference and dtype mapping now live on ArrowTableFormat, with public ColumnFormat and BaseTableFormat.columns. Shared conversion no longer requires private imports.

@wordsworthc wordsworthc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Sergio, looks a lot tidier now. One remaining comment regarding the readme, other than that I think we're ready to go 🎉

Comment thread packages/evo-objects/README.md Resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants