Skip to content

PolarsCursor fails with the deprecated dtypes reader option #1111

Closed

Description

@laughingman7743

Problem

PolarsCursor.execute(..., dtypes={...}) fails when the result is read as CSV.
The result set always passes its own schema_overrides (the column types from Athena's metadata) to polars.read_csv() / polars.scan_csv(), and Polars rejects receiving both the deprecated dtypes name and schema_overrides:

TypeError: `'read_csv'` received both `'dtypes'` and `'schema_overrides'` as arguments; `'dtypes'` was deprecated in version 0.20.31, use `'schema_overrides'` instead

PyAthena raises it as OperationalError after the query has run.
schema_overrides= works, so the only effect is on callers still using the alias deprecated since Polars 0.20.31 (PyAthena requires polars>=1.39.0).

The other code paths that special-case schema_overrides given to execute() (_get_timestamp_dtypes(), the new_columns handling in _read_csv() and the chunked CSV reader in pyathena/polars/result_set.py) do not recognize dtypes either.

Options

  1. Treat dtypes as schema_overrides in the result set (drop the default schema_overrides and apply the same new_columns/timestamp rules).
  2. Document schema_overrides as the supported name and leave the deprecated alias to Polars.

Found while reviewing #1102.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions