Backport #1185 and #1169 to 3.x

[laughingman7743]

WHAT

This PR backports three master fixes and one prerequisite commit to the 3.x maintenance branch, in master merge order.

Master change	3.x handling
#798, commit c61c361 only	Prerequisite of #1185, by maintainer decision. AsyncAdapt_pyathena_connection.cursor() takes an optional cursor class and maps Cursor to AioCursor. It takes no **kwargs, which came with #777 (not on 3.x). The rest of #798 builds on metadata work that 3.x does not have.
#1185 (119cc70, by @greg-om)	Cherry-picked with -x; the author is kept. do_ping() opens the API cursor with dbapi_connection.cursor(Cursor), because Connection._internal_cursor() comes from #1102 (a 4.0.0 breaking change). It has no @override, which 3.x does not have. The unit test uses the 3.x adapter names AsyncAdapt_pyathena_connection and AsyncAdapt_pyathena_cursor.
#1169 (4 commits)	0c6a0b2: only its own delta. _read_csv() opens the result file with fsspec.open() and PyAthena's storage options and passes the bytes to pl.read_csv(). The dev group adds the Polars 2 marker pair; uv.lock changes only polars and polars-runtime-32 (Python 3.14 resolves 2.0.0, older versions keep 1.36.1). The master tests it adjusts do not exist on 3.x. 6ec3e00 (docs sentence and test comments that 3.x does not have) is skipped as empty. a473e9c: docs/polars.md is unchanged for the same reason, and test_nested_type_description_unload runs without chunksize only (see below). 22de454 applies unchanged.
#1200 (3 commits)	Fixes #1199 in #1185's override. do_ping() passes DefaultDialect.do_ping() a _PingConnection proxy whose cursor() opens the API cursor and whose other attributes are the pinged connection's, so SQLAlchemy keeps its own handling of a failed ping. Before 2.0.5 that handling is inside do_ping() and calls is_disconnect() with the connection, which matters on 3.x for SQLAlchemy 1.4 as well. The proxy's cursor() uses self._connection.cursor(Cursor) instead of _internal_cursor(Cursor), and the test uses the 3.x adapter names and adds the OperationalError import.

Each commit message records its conflict resolution.

Release notes

• SQLAlchemy: with unload=true, the connection ping (pool_pre_ping, and connection tests such as Apache Superset's) no longer fails with MISSING_COLUMN_NAME. The ping runs SELECT 1 through a plain Cursor instead of the connection's default cursor, while SQLAlchemy keeps its own handling of a failed ping, including is_disconnect() before SQLAlchemy 2.0.5 (fix: run ping with plain cursor #1185 by @greg-om, reported in ping not working with UNLOAD=true #1189; SQLAlchemy ping no longer detects disconnects before SQLAlchemy 2.0.5 #1199/Let DefaultDialect.do_ping() run the ping with the API cursor #1200).
• PolarsCursor and AioPolarsCursor work with Polars 2.0 (PolarsCursor fails with Polars 2.0 #1168). Before, every query read from a CSV result file without chunksize failed with OperationalError (invalid value for 'connection'). The CSV result file is now read through PyAthena's S3 filesystem and passed to Polars as bytes, with Polars 1.x as well.
• Polars 2.0 reads an UNLOAD map column as Map, which the cursor description now reports as map. Before, it was string, or array/row when the map's values were lists or structs. With Polars 1.x, a map stays array. A struct containing a list is now described as row instead of array.
• Known limitation, unchanged: a chunked UNLOAD result of PolarsCursor describes its columns as the UNLOAD statement's rows bigint column. Master fixed this in Describe chunked Polars UNLOAD results with the Parquet schema #1012, which is not on 3.x.

WHY

#1185 fixes #1189 on master; 3.x has the same defect, because its dialect uses SQLAlchemy's default do_ping(), which opens the connection's default cursor with cursor_kwargs (including unload).
3.x's polars extra requires polars>=1.0.0 without an upper bound, so a fresh install of PyAthena[polars]<4 resolves Polars 2.0.0, and 3.x passed connection in the read_csv() storage options just as master did before #1169.
#1200 fixes #1199, a defect of #1185's override found by the independent review of this PR, and is needed with it. The maintainer chose #1185 and #1169 from the master changes after v3.39.0; the others are 4.x-only, performance work, or need prerequisites that 3.x does not have.

TEST

Tested commits: 90a9922 (head) for the #1200 port; a2e6899, before the #1200 commits, for the rest.

At 90a9922:

• just lint (Python 3.14): passed. tests/pyathena/sqlalchemy/test_base.py -k do_ping (offline): 4 passed, including test_do_ping_keeps_default_dialect_ping for the sync and async dialects.
• SQLAlchemy ping no longer detects disconnects before SQLAlchemy 2.0.5 #1199's reproductions (Python 3.11, uv run --isolated --with sqlalchemy==<version>), with an is_disconnect() override that reads connection.schema_name: the pre-ping reconnects with SQLAlchemy 1.4.54, 2.0.4, and 2.0.46. At a2e6899 it raised OperationalError with 1.4.54 and 2.0.4. With a handle_error listener instead, 2.0.4 and 2.0.46 reconnect. 1.4.54 raises on this branch and on the base alike, since 1.4's do_ping() does not consult the listeners. SQLAlchemy 1.3.24 cannot import pyathena.sqlalchemy on the base either (ImportError: InternalTraversal).
• Both test_ping_with_unload tests (live AWS, Python 3.14): 2 passed; do_ping() on a live awsathena+pandas engine with unload=true returns True.

At a2e6899:

• just lint (Python 3.14): passed.
• PyAthena Polars suites, live AWS, uv run --env-file .env pytest -n 4 tests/pyathena/polars tests/pyathena/aio/polars plus tests/pyathena/arrow/test_cursor.py::TestArrowCursor::test_complex_unload_as_polars:
  • Python 3.14 with Polars 2.0.0: 112 passed.
  • Python 3.13 with Polars 1.36.1: 110 passed and 1 skipped (test_get_athena_type_map, since Polars 1.x has no Map type), with test_iter_chunks failing in 2 of 2 runs.
    The failure is a panic inside Polars 1.36.1 (polars-io/src/file_cache/entry.rs:206, No such file or directory), which collect_batches() swallows, so the chunked scan_csv() read yields no chunks.
    The panic means the downloaded data file was gone right after the download. A likely cause is the cache shared by the xdist workers in the temporary directory: Polars 1.36.1 evicts an entry whose metadata file is missing or unreadable (EvictionCandidate::update_ttl() sets its TTL to 0), so one worker's eviction can delete a file that another is still writing. This was inferred from the Polars source, not traced in the run.
    This path (_iter_csv_chunks()) is not changed here. The test passes alone, the whole test_cursor.py passes serially (55 passed), and the same parallel run passes (105 passed) with the two new tests (test_util.py and test_nested_type_description_unload) deselected; the base origin/3.x passed the same parallel run 3 of 3 times.
    The release gate runs Python 3.10–3.13 with this Polars release and -n 8, so this race can fail a run there regardless of this PR.
• tests/pyathena/sqlalchemy and tests/pyathena/aio/sqlalchemy, live AWS, Python 3.14: 521 passed, including the new test_do_ping_uses_api_cursor_without_unload and both test_ping_with_unload tests.
• Reproduced on the base origin/3.x (5970145), Python 3.14:
  • With Polars 2.0.0 (uv run --with polars==2.0.0), TestPolarsCursor::test_fetchone fails with OperationalError: invalid value for 'connection': '<pyathena.connection.Connection object at ...>' (expected str).
  • AthenaDialect.do_ping() on an awsathena+pandas engine with unload=true raises OperationalError: MISSING_COLUMN_NAME: line 1:1: Column name not specified at position 1; on this branch it returns True.
• uv lock --check: passed.

Not run locally: the full just test pyathena suite, the SQLAlchemy compliance suites (just test sqla, just test sqla-async), and Python 3.10–3.12; the AWS CI of this PR and the release gate's full matrix cover them.

🤖 Generated with Claude Code

[laughingman7743]

Self-review round one (implementation behavior): CLEAN.

Scope: base 5970145, head a2e6899; every changed file (dialect do_ping(), the async adapter's cursor(), AthenaPolarsResultSet._read_csv(), get_athena_type(), the dev group and uv.lock, and the new and changed tests).

• Behavior and failure paths: do_ping() returns True or raises, like SQLAlchemy's default on 2.x. On 1.x the default also catches dbapi.Error for is_disconnect(), which this dialect does not override, so the error propagates there too. Hypothesis checked and rejected: a DataFrame cursor's arraysize above 1000 in cursor_kwargs making the API cursor raise. On 3.x, Connection.cursor() and AioConnection.cursor() ignore arraysize from cursor_kwargs for every cursor class (measured arraysize == 1000 for PandasCursor, Cursor, AioPandasCursor, and AioCursor with 5000 and 0), so the ping cannot fail where the default cursor succeeded. unload and chunksize are accepted and ignored by Cursor/AioCursor.
• The async adapter's cursor() with no argument passes None to AioConnection.cursor(), which picks the default cursor class as before; SQLAlchemy never passed arguments to it on 3.x, since it accepted none.
• Data and resource boundaries: _read_csv() keeps the output_location check and wraps fsspec.open() failures in OperationalError with the same log line. The bytes are read whole, as Polars 1.x did for an fsspec path. A storage_options key in the execute kwargs raised TypeError (duplicate keyword) before and now reaches pl.read_csv(); not a regression, and 3.x never supported it.
• get_athena_type(): Polars 1.x still describes an UNLOAD map (list of structs) as array; a struct containing a list changes from array to row, which is release-noted.
• Tests: test_do_ping_uses_api_cursor_without_unload fails on the base (the default do_ping() opens PandasCursor/AioPandasCursor, whose execute() is not patched). The live reproduction on the base and the branch is recorded in the PR body.

Out of scope, recorded only: with Polars 1.36.1, _iter_csv_chunks() can yield no chunks when Polars' file cache panics in a collect_batches() worker (see the PR body). That is pre-existing on 3.x and unaffected by this diff.

[laughingman7743]

Self-review round two (claims, callers, operations): FINDINGS, repaired in the PR body only; no code change.

Scope: base 5970145, head a2e6899; the PR body, the five commit messages, the changed comments and docstring, and docs/polars.md and docs/sqlalchemy.md on 3.x.

Finding (repaired): the release note said an UNLOAD map was described as string before and map now. With Polars 2.0 on the base, a map whose values are lists or structs was described as array or row, because str(dtype) is Map(String, List(Int32)) or Map(String, Struct({'a': Int32})) and the old substring checks matched it. The note now states all three previous results. The TEST section's account of the Polars 1.36.1 file-cache panic is narrowed to what was observed. The eviction explanation is marked as inferred from the Polars source (EvictionCandidate::update_ttl()), not traced in the run.

Claims checked and held:

• **kwargs and the adapter's cursor_kwargs/retry_config properties come from Fix metadata reflection throttling and reuse listed metadata #777 (commits 7932642, 922b295, and 9d68b59 all belong to Fix metadata reflection throttling and reuse listed metadata #777). Connection._internal_cursor() comes from BREAKING: Reject unknown cursor keyword arguments #1102 (09f7fb3). 3.x has no pyathena.util.override.
• uv.lock: only the polars and polars-runtime-32 entries change. Polars 2.0.0 requires Python >=3.10, so every Python version 3.x supports resolves it for a fresh PyAthena[polars]<4 install.
• The tests 0c6a0b2 adjusts do not exist on 3.x (TestAthenaPolarsResultSet, test_new_columns_with_schema_overrides, test_csv_read_kwargs_replace_defaults, test_txt_new_columns_with_schema_overrides, test_storage_options_replace_defaults: 0 matches).
• The chunked UNLOAD path (_create_dataframe_iterator() → _iter_parquet_chunks()) never calls _read_parquet_schema(), so it keeps the UNLOAD statement's rows metadata; Describe chunked Polars UNLOAD results with the Parquet schema #1012 ("Describe chunked Polars UNLOAD results with the Parquet schema") is the master fix.
• Existing callers: Apache Superset's connection test calls engine.dialect.do_ping(engine.raw_connection()) (superset/commands/database/utils.py). That passes a _ConnectionFairy, whose cursor(*args, **kwargs) forwards the cursor class to the DBAPI connection (SQLAlchemy 2.0.46, pool/base.py). SQLAlchemy 1.4's default do_ping() re-raises a non-disconnect error, and this dialect does not override is_disconnect(), so error propagation is unchanged.
• Operations: the 3.x release gate runs just tox → just test pyathena (pytest -n 8) on Python 3.10–3.14, so Python 3.10–3.13 keep Polars 1.36.1 as before.
• Docs: docs/polars.md still holds (PolarsCursor uses pl.read_csv/pl.read_parquet, and PyAthena's S3FileSystem for S3 access); docs/sqlalchemy.md does not mention pings.
• Evidence: every result in the TEST section was produced at a2e6899, including the base comparisons, which ran on 5970145.

[laughingman7743]

Independent review (relayed): Codex CLI 0.162.1, model gpt-6-astra, reasoning effort max, codex exec -s read-only, session 01a14263-33c1-7631-afb9-db0693f4b040. Static review only: no edits, builds, tests, GitHub, or network access. Snapshot: detached worktree at head a2e6899, base 5970145. The prompt gave the intended behavior without the PR number, description, commit messages, or prior findings. The snapshot and the PR worktree were unchanged afterwards.

Coverage reported: the full 11-file diff (sync/async ping, pool-proxy forwarding, cursor options, CSV error handling/resource release/memory/parser arguments, nested/map metadata, tests and fixtures, dependency/lock consistency), compared against cached SQLAlchemy 1.4.54, 2.0.0, and 2.0.46, and Polars 1.36.1 and 2.0.0.

Result: FINDINGS (1).

P2 — Preserve older SQLAlchemy disconnect handling — pyathena/sqlalchemy/base.py:293. Introduced by this range.
SQLAlchemy 1.4.54 and 2.0.0 handle recognized disconnects inside DefaultDialect.do_ping(). This override replaces that handling with an unconditional execute-and-propagate path.
Concrete scenario: SQLAlchemy 2.0.0, pool_pre_ping=True, a pooled connection with expired assumed-role credentials, and a handle_error listener marking the resulting DBAPI error as a disconnect. Previously, ping returned False, allowing the pool to reconnect and obtain fresh credentials. Now the exception escapes ping and checkout fails. SQLAlchemy 1.4 applications customizing is_disconnect() have the same regression.
The added success-path tests and locked SQLAlchemy 2.0.46 would miss this because that version handles errors outside do_ping().
No additional introduced defect identified in the Polars or lock changes. Compatibility at the earliest SQLAlchemy/Polars floors was not fully verified.

Author verification: confirmed. DefaultDialect.do_ping() in SQLAlchemy 1.4.54 and 2.0.0 catches dbapi.Error, consults is_disconnect() and (2.0.0) the handle_error events, and returns False for a disconnect. From 2.0.5 (rel_2_0_5; absent in 2.0.0–2.0.4), the pool calls _do_ping_w_event(), which wraps do_ping() and keeps that handling for an overriding dialect. This dialect does not override is_disconnect(), so the affected case is SQLAlchemy < 2.0.5 with a handle_error listener (or subclass) that marks the ping's error as a disconnect. The same override is on master (#1185, extra floor sqlalchemy>=2.0.0), so master is affected on 2.0.0–2.0.4. Repair pending a maintainer decision; this PR stays Draft.

[laughingman7743]

Repair: master fixed this finding in #1200 (issue #1199, merged as 1f6864a), and its three commits are ported here as cb34998, e5c4fc4, and 90a9922 (head 90a9922).

do_ping() now passes DefaultDialect.do_ping() a _PingConnection proxy. Its cursor() opens the API cursor with self._connection.cursor(Cursor) (3.x has no _internal_cursor()), and every other attribute is the pinged connection's. On SQLAlchemy 1.4 and 2.0.0–2.0.4, a failed ping goes through SQLAlchemy's own is_disconnect() handling again; from 2.0.5 it goes through _do_ping_w_event() as before.

Repair self-review (range a2e6899..90a9922, base 5970145):

• Implementation behavior: CLEAN. The async adapter's cursor(Cursor) maps to AioCursor (the ported Read the dialect's own queries through an API cursor and resolve federated absence from information_schema #798 commit). The proxy delegates schema_name, and the adapter's driver_connection (SQLAlchemy AdaptedConnection). On SQLAlchemy 1.4.54, an is_disconnect() override that reads connection.schema_name makes the pre-ping reconnect. At a2e6899 the same override raised OperationalError with 1.4.54 and 2.0.4. A handle_error-only listener behaves on 1.4.54 exactly as on the base (it raises), because 1.4's do_ping() does not consult listeners. The 3.x test adds only the OperationalError import and the 3.x adapter names.
• Claims: CLEAN. The PR body lists the Let DefaultDialect.do_ping() run the ping with the API cursor #1200 row, the extended release note, and which results ran at 90a9922 and which at a2e6899. SQLAlchemy 1.3.24 cannot import pyathena.sqlalchemy on the base either (ImportError: InternalTraversal), so 1.3 is not a supported runtime in practice; recorded, not changed here.

Validation at 90a9922: just lint passed; -k do_ping 4 passed; both live test_ping_with_unload tests passed.

An independent follow-up review of a2e6899..90a9922 is next.

[laughingman7743]

Independent follow-up review (relayed): Codex CLI 0.162.1, model gpt-6-astra, reasoning effort max, codex exec -s read-only, session 01a14530-25e6-71e0-86a7-0423d1949335. Static review only. Snapshot: detached worktree at 90a9922; ranges a2e68991..90a9922f and 59701452..90a9922f. The snapshot and the PR worktree were unchanged afterwards.

Coverage reported: sync/async execution, pool-proxied direct calls, delegation and recursion, DataFrame cursor_kwargs, cursor cleanup, typing, and the regression tests, against cached SQLAlchemy 1.4.54, 2.0.0, 2.0.4, 2.0.5, and 2.1.1.

Result: CLEAN.

The previous finding is resolved by delegating to the parent (pyathena/sqlalchemy/base.py:324). SQLAlchemy 1.4: restores the parent's is_disconnect() handling and False return for recognized disconnects. 2.0.0–2.0.4: also restores the handle_error listener processing. 2.0.5+: preserves the pool's outer _do_ping_w_event() handling, which receives the original connection.
The proxy forwards cursor(Cursor) through pool proxies without re-entering the dialect or checking out another connection; the async adapter maps it to AioCursor. DataFrame options such as unload, engine, and chunksize do not activate DataFrame behavior in the API cursor. The guarded type import and string-form cast introduce no SQLAlchemy 1.4 runtime dependency.
The new regression test distinguishes the revision: both cases would propagate the injected OperationalError on the previous head instead of returning False.
Direct do_ping() calls on 2.0.5+ keep upstream semantics (errors propagate; the pool's outer handler classifies disconnects). Not a regression.

The independent review of this PR is complete at 90a9922.