[laughingman7743]

WHAT

Removes the Polars 1.x whole-read pyarrow fallback of #1233. On Polars 1.x, a whole UNLOAD read without Polars read options read a result with timestamps through pyarrow and PyAthena's S3FileSystem, because Polars 1.x reads INT96 at nanoseconds whatever the read schema. Polars 1.x now reads the result with pl.read_parquet() in every case: the timestamps are read at nanoseconds, cast to microseconds, and the warning says that values outside 1677-09-21 to 2262-04-11 are wrong and recommends Polars 2.0 or later. The chunked read through ParquetChunkReader (with pyarrow installed and no read options) stays, as it bounds memory use; its warning still mentions that read as a remedy.

_read_parquet_with_pyarrow() and _reads_with_pyarrow() are removed (the chunked read's condition is inlined at its only caller), and _warn_nanosecond_read() logs one message for the whole read and the chunked scan, naming both remedies: Polars 2.0 or later, or a chunked read with pyarrow installed and no read options. docs/polars.md no longer lists this read among the reads through PyAthena's filesystem, and says that the chunked read with pyarrow has neither the cast nor the warning.

Tests: test_unload_timestamps_polars_1 (renamed from test_unload_timestamps_with_read_options_polars_1, existing IDs kept) gains the whole read without options as whole-no-options, and asserts the warning's consequence; the whole case of test_unload_nested_timestamps and the unload case of test_temporal_types are skipped on Polars 1.x, as their years outside 1677-2262 are now out of its reach, like the scanned chunks before; test_unload_nested_timestamps_polars_1 asserts the cast of timestamps inside arrays, rows, and maps with in-range values on Polars 1.x.

Nothing of this is released (#1233 is in 4.2.0), so there is no compatibility consequence. The 4.2.0 release note on UNLOAD timestamps should say: Polars 2.0 or later reads them at microseconds; Polars 1.x reads them at nanoseconds and casts them, with wrong values outside 1677-09-21 to 2262-04-11 and a warning, except for chunked reads with pyarrow and no read options.

WHY

Closes #1280. PyAthena does not switch readers on the user's behalf to work around a dependency's limitation (decided on #1235).

TEST

Tested at 9aa7419; 259cdb33 adds test_unload_nested_timestamps_polars_1 only, which passed on Polars 1.44.2 and 1.39.0 (and is skipped on 2.0.0), and fails (2/2) with the nested cast disabled.

• just lint, just docs lint: passed.
• uv run --env-file .env pytest -n 4 tests/pyathena/polars tests/pyathena/aio/polars (locked environment: Python 3.15, Polars 2.0.0): 179 passed, 4 skipped.
• The same suites with --python 3.13 --isolated --with polars==1.44.2 and --with polars==1.39.0: 177 passed, 6 skipped on each.
• The same suites with --python 3.14 --isolated (Polars 2.0.0, pyarrow 25.0.1) at 259cdb3: 179 passed, 6 skipped (the Polars 1.x-only cases).
• Mutation check on Polars 1.44.2: with the source change reverted (fallback restored), test_unload_timestamps_polars_1[whole-no-options] fails on the warning's consequence, and the other cases pass.
• The --with polars==1.x overlay on the Python 3.15 environment, whose base holds Polars 2.0.0, gives unrelated AttributeError: 'NoneType' object has no attribute '_s' errors at collection; the isolated Python 3.13 runs above avoid that.

🤖 Generated with Claude Code

[laughingman7743]

Self-review round 1 (behavior and implementation), base d7f6710f5b7a8e2bcb736f840eac13d102e87b7c, head 9aa7419dbe963c0c5cc790d63a471c9e9e219cf4 (findings recorded against 6e1e25f4ec37c296d47d6ae294f3761f940bafd8, repaired in 9aa7419d).

Covered: pyathena/polars/result_set.py _read_parquet() (the removed pyarrow branch, the native read, cast, and warning for Polars 1.x), the removed _read_parquet_with_pyarrow() and _reads_with_pyarrow(), _iter_parquet_chunks() / _read_parquet_chunks() / _scan_parquet_chunks() (the chunked pyarrow read stays), _warn_nanosecond_read(), the docstrings of _int96_read_options() and _output_dtypes(), the aio Polars cursors (same result set), docs/polars.md (the filesystem reads, the UNLOAD timestamps section, the S3 read options section), and the three changed tests. Live matrix on Polars 2.0.0 (locked, Python 3.15), 1.44.2, and 1.39.0 (Python 3.13). Also ran /code-review high on the PR (7 candidates).

Result: FINDINGS, repaired in 9aa7419d:

• R1-1 (code-review) _warn_nanosecond_read(): the whole read and the chunked scan logged different remedies, and the whole read's omitted the chunked read with pyarrow that docs/polars.md describes as reading the timestamps correctly. One message now names both remedies, assembled in one place.
• R1-2 (code-review) tests/pyathena/polars/test_cursor.py: the new whole-read case had taken the existing ID whole, against the rule to preserve test IDs. The existing cases keep their IDs and the new case is whole-no-options; the mutation check was repeated (fails with the fallback restored).
• R1-3 (code-review) _reads_with_pyarrow() had one caller left and a docstring that said "in chunks" without checking chunksize; inlined at the caller in _iter_parquet_chunks().
• R1-4 (code-review) docs/polars.md did not say that the chunked read with pyarrow has neither the cast nor the warning; a sentence says so.

[laughingman7743]

Round 1, code-review candidates checked and not changed:

• test_temporal_types[unload] is skipped as a whole on Polars 1.x rather than per column: the query's years outside 1677-2262 are out of Polars 1.x's reach by design, and a per-column exception would write Polars 1.x's defect into the contract test. CI runs Polars 2.0 (the dev group pins polars>=2.0.0 on Python 3.14+ and the Test workflow runs 3.15), so the UNLOAD coverage of dates and text fallbacks stays in CI; test_unload_timestamps_polars_1 covers the Polars 1.x UNLOAD read with in-range timestamps.
• test_unload_nested_timestamps[chunks] has no pyarrow mark and would fail on Polars 1.x without pyarrow: pre-existing, and pyarrow is in the dev dependency group, so the test environment always has it.

[laughingman7743]

Self-review round 2 (claims, callers, operations), base d7f6710f5b7a8e2bcb736f840eac13d102e87b7c, head 9aa7419dbe963c0c5cc790d63a471c9e9e219cf4.

Claims checked against evidence:

• "Nothing of this is released": git tag --contains 4ac04db3 (the Read UNLOAD timestamps at microseconds #1233 merge) lists no tag; the latest tag is v4.1.0.
• "CI runs Polars 2.0": the dev group pins polars>=2.0.0; python_version >= '3.14' and polars<2.0.0; python_version < '3.14' (pyproject.toml), and the Draft run of this PR ran test / run (3.15). The Polars 1.x paths are therefore exercised only by the local matrix recorded in the PR body (1.44.2 and 1.39.0 on Python 3.13).
• The warning's remedy "read the result in chunks with pyarrow installed and no read options": _iter_parquet_chunks() takes the ParquetChunkReader path exactly under that condition, and the reader opens files with coerce_int96_timestamp_unit="us" (pyathena/arrow/reader.py:169), so the chunks are at microseconds without the cast or the warning, as docs/polars.md now says.
• Existing callers on Polars 1.x: a whole UNLOAD read without options now goes through pl.read_parquet() with the frozen-credential storage options, as every other Polars read does, instead of PyAthena's S3FileSystem; the output dtypes are unchanged. Live check on Polars 1.44.2 with temporal.nested_query(): the schema is Datetime("us") at every nesting level (top level, List, Struct, List of key-value Structs, nested), and the warning above is logged once. The values of years 1, 1600, and 9999 are wrong there, as documented; in-range values are asserted by test_unload_timestamps_polars_1[whole-no-options].
• Existing callers on Polars 2.0: unaffected (_int96_read_options() returns the read schema and no warning is logged); the locked suites pass (179 passed, 4 skipped).
• AWS operations: the Polars 1.x whole read no longer makes the S3 requests of PyAthena's filesystem (one HeadObject/ranged GetObject set per file); Polars' object store reads the files instead, as for Polars 2.0. No retries or extra requests added.
• Documentation: the three changed passages of docs/polars.md were walked through against _read_parquet(), _iter_parquet_chunks(), and _csv_storage_options; the reference {ref}polars-unload-options on line 14 still resolves to the rewritten section.
• Evidence: all results in the PR body were obtained at 9aa7419d (lint, docs lint, the suites on 2.0.0/1.44.2/1.39.0, and the mutation check of [whole-no-options]), and the skipped cases are listed with their reasons.

Result: CLEAN within this scope. One limit: the --with polars==1.x overlay on the Python 3.15 environment (whose base holds Polars 2.0.0) is unusable for the matrix; the isolated Python 3.13 runs replace it, as the PR body records.

[laughingman7743]

Independent review (relayed), base d7f6710f5b7a8e2bcb736f840eac13d102e87b7c, head 9aa7419dbe963c0c5cc790d63a471c9e9e219cf4.

Reviewer: Codex CLI 0.163.0, model gpt-6-astra, reasoning effort max, session 01a14e45-c79d-75b3-9301-7561bc25f209, read-only sandbox, stdin closed, in a detached checkout of the head. The package held the literal diff, AGENTS.md, and the testing guide's "Choose what to test" section; no PR number, PR body, commit messages, prior findings, or agent memory. Static review only: no builds, tests, or GitHub access. The snapshot and the PR worktree were unchanged afterwards.

Covered (reviewer's list): the diff; whole and chunked reads across Polars 1.x/2.0, optional pyarrow, and read options; scalar and nested timestamps, schema overrides, warnings, manifest selection, and error propagation through the sync, async, and aio cursors; the changed and surrounding documentation, temporal fixtures, test coverage and skips, and removed-path references and imports.

Result: FINDINGS (4), verified by the author:

1. "Whole reads can omit partitions" (_prepare_parquet_location() derives the unload location from the first manifest entry when the formatter gave none; a user-written partitioned UNLOAD would lose other partitions; before, the Polars 1.x pyarrow fallback read every manifest entry). Rejected as a regression: the derivation is the existing logic of this result set, of ArrowResultSet._read_parquet() (pyathena/arrow/result_set.py:396), and of the Polars 2.0 and read-option paths; PyAthena's unload option wraps a SELECT in its own UNLOAD without partitioned_by, so the scenario is not a supported one, and this PR makes Polars 1.x read as Polars 2.0 does.
2. schema=None given to execute() disables the timestamp handling and the warning ("schema" in self._kwargs). Pre-existing, outside this PR's scope; reported to the maintainer for a decision.
3. use_pyarrow=True on Polars 2.0 fails on the schema= that _int96_read_options() adds. Pre-existing and known: recorded in Pre-existing UNLOAD read defects found in the #1214 review #1235 as reachable only once read_parquet(use_pyarrow=True) cannot read a remote directory or a list of remote files pola-rs/polars#30038 is fixed, since the prefix read fails first.
4. The skips left the Polars 1.x native cast of nested timestamps without a test with in-range values. Accepted: test_unload_nested_timestamps_polars_1 (whole and scanned chunks) asserts Datetime("us") inside arrays, rows, and maps and the values of a 2020 timestamp and NULL; with the nested cast disabled both cases fail (2/2). Repaired in 259cdb33. The repair was checked under both self-review perspectives: it adds a test only, its docstring claim (values within 1677-2262) matches the query, and it passes on Polars 1.44.2 and 1.39.0 and is skipped on 2.0.0. The loss of the UNLOAD date-boundary coverage on Polars 1.x from the test_temporal_types[unload] skip stays as recorded in round 1 (CI runs Polars 2.0).

[laughingman7743]

Independent follow-up (relayed) on the repair 9aa7419dbe963c0c5cc790d63a471c9e9e219cf4..259cdb334756717b4ec82c108ba606811707596f (the added test only). Reviewer: Codex CLI 0.163.0, model gpt-6-astra, reasoning effort max, session 01a14e53-bde4-74c0-9b61-da2efc9bac23, read-only sandbox, detached checkout of 259cdb33, given the literal patch and the testing guide's section; static review only.

Result: CLEAN. Reviewer's notes: both cases exercise the native Polars 1.x cast (low_memory=True keeps the scan off the pyarrow chunk reader); the nested schema assertions fail if _cast() or the recursive to_microseconds() is removed, even where the Python values would still compare equal; the value assertions also check the scaling and NULLs; the skip condition and docstring are correct; the existing nested coverage skips native Polars 1.x reads, so the test adds distinct evidence.