Read UNLOAD timestamps at microseconds by laughingman7743 · Pull Request #1233 · pyathena-dev/PyAthena

laughingman7743 · 2026-10-17T03:27:05Z

WHAT

UNLOAD results now read Athena's timestamps at microseconds in PandasCursor, ArrowCursor, and PolarsCursor, at every nesting level (array, row, map) and in whole and chunked reads. Values outside 1677-09-21 to 2262-04-11 are no longer wrapped or saturated, and the dtype is datetime64[us], timestamp[us], or Datetime("us"), as for CSV results.

PandasCursor / ArrowCursor: pyarrow reads INT96 with coerce_int96_timestamp_unit="us". For PandasCursor, coerce_int96_timestamp_unit given to execute() replaces it, as other pandas.read_parquet() options do. ParquetChunkReader, which the chunked reads of PandasCursor and PolarsCursor use, does the same.
PolarsCursor with Polars 2.0 and later: the read schema is the file schema with Datetime("us"), which makes Polars decode INT96 at microseconds. Polars 2.0 still saturates INT96 values inside a Map (reported as INT96 timestamps inside a Map saturate although the schema requests Datetime("us") pola-rs/polars#30031), so maps that hold timestamps are read as lists of key-value structs and cast back to Map. The casts work by column name, as pl.col() would read names such as "*" as selectors. A schema given to execute() is used as given.
PolarsCursor with Polars 1.x: Polars 1.x decodes INT96 only at nanoseconds (1.39.0 wraps, 1.44.2 saturates).
- Without read options given to execute() and with pyarrow installed, a whole read of a result with timestamps goes through pyarrow and PyAthena's S3FileSystem, and logs a warning that recommends Polars 2.0.
- Chunked reads without read options already use ParquetChunkReader.
- With use_pyarrow=True, Polars reads with pyarrow, and coerce_int96_timestamp_unit="us" is added to pyarrow_options unless they set it.
- Otherwise (no pyarrow, or other read options such as storage_options or columns), Polars 1.x reads natively as before. The timestamps are cast to us, and a warning says that values outside 1677-09-21 to 2262-04-11 are wrong. It is logged only when the columns read hold timestamps.
- Results without timestamps read as before.
The tests query Athena through each cursor's public API. tests/pyathena/temporal.nested_query() UNLOADs years 1, 1600, and 9999 and NULL:
- at the top level;
- in an array, a row, and a map named "*";
- in a map in a row in an array.
The tests check the microsecond types first, then the values, in ArrowCursor, PandasCursor, and PolarsCursor, whole and chunked. They also check the documented options: PandasCursor's coerce_int96_timestamp_unit, PolarsCursor's schema on Polars 2.0, and PolarsCursor's read options and warning on Polars 1.x.
The two UNLOAD dtype expectations in tests/pyathena/arrow/test_cursor.py change from ns to us. The pandas, Arrow, and Polars docs state the microsecond unit and the Polars 1.x behavior.

Years outside 1 to 9999 are out of scope (#1212, contract item 3); they pass through as pandas, pyarrow, or Polars return them.

The tests of #1226: test_temporal_types[unload] now passes for ArrowCursor and PolarsCursor, so their #1214 xfails are removed. PandasCursor's stays an xfail, for #1219 only.

Closes #1214

WHY

#1214 (part of #1212): UNLOAD writes TIMESTAMP(3) as INT96, and the readers decoded it at nanoseconds, so timestamps outside the nanosecond range came back as silently wrong values (1600-01-02 03:04:05.123 → 2184-07-22 02:38:38.832551616, or 2262-04-11 23:47:16.854775807 with Polars). Under #1212, UNLOAD results also need the same us dtype as CSV results; that dtype work moved from #1219 to #1214.

Measured facts (2026-10-17):

A live UNLOAD writes INT96 at the top level and inside arrays, rows, and maps, with no ARROW:schema metadata.
Polars 2.0.0 decodes INT96 at the unit of the read schema since fix: Respect Parquet schema when reading Int96 timestamps pola-rs/polars#29208, but not inside a Map. That upstream int96_options() has no Map case, and it is unchanged on main.

Behavior changes for the 4.2.0 release notes:

UNLOAD timestamp dtypes change from ns to us in all three cursors.
Timestamps outside 1677-2262 are now correct.
With Polars 1.x, whole reads of UNLOAD results with timestamps go through pyarrow when no read options are given and pyarrow is installed. These reads are slower and use more memory (see TEST), and log a warning. Other Polars 1.x reads keep the nanosecond values, cast to us, and log a warning that values outside 1677-2262 are wrong.
The private AthenaPolarsResultSet._read_parquet(), _read_parquet_schema(), and _iter_parquet_chunks() change their signatures; _read_parquet_schema() returns the Polars schema.

TEST

Head 1dc9b28, on master 7e5d9ee (which includes #1226, #1231, #1234, and #1236).

just lint: passed. just docs lint: 0 errors.
Live AWS, new tests (test_unload_nested_timestamps, test_unload_int96_unit_option, test_unload_schema_option, test_unload_timestamps_with_read_options_polars_1):
- Python 3.15 / Polars 2.0.0: 9 passed, 3 skipped (the Polars 1.x cases).
- Python 3.13 / Polars 1.44.2: 9 passed, 3 skipped (the Polars 2.0 cases).
- On master, test_unload_nested_timestamps fails in all 6 cases. The precedence tests pass on master by design: they guard that the new defaults do not override user options.
Live AWS at 043e295, pytest -n 4 on tests/pyathena/{arrow,polars,pandas}/ and tests/pyathena/aio/{arrow,pandas,polars}, Python 3.15 / Polars 2.0.0: 986 passed, 11 skipped, 2 xfailed (Make PandasCursor's temporal dtypes and fetched types depend only on the Athena type #1219). On Python 3.13 / Polars 1.44.2, the Polars suites plus the Arrow and pandas unload and temporal tests: 55 passed, 3 skipped, 2 xfailed.
test_temporal_types[unload] (Test the temporal types of each synchronous cursor #1226) passes for ArrowCursor and PolarsCursor without its #1214 xfail. PandasCursor's stays an xfail for #1219: with --runxfail, only the date columns differ.
Offline, the touched test_result_set.py files on Python 3.15 / Polars 2.0.0: 368 passed, 10 skipped.
The Python 3.11-3.13 (Polars 1.x) AWS jobs run in the Release workflow and on dispatch, not on pull requests.

Performance: a local 5M-row file (INT96 timestamps and strings), median of 5 reads, elapsed time / peak RSS, range over 3 alternating runs:

Path	Before	After
Polars 2.0 whole	21-45 ms / 364-378 MiB	21-35 ms / 375-385 MiB
ArrowCursor	33-38 ms / 449-466 MiB	33-34 ms / 466-482 MiB
PandasCursor whole	36-40 ms / 592-597 MiB	35-42 ms / 538-599 MiB
Polars 1.x whole, with timestamps (native → pyarrow)	17-19 ms / 339-354 MiB	60-78 ms / 593-671 MiB

The Polars 1.x row is the pyarrow read of a whole result without read options. It is about 4x slower and uses about 1.8x memory, and its warning recommends Polars 2.0. Polars 1.x reads with read options stay native.

Not run:
- The full just test pyathena. AWS CI runs it when the PR is marked Ready.
- S3-backed timing of the Polars 1.x fallback, which reads through PyAthena's S3FileSystem instead of Polars' object_store.

Pre-existing defects found in review, not changed here, filed as #1235:

With use_pyarrow=True, Polars opens an UNLOAD prefix on S3 as one file and fails.
pl.read_parquet()-only options (columns, use_pyarrow, pyarrow_options) break chunked scans.
A manifest body read that times out escapes OperationalError (pyathena/result_set.py).
At the minimum versions (pyarrow 22.0.0, Polars 1.39.0, pandas 3.0.0), 5 cases of test_read_csv_with_pyarrow_matches_pandas fail, as on master.

🤖 Generated with Claude Code

laughingman7743 · 2026-10-17T03:30:49Z

            _logger.exception(f"Failed to read schema from {self._unload_location}.")
            raise OperationalError(*e.args) from e

+    def _int96_read_options(self, schema: pl.Schema) -> dict[str, Any] | None:


Self-review round 1 (behavior, data boundaries, simplicity, tests). Base 18c1eea, head 1b1161c.

Covered: every changed file; the pandas whole read (read_parquet kwargs precedence, filesystem/storage_options branches) and chunk read (ParquetChunkReader, _PARQUET_CHUNK_OPTIONS keeps a user coerce_int96_timestamp_unit on the whole read); ArrowCursor _read_parquet and as_polars(); PolarsCursor whole, chunked (ParquetChunkReader and scan_parquet), metadata (_read_parquet_schema now read once, before the data), user schema, use_pyarrow/pyarrow_options precedence, Polars 1.x fallback and NotSupportedError; async/aio cursors share these result sets (live aio suites ran).

Result: FINDINGS, two, repaired in 3a675e2 (see the threads below). Checked and not defects: a use_pyarrow kwarg on a scanned chunk read still fails as before (scan_parquet has no use_pyarrow); schema together with use_pyarrow=True is rejected by Polars, which is why use_pyarrow gets pyarrow_options instead of a schema.

laughingman7743 · 2026-10-17T03:30:49Z

+                coerce_int96_timestamp_unit="us",
+            ).read(use_threads=True)
+            # Not rechunked, as Polars does not rechunk its own reads.
+            return cast("pl.DataFrame", pl.from_arrow(table, rechunk=False))


Round 1 finding (repaired in 3a675e2): pl.from_arrow(table) rechunks by default, adding a copy of the whole result to the peak memory of the Polars 1.x fallback. Measured on a local 5M-row INT96 file: about 70-120 MiB less peak RSS with rechunk=False. Polars' own reads and the chunk path do not rechunk, so the fallback now matches them.

laughingman7743 · 2026-10-17T03:30:49Z

+        [(None, {}, False), (3, {}, False), (None, {"low_memory": True}, True)],
+        ids=["whole-without-pyarrow", "chunks-without-pyarrow", "whole-read-options"],
+    )
+    def test_read_parquet_int96_polars_1(self, tmp_path, chunksize, kwargs, pyarrow):


Round 1 finding (repaired in 3a675e2): the read-options case patched find_spec to report no pyarrow, so it could pass because pyarrow was missing rather than because of the read options. It now runs with pyarrow reported as installed.

laughingman7743 · 2026-10-17T03:42:16Z


 See {ref}`arrow-unload-options` for more information.

+Timestamps of UNLOAD results are read at microseconds, as `Datetime("us")`.


Self-review round 2 (claims, existing callers, operators, docs readers, evidence). Base 9e2bf6e (after the rebase onto #1226), head 5e4fac3.

Claims re-checked against evidence:

INT96 at every nesting level, no ARROW:schema: two live UNLOADs; the committed fixture is the second.

Polars 1.39.0 wraps and 1.44.2 saturates: re-measured on the real Athena fixture. The earlier evidence used pyarrow-written files.

schema cannot be combined with use_pyarrow=True: measured on Polars 1.39.0, 1.44.2, and 2.0.0.

Map saturation in Polars 2.0: int96_options() on Polars main has no Map case.

pyarrow 22's INT96 writer wraps, while its reader is correct on the Athena fixture.

Performance: re-measured after the rechunk repair, 3 alternating runs; the PR table is updated.

FINDINGS, repaired in 5e4fac3 and the PR body:

The two polars.md statements outside the diff, which list the reads that use PyAthena's S3FileSystem, omitted the Polars 1.x fallback (see the thread below).

The PR body still named the pre-rebase tested commit and the pre-repair Polars 1.x numbers.

The PR body did not state that storage_options alone triggers NotSupportedError on Polars 1.x with timestamps, or that the private Polars reader signatures change.

Deferred: at the minimum versions, 5 test_read_csv_with_pyarrow_matches_pandas cases fail identically on master 9e2bf6e (CSV path, pre-existing, outside this change).
Not measured: S3-backed timing and request counts of the Polars 1.x fallback, which goes through PyAthena's S3FileSystem and its retries instead of object_store.

laughingman7743 · 2026-10-17T03:42:16Z

 ```

-They apply to the reads through PyAthena's filesystem: CSV results read without the chunksize option, which use `"bytes"` and 5 MiB blocks by default, and the Parquet files of UNLOAD results read with the chunksize option when pyarrow is installed and `execute()` is given no Polars read options.
+They apply to the reads through PyAthena's filesystem: CSV results read without the chunksize option and, with Polars 1.x, the Parquet files of UNLOAD results with timestamps read without it, which use `"bytes"` and 5 MiB blocks by default, and the Parquet files of UNLOAD results read with the chunksize option when pyarrow is installed and `execute()` is given no Polars read options.


Round 2 finding (repaired in 5e4fac3): the Polars 1.x fallback reads the data manifest through S3FileSystem(**self._csv_storage_options), so block_size and cache_type apply to it with the CSV defaults ("bytes", 5 MiB). The intro and this paragraph listed only the CSV and chunked UNLOAD reads.

laughingman7743 · 2026-10-17T04:09:20Z

+        }
+
+    @staticmethod
+    def _cast(df: pl.DataFrame, dtypes: dict[str, Any]) -> pl.DataFrame:


Independent review (relayed): Codex codex exec -s read-only, static review, no tests run. Base 9e2bf6e, head 5e4fac3, detached snapshot, prompt without PR framing. Result: FINDINGS.

P2 regression: Map restore with pl.col(name) treats names such as "*" or ^...$ as selectors. On Polars 2.0, an UNLOAD map column named "*" would make the cast hit every column. Verified (pl.col("*").cast(Map) raises InvalidOperationError). Fixed in ba6780e: cast by df.get_column(name) Series; test_read_parquet_int96_map_named_like_a_selector covers it.

P2 regression: Polars 1.x raised NotSupportedError with any read option, including columns=["id"] that excludes timestamps. Fixed in ba6780e by the maintainer's choice: Polars 1.x reads natively at ns, casts to us, and warns only when the read columns hold timestamps (test_read_parquet_int96_polars_1[no-timestamps]).

Pre-existing: use_pyarrow=True with chunksize routes to scan_parquet, which has no use_pyarrow. This was already the case on master. Deferred, not part of UNLOAD timestamps outside 1677-2262 are read as wrong values #1214.

laughingman7743 · 2026-10-17T04:09:20Z

+        }
+
+    @staticmethod
+    def _output_dtypes(schema: pl.Schema, options: dict[str, Any] | None) -> dict[str, Any]:


/code-review (high), the same head 5e4fac3 — 7 findings, dispositions:

Polars 1.x NotSupportedError with read options: fixed (ba6780e), as above.

holds_datetimes() matched any Datetime, not only ns: fixed (ba6780e). The check is now to_microseconds(dtype) != dtype, and the helper is removed.

The scan path with use_pyarrow: pre-existing, as above; the dead NotSupportedError gate it mentioned no longer exists.

Timestamp-free maps were read as lists of structs: fixed (ba6780e). to_microseconds(maps=False) keeps a Map without ns Datetimes.

The fallback's fsspec cache for large files: deferred. It is unmeasured on S3, and the PR states so.

The docs implied a warning for chunked reads: fixed. polars.md now lists each Polars 1.x path.

Two Polars 2 checks (__version__ versus getattr(pl, "Map")): kept. They test different features: the INT96 read-schema fix (fix: Respect Parquet schema when reading Int96 timestamps pola-rs/polars#29208) and the Map dtype.

/simplify (reuse, simplification, efficiency, altitude), head ba6780e — applied in 4f2a7f2:

the output dtypes are worked out once from the file schema, instead of per chunk from df.schema, which Polars rebuilds on each access;

a user schema selects no read arguments;

one _reads_with_pyarrow() serves the whole and chunked reads, and one warning helper serves both;

the tests share INT96_ARROW_TYPES, and _read_unload_result(table=) writes tables in place.

Skipped:

parse_output_location() in the fallback: it breaks local-path tests, and the manifest is always s3://;

a shared whole-read helper with ArrowCursor: the filesystems differ;

routing whole reads through pyarrow like chunks: it would slow Polars 2;

merging the pandas test helpers: the gain is small.

Self-review of these repairs. Round 1, behavior: Polars 1.x paths (pyarrow fallback, native with options, no pyarrow, columns projection), the user schema on both majors, use_pyarrow precedence, and Maps named like selectors. Round 2, claims: polars.md now matches each path, and the PR body is updated. Validation: offline suites on Polars 2.0.0 / Py3.15, Polars 1.44.2 / Py3.13, and the minimums (only the pre-existing 5 CSV failures); live Polars suites on both majors at ba6780e (225 and 223 passed). A live rerun at 4f2a7f2 is in progress.

laughingman7743 · 2026-10-17T04:22:43Z

-            return pl.read_parquet(
-                self._unload_location,
-                **self._read_kwargs(lambda: self._get_parquet_storage_options(self.connection)),
+            kwargs = self._read_kwargs(


Independent review 2 (relayed): Codex codex exec -s read-only, a full static review of the expanded Polars 1.x contract; no tests run. Base 9e2bf6e, head 4f2a7f2. Result: FINDINGS.

P2 regression: the whole read computed _read_kwargs() (storage options, which can refresh credentials) outside the OperationalError boundary, so a credential failure escaped untranslated. Fixed in 0650d33: the preparation is inside the try, and the scan works out its arguments inside its own. test_read_parquet_storage_options_error covers both; it fails at 4f2a7f2 for the whole read.

Pre-existing: use_pyarrow=True cannot read an UNLOAD prefix from S3. Verified live on master 9e2bf6e with Polars 2.0.0: FileNotFoundError on the prefix, wrapped as OperationalError. The new docs advertised it, so 0650d33 removes the pyarrow_options merge, its test, and the docs bullet rather than keeping an unreachable path.

Pre-existing: read options that only pl.read_parquet() takes (columns, use_pyarrow, pyarrow_options) break chunked scans, as on master. Deferred: Polars option routing, not INT96.

laughingman7743 · 2026-10-17T04:30:44Z

        import polars as pl

        try:
+            options = self._int96_read_options(schema)


Independent follow-up (relayed): Codex codex exec -s read-only, static review. Range 4f2a7f2..0650d33, same base 9e2bf6e. Result: no regression introduced by the patch. Two pre-existing findings:

pyathena/result_set.py:893: a manifest body read that times out after the response escapes OperationalError. This is shared result-set code, untouched here. Deferred (pre-existing).

docs/polars.md:411 still names use_pyarrow=True as the fsspec route. 3793b71 removed the clause, and a second narrow Codex follow-up on that docs-only diff found the original sentence the accurate one: it describes where Polars sends storage_options, which holds for use_pyarrow=True. 0bbaaab reverts it, so the head tree equals the reviewed 0650d33. That use_pyarrow=True cannot read an UNLOAD prefix stays a separate, pre-existing defect.

Live at 0650d33: tests/pyathena/polars/ and tests/pyathena/aio/polars gave 225 passed, 4 skipped on Polars 2.0.0 / Py3.15, and 223 passed, 6 skipped on Polars 1.44.2 / Py3.13.

laughingman7743 · 2026-10-17T07:38:09Z

            check_dtypes=True,
        )

+    @pytest.mark.parametrize(("chunksize", "kwargs"), _UNLOAD_READS)


Rebase onto 59169c5 (#1231, #1234), relayed independent follow-up: Codex codex exec -s read-only, static review. git range-diff 9e2bf6ee..0bbaaab4 59169c5e..25c8890a: commits 2-9 are unchanged. Commit 1 changed only by the union resolution of additive conflicts in this file, which keeps both the #1234 CSV tests and the UNLOAD tests. 25c8890 restores the blank line that the union dropped. Codex also checked whether the upstream CSV timestamp changes (_get_csv_schema_overrides, _to_datetimes(strict=)) and tests/pyathena/util.py interact with the UNLOAD Parquet paths. Result: CLEAN.

Self-review of the rebase: the upstream Polars changes are confined to CSV reads; _read_kwargs() and the Parquet methods are untouched upstream.

Validation at 25c8890:

just lint passed.

Offline (+tests/pyathena/polars/test_converter.py):

Py3.15 / Polars 2.0.0: 395 passed, 14 skipped.

Py3.13 / Polars 1.44.2: 393 passed, 16 skipped.

Minimum versions: 388 passed, plus the pre-existing 5 (Pre-existing UNLOAD read defects found in the #1214 review #1235).

Live:

Polars 2.0.0: Polars suites plus the Arrow and pandas test_temporal_types, 238 passed, 4 skipped, 3 xfailed (Read CSV dates and NULL temporal values in ArrowCursor as the contract defines #1216, Make PandasCursor's temporal dtypes and fetched types depend only on the Athena type #1219).

Polars 1.44.2: Polars suites, 235 passed, 6 skipped.

Athena's UNLOAD writes timestamps as Parquet INT96, which the readers decoded at nanoseconds, so values outside 1677-09-21 to 2262-04-11 were wrapped or saturated, and the dtype was ns instead of the us of CSV results. PandasCursor and ArrowCursor read INT96 at microseconds with pyarrow's coerce_int96_timestamp_unit, which an execute() option given to pandas.read_parquet() still replaces. ParquetChunkReader does the same for the chunked reads of PandasCursor and PolarsCursor. Polars 2.0 reads INT96 at the unit of the read schema, so PolarsCursor reads with the file schema in microseconds. Map values still saturate in Polars 2.0, so maps are read as lists of key-value structs and cast back. Polars 1.x reads INT96 only at nanoseconds, so a result with timestamps is read with pyarrow and a warning recommends Polars 2.0; it raises NotSupportedError without pyarrow or with Polars read options, except use_pyarrow=True, which gets coerce_int96_timestamp_unit in pyarrow_options. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

pl.from_arrow() rechunks by default, which added a copy of the result to the peak memory of the fallback; Polars does not rechunk its own reads. The Polars 1.x test now also covers read options with pyarrow installed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The UNLOAD timestamps of ArrowCursor and PolarsCursor now match the contract; PandasCursor's UNLOAD dates remain #1219. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Polars 1.x raised NotSupportedError for a result with timestamps read with Polars read options, such as storage_options, which read in-range timestamps correctly before. It now reads them natively at nanoseconds, casts them to microseconds, and warns that values outside 1677-09-21 to 2262-04-11 are wrong; the warning follows the columns actually read. Only nanosecond Datetimes select the INT96 handling, and only maps that hold them are read as lists of structs. Columns are cast by name with Series, as pl.col() reads names such as "*" as selectors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The casts after a read follow from the file schema and the read arguments, so they are worked out once per read instead of per chunk from each DataFrame's schema, and a schema given to execute() now selects no read arguments. The whole and chunked reads share the pyarrow condition and the warning, and the tests share the fixture's Arrow types and write their own tables in place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The storage options of the whole Polars read, which can refresh credentials, were computed outside the OperationalError boundary; the chunked scan now also works out its read arguments inside its own. use_pyarrow=True cannot read an UNLOAD prefix from S3 (Polars opens it as one file through fsspec), so the pyarrow_options merge for it is removed with its documentation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Polars' use_pyarrow=True opens an UNLOAD prefix from S3 as one file and fails, and the chunked CSV and UNLOAD scans do not take it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The sentence describes where Polars sends storage_options, which holds for use_pyarrow=True; that use_pyarrow=True cannot read an UNLOAD prefix is a separate, pre-existing defect. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The tests that patched private result set methods, asserted helper dtypes, or read a stored file are replaced by queries through each cursor's public API. They check the timestamps that Athena's UNLOAD writes inside arrays, rows, and a map named "*", in whole and chunked reads, and the documented options and warning: coerce_int96_timestamp_unit for PandasCursor, and for PolarsCursor a schema given to execute() on Polars 2.0 and the warning of a read with read options on Polars 1.x. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Milliseconds hold the values of the timestamps that UNLOAD writes, so the nested tests now assert the microsecond types before the values, and the Polars Map types before casting them. The query adds the top-level timestamp and a map in a row in an array. The Polars 1.x test uses the columns option, whose effect shows which reader ran, covers scanned chunks, and shows no warning for a read without timestamps; the schema option is also checked for scanned chunks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

laughingman7743 · 2026-10-17T09:11:42Z

+        indirect=["polars_cursor"],
+        ids=["whole", "chunks", "scanned-chunks"],
+    )
+    def test_unload_nested_timestamps(self, polars_cursor, kwargs):


Test rework, after the maintainer's note that the AI-written tests were of poor quality. The rework is in 043e295 and 1dc9b28, rebased onto 7e5d9ee (#1236).

Removed, because they patched private result-set methods, asserted helper dtypes, or read a stored file:

the INT96 tests in tests/pyathena/{arrow,pandas,polars}/test_result_set.py;

the to_microseconds / holds_datetimes tests;

the ParquetChunkReader fixture test, with tests/resources/unload/int96_timestamps.parquet.

They are replaced by queries against Athena through each cursor's public API:

test_unload_nested_timestamps in ArrowCursor, PandasCursor (whole, chunks), and PolarsCursor (whole, chunks, scanned chunks on Polars 2.0). It runs temporal.nested_query(), which holds years 1, 1600, and 9999 and NULL at the top level and in an array, a row, a map named "*", and a map in a row in an array. It asserts the microsecond types, then the values.

PandasCursor coerce_int96_timestamp_unit precedence.

PolarsCursor schema precedence on Polars 2.0 (whole, scanned).

PolarsCursor read options on Polars 1.x (whole with columns, scanned chunks, a projection without timestamps): which reader ran, the cast to us, and whether a warning is logged.

The nested tests fail on master (6 failures). The precedence tests pass on master by design: they guard that the new defaults do not override user options.

Relayed independent reviews (Codex codex exec -s read-only, static). The first, over 7e5d9ee..043e295: FINDINGS. The nested tests checked values but not units, deeper nesting was missing, the pandas top-level timestamps were hidden by the #1219 xfail, the Polars 1.x reader path was indistinguishable, and chunked schema precedence was untested. All are fixed in 1dc9b28. The changes it named in the existing test_result_set.py tests are pre-existing patterns, left to #1239. The follow-up over 043e295..1dc9b28: CLEAN.

Live validation:

At 043e295, tests/pyathena/{arrow,polars,pandas}/ and aio/{arrow,pandas,polars} on Polars 2.0.0: 986 passed, 11 skipped, 2 xfailed (Make PandasCursor's temporal dtypes and fetched types depend only on the Athena type #1219).

At 1dc9b28, the new tests: Polars 2.0.0 / Py3.15, 9 passed, 3 skipped; Polars 1.44.2 / Py3.13, 9 passed, 3 skipped.

laughingman7743 commented Oct 17, 2026

View reviewed changes

laughingman7743 force-pushed the fix/1214-unload-int96-us branch from 3a675e2 to 9218258 Compare October 17, 2026 03:34

laughingman7743 commented Oct 17, 2026

View reviewed changes

laughingman7743 marked this pull request as ready for review October 17, 2026 04:34

laughingman7743 mentioned this pull request Oct 17, 2026

Pre-existing UNLOAD read defects found in the #1214 review #1235

Closed

laughingman7743 marked this pull request as draft October 17, 2026 07:28

laughingman7743 force-pushed the fix/1214-unload-int96-us branch from 0bbaaab to 25c8890 Compare October 17, 2026 07:37

laughingman7743 commented Oct 17, 2026

View reviewed changes

laughingman7743 marked this pull request as ready for review October 17, 2026 07:42

laughingman7743 marked this pull request as draft October 17, 2026 07:54

laughingman7743 marked this pull request as ready for review October 17, 2026 08:09

This was referenced Oct 17, 2026

test_write_failed_part_while_waiting fails intermittently since queued parts are cancelled #1237

Closed

Audit the test suite for unnecessary and flaky tests #1239

Closed

laughingman7743 and others added 12 commits October 17, 2026 17:52

Expect the UNLOAD temporal types to pass

7633e22

The UNLOAD timestamps of ArrowCursor and PolarsCursor now match the contract; PandasCursor's UNLOAD dates remain #1219. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

List the Polars 1.x UNLOAD reads that use PyAthena's filesystem

b9287d7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Drop the use_pyarrow route from the Polars S3 options docs

9c0b3f2

Polars' use_pyarrow=True opens an UNLOAD prefix from S3 as one file and fails, and the chunked CSV and UNLOAD scans do not take it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Separate the Polars tests that the rebase joined

bc55d4b

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

laughingman7743 force-pushed the fix/1214-unload-int96-us branch from 25c8890 to 1dc9b28 Compare October 17, 2026 09:11

laughingman7743 commented Oct 17, 2026

View reviewed changes

laughingman7743 merged commit 4ac04db into master Oct 17, 2026
9 checks passed

laughingman7743 deleted the fix/1214-unload-int96-us branch October 17, 2026 10:20

Navigation Menu

Repository navigation

Read UNLOAD timestamps at microseconds - #1233

Read UNLOAD timestamps at microseconds#1233
laughingman7743 merged 12 commits into
masterfrom
fix/1214-unload-int96-us

laughingman7743 commented Oct 17, 2026 •

edited

Loading

Uh oh!

laughingman7743 Oct 17, 2026

Uh oh!

laughingman7743 Oct 17, 2026

Uh oh!

laughingman7743 Oct 17, 2026

Uh oh!

laughingman7743 Oct 17, 2026

Uh oh!

laughingman7743 Oct 17, 2026

Uh oh!

laughingman7743 Oct 17, 2026

Uh oh!

laughingman7743 Oct 17, 2026

Uh oh!

laughingman7743 Oct 17, 2026

Uh oh!

laughingman7743 Oct 17, 2026

Uh oh!

laughingman7743 Oct 17, 2026

Uh oh!

laughingman7743 Oct 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant


		See {ref}`arrow-unload-options` for more information.

		Timestamps of UNLOAD results are read at microseconds, as `Datetime("us")`.

Conversation

laughingman7743 commented Oct 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

WHAT

WHY

TEST

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

laughingman7743 commented Oct 17, 2026 •

edited

Loading