Skip to content
1 change: 1 addition & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ sqlalchemy

filesystem
null_handling
temporal_types
contributing
testing
```
Expand Down
2 changes: 2 additions & 0 deletions docs/sqlalchemy.md
Original file line number Diff line number Diff line change
Expand Up @@ -737,6 +737,8 @@ engine_arrow = create_engine(
)
```

(ddl-and-cast-types)=

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of the added labels (base d36a1f82a58592738eb56618cfa2ff4c8128cdaa, range 7053f962..f154c17d).

Self-review, both perspectives: each label sits on the heading that its link's text names ("DDL and CAST types", "Assume role provider"), and each name is defined once in docs/. just docs lint passes. A clean sphinx-build -E has no xref_missing warnings, and the built pages have id="ddl-and-cast-types" and id="assume-role-provider". Result: CLEAN.

Independent follow-up (relayed): Codex CLI (codex exec -s read-only), session 01a14f11-0147-73f0-8ea1-54245462de0d, run on a detached snapshot at f154c17de024e43bb9b858a8aca59d1decc8676d. Static review only. Covered: the diff, both heading and link contexts, the MyST configuration (docs/conf.py:163), and every reference to both names in the repository. Result: CLEAN.


## DDL and CAST types

Athena parses DDL statements such as `CREATE TABLE` with Hive type syntax, and queries with Trino type syntax.
Expand Down
117 changes: 117 additions & 0 deletions docs/temporal_types.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
<!--
Copyright 2026 The PyAthena authors

Licensed under the MIT License.
See LICENSE or https://opensource.org/licenses/MIT.

SPDX-License-Identifier: MIT
-->

(temporal_types)=

# Temporal Types

This section documents how each cursor returns the Athena temporal types:
`timestamp`, `timestamp with time zone`, `date`, `time`, and `time with time zone`.
The asynchronous and aio variants of each cursor return the same types as the cursor.
`SparkCursor` and its variants are not covered.

## Types by Cursor

| Athena type | Fetch methods | `PandasCursor` | `ArrowCursor` | `PolarsCursor` |
| --- | --- | --- | --- | --- |
| `timestamp` | naive `datetime` | `datetime64[us]` | `timestamp[us]` | `Datetime("us")` |
| `timestamp with time zone` | aware `datetime` | `datetime64[us, UTC]` | `string` | `String` |
| `date` | `date` | `datetime64[us]` | `date32` | `Date` |
| `time` | naive `time` | `object` (`time`) | `string` | `String` |
| `time with time zone` | aware `time` | `object` (aware `time`) | `string` | `String` |

The fetch methods are `fetchone()`, `fetchmany()`, `fetchall()`, and iteration of the cursor.
They return the types in the "Fetch methods" column in every cursor, including `Cursor`, `DictCursor`, and `S3FSCursor`.
`PandasCursor` returns `timestamp with time zone` values as aware `datetime` objects in UTC, as its DataFrame holds them.

The type of a column depends only on its Athena type and the cursor.
It is the same for every value, for whole and chunked reads, for results read with `unload=True`, and for {ref}`managed query result storage <managed-query-result-storage>`.
Options given to `execute()` and custom converters can change it, as each cursor's page describes:
{ref}`pandas-cursor`, {ref}`arrow-cursor`, and {ref}`polars-cursor`.

`ArrowCursor` and `PolarsCursor` keep `timestamp with time zone`, `time`, and `time with time zone` columns as Athena's text, such as `2020-01-02 03:04:05.123 Asia/Tokyo` and `03:04:05.123+05:30`.
Their fetch methods convert these values to `datetime` and `time` objects.

## Supported Years

Typed values cover years 1 to 9999.
PyAthena does not check for other years.
For them, the result is whatever Python, pandas, pyarrow, or Polars returns or raises.

`PandasCursor` holds `timestamp with time zone` values in UTC, so for them the range applies to the year in UTC.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review (relayed). Reviewer: Codex CLI (codex exec -s read-only), session 01a14ee8-8552-7d80-9d4d-10168c333be8. Base d36a1f82a58592738eb56618cfa2ff4c8128cdaa, head 4da93d3253ca094d43c67e3f90f89939767f38f3, run on a detached snapshot. The prompt had no PR number, description, or prior findings. Static review only: no edits, builds, tests, or network. Covered: all claims of the new page; converter and fetch paths of the sync, async, and aio cursors; CSV, managed storage, chunked, and nested UNLOAD results; the temporal test expectations; dependency floors; related docs, labels, the toctree, and doc conventions. Result: FINDINGS (2).

Finding 1 [P2], at docs/temporal_types.md:43 at the reviewed head: "Typed values cover years 1 to 9999" needs a condition for the PandasCursor fetch methods. tests/pyathena/pandas/test_result_set.py:1177 converts 0001-01-01 00:00:00.000 Asia/Tokyo to 0000-12-31T14:41:01 UTC, and fetching calls to_pydatetime() (pyathena/pandas/result_set.py:726), which cannot represent year 0.

Verified: to_pydatetime() on year 0 or year 10000 datetime64[us, UTC] values raises ValueError: year must be in 1..9999. The DataFrame holds them. This is within contract item 3 of #1212 (out-of-range years pass through library errors), so there is no code change, but the page overstated the range. Repaired in ed3553d (this line): for PandasCursor timestamp with time zone, the range applies to the year in UTC, with the Asia/Tokyo example.

For example, the fetch methods raise an error for `0001-01-01 00:00:00.000 Asia/Tokyo`, which is in year 0 in UTC.

## Precision

Microseconds are the precision of every typed value.
Fractional digits beyond six are truncated, not rounded, including before 1970:
the `timestamp(9)` value `1969-12-31 23:59:59.987654321` is read as `datetime(1969, 12, 31, 23, 59, 59, 987654)`.
Columns kept as text keep every fractional digit.

`cursor.description` reports precision 3 for every `timestamp` column, whatever precision the column has.

## NULL

SQL NULL is `None` in the fetch methods.
In `PandasCursor` DataFrames, it is `NaT` in the `datetime64` columns and `None` in the `object` columns.
In Arrow Tables and Polars DataFrames, it is `null`, including the columns kept as text.
See {ref}`null_handling` for the other types.

## Time Zones

A `timestamp with time zone` value has a UTC offset, such as `+05:30`, or a time zone name, such as `UTC` or `Asia/Tokyo`.
The fetch methods of the cursors other than `PandasCursor` return a value with an offset in a fixed-offset `datetime.timezone`, and a value with a zone name in `zoneinfo.ZoneInfo`.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round two (claims; base d36a1f82a58592738eb56618cfa2ff4c8128cdaa, head 5f077b28). Result: FINDINGS, repaired in 4da93d3.

Claims checked, with evidence:

  • UNLOAD: NOT_SUPPORTED: Unsupported Hive type for time, time with time zone, and timestamp with time zone, and Incorrect timestamp precision for TIMESTAMP(0)/(6), were checked on live Athena (ArrowCursor, unload=True). The three casts in the example run on live Athena with and without UNLOAD. The UNLOAD format is Parquet (formatter.py:164).
  • description precision 3 was checked on live Athena: TIMESTAMP(0), (6), (9), and (12) all report 3.
  • NULL as None in PandasCursor object time columns was checked on live Athena. My first draft said NaT and was corrected before the PR was published.
  • tzdata>=2024.2 is in pyproject.toml:15. The zoneinfo lookup order (TZPATH, then the tzdata package) follows the Python docs. The time zone rule skew and the Iceberg UTC read-back are the Define time zone conversion for values whose zone cannot be represented #1220 measurements.
  • The PR body says "Documentation only; no AWS test jobs apply". This holds because test.yaml has paths-ignore: docs/**, **.md.

Finding 2 (this line): "The fetch methods return ... zoneinfo.ZoneInfo" contradicted the earlier statement that PandasCursor returns these values in UTC. It now names the cursors other than PandasCursor.

Finding 3 (PR body): the body said that the only remaining xref_missing warning was #assume-role-provider. A clean sphinx-build -E also shows the existing ddl-and-cast-types warning in docs/sqlalchemy.md. The body is corrected. Both warnings predate this PR and are outside its scope.

Documentation reader: null_handling.md (Unload Limitations), usage.md (the timestamp(6) UNLOAD note), and the s3fs.md type table do not contradict the page. The Polars 1.x UNLOAD details are linked rather than repeated, because #1282 changes them.

`time with time zone` values always have an offset.

Athena writes a value with a zone name as the local time in that zone.
A local time that a change of the zone's UTC offset, such as daylight saving time, repeats or skips does not identify one instant.
For such a value, and for a zone name that Python's time zone database does not know, the cursors raise `DataError` naming the column when they convert the value.
Some reads convert values before the fetch methods are called, for example `execute()` for the first page of `Cursor` and `DictCursor` results and the chunk reads of `PandasCursor`, so the error is raised there.

```text
DataError: Column 'col': '2020-11-01 01:30:00.000 America/New_York' is ambiguous: the local time occurs twice in America/New_York. Convert the value to UTC in SQL, for example with AT TIME ZONE 'UTC'.
```

Python looks up zone names in the system time zone database first and then in the `tzdata` package, which PyAthena requires at version 2024.2 or later.
For an unknown zone name, update the `tzdata` package or the system time zone database.

Athena's time zone rules can differ from Python's for some zones and years.
For such a value, the instant that PyAthena returns differs from the one Athena holds, and PyAthena cannot detect it.
To read the exact instant, convert the value in SQL:

```sql
SELECT
col_timestamp_tz AT TIME ZONE 'UTC' AS col_utc,
to_unixtime(col_timestamp_tz) AS col_epoch_seconds
FROM your_table
```

Values with a zone name come from SQL expressions, such as `AT TIME ZONE`, `at_timezone()`, `with_timezone()`, and `from_unixtime()` with a zone.
Iceberg `timestamp with time zone` columns read back in UTC.

## UNLOAD

With `unload=True`, Athena writes the result as Parquet, which supports only some temporal types:

- `timestamp` columns must be `TIMESTAMP(3)`. Other precisions fail with `NOT_SUPPORTED: Incorrect timestamp precision`.
- `time`, `time with time zone`, and `timestamp with time zone` columns fail with `NOT_SUPPORTED: Unsupported Hive type`.
- `date` columns are supported.

Cast such columns in the query:

```sql
SELECT
CAST(col_timestamp_6 AS TIMESTAMP(3)) AS col_timestamp,
CAST(col_timestamp_tz AT TIME ZONE 'UTC' AS TIMESTAMP(3)) AS col_timestamp_utc,
CAST(col_time AS VARCHAR) AS col_time
FROM your_table
```

Timestamps are read at microseconds, including those inside arrays, rows, and maps, so years outside 1677 to 2262 are read correctly.
For Polars 1.x, see {ref}`polars-unload-options`.
4 changes: 4 additions & 0 deletions docs/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,8 @@ This is a breaking change for `work_group=""`, which previously read `AWS_ATHENA
Omit `work_group` or pass `None` to retain that fallback.
The behavior of `s3_staging_dir=""` is unchanged, including its use with managed query result storage.

(managed-query-result-storage)=

## Managed query result storage

When using a workgroup with [managed query result storage](https://docs.aws.amazon.com/athena/latest/ug/managed-results.html) enabled,
Expand Down Expand Up @@ -860,6 +862,8 @@ cursor = connect(profile_name="YOUR_PROFILE_NAME",
region_name="us-west-2").cursor()
```

(assume-role-provider)=

#### Assume role provider

```python
Expand Down
Loading