Repository navigation
Document the temporal type contract (GH-1221) - #1284
Conversation
Add docs/temporal_types.md with the types each cursor returns for the Athena temporal types, the supported years, precision, NULL, time zone errors and rule differences, and the UNLOAD limits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
82961e9 to
5f077b2
Compare
…H-1221) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
|
||
| Athena writes a value with a zone name as the local time in that zone. | ||
| A local time that a change of the zone's UTC offset, such as daylight saving time, repeats or skips does not identify one instant. | ||
| For such a value, and for a zone name that Python's time zone database does not know, the cursors raise `DataError` naming the column where they convert the value: |
There was a problem hiding this comment.
Self-review round one (implementation behavior; base d36a1f82a58592738eb56618cfa2ff4c8128cdaa, head 9f402cc9eb349c9cc383a34a17250d0788a17376). Result: FINDINGS, repaired.
Covered: every statement of docs/temporal_types.md against the code on master. That is the table against DefaultArrowTypeConverter / DefaultPolarsTypeConverter / _DEFAULT_CONVERTERS and the test_temporal_types expectations of the pandas, Arrow, and Polars tests. The S3FS converter uses _DEFAULT_CONVERTERS. AthenaResultSet._convert_values() adds the Column 'name': prefix. The truncation is in _parse_datetime() / _parse_time(). _to_datetime_with_tz() produces the ambiguous/nonexistent message. Iteration is BaseCursor.__iter__. The toctree entry and the usage.md label are also covered.
Finding 1: the page said that "the fetch methods and PandasCursor" raise DataError for time zone values. Cursor and DictCursor convert the first page in execute() (AthenaResultSet construction), so a caller wrapping only fetchone() would miss it. Repaired in 5f077b2 (this line): the error is raised where the value is converted, which is the fetch methods, execute() for the first page of Cursor/DictCursor, and the reads of PandasCursor.
No tests: documentation only.
| ## Time Zones | ||
|
|
||
| A `timestamp with time zone` value has a UTC offset, such as `+05:30`, or a time zone name, such as `UTC` or `Asia/Tokyo`. | ||
| The fetch methods of the cursors other than `PandasCursor` return a value with an offset in a fixed-offset `datetime.timezone`, and a value with a zone name in `zoneinfo.ZoneInfo`. |
There was a problem hiding this comment.
Self-review round two (claims; base d36a1f82a58592738eb56618cfa2ff4c8128cdaa, head 5f077b28). Result: FINDINGS, repaired in 4da93d3.
Claims checked, with evidence:
- UNLOAD:
NOT_SUPPORTED: Unsupported Hive typefortime,time with time zone, andtimestamp with time zone, andIncorrect timestamp precisionforTIMESTAMP(0)/(6), were checked on live Athena (ArrowCursor,unload=True). The three casts in the example run on live Athena with and without UNLOAD. The UNLOAD format is Parquet (formatter.py:164). descriptionprecision 3 was checked on live Athena:TIMESTAMP(0),(6),(9), and(12)all report 3.- NULL as
Nonein PandasCursorobjecttime columns was checked on live Athena. My first draft saidNaTand was corrected before the PR was published. tzdata>=2024.2is inpyproject.toml:15. The zoneinfo lookup order (TZPATH, then the tzdata package) follows the Python docs. The time zone rule skew and the Iceberg UTC read-back are the Define time zone conversion for values whose zone cannot be represented #1220 measurements.- The PR body says "Documentation only; no AWS test jobs apply". This holds because
test.yamlhaspaths-ignore: docs/**, **.md.
Finding 2 (this line): "The fetch methods return ... zoneinfo.ZoneInfo" contradicted the earlier statement that PandasCursor returns these values in UTC. It now names the cursors other than PandasCursor.
Finding 3 (PR body): the body said that the only remaining xref_missing warning was #assume-role-provider. A clean sphinx-build -E also shows the existing ddl-and-cast-types warning in docs/sqlalchemy.md. The body is corrected. Both warnings predate this PR and are outside its scope.
Documentation reader: null_handling.md (Unload Limitations), usage.md (the timestamp(6) UNLOAD note), and the s3fs.md type table do not contradict the page. The Polars 1.x UNLOAD details are linked rather than repeated, because #1282 changes them.
…GH-1221) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| PyAthena does not check for other years. | ||
| For them, the result is whatever Python, pandas, pyarrow, or Polars returns or raises. | ||
|
|
||
| `PandasCursor` holds `timestamp with time zone` values in UTC, so for them the range applies to the year in UTC. |
There was a problem hiding this comment.
Independent review (relayed). Reviewer: Codex CLI (codex exec -s read-only), session 01a14ee8-8552-7d80-9d4d-10168c333be8. Base d36a1f82a58592738eb56618cfa2ff4c8128cdaa, head 4da93d3253ca094d43c67e3f90f89939767f38f3, run on a detached snapshot. The prompt had no PR number, description, or prior findings. Static review only: no edits, builds, tests, or network. Covered: all claims of the new page; converter and fetch paths of the sync, async, and aio cursors; CSV, managed storage, chunked, and nested UNLOAD results; the temporal test expectations; dependency floors; related docs, labels, the toctree, and doc conventions. Result: FINDINGS (2).
Finding 1 [P2], at docs/temporal_types.md:43 at the reviewed head: "Typed values cover years 1 to 9999" needs a condition for the PandasCursor fetch methods. tests/pyathena/pandas/test_result_set.py:1177 converts 0001-01-01 00:00:00.000 Asia/Tokyo to 0000-12-31T14:41:01 UTC, and fetching calls to_pydatetime() (pyathena/pandas/result_set.py:726), which cannot represent year 0.
Verified: to_pydatetime() on year 0 or year 10000 datetime64[us, UTC] values raises ValueError: year must be in 1..9999. The DataFrame holds them. This is within contract item 3 of #1212 (out-of-range years pass through library errors), so there is no code change, but the page overstated the range. Repaired in ed3553d (this line): for PandasCursor timestamp with time zone, the range applies to the year in UTC, with the Asia/Tokyo example.
| Athena writes a value with a zone name as the local time in that zone. | ||
| A local time that a change of the zone's UTC offset, such as daylight saving time, repeats or skips does not identify one instant. | ||
| For such a value, and for a zone name that Python's time zone database does not know, the cursors raise `DataError` naming the column when they convert the value. | ||
| Depending on the cursor and the read, that is in `execute()`, such as for the first page of `Cursor` and `DictCursor` results, or in the fetch methods. |
There was a problem hiding this comment.
Independent review (relayed, same run), finding 2 [P2], at docs/temporal_types.md:72 at the reviewed head: the error-timing list named execute() only for Cursor/DictCursor. S3FSCursor with managed query result storage converts every row in execute() (pyathena/s3fs/cursor.py:208 builds the result set, and pyathena/s3fs/result_set.py:133 fetches and converts all rows).
Verified in the code. Repaired in ed3553d (this line): the page now says the error is raised when the value is converted, which depends on the cursor and the read: execute(), such as for the first page of Cursor/DictCursor, or the fetch methods. Listing every case invites further omissions, such as managed results and the async futures, so the list is gone.
Repair self-review (both perspectives): just docs lint passes, and the incremental sphinx-build has no warnings on the page. The year sentence was reordered so that "other years" still refers to years outside 1-9999.
There was a problem hiding this comment.
Independent follow-up review (relayed). Reviewer: Codex CLI (codex exec -s read-only), session 01a14eed-e39f-74d1-bb08-ea445b627837. Range 4da93d32..ed3553dd, run on a detached snapshot at ed3553ddb0b688f226ae8d63937d767e9f064db8. Static review only. Covered: the repair diff, the whole page, the related docs/pandas.md sections, pandas UTC conversion and chunk reads, the Cursor/DictCursor prefetch, the S3FS managed-result conversion, and the related tests.
The UTC-year repair resolves finding 1. The broadened wording covers the managed S3FS path of finding 2. Result: FINDINGS (1).
Finding [P2], at docs/temporal_types.md:75: the new wording names only execute() and the fetch methods, but chunked PandasCursor reads raise the column-naming DataError when a chunk is read through iter_chunks() or as_pandas(). Evidence: pyathena/pandas/result_set.py:597 and :1209, and tests/pyathena/pandas/test_result_set.py:1078 (test_time_zone_errors, which raises in the chunk read).
Verified against test_time_zone_errors, whose docstring says that a chunked read raises "when it reads the chunk with the value". Repaired in 35b149d: the sentence adds "or, for chunked PandasCursor reads, when the chunk with the value is read, such as by iter_chunks()". Repair self-review: PolarsCursor chunks keep timestamp with time zone as text, so its errors come from the fetch methods, which the sentence already names, and ArrowCursor has no chunked read. just docs lint passes. A second independent follow-up on ed3553dd..35b149d5 follows.
There was a problem hiding this comment.
Independent follow-up review 2 (relayed). Reviewer: Codex CLI (codex exec -s read-only), session 01a14ef1-6bcd-7aa1-8106-cd422e9cd0df. Range ed3553dd..35b149d5, run on a detached snapshot at 35b149d5a55b37c55a796fd5c3e561547a93a396. Static review only. Covered: the shared time zone conversion and column naming, Cursor/DictCursor pagination, S3FSCursor CSV and managed results, whole and chunked Pandas/Polars reads, ArrowCursor, the async and aio variants, and docs/pandas.md. The PandasCursor clause is accurate. Result: FINDINGS (2).
- [P2] The
AsyncCursorfamily (pyathena/async_cursor.py:343,:176) builds the result set in the worker, so the error surfaces throughfuture.result(), which is neitherexecute()nor a fetch method. - [P3]
PolarsDataFrameIterator.itertuples()/iterrows()(pyathena/polars/result_set.py:156,:224), reached throughiter_chunks(), convert values outside the fetch methods.
Both verified in the code. Each list repair turned up another path, so 7053f96 replaces the list with a non-exhaustive statement: the error is raised when the value is converted, and some reads convert before the fetch methods are called, for example execute() for the first page of Cursor/DictCursor and the chunk reads of PandasCursor. Repair self-review: both examples are true (result_set.py first-page prefetch, and test_time_zone_errors), and the sentence no longer claims a complete list. just docs lint passes. A third independent follow-up on 35b149d5..7053f962 follows.
There was a problem hiding this comment.
Independent follow-up review 3 (relayed). Reviewer: Codex CLI (codex exec -s read-only), session 01a14ef6-c59b-7f61-816c-94bfa4d9445a. Range 35b149d5..7053f962, run on a detached snapshot at 7053f9628cc876c736aa429aac30d0726ee392b3. Static review only. Covered: the shared time zone parsing and column-naming DataError; Cursor, DictCursor, S3FSCursor, PandasCursor, ArrowCursor, and PolarsCursor, including the async futures and aio variants; first-page conversion, whole and chunked reads, managed results, and Polars itertuples()/iterrows(); and the repaired paragraph, the error example, the rest of the page, and docs/pandas.md.
Result: CLEAN. Both examples are correct, and "Some reads" / "for example" leave the list open. The general rule covers futures and the Polars row iterators. Nothing contradicts docs/pandas.md.
…H-1221) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
9551663 to
35b149d
Compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/usage.md links to #assume-role-provider and docs/sqlalchemy.md to #ddl-and-cast-types, but MyST generates no heading anchors here, so neither link had a target. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| ) | ||
| ``` | ||
|
|
||
| (ddl-and-cast-types)= |
There was a problem hiding this comment.
Review of the added labels (base d36a1f82a58592738eb56618cfa2ff4c8128cdaa, range 7053f962..f154c17d).
Self-review, both perspectives: each label sits on the heading that its link's text names ("DDL and CAST types", "Assume role provider"), and each name is defined once in docs/. just docs lint passes. A clean sphinx-build -E has no xref_missing warnings, and the built pages have id="ddl-and-cast-types" and id="assume-role-provider". Result: CLEAN.
Independent follow-up (relayed): Codex CLI (codex exec -s read-only), session 01a14f11-0147-73f0-8ea1-54245462de0d, run on a detached snapshot at f154c17de024e43bb9b858a8aca59d1decc8676d. Static review only. Covered: the diff, both heading and link contexts, the MyST configuration (docs/conf.py:163), and every reference to both names in the repository. Result: CLEAN.
WHAT
Adds
docs/temporal_types.md(Advanced Topics, after NULL handling), which documents the temporal type contract of #1212 as it is on master:timestamp,timestamp with time zone,date,time, andtime with time zone(the contract table of Define and enforce the contract for Athena temporal types across cursors #1212), and that the type depends only on the Athena type and the cursor, withexecute()options and custom converters left to each cursor's page.timestamp with time zone, the range applies to the year in UTC.descriptionreports precision 3.DataErrornaming the column for repeated, skipped, or unknown zones (Define time zone conversion for values whose zone cannot be represented #1220, Name the column in DataError for time zone values in every cursor #1217), raised when the value is converted, which can be before the fetch methods, thetzdatalookup order and floor, and that Athena's and Python's rules can differ undetectably, withAT TIME ZONE 'UTC'andto_unixtime()for exact instants.TIMESTAMP(3)anddateamong the temporal types, the casts that make other columns writable, microsecond reads, and a link to the Polars 1.x section ofdocs/polars.md.docs/usage.mdgains the label(managed-query-result-storage)=, which the new page links to. The existing link#managed-query-result-storagein its Environment variables section had no target and now resolves.Two other existing links had no target, because MyST generates no heading anchors in these docs. They get labels with the names they link to:
(assume-role-provider)=indocs/usage.mdand(ddl-and-cast-types)=indocs/sqlalchemy.md.WHY
Part of #1221 (under #1212): the documentation part. The 4.2.0 release notes are drafted on #1221, and the benchmarks run in the final combined benchmark before the 4.2.0 release. The strict xfails of the conformance tests are already gone from master.
The Polars 1.x UNLOAD behavior is changing in #1282, so this page links to
docs/polars.md, which #1282 updates, instead of repeating it.TEST
At 9f402cc, with the review repairs up to 7053f96 re-checked by
just docs lintandsphinx-build(the docs workflow's build passes at 7053f96):just lintandjust docs lint: pass.uv run sphinx-build -b html docs <tmp>: the new page builds and its references resolve (just docs buildbuilds only master and tags through sphinx-multiversion). At f154c17, a clean build (-E) has noxref_missingwarnings, and the built pages have theidsmanaged-query-result-storage,assume-role-provider, andddl-and-cast-types.unload=True), for the UNLOAD section:time,time with time zone, andtimestamp with time zonefail withNOT_SUPPORTED: Unsupported Hive type: ...;TIMESTAMP(0)andTIMESTAMP(6)fail withNOT_SUPPORTED: Incorrect timestamp precision for timestamp(N); ...;TIMESTAMP(3),date, and the casts in the example (CAST(x AT TIME ZONE 'UTC' AS TIMESTAMP(3)),to_unixtime(),CAST(time AS VARCHAR),CAST(timestamp(6) AS TIMESTAMP(3))) succeed.descriptionreports precision 3 forTIMESTAMP(0),(6),(9), and(12)columns.timeandtime with time zonevalues areNonein theirobjectcolumns.test_temporal_typesconformance tests on master.Documentation only; no AWS test jobs apply.
🤖 Generated with Claude Code