Repository navigation
fix(deps): update dependency pyathena to v4 - #401
Conversation
|
Important Review skippedAuto reviews are limited based on label configuration. 🏷️ Required labels (at least one): 1
Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #401 +/- ##
=======================================
Coverage 58.32% 58.32%
=======================================
Files 199 199
Lines 9956 9956
=======================================
Hits 5807 5807
Misses 4149 4149 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
fe54bf5 to
81dc76e
Compare
81dc76e to
b2540e0
Compare
b2540e0 to
24f7ca7
Compare
Note
This PR body was truncated due to platform limits.
This PR contains the following updates:
<4→<5Release Notes
pyathena-dev/PyAthena (pyathena)
v4.2.0Compare Source
Breaking changes
ProgrammingError("Cursor is closed.")fromexecute(),executemany(), the fetch methods, iteration,cancel(), the metadata methods, and its other methods (#1274). Before, fetches returnedNoneor[]andexecute()ran a new query. Cursors have a newis_closedproperty.ProgrammingErrorfrom its fetch methods, iteration,as_pandas(),as_arrow(),as_polars(), anditer_chunks(), instead of returningNone,[], or an empty DataFrame or table (#1274).next()on an asyncio cursor or result set raisesTypeErrorinstead of returning an un-awaited coroutine (#1274).fetchone(),fetchmany(), andfetchall()calls raiseProgrammingError(#1238). Before, the pandas and Polars result sets returned no more rows, and the other result sets continued after the skipped rows. Invalid arguments, such as a non-integerfetchmany()size, do not count as a failed fetch.threadsafety = 2declares (#1238). This includescancel()from a timer thread andclose()during a fetch. To stop a query from another thread, use its query ID.AioPandasCursor,AioArrowCursor,AioPolarsCursor, orAioS3FSCursordrops the result set from the cursor, so later fetches raiseProgrammingError("No result set.")(#1238). A cancelledas_*()call of these cursors does the same (#1286).AioPandasCursor.as_pandas(),AioPolarsCursor.as_polars()/as_arrow(), andAioArrowCursor.as_arrow()/as_polars()are coroutines: useawait cursor.as_pandas()(#1286).AioPandasCursor.as_pandas()always returns a DataFrame. To read chunks one at a time, use the new asynciter_chunks()ofAioPandasCursorandAioPolarsCursor.close()ofAsyncCursoror one of its variants (dict, pandas, Arrow, Polars, S3FS), the futures ofexecute(),description(), andpoll()that still wait for their query raiseProgrammingError(#1279, #1300). The query is not cancelled. A result set built after the close is closed before its future resolves. The futures ofcancel()andquery_execution()still run. A result set whose future resolved before the close stays open and belongs to the caller.AsyncCursor.description(query_id)reads only the result set metadata, or the Parquet schema of an UNLOAD result, and no longer reads the result rows (#1300). It uses the new metadata reader classes, such asAthenaResultSetMetadataReader. A custom result set class that overrides thedescriptionproperty no longer changes whatdescription(query_id)returns; make that change in its metadata reader class instead.AsyncPandasCursorandAioPandasCursor.timestampis alwaysdatetime64[us], and digits beyond microseconds are truncated. It was[ns]for 7 or more fractional digits,[s]when all values were NULL or withengine="pyarrow"and no fraction,[ns]withengine="pyarrow"and a fraction, andstrwhen a column also held values outside the nanosecond range.timestamp with time zoneisdatetime64[us, UTC]. It had the data's zone, or wasobjectfor several zones or an all-NULL column.datefrom UNLOAD results isdatetime64[us], as from CSV results. It wasobject(date).datetime,date, and awaredatetimein UTC, andNonefor NULL. They returnedpd.TimestampandNaT.parse_dates,dtype, orconvertersis given toexecute(), PyAthena leaves the temporal columns to pandas, and the fetch methods return pandas' values for them.engine="pyarrow"is now also used for results withtimestamp with time zonecolumns.timestamp with time zonevalue raisesDataErrorwhen its zone name is unknown, or when its zone repeats or skips its local time (daylight saving time) (#1224). Before, it returned a naivedatetimeor the earlier instant. Convert such values to UTC in SQL (AT TIME ZONE 'UTC') or readto_unixtime().tzdata>=2024.2is required.S3File.multipart_uploadis read-only, andS3File.multipart_upload_partsis removed (#1232).S3File.completed_partsreturns the uploaded or copied parts in part-number order.commit()raisesS3MultipartStateErrorafterdiscard()or after a failed write or commit. Before, it uploaded the buffered data after a discard and did nothing after a failure.get_std_out()/get_std_error(), raisesOperationalErrorinstead of the botocore exception (#1278). Callers that catch botocore errors there must catchOperationalError.fetchmany()andfetchall()now take rows through_take_rows(). A subclass whosefetchone()does not use the row buffer must override_take_rows()as well (#1195). Several private result set helpers moved to the metadata reader classes or were removed (#1300).Behavior changes
datetime64[us],timestamp[us], andDatetime("us"), as for CSV results. Values outside 1677-09-21 to 2262-04-11 are now correct; before, they were wrapped or saturated.coerce_int96_timestamp_unitgiven toexecute()still replaces the unit.coerce_int96_timestamp_unit=Noneis the same as not giving it (#1291).schemagiven toexecute()is used as given.execute()read them correctly (#1282).datefrom CSV results asdate32instead oftimestamp[ms], as from UNLOAD results (#1236).as_polars()givespl.Dateinstead ofpl.Datetime("ms").time,time with time zone, andtimestamp with time zonevalues of CSV results as null instead of''(#1236). The fetch methods still returnNone.DataErrornaming the column (Column 'name': ...) (#1273, #1277). PandasCursor whole reads raise it instead of anOperationalErrorthat wrapped it. ADataErrorfrom a function given inconvertersis still wrapped inOperationalError.ignore_errors=Truetruncates a CSV timestamp with more than nine fractional digits instead of returning null (#1234). Text that is not a timestamp is still null.new_columnsis given together withschema_overrides, it must name every column.map, and its values are dicts.ArrowCursor.as_polars()returns map columns aspl.Map.rowinstead ofarray.PolarsCursor,AioPolarsCursor, andAsyncPolarsCursor, theHeadObjectsize check runs when the result set is created, and its error is raised there instead of at the first fetch (#1190).iter_chunks()oras_pandas(), raiseProgrammingErrorat their next read once their result set is closed, for example by another query on the cursor (#1286). Before, they ended without an error.[]. For a chunked pandas read whose files hold no rows,engine="c"now raisesProgrammingError.to_sql()with the defaultflavor="spark"creates tables whose column names have spaces, tabs, newlines, and,;{}()=replaced with_, matching the names in the Parquet files (#1299). Such columns used to read as NULL. Tables created before keep the old names, and appending does not change them; recreate them, for example withif_exists="replace".AioCursorpages, and object construction (#1223). A request already sent still runs to its end. The cleanup requests (StopQueryExecution,StopCalculationExecution,TerminateSession) keep their retries.AioS3FileSystemcoroutine stops the retries of its S3 requests (#1231). Cancelling a chunked UNLOAD read ofAioPandasCursororAioPolarsCursorstops the retries of the requests that open its Parquet files (#1297). Closing a chunked UNLOAD result no longer waits for the retries of a Parquet file being opened ahead (#1297).AioS3FileSystemruns its blocking calls, including multipart copies, in a pool of up tomax_workersthreads of the filesystem (#1184). Each of its files uses a pool of up tomax_workersthreads. The event loop's default executor no longer limits them, so a process can use more threads than before. Amax_workersthat is not positive raisesValueErrorwhen the filesystem is created. AfterS3AioExecutor.shutdown(),submit()raisesRuntimeError.S3Filemultipart writes keep at mostmax_workersparts in flight, counting the part copies of an append, and a write waits until a part finishes (#1165, #1232). A part that failed is raised by the nextwrite(),flush(), orclose(), which closes the file without committing it (#1176, #1183, #1232). Before, it was raised only byclose()orcommit(). Inside an fsspec transaction, the error is raised by the write. If the caller catches it, the transaction fails when it commits. Committing again after a successful commit sends no request.chmodstops after the first failed update, and the keys it has not updated keep their ACL (#1231).S3Executor, the frames belowS3Core.call()or the executor no longer hold local variables, for example inpdb.pm()(#1192).executemany()closes only the result set after a failure, not the cursor (#1274).Benchmark
The benchmark of this release ran on commit
9e99caa8(#1288), whose package code is identical to this release: the release commitbaee716cdiffers from it only in one test file (#1311). Its history entries arebenchmarks/history/2026-10-19-9e99caa8.mdandbenchmarks/history/2026-10-19-9e99caa8-filesystem.md(#1310). Highlights:pandas.read_sql_query()through the result-file drivers is faster, because the row output of PandasCursor and PolarsCursor is faster (#1167, #1177). For 10 million flat rows,awsathena+pandaswithunload=truetook 50.7 seconds instead of 162.6, andawsathena+polarswithunload=true48.7 seconds instead of 92.0.AioS3FileSystemwrites and copies as fast asS3FileSystemwith 40 workers (#1184): a 6 GiB multipart copy ran at 1,889–1,925 MiB/s instead of 521–540.chunksize=100000.Known issues
What's Changed
New Contributors
Full Changelog: pyathena-dev/PyAthena@v4.1.0...v4.2.0
v4.1.0Compare Source
Breaking changes
pyathena.aio.sqlalchemy.baseare renamed, with no aliases for the old names:AsyncAdapt_pyathena_cursor→AthenaAioAdaptedCursor,AsyncAdapt_pyathena_connection→AthenaAioAdaptedConnection,AsyncAdapt_pyathena_dbapi→AthenaAioAdaptedDBAPI(#1151). Code that imports the old names must use the new ones. Engines created with theawsathena+aio*URLs are unaffected.Behavior changes
jsoncolumns with pandas' defaultlow_memory=Trueand converts thejsoncolumns after parsing (#1141). pandas now applies these options to the JSON text instead of the converted value:parse_datesnaming ajsoncolumn no longer parses a JSON string such as"2020-01-01", which keeps its quotes; cast in SQL instead.engine="python"and NA options such askeep_default_na=True, a JSON string that matches an NA token, such as"NA", stays a string; SQL NULL is stillNone.Benchmark
The benchmark of this release is in
benchmarks/history/2026-10-13-437276e1.mdandbenchmarks/history/2026-10-13-437276e1-filesystem.md(#1157). Highlights:What's Changed
Full Changelog: pyathena-dev/PyAthena@v4.0.0...v4.1.0
v4.0.0Compare Source
v4.0.0 changes the project's copyright notices. Please read the Copyright and license notices section first and review NOTICE. The license itself is still the MIT License.
v4.0.0 is a major release from
master, which was branched from the 3.x line at v3.36.0. It drops Python 3.10, raises the minimum versions of the core and optional dependencies, makes connection and cursor settings keyword-only, and rejects arguments that used to be ignored silently. It also contains a large number of fixes and behavior changes across the cursors, the SQLAlchemy dialect, Spark, and the S3 filesystem.Read Breaking changes before upgrading. The usage guide has migration sections for the keyword-only arguments, the worker settings, and the environment fallbacks ("... in PyAthena 4.0" in
docs/usage.md), and the Spark and S3FS guides have their own 4.0 sections. Users who need Python 3.10, pandas 2, an older pyarrow or Polars, or SQLAlchemy 1.x can stay on the 3.x maintenance line (latest v3.39.0).Several fixes in this release were already backported to 3.x in v3.37.0, v3.38.0, and v3.39.0. They are listed under Already in 3.x releases at the end and marked in "What's Changed"; users upgrading from v3.39.0 already have them.
Thanks to @mvanhorn for the STRUCT DDL fix (#870), to @aminghadersohi for the bare
awsathena://scheme fix (#839) and for reporting #854, #855, and #857, and to @innicoder for reporting the memory issue #417, which #1001 addresses.Copyright and license notices
LICENSEchanges from "laughingman7743" to "The PyAthena authors" (Copyright 2017 The PyAthena authors). The license terms are unchanged (MIT) (#809, #790).MITwithLICENSEandNOTICEas license files, so both files are included in the sdist and wheel (#809).Copyright <year> The PyAthena authorswithSPDX-License-Identifier: MIT. Existing files written solely by the maintainer received it, new original files must carry it, and a check enforces the headers across the repository (#805, #808).NOTICEfile acknowledges PyHive, which informed PyAthena's early development, and lists the parts that still follow it, with PyHive's Apache License 2.0 notice. It also records the other third-party material (the GitHub Actions OIDC CloudFormation template andstrtobool) and explains which files do not carry the header, such as files with contributions from other contributors, whose authorship is recorded in the Git history (#802). The per-file classification is recorded in #790.NOTICEtogether withLICENSE.Breaking changes
Python and dependencies
boto3>=1.43.31andbotocore>=1.43.31(was 1.41.2). botocore 1.43.31 is the first release with the S3 annotation operations used by multipart copies.fsspec>=2026.9.0(no lower bound before).tenacity>=8.0.1(was 4.1.0). Older releases breakretry_api_call(): releases before 6.1.0 fail on Python 3.11+, and 8.0.0 raisesTypeErrorfromafter_logon the first retry. (Also in v3.39.0.)python-dateutilis no longer a dependency;tzdatais required.sqlalchemy:sqlalchemy>=2.0.0(was 1.0.0). Environments pinned to SQLAlchemy 1.4 cannot installpyathena[sqlalchemy]4.0.0.pandas:pandas>=3.0.0.arrow:pyarrow>=22.0.0.polars:polars>=1.39.0(was 1.0.0). This applies even to users who never setchunksize: Polars 1.34–1.36 silently truncate chunked reads on errors, and Polars before 1.39 fill a local file cache that is not released.Development Status :: 5 - Production/Stable(#904, #865).Connections and cursors
TypeError.execute()keepsoperationandparameterspositional and requires keywords after them.connection.cursor(CursorClass)still works (#1101, #1092).cursor_kwargs, raiseTypeError.execute()of the default, dict, Arrow, S3FS, and Spark cursors raisesTypeErrorfor unknown keywords before changing cursor state or starting a query. The pandas and Polars cursors still pass unusedexecute()keywords to the reader, which rejects unknown ones when it reads the results.connection.cursor(OtherCursor)raises ifcursor_kwargscontain options thatOtherCursordoes not accept. A cursor subclass that passes its own settings fromcursor_kwargson to the base constructor must consume them first (#1102, #1093).PandasCursor,PolarsCursor,AioPandasCursor, andAioPolarsCursoris renamed frommax_workerstos3_max_workers, in the constructor and inexecute(). InAsyncPandasCursorandAsyncPolarsCursor,max_workerssizes only the cursor's thread pool ands3_max_workerssets the S3 read workers; pass both toAsyncPolarsCursorto keep its former values. The old name raisesTypeError(inexecute(), when the result set is created after the query has run) (#1103, #1094).arraysizepassed at cursor creation now takes effect and is validated:Cursor,DictCursor,AioCursor,AioDictCursor,AsyncCursor, andAsyncDictCursorraiseProgrammingErrorfor values above 1000, and every SQL cursor raises it for zero or negative values. Before, the value was ignored byCursor,DictCursor,AioCursor, andAioDictCursor(since v2.0.0), so a small value now means moreGetQueryResultscalls. The Arrow, pandas, Polars, and S3FS cursors still accept values above 1000 (#901, #897).connect(endpoint_url=...)no longer applies to S3. Setups that serve Athena and S3 from one endpoint (for example LocalStack) needAWS_ENDPOINT_URL_S3orAWS_ENDPOINT_URL. This affects the result sets,S3FileSystem(connection=...), Polarsstorage_options, the Spark cursors, andto_sql()(#1058, #1011).work_groupfollows the same environment fallback rule ass3_staging_dir: only an omitted argument orNonereadsAWS_ATHENA_WORK_GROUP; an explicit empty string skips the fallback and means no work group (#1108, #1096).DefaultCSVReader(S3FS cursor) is renamed toEmptyStringAsNullCSVReader, without an alias (#1107, #1097).pyathena.formatter._escape_presto; use_escape_trino.BaseCursor._execute()/AioBaseCursor._execute(); passoptions=ExecuteOptions(...).pyathena.util.strtobool()returnsboolinstead ofint; callint(strtobool(value))where an integer is required.SQLAlchemy
pyathena.sqlalchemy.types.get_double_typeis removed; usesqlalchemy.types.DOUBLE(#860).AthenaDialectdeclares SQLAlchemy 2.0'stype_compiler_clsinstead of the legacytype_compiler, so code that readsAthenaDialect.type_compileron the class no longer finds the compiler class (#895, #894).TypeEngine.compile(dialect)rendersAthenaStructasSTRUCT<name:type>instead ofROW(name type), andInteger(including subclasses andTypeDecorators over it) asINTin all DDL.Integer().with_variant(OtherType(), "awsathena")renders the variant inCREATE TABLE.JSONcolumn, anAthenaStructwithout fields, andNullTypein DDL raiseCompileErrorinstead of emitting DDL that Athena rejects.CAST(... AS JSON)is unchanged.CLOB/NCLOBrenderSTRINGinstead ofBINARY.Spark
SparkCursor,AsyncSparkCursor, andAioSparkCursorraiseNotSupportedErrorfor anyparametersvalue other thanNone, and theirexecute()no longer takeswork_group(TypeError, even forNone). Before, both were silently ignored (#1108, #1096).session_idno longer terminates that session. Passterminate_session_on_close=Trueto keep the previous behavior (#844, #793, #796).S3 filesystem
S3FileSystem.PATTERN_PATHandS3FileSystem.parse_path()/AioS3FileSystem.parse_path()are removed. UseS3Path.parse(path)and itsbucket,key, andversion_idattributes (#1100, #1052).S3FileSystem.OBJECT_ACLS,BUCKET_ACLS, and theMULTIPART_UPLOAD_*limits are removed; useS3Core.OBJECT_ACLS/S3Core.BUCKET_ACLS(for examplefs.core.OBJECT_ACLS) (#1090).max_workerkeyword ofopen()andpipe_file()is removed; passmax_workers(#948, #925).open()for writing or appending (wb,ab,xb) raisesValueErrorfor a path with?versionId=(#958, #936).find(maxdepth=n)listsnlevels instead ofn + 1, andmaxdepth=0raisesValueError.rm(),glob(),du(),expand_path(),copy(), andget()passmaxdepththroughfind(), sorm(path, recursive=True, maxdepth=1)no longer deletes the objects one level below the path. Callers relying on the old numbering needmaxdepth + 1(#956, #933).S3FileSystem.request_kwargsis a read-only property that returns a copy; passrequester_pays=Trueto the constructor instead of changing it (#1060).S3Filecallssubmit_to()on its executor. An executor passed toS3Filethat does not subclassS3Executormust provide it (#1113).Dependencies and packaging
S3Core(S3FileSystem.core,AioS3FileSystem.core) with typed listing entries, pages, and delete/multipart/copy/tagging/ACL/presign requests;S3Path;S3MultipartWriter;S3Executor.submit_to();connect(s3_config=...)andConnection.s3_client;TERMINAL_STATESon the query and calculation execution models (#1060, #1052, #1085, #1113, #1058, #881).pyathena/has a Google-style docstring, checked with ruff (#919).Connections and cursors
kill_on_interrupt=True, an interrupt duringexecute()of a SQL cursor now propagates asKeyboardInterrupt/asyncio.CancelledErrorafter the query is stopped, instead of surfacing asOperationalErroror being lost when the query succeeded.asyncio.wait_for()now raisesTimeoutError. An interrupt while a query is being started waits forStartQueryExecutionand stops the query it started (#853, #840).cursor()argument now overrides the same key in the connection'scursor_kwargs; before,cursor_kwargswon silently (#872, #859).dict_typepassed to one dict cursor no longer changes the row type of other dict cursors, anddict_type=Noneuses the default (#930, #923).Connection.close()closes the network connections of its clients. SQLAlchemy URLs withuse_ssl=falsenow disable TLS, asconnect(use_ssl=False)does (#1058, #1011).datecolumns aredatetime64,array/map/roware Athena text, a result without rows has its columns, andresult_set_type_hintsno longer applies to these cursors on managed storage (#1061, #1028, #1042).ArrowCursorreadstimestampcolumns astimestamp[us]instead oftimestamp[ms]; the Arrow and (unchunked) Polars cursorsConfiguration
📅 Schedule: (UTC)
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.
This PR was generated by Mend Renovate. View the repository job log.