Repository navigation
Remove and rewrite unnecessary tests in the Polars, Arrow, and S3FS cursor tests #1246
Description
Summary
Part of #1239. Files: polars/, arrow/, s3fs/ (test_cursor.py, test_async_cursor.py, test_result_set.py, test_converter.py, test_reader.py, test_util.py). At master 59169c5 with the cross-check corrections: 289 test functions: 212 KEEP, 55 REMOVE, 6 MERGE, 16 REWRITE. #1238 deletes 4 of the REMOVE rows; the tables list 51 REMOVE, 6 MERGE, 16 REWRITE. Applying them saves about 44 Athena queries per run: 17 in the S3FS per-reader tests, 14 in the Polars chunk tests, 7 in the ArrowCursor as_polars tests. The result-set, converter, and reader rows are offline.
Paths are relative to tests/pyathena/. [original] marks a test that existed at 21cbf05 (v3.15.0).
Order
- After Break result sets after failed or cancelled fetches and drop the cross-thread guards #1238 (06adf23). It deletes
polars/test_result_set.py::TestAthenaPolarsResultSet::{test_close_during_fetchall, test_close_during_fetchall_generator, test_close_during_first_read}andTestPolarsDataFrameIterator::test_close_during_read, modifies the KEEPTestAthenaPolarsResultSet::test_fetch_after_conversion_error, and addsTestAthenaPolarsResultSet::{test_fetch_after_failed_chunk_read, test_fetchone_interrupted_while_counting}andTestPolarsDataFrameIterator::test_read_after_failed_read. Section 3 is rebased onto it. - After Read UNLOAD timestamps at microseconds #1233 (1dc9b28). It modifies
arrow/test_cursor.py::TestArrowCursor::{test_complex_as_polars, test_complex_unload_as_polars}(section 5, REMOVE still holds; onlycol_datemoves topl.Date), their coverstest_complex_as_arrow,test_complex_unload_as_arrow, andtest_temporal_types, andpolars/test_result_set.py::TestAthenaPolarsResultSet::{test_storage_options_replace_defaults, test_iter_parquet_chunks_raises_when_read_fails_partway}(section 3). It changes the private_read_parquet/_iter_parquet_chunkssignatures to take apl.Schema, which the section 3 REWRITE no longer calls. It also changespolars/test_cursor.py::TestPolarsCursor::test_temporal_typesand adds tests that were not audited:TestArrowCursor::test_unload_nested_timestamps,TestPolarsCursor::{test_unload_nested_timestamps, test_unload_schema_option, test_unload_timestamps_with_read_options_polars_1}. - Read CSV dates as date32 and temporal text NULL as null in ArrowCursor #1236 is merged (master 7e5d9ee). It changes
col_datetopa.date32()inTestArrowCursor::test_complex_as_arrowand topl.Dateintest_complex_as_polars, removes the#1216xfail fromtest_temporal_types, and addsTestArrowCursor::test_temporal_text_null_with_binaryandarrow/test_converter.py::test_to_null(not audited). Re-checked at 7e5d9ee: the section 5 REMOVE oftest_complex_as_polarsstill holds, because the Polars dtypes follow from the Arrow table thattest_complex_as_arrowasserts. s3fs/test_cursor.py::TestS3FSCursor::test_on_start_query_execution(section 9) is removed with or after the top-level sub-issue's REWRITE oftest_connection.py::test_cursor_accepts_connection_callbacks, which must keep running the callback for every class inCURSOR_CLASSES.- Covers within this sub-issue that are MERGE/REWRITE targets:
polars/test_converter.py::test_to_datetimes_not_strictneeds the section 3 REWRITE oftest_read_csv_ignore_errorsandtest_read_csv_truncates_timestamps(keep the non-strict read and theOperationalError);TestArrowCursor::test_many_as_polarsneeds the rewrittenTestArrowCursor::test_as_polars;TestS3FSCursor::test_quoted_string_with_commamerges into the rewrittenTestS3FSCursor::test_contain_tab_character.
Findings
1. polars/test_cursor.py
| test | change | criteria | reason | covered by |
|---|---|---|---|---|
TestPolarsCursor::test_as_arrow |
REMOVE | 2, 6 | as_arrow() is as_polars().to_arrow(), so this checks Polars' to_arrow (2 queries). |
TestPolarsCursor::test_whole_result_reused (asserts as_arrow() values) |
TestPolarsCursor::test_iter_chunks, TestPolarsCursor::test_iter_chunks_many_rows |
REMOVE | 2 | Row totals and "is a DataFrame" checks of CSV chunking with another chunksize add nothing to the consistency test. | TestPolarsCursor::test_iter_chunks_data_consistency (with the merged size bound) |
TestPolarsCursor::test_iter_chunks_without_chunksize |
REMOVE | 2 | Without a chunksize, iter_chunks() yields one whole chunk, which is already asserted. |
TestPolarsCursor::test_whole_result_reused ([3] chunk heights) |
TestPolarsCursor::test_iter_chunks_unload |
REMOVE | 2 | A chunked UNLOAD row total is weaker than the exact sizes, schema, and equality checks of the cover. | TestPolarsCursor::test_unload_chunks |
TestPolarsCursor::test_iter_chunks_chunk_sizes |
MERGE | 2 | Its only extra assertion is chunk.height <= chunksize. |
into TestPolarsCursor::test_iter_chunks_data_consistency |
TestPolarsCursor::test_fetchone_with_chunksize, TestPolarsCursor::test_fetchmany_with_chunksize, TestPolarsCursor::test_fetchall_with_chunksize, TestPolarsCursor::test_iterator_with_chunksize, TestPolarsCursor::test_fetchone_with_chunksize_unload, TestPolarsCursor::test_iterator_with_chunksize_unload |
REMOVE | 2 | Row-count-only checks of fetching across chunks; the cover checks values and rownumber for CSV and UNLOAD (6 queries). |
TestPolarsCursor::test_fetch_across_chunks; description: TestPolarsCursor::test_description_with_chunksize_unload; iteration: TestPolarsCursor::test_iterator |
2. polars/test_async_cursor.py
| test | change | criteria | reason | covered by |
|---|---|---|---|---|
TestAsyncPolarsCursor::test_as_arrow |
REMOVE | 2, 6 | The same Polars to_arrow wrapper through the async future (2 queries). |
polars/test_cursor.py::TestPolarsCursor::test_whole_result_reused |
TestAsyncPolarsCursor::test_read_options |
REWRITE | 1, 6 | Keeps the option-plumbing assertions and drops cursor._executor._max_workers == 1, which only checks that the constructor stored its argument. |
— |
3. polars/test_result_set.py
#1238 deletes TestAthenaPolarsResultSet::test_close_during_fetchall, TestAthenaPolarsResultSet::test_close_during_fetchall_generator, TestAthenaPolarsResultSet::test_close_during_first_read, and TestPolarsDataFrameIterator::test_close_during_read; they are not listed.
| test | change | criteria | reason | covered by |
|---|---|---|---|---|
TestAthenaPolarsResultSet::test_fetchmany_size_not_integer |
REMOVE | 2 | The size is validated by the base AthenaResultSet.fetchmany (operator.index) before the Polars _take_rows runs. |
test_result_set.py::TestAthenaResultSet::test_fetchmany_size_not_integer |
TestPolarsDataFrameIterator::test_itertuples_without_column_names |
REMOVE | 7 | Exercises the repeat((), length) branch for a chunk with rows but no column names, which Athena results never have. |
— |
TestAthenaPolarsResultSet::test_storage_options_replace_defaults, TestAthenaPolarsResultSet::test_csv_storage_options_replace_defaults |
MERGE | 1, 5 | They assert the kwargs of mocked polars.scan_csv/read_parquet/scan_parquet/fsspec.open and that private properties are not computed; the contract that storage_options replace PyAthena's is observable end to end. |
into polars/test_cursor.py::TestPolarsCursor::test_as_polars_with_read_kwargs (parametrized over unload and chunksize) |
TestAthenaPolarsResultSet::test_iter_csv_chunks_raises_when_read_fails_partway, TestAthenaPolarsResultSet::test_iter_parquet_chunks_raises_when_read_fails_partway, TestAthenaPolarsResultSet::test_csv_read_kwargs_replace_defaults, TestAthenaPolarsResultSet::test_txt_new_columns_with_schema_overrides, TestAthenaPolarsResultSet::test_read_csv_truncates_timestamps, TestAthenaPolarsResultSet::test_read_csv_ignore_errors |
REWRITE | 1 | Build the result set through its constructor over a local output location, patching only the _pre_fetch metadata and S3FileSystem with LocalFileSystem, and assert through as_polars()/iter_chunks(): a failed read raises instead of truncating, read kwargs replace the defaults, schema_overrides apply to new_columns, timestamps truncate (OperationalError on a strict failure), and ignore_errors reads text; drop scan_parquet.called is bool(read_options) and the __cause__ AttributeError check. |
— |
TestPolarsDataFrameIterator::test_itertuples_nested_value_python_does_not_hold |
REWRITE | 8 | Asserts that the rows before the bad nested value are returned and that the next row raises, without naming pl.exceptions.PanicException, Polars' internal failure type. |
— |
4. polars/test_converter.py
| test | change | criteria | reason | covered by |
|---|---|---|---|---|
test_to_datetimes_not_strict |
REMOVE | 2, 8 | The non-strict path is covered through the reader, and the strict half asserts Polars' InvalidOperationError. |
polars/test_result_set.py::TestAthenaPolarsResultSet::test_read_csv_ignore_errors; strict failure: polars/test_result_set.py::TestAthenaPolarsResultSet::test_read_csv_truncates_timestamps (both rewritten in section 3) |
5. arrow/test_cursor.py
| test | change | criteria | reason | covered by |
|---|---|---|---|---|
TestArrowCursor::test_as_polars |
REWRITE | 4, 6 | ArrowCursor.as_polars() is pl.from_arrow(self._table); keep one CSV query that asserts the Polars values and drop the unload parameter (2 → 1 query). |
— |
TestArrowCursor::test_many_as_polars |
REMOVE | 2, 4, 6 | The 10,000-row values are asserted on the Arrow table, and the Polars side is pl.from_arrow (2 queries). |
TestArrowCursor::test_many_as_arrow [original], TestArrowCursor::test_as_polars (rewritten) |
TestArrowCursor::test_complex_as_polars, TestArrowCursor::test_complex_unload_as_polars |
REMOVE | 6, 8, 2 | They assert the Polars dtypes and values that pl.from_arrow makes from the Arrow table, which tests Polars; test_complex_unload_as_polars failed once on a Polars-version Map dtype (#1239). |
TestArrowCursor::test_complex_as_arrow [original], TestArrowCursor::test_complex_unload_as_arrow [original] |
TestArrowCursor::test_timeout_parameters, TestArrowCursor::test_timeout_parameters_float |
REMOVE | 1, 6, 4 | They assert the private _connect_timeout/_request_timeout of the cursor and result set; the float variant tests Python (2 queries). |
TestArrowCursor::test_read_options (offline, same plumbing with the execute() override) |
TestArrowCursor::test_fetch_converts_each_value_once |
REWRITE | 1 | Regression for d1dee80 (values converted twice): assert the rows of a non-idempotent converter, e.g. lambda v: v + "!" gives [("a!",), ("b!",)], instead of counting converter calls. |
— |
6. arrow/test_async_cursor.py
| test | change | criteria | reason | covered by |
|---|---|---|---|---|
TestAsyncArrowCursor::test_binary_null_vs_empty |
REMOVE | 2, 4 | Same AthenaArrowResultSet and query as the sync test; the async cursor adds no conversion code. |
arrow/test_cursor.py::TestArrowCursor::test_binary_null_vs_empty |
7. arrow/test_result_set.py
| test | change | criteria | reason | covered by |
|---|---|---|---|---|
TestAthenaArrowResultSet::test_fetch_after_close |
REMOVE | 5, 2 | The query execution is FAILED, so the result set has no rows and the test passes even if close() does nothing. |
test_result_set.py::TestAthenaResultSet::test_fetch_after_close |
TestAthenaArrowResultSet::test_fetch_across_batches |
MERGE | 2, 1 | Its unique part is mixing methods (fetchmany(2) then fetchone()) and the calls after exhaustion (fetchmany() == [], fetchall() == [], fetchone() is None), which depend on Arrow's own _fetch. |
into TestAthenaArrowResultSet::test_fetch_across_empty_batches (add the mixed and after-exhaustion asserts) |
TestAthenaArrowResultSet::test_open_csv_closes_file_when_stream_fails |
REMOVE | 1, 7 | Calls the private _open_csv on an __new__ instance and asserts file.close.assert_called_once_with() on a mock. |
— |
TestAthenaArrowResultSet::test_read_csv_timestamps_with_the_same_name |
MERGE | 1 | Positional typing of same-named TIMESTAMP columns, tested by bypassing __init__ and patching _fetch_all_rows_as_csv; add a duplicated TIMESTAMP column to the integration query, whose managed parameter already reads through GetQueryResults. |
into arrow/test_cursor.py::TestArrowCursor::test_duplicate_column_names |
8. arrow/test_reader.py
| test | change | criteria | reason | covered by |
|---|---|---|---|---|
TestParquetChunkReader::test_releases_files_opened_ahead |
REWRITE | 1 | Asserts that more files were opened than received and that all are closed, instead of the exact count 4 if corrupt else 2, which pins a lookahead of one; the close parameter folds into the cover. |
close parameter: TestParquetChunkReader::test_close |
TestParquetChunkReader::test_open_failure_opens_no_further_file |
REMOVE | 1, 7 | Patches the private _open and asserts which paths were attempted after a failure. |
— |
TestParquetChunkReader::test_prefetched_file |
REWRITE | 1 | Keeps the one-GET request contract for a whole-read file (fetched == [(0, size)]) and drops fs.opened == [(None, "none")], the internal _open arguments. |
— |
TestParquetChunkReader::test_streamed_file_row_groups |
REWRITE | 1, 8 | Asserts laziness instead of ranges that depend on PyArrow's pre-buffer coalescing: after the first chunk fewer bytes than the file size are fetched, and the total is 300 rows. | — |
9. s3fs/test_cursor.py
| test | change | criteria | reason | covered by |
|---|---|---|---|---|
TestS3FSCursor::test_query_id |
REMOVE | 2, 6 | query_id is None before execute() and set after, in base cursor code. |
TestS3FSCursor::test_on_start_query_execution_execute (callback_query_id == cursor.query_id) |
TestS3FSCursor::test_description_without_execute |
REMOVE | 2, 6 | description is inherited unchanged from the base cursor. |
test_cursor.py::TestCursor::test_description_initial [original] |
TestS3FSCursor::test_description_with_select |
REMOVE | 2 | A one-column description is a subset of the 19-column description already asserted. | TestS3FSCursor::test_complex, TestS3FSCursor::test_show_columns |
TestS3FSCursor::test_executemany |
REWRITE | — | Permanently @pytest.mark.skip because it needs a nonexistent insert_test table; use the empty_table fixture and assert the inserted rows as arrow/test_cursor.py::TestArrowCursor::test_executemany does, or delete it. |
— |
TestS3FSCursor::test_on_start_query_execution |
REMOVE | 2, 4 | The cursor-level callback is stored and called by shared common.py code, not S3FS code. |
test_connection.py::test_cursor_accepts_connection_callbacks (rewritten by the top-level sub-issue), test_cursor.py::TestCursor::test_connection_level_callback |
TestS3FSCursor::test_contain_tab_character |
REWRITE | 4 | One SELECT with the tab, newline, and tab+newline columns, asserting each value for both readers (4 → 1 query per reader); the docstring's tab-delimiter claim is wrong, since CSV results use commas. | — |
TestS3FSCursor::test_quoted_string_with_comma |
MERGE | 2, 4 | 'a,b,c' becomes another column of the rewritten special-character query. |
into TestS3FSCursor::test_contain_tab_character (rewritten) |
TestS3FSCursor::test_basic_query_with_reader, TestS3FSCursor::test_multiple_columns_with_reader |
REMOVE | 2, 4 | Both readers returning a basic row is already proven on a real query for both readers (2 queries each). | TestS3FSCursor::test_null_vs_empty_string; default reader: TestS3FSCursor::test_complex |
TestS3FSCursor::test_null_with_empty_string_as_null_reader, TestS3FSCursor::test_null_with_athena_reader, TestS3FSCursor::test_empty_string_with_empty_string_as_null_reader, TestS3FSCursor::test_empty_string_with_athena_reader, TestS3FSCursor::test_mixed_values_with_athena_reader |
REMOVE | 2, 4 | Each checks one cell of the NULL vs '' matrix that one parametrized query checks in full (5 queries). |
TestS3FSCursor::test_null_vs_empty_string |
10. s3fs/test_async_cursor.py
| test | change | criteria | reason | covered by |
|---|---|---|---|---|
TestAsyncS3FSCursor::test_null_vs_empty_string |
REMOVE | 2, 4 | Same AthenaS3FSResultSet and readers as the sync test; the csv_reader plumbing to the async result set is covered offline (2 queries). |
s3fs/test_cursor.py::TestS3FSCursor::test_null_vs_empty_string, TestAsyncS3FSCursor::test_read_options |
TestAsyncS3FSCursor::test_complex, TestAsyncS3FSCursor::test_show_columns |
REMOVE | 2, 4 | Identical queries and assertions to the sync tests on the same result-set class. | s3fs/test_cursor.py::TestS3FSCursor::test_complex, s3fs/test_cursor.py::TestS3FSCursor::test_show_columns |
11. s3fs/test_result_set.py
| test | change | criteria | reason | covered by |
|---|---|---|---|---|
TestAthenaS3FSResultSet::test_fetch_across_batches |
REMOVE | 2 | fetchmany/fetchone/fetchall/rownumber over a mocked CSV are asserted by the cover across 8 parameters. |
TestAthenaS3FSResultSet::test_reader_null_conversion |
TestAthenaS3FSResultSet::test_get_csv_open_options |
REWRITE | 1, 6 | Asserts only the user-visible precedence (prefetch on for PyAthena's filesystems, off with block_size or a user subclass) instead of calling the private _get_csv_open_options and comparing it with the _PREFETCH_OPTIONS constant. |
— |
12. s3fs/test_reader.py
| test | change | criteria | reason | covered by |
|---|---|---|---|---|
TestEmptyStringAsNullCSVReader::test_basic_parsing, TestEmptyStringAsNullCSVReader::test_tab_delimiter, TestEmptyStringAsNullCSVReader::test_quoted_field_with_comma, TestEmptyStringAsNullCSVReader::test_quoted_field_with_newline, TestEmptyStringAsNullCSVReader::test_escaped_quote, TestEmptyStringAsNullCSVReader::test_empty_file |
REMOVE | 6 | EmptyStringAsNullCSVReader wraps Python's csv.reader, so these test the standard library. |
basic and close: TestEmptyStringAsNullCSVReader::test_context_manager; tab: s3fs/test_result_set.py::TestAthenaS3FSResultSet::test_reader_null_conversion[.txt-EmptyStringAsNullCSVReader] |
TestEmptyStringAsNullCSVReader::test_implements_iterator_protocol, TestAthenaCSVReader::test_implements_iterator_protocol |
REMOVE | 6 | isinstance(reader, Iterator) asserts the class declaration. |
— |
TestEmptyStringAsNullCSVReader::test_close, TestAthenaCSVReader::test_close |
REMOVE | 2 | __exit__ calls close(), and the cover asserts that iteration stops afterwards. |
TestEmptyStringAsNullCSVReader::test_context_manager, TestAthenaCSVReader::test_context_manager |
TestAthenaCSVReader::test_no_trailing_null |
REMOVE | 2 | Identical input and output ("a,b,c\n") to a row of the basic test. |
TestAthenaCSVReader::test_basic_parsing |
TestAthenaCSVReader::test_quoted_field_with_newline |
REMOVE | 2 | One embedded newline is subsumed by the multi-newline and multi-row cases. | TestAthenaCSVReader::test_quoted_field_with_multiple_newlines, TestAthenaCSVReader::test_multiple_rows_with_multiline_field |
Notes
- Flaky tests named in Audit the test suite for unnecessary and flaky tests #1239:
arrow/test_cursor.py::TestArrowCursor::test_complex_unload_as_polarsis REMOVE (section 5).polars/test_result_set.py::TestPolarsDataFrameIterator::test_itertuples_value_python_does_not_holdis KEEP; since 90c94d9 it matches only"year", which both Python messages contain.polars/test_cursor.py::TestPolarsCursor::test_null_vs_empty_stringis KEEP; its one failure (run 38560114057) belongs to the flakiness classification. - The per-cursor copies of the original Arrow suite in the Polars and S3FS cursors (sync and async: fetch methods and iterator over CSV and UNLOAD,
arraysize,description,query_execution,poll,bad_query,cancel,open_close,no_ops,empty_result,executemany,show_columns) are KEEP because they follow the original per-cursor shape. They are the largest remaining pool of queries, about 40 for Polars alone. test_cancelin all 6 cursors uses a random 5-10 s sleep and a cross join (criteria 3 and 4). It is KEEP because cross-thread cancel is a supported contract.test_read_optionsin the Polars, Arrow, and S3FS cursors (sync and async) patches the private_execute/_polland asserts the result-set constructor kwargs. It is KEEP as the only check thatblock_size,cache_type,s3_max_workers,chunksize, the timeouts, andcsv_readerreach the reader, withexecute()overriding the cursor.arrow/test_cursor.py::TestArrowCursor::test_iceberg_table[original] runs about 12 queries (DDL, DML, MERGE, VACUUM) through the base cursor path.arrow/test_result_set.py::TestAthenaArrowResultSet::test_create_s3_file_system_uses_session_credentialsbypasses__init__and reads PyArrow's__reduce__internals (criteria 1 and 8). It is KEEP as the only check for the ArrowCursor's native S3 filesystem ignores the connection's profile and role credentials #1115 credentials bug.polars/test_result_set.py::TestPolarsDataFrameIterator::test_itertuples_large_chunkuses the private_ROW_BLOCK_SIZE; integration results never cross a 10,000-row block.- The
match=text inarrow/test_reader.py::TestParquetChunkReader::{test_columns_not_top_level, test_invalid_chunksize}, the Polars and S3FStest_ambiguous_time_zone_value, ands3fs/test_result_set.py::TestAthenaS3FSResultSet::test_reader_failure_closes_fileis not a documented contract; the exception types stay asserted and thematch=arguments can be dropped. - Unused after the change: in
polars/test_result_set.py, the patch targetspyathena.polars.result_set.fsspec.openandto_column_infoafter the storage-options MERGE (Break result sets after failed or cancelled fetches and drop the cross-thread guards #1238 already removes thethreading,ThreadPoolExecutor, andBytesIOimports, and itstest_read_after_failed_readkeepsinspect);datetimeinarrow/test_result_set.py;datetimeandDecimalins3fs/test_async_cursor.py;collections.abc.Iteratorins3fs/test_reader.py.ImmediateExecutor,_RecordingFile, and_RecordingFileSysteminarrow/test_reader.pystay in use. No fixture ortests/pyathena/util.pyhelper becomes unused. - Each row's cover is re-checked before the row is applied. A row whose cover does not hold becomes MERGE or KEEP.
Activity
Check that storage options reach chunked CSV reads and that fetches stop
Read the local Polars results through GetQueryResults and as_polars()
Check the read options and conversions that the removed tests protected
Test the cleanup and routing that the follow-up review found untested
Read the S3FS test metadata through GetQueryResults
laughingman7743 commentedon Oct 18, 2026
The row of section 9 that #1252 kept until #1243,
s3fs/test_cursor.py::TestS3FSCursor::test_on_start_query_execution, is removed by #1261: since #1253,test_connection.py::test_cursor_accepts_connection_callbacksruns the connection's callbacks throughS3FSCursor.Check the callback that the connection's cursor_kwargs give a cursor
Run the callbacks given through cursor_kwargs for every cursor class
Check that cursor_kwargs callbacks win over the connection's
Keep a case of callbacks given only through cursor_kwargs
Remove the S3FS cursor-level callback test
Check the callback that the connection's cursor_kwargs give a cursor
Run the callbacks given through cursor_kwargs for every cursor class
Check that cursor_kwargs callbacks win over the connection's
Keep a case of callbacks given only through cursor_kwargs