Skip to content

Remove and rewrite unnecessary tests in the Polars, Arrow, and S3FS cursor tests #1246

Closed

Description

@laughingman7743

Summary

Part of #1239. Files: polars/, arrow/, s3fs/ (test_cursor.py, test_async_cursor.py, test_result_set.py, test_converter.py, test_reader.py, test_util.py). At master 59169c5 with the cross-check corrections: 289 test functions: 212 KEEP, 55 REMOVE, 6 MERGE, 16 REWRITE. #1238 deletes 4 of the REMOVE rows; the tables list 51 REMOVE, 6 MERGE, 16 REWRITE. Applying them saves about 44 Athena queries per run: 17 in the S3FS per-reader tests, 14 in the Polars chunk tests, 7 in the ArrowCursor as_polars tests. The result-set, converter, and reader rows are offline.

Paths are relative to tests/pyathena/. [original] marks a test that existed at 21cbf05 (v3.15.0).

Order

  • After Break result sets after failed or cancelled fetches and drop the cross-thread guards #1238 (06adf23). It deletes polars/test_result_set.py::TestAthenaPolarsResultSet::{test_close_during_fetchall, test_close_during_fetchall_generator, test_close_during_first_read} and TestPolarsDataFrameIterator::test_close_during_read, modifies the KEEP TestAthenaPolarsResultSet::test_fetch_after_conversion_error, and adds TestAthenaPolarsResultSet::{test_fetch_after_failed_chunk_read, test_fetchone_interrupted_while_counting} and TestPolarsDataFrameIterator::test_read_after_failed_read. Section 3 is rebased onto it.
  • After Read UNLOAD timestamps at microseconds #1233 (1dc9b28). It modifies arrow/test_cursor.py::TestArrowCursor::{test_complex_as_polars, test_complex_unload_as_polars} (section 5, REMOVE still holds; only col_date moves to pl.Date), their covers test_complex_as_arrow, test_complex_unload_as_arrow, and test_temporal_types, and polars/test_result_set.py::TestAthenaPolarsResultSet::{test_storage_options_replace_defaults, test_iter_parquet_chunks_raises_when_read_fails_partway} (section 3). It changes the private _read_parquet/_iter_parquet_chunks signatures to take a pl.Schema, which the section 3 REWRITE no longer calls. It also changes polars/test_cursor.py::TestPolarsCursor::test_temporal_types and adds tests that were not audited: TestArrowCursor::test_unload_nested_timestamps, TestPolarsCursor::{test_unload_nested_timestamps, test_unload_schema_option, test_unload_timestamps_with_read_options_polars_1}.
  • Read CSV dates as date32 and temporal text NULL as null in ArrowCursor #1236 is merged (master 7e5d9ee). It changes col_date to pa.date32() in TestArrowCursor::test_complex_as_arrow and to pl.Date in test_complex_as_polars, removes the #1216 xfail from test_temporal_types, and adds TestArrowCursor::test_temporal_text_null_with_binary and arrow/test_converter.py::test_to_null (not audited). Re-checked at 7e5d9ee: the section 5 REMOVE of test_complex_as_polars still holds, because the Polars dtypes follow from the Arrow table that test_complex_as_arrow asserts.
  • s3fs/test_cursor.py::TestS3FSCursor::test_on_start_query_execution (section 9) is removed with or after the top-level sub-issue's REWRITE of test_connection.py::test_cursor_accepts_connection_callbacks, which must keep running the callback for every class in CURSOR_CLASSES.
  • Covers within this sub-issue that are MERGE/REWRITE targets: polars/test_converter.py::test_to_datetimes_not_strict needs the section 3 REWRITE of test_read_csv_ignore_errors and test_read_csv_truncates_timestamps (keep the non-strict read and the OperationalError); TestArrowCursor::test_many_as_polars needs the rewritten TestArrowCursor::test_as_polars; TestS3FSCursor::test_quoted_string_with_comma merges into the rewritten TestS3FSCursor::test_contain_tab_character.

Findings

1. polars/test_cursor.py

test change criteria reason covered by
TestPolarsCursor::test_as_arrow REMOVE 2, 6 as_arrow() is as_polars().to_arrow(), so this checks Polars' to_arrow (2 queries). TestPolarsCursor::test_whole_result_reused (asserts as_arrow() values)
TestPolarsCursor::test_iter_chunks, TestPolarsCursor::test_iter_chunks_many_rows REMOVE 2 Row totals and "is a DataFrame" checks of CSV chunking with another chunksize add nothing to the consistency test. TestPolarsCursor::test_iter_chunks_data_consistency (with the merged size bound)
TestPolarsCursor::test_iter_chunks_without_chunksize REMOVE 2 Without a chunksize, iter_chunks() yields one whole chunk, which is already asserted. TestPolarsCursor::test_whole_result_reused ([3] chunk heights)
TestPolarsCursor::test_iter_chunks_unload REMOVE 2 A chunked UNLOAD row total is weaker than the exact sizes, schema, and equality checks of the cover. TestPolarsCursor::test_unload_chunks
TestPolarsCursor::test_iter_chunks_chunk_sizes MERGE 2 Its only extra assertion is chunk.height <= chunksize. into TestPolarsCursor::test_iter_chunks_data_consistency
TestPolarsCursor::test_fetchone_with_chunksize, TestPolarsCursor::test_fetchmany_with_chunksize, TestPolarsCursor::test_fetchall_with_chunksize, TestPolarsCursor::test_iterator_with_chunksize, TestPolarsCursor::test_fetchone_with_chunksize_unload, TestPolarsCursor::test_iterator_with_chunksize_unload REMOVE 2 Row-count-only checks of fetching across chunks; the cover checks values and rownumber for CSV and UNLOAD (6 queries). TestPolarsCursor::test_fetch_across_chunks; description: TestPolarsCursor::test_description_with_chunksize_unload; iteration: TestPolarsCursor::test_iterator

2. polars/test_async_cursor.py

test change criteria reason covered by
TestAsyncPolarsCursor::test_as_arrow REMOVE 2, 6 The same Polars to_arrow wrapper through the async future (2 queries). polars/test_cursor.py::TestPolarsCursor::test_whole_result_reused
TestAsyncPolarsCursor::test_read_options REWRITE 1, 6 Keeps the option-plumbing assertions and drops cursor._executor._max_workers == 1, which only checks that the constructor stored its argument. —

3. polars/test_result_set.py

#1238 deletes TestAthenaPolarsResultSet::test_close_during_fetchall, TestAthenaPolarsResultSet::test_close_during_fetchall_generator, TestAthenaPolarsResultSet::test_close_during_first_read, and TestPolarsDataFrameIterator::test_close_during_read; they are not listed.

test change criteria reason covered by
TestAthenaPolarsResultSet::test_fetchmany_size_not_integer REMOVE 2 The size is validated by the base AthenaResultSet.fetchmany (operator.index) before the Polars _take_rows runs. test_result_set.py::TestAthenaResultSet::test_fetchmany_size_not_integer
TestPolarsDataFrameIterator::test_itertuples_without_column_names REMOVE 7 Exercises the repeat((), length) branch for a chunk with rows but no column names, which Athena results never have. —
TestAthenaPolarsResultSet::test_storage_options_replace_defaults, TestAthenaPolarsResultSet::test_csv_storage_options_replace_defaults MERGE 1, 5 They assert the kwargs of mocked polars.scan_csv/read_parquet/scan_parquet/fsspec.open and that private properties are not computed; the contract that storage_options replace PyAthena's is observable end to end. into polars/test_cursor.py::TestPolarsCursor::test_as_polars_with_read_kwargs (parametrized over unload and chunksize)
TestAthenaPolarsResultSet::test_iter_csv_chunks_raises_when_read_fails_partway, TestAthenaPolarsResultSet::test_iter_parquet_chunks_raises_when_read_fails_partway, TestAthenaPolarsResultSet::test_csv_read_kwargs_replace_defaults, TestAthenaPolarsResultSet::test_txt_new_columns_with_schema_overrides, TestAthenaPolarsResultSet::test_read_csv_truncates_timestamps, TestAthenaPolarsResultSet::test_read_csv_ignore_errors REWRITE 1 Build the result set through its constructor over a local output location, patching only the _pre_fetch metadata and S3FileSystem with LocalFileSystem, and assert through as_polars()/iter_chunks(): a failed read raises instead of truncating, read kwargs replace the defaults, schema_overrides apply to new_columns, timestamps truncate (OperationalError on a strict failure), and ignore_errors reads text; drop scan_parquet.called is bool(read_options) and the __cause__ AttributeError check. —
TestPolarsDataFrameIterator::test_itertuples_nested_value_python_does_not_hold REWRITE 8 Asserts that the rows before the bad nested value are returned and that the next row raises, without naming pl.exceptions.PanicException, Polars' internal failure type. —

4. polars/test_converter.py

test change criteria reason covered by
test_to_datetimes_not_strict REMOVE 2, 8 The non-strict path is covered through the reader, and the strict half asserts Polars' InvalidOperationError. polars/test_result_set.py::TestAthenaPolarsResultSet::test_read_csv_ignore_errors; strict failure: polars/test_result_set.py::TestAthenaPolarsResultSet::test_read_csv_truncates_timestamps (both rewritten in section 3)

5. arrow/test_cursor.py

test change criteria reason covered by
TestArrowCursor::test_as_polars REWRITE 4, 6 ArrowCursor.as_polars() is pl.from_arrow(self._table); keep one CSV query that asserts the Polars values and drop the unload parameter (2 → 1 query). —
TestArrowCursor::test_many_as_polars REMOVE 2, 4, 6 The 10,000-row values are asserted on the Arrow table, and the Polars side is pl.from_arrow (2 queries). TestArrowCursor::test_many_as_arrow [original], TestArrowCursor::test_as_polars (rewritten)
TestArrowCursor::test_complex_as_polars, TestArrowCursor::test_complex_unload_as_polars REMOVE 6, 8, 2 They assert the Polars dtypes and values that pl.from_arrow makes from the Arrow table, which tests Polars; test_complex_unload_as_polars failed once on a Polars-version Map dtype (#1239). TestArrowCursor::test_complex_as_arrow [original], TestArrowCursor::test_complex_unload_as_arrow [original]
TestArrowCursor::test_timeout_parameters, TestArrowCursor::test_timeout_parameters_float REMOVE 1, 6, 4 They assert the private _connect_timeout/_request_timeout of the cursor and result set; the float variant tests Python (2 queries). TestArrowCursor::test_read_options (offline, same plumbing with the execute() override)
TestArrowCursor::test_fetch_converts_each_value_once REWRITE 1 Regression for d1dee80 (values converted twice): assert the rows of a non-idempotent converter, e.g. lambda v: v + "!" gives [("a!",), ("b!",)], instead of counting converter calls. —

6. arrow/test_async_cursor.py

test change criteria reason covered by
TestAsyncArrowCursor::test_binary_null_vs_empty REMOVE 2, 4 Same AthenaArrowResultSet and query as the sync test; the async cursor adds no conversion code. arrow/test_cursor.py::TestArrowCursor::test_binary_null_vs_empty

7. arrow/test_result_set.py

test change criteria reason covered by
TestAthenaArrowResultSet::test_fetch_after_close REMOVE 5, 2 The query execution is FAILED, so the result set has no rows and the test passes even if close() does nothing. test_result_set.py::TestAthenaResultSet::test_fetch_after_close
TestAthenaArrowResultSet::test_fetch_across_batches MERGE 2, 1 Its unique part is mixing methods (fetchmany(2) then fetchone()) and the calls after exhaustion (fetchmany() == [], fetchall() == [], fetchone() is None), which depend on Arrow's own _fetch. into TestAthenaArrowResultSet::test_fetch_across_empty_batches (add the mixed and after-exhaustion asserts)
TestAthenaArrowResultSet::test_open_csv_closes_file_when_stream_fails REMOVE 1, 7 Calls the private _open_csv on an __new__ instance and asserts file.close.assert_called_once_with() on a mock. —
TestAthenaArrowResultSet::test_read_csv_timestamps_with_the_same_name MERGE 1 Positional typing of same-named TIMESTAMP columns, tested by bypassing __init__ and patching _fetch_all_rows_as_csv; add a duplicated TIMESTAMP column to the integration query, whose managed parameter already reads through GetQueryResults. into arrow/test_cursor.py::TestArrowCursor::test_duplicate_column_names

8. arrow/test_reader.py

test change criteria reason covered by
TestParquetChunkReader::test_releases_files_opened_ahead REWRITE 1 Asserts that more files were opened than received and that all are closed, instead of the exact count 4 if corrupt else 2, which pins a lookahead of one; the close parameter folds into the cover. close parameter: TestParquetChunkReader::test_close
TestParquetChunkReader::test_open_failure_opens_no_further_file REMOVE 1, 7 Patches the private _open and asserts which paths were attempted after a failure. —
TestParquetChunkReader::test_prefetched_file REWRITE 1 Keeps the one-GET request contract for a whole-read file (fetched == [(0, size)]) and drops fs.opened == [(None, "none")], the internal _open arguments. —
TestParquetChunkReader::test_streamed_file_row_groups REWRITE 1, 8 Asserts laziness instead of ranges that depend on PyArrow's pre-buffer coalescing: after the first chunk fewer bytes than the file size are fetched, and the total is 300 rows. —

9. s3fs/test_cursor.py

test change criteria reason covered by
TestS3FSCursor::test_query_id REMOVE 2, 6 query_id is None before execute() and set after, in base cursor code. TestS3FSCursor::test_on_start_query_execution_execute (callback_query_id == cursor.query_id)
TestS3FSCursor::test_description_without_execute REMOVE 2, 6 description is inherited unchanged from the base cursor. test_cursor.py::TestCursor::test_description_initial [original]
TestS3FSCursor::test_description_with_select REMOVE 2 A one-column description is a subset of the 19-column description already asserted. TestS3FSCursor::test_complex, TestS3FSCursor::test_show_columns
TestS3FSCursor::test_executemany REWRITE — Permanently @pytest.mark.skip because it needs a nonexistent insert_test table; use the empty_table fixture and assert the inserted rows as arrow/test_cursor.py::TestArrowCursor::test_executemany does, or delete it. —
TestS3FSCursor::test_on_start_query_execution REMOVE 2, 4 The cursor-level callback is stored and called by shared common.py code, not S3FS code. test_connection.py::test_cursor_accepts_connection_callbacks (rewritten by the top-level sub-issue), test_cursor.py::TestCursor::test_connection_level_callback
TestS3FSCursor::test_contain_tab_character REWRITE 4 One SELECT with the tab, newline, and tab+newline columns, asserting each value for both readers (4 → 1 query per reader); the docstring's tab-delimiter claim is wrong, since CSV results use commas. —
TestS3FSCursor::test_quoted_string_with_comma MERGE 2, 4 'a,b,c' becomes another column of the rewritten special-character query. into TestS3FSCursor::test_contain_tab_character (rewritten)
TestS3FSCursor::test_basic_query_with_reader, TestS3FSCursor::test_multiple_columns_with_reader REMOVE 2, 4 Both readers returning a basic row is already proven on a real query for both readers (2 queries each). TestS3FSCursor::test_null_vs_empty_string; default reader: TestS3FSCursor::test_complex
TestS3FSCursor::test_null_with_empty_string_as_null_reader, TestS3FSCursor::test_null_with_athena_reader, TestS3FSCursor::test_empty_string_with_empty_string_as_null_reader, TestS3FSCursor::test_empty_string_with_athena_reader, TestS3FSCursor::test_mixed_values_with_athena_reader REMOVE 2, 4 Each checks one cell of the NULL vs '' matrix that one parametrized query checks in full (5 queries). TestS3FSCursor::test_null_vs_empty_string

10. s3fs/test_async_cursor.py

test change criteria reason covered by
TestAsyncS3FSCursor::test_null_vs_empty_string REMOVE 2, 4 Same AthenaS3FSResultSet and readers as the sync test; the csv_reader plumbing to the async result set is covered offline (2 queries). s3fs/test_cursor.py::TestS3FSCursor::test_null_vs_empty_string, TestAsyncS3FSCursor::test_read_options
TestAsyncS3FSCursor::test_complex, TestAsyncS3FSCursor::test_show_columns REMOVE 2, 4 Identical queries and assertions to the sync tests on the same result-set class. s3fs/test_cursor.py::TestS3FSCursor::test_complex, s3fs/test_cursor.py::TestS3FSCursor::test_show_columns

11. s3fs/test_result_set.py

test change criteria reason covered by
TestAthenaS3FSResultSet::test_fetch_across_batches REMOVE 2 fetchmany/fetchone/fetchall/rownumber over a mocked CSV are asserted by the cover across 8 parameters. TestAthenaS3FSResultSet::test_reader_null_conversion
TestAthenaS3FSResultSet::test_get_csv_open_options REWRITE 1, 6 Asserts only the user-visible precedence (prefetch on for PyAthena's filesystems, off with block_size or a user subclass) instead of calling the private _get_csv_open_options and comparing it with the _PREFETCH_OPTIONS constant. —

12. s3fs/test_reader.py

test change criteria reason covered by
TestEmptyStringAsNullCSVReader::test_basic_parsing, TestEmptyStringAsNullCSVReader::test_tab_delimiter, TestEmptyStringAsNullCSVReader::test_quoted_field_with_comma, TestEmptyStringAsNullCSVReader::test_quoted_field_with_newline, TestEmptyStringAsNullCSVReader::test_escaped_quote, TestEmptyStringAsNullCSVReader::test_empty_file REMOVE 6 EmptyStringAsNullCSVReader wraps Python's csv.reader, so these test the standard library. basic and close: TestEmptyStringAsNullCSVReader::test_context_manager; tab: s3fs/test_result_set.py::TestAthenaS3FSResultSet::test_reader_null_conversion[.txt-EmptyStringAsNullCSVReader]
TestEmptyStringAsNullCSVReader::test_implements_iterator_protocol, TestAthenaCSVReader::test_implements_iterator_protocol REMOVE 6 isinstance(reader, Iterator) asserts the class declaration. —
TestEmptyStringAsNullCSVReader::test_close, TestAthenaCSVReader::test_close REMOVE 2 __exit__ calls close(), and the cover asserts that iteration stops afterwards. TestEmptyStringAsNullCSVReader::test_context_manager, TestAthenaCSVReader::test_context_manager
TestAthenaCSVReader::test_no_trailing_null REMOVE 2 Identical input and output ("a,b,c\n") to a row of the basic test. TestAthenaCSVReader::test_basic_parsing
TestAthenaCSVReader::test_quoted_field_with_newline REMOVE 2 One embedded newline is subsumed by the multi-newline and multi-row cases. TestAthenaCSVReader::test_quoted_field_with_multiple_newlines, TestAthenaCSVReader::test_multiple_rows_with_multiline_field

Notes

  • Flaky tests named in Audit the test suite for unnecessary and flaky tests #1239: arrow/test_cursor.py::TestArrowCursor::test_complex_unload_as_polars is REMOVE (section 5). polars/test_result_set.py::TestPolarsDataFrameIterator::test_itertuples_value_python_does_not_hold is KEEP; since 90c94d9 it matches only "year", which both Python messages contain. polars/test_cursor.py::TestPolarsCursor::test_null_vs_empty_string is KEEP; its one failure (run 38560114057) belongs to the flakiness classification.
  • The per-cursor copies of the original Arrow suite in the Polars and S3FS cursors (sync and async: fetch methods and iterator over CSV and UNLOAD, arraysize, description, query_execution, poll, bad_query, cancel, open_close, no_ops, empty_result, executemany, show_columns) are KEEP because they follow the original per-cursor shape. They are the largest remaining pool of queries, about 40 for Polars alone.
  • test_cancel in all 6 cursors uses a random 5-10 s sleep and a cross join (criteria 3 and 4). It is KEEP because cross-thread cancel is a supported contract.
  • test_read_options in the Polars, Arrow, and S3FS cursors (sync and async) patches the private _execute/_poll and asserts the result-set constructor kwargs. It is KEEP as the only check that block_size, cache_type, s3_max_workers, chunksize, the timeouts, and csv_reader reach the reader, with execute() overriding the cursor.
  • arrow/test_cursor.py::TestArrowCursor::test_iceberg_table [original] runs about 12 queries (DDL, DML, MERGE, VACUUM) through the base cursor path.
  • arrow/test_result_set.py::TestAthenaArrowResultSet::test_create_s3_file_system_uses_session_credentials bypasses __init__ and reads PyArrow's __reduce__ internals (criteria 1 and 8). It is KEEP as the only check for the ArrowCursor's native S3 filesystem ignores the connection's profile and role credentials #1115 credentials bug.
  • polars/test_result_set.py::TestPolarsDataFrameIterator::test_itertuples_large_chunk uses the private _ROW_BLOCK_SIZE; integration results never cross a 10,000-row block.
  • The match= text in arrow/test_reader.py::TestParquetChunkReader::{test_columns_not_top_level, test_invalid_chunksize}, the Polars and S3FS test_ambiguous_time_zone_value, and s3fs/test_result_set.py::TestAthenaS3FSResultSet::test_reader_failure_closes_file is not a documented contract; the exception types stay asserted and the match= arguments can be dropped.
  • Unused after the change: in polars/test_result_set.py, the patch targets pyathena.polars.result_set.fsspec.open and to_column_info after the storage-options MERGE (Break result sets after failed or cancelled fetches and drop the cross-thread guards #1238 already removes the threading, ThreadPoolExecutor, and BytesIO imports, and its test_read_after_failed_read keeps inspect); datetime in arrow/test_result_set.py; datetime and Decimal in s3fs/test_async_cursor.py; collections.abc.Iterator in s3fs/test_reader.py. ImmediateExecutor, _RecordingFile, and _RecordingFileSystem in arrow/test_reader.py stay in use. No fixture or tests/pyathena/util.py helper becomes unused.
  • Each row's cover is re-checked before the row is applied. A row whose cover does not hold becomes MERGE or KEEP.

Activity

  1. laughingman7743 commented on Oct 18, 2026

    @laughingman7743
    MemberAuthor

    The row of section 9 that #1252 kept until #1243, s3fs/test_cursor.py::TestS3FSCursor::test_on_start_query_execution, is removed by #1261: since #1253, test_connection.py::test_cursor_accepts_connection_callbacks runs the connection's callbacks through S3FSCursor.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions