Skip to content

Remove, merge, and trim unnecessary tests in the pandas tests #1244

Closed
#1256
Closed
Remove, merge, and trim unnecessary tests in the pandas tests#1244
#1256
Labels
enhancement

Description

@laughingman7743
laughingman7743
opened on Oct 17, 2026

Summary

Part of #1239. Files: pandas/test_cursor.py, pandas/test_async_cursor.py, aio/pandas/test_cursor.py, pandas/test_result_set.py, pandas/test_util.py, pandas/test_converter.py (pandas/test_reader.py has no findings). At master 59169c5 with the cross-check corrections: 194 test functions: 88 KEEP, 50 REMOVE, 9 MERGE, 47 REWRITE. #1238 deletes 4 of the REMOVE tests, so this PR removes 46. The changes save about 150 Athena queries and 5 external tables per full run: about 25 queries from sync REMOVE/MERGE, about 38 from sync parameter trims, about 42 queries and 5 external tables from async REMOVE, about 25 from async parameter trims, about 7 from aio, and up to 20 from the test_binary_csv_stream_close* rewrite. The test_result_set.py, test_converter.py and aio offline rows save no AWS calls.

Paths are relative to tests/pyathena/. [original] marks a test that existed at 21cbf05 (v3.15.0).

Two facts apply to many rows:

  • engine="auto" and engine="pyarrow" are the same UNLOAD path. fastparquet was removed in 104c0d0; _get_parquet_engine() returns "pyarrow" for "auto". Every parametrization with both ({"unload": True}, "auto") and ({"unload": True}, "pyarrow") runs the same query twice. DDL (SHOW COLUMNS, CREATE EXTERNAL TABLE) is never wrapped in UNLOAD, and a .txt result always uses the C engine.
  • AsyncPandasCursor and AioPandasCursor read through the same AthenaPandasResultSet as PandasCursor. The async and aio files keep the cursor-specific API: the execute() future or coroutine, poll, query_execution, description, cancel, as_pandas, and read-option forwarding.

Order

  1. Break result sets after failed or cancelled fetches and drop the cross-thread guards #1238 (06adf23) merges first. It deletes pandas/test_result_set.py::TestPandasDataFrameIterator::test_close_during_read and TestAthenaPandasResultSet::{test_close_during_fetchall, test_close_during_fetchall_generator, test_iter_parquet_chunks_after_close}. It modifies TestAthenaPandasResultSet::test_fetch_after_conversion_error (KEEP) and pandas/test_cursor.py::TestPandasCursor::test_binary_csv_stream_close_on_read_error (REWRITE below): reads and fetches after the failed read raise ProgrammingError instead of StopIteration/None. It adds TestAthenaPandasResultSet::{test_fetch_after_failed_chunk_read, test_fetchone_interrupted_while_counting} and TestPandasDataFrameIterator::test_read_after_failed_read, which this audit did not review.
  2. Read UNLOAD timestamps at microseconds #1233 (1dc9b28) merges first. It modifies TestAthenaPandasResultSet::test_read_parquet_filesystem (REMOVE below; it adds coerce_int96_timestamp_unit="us" to the expected kwargs) and TestPandasCursor::test_temporal_types (KEEP). It adds TestPandasCursor::{test_unload_int96_unit_option, test_unload_nested_timestamps}. test_unload_int96_unit_option covers the INT96 option of the removed test_read_parquet_filesystem.
  3. Rows that depend on the aio sub-issue:
    • TestAioPandasCursor::test_cancel_initial is removed only after the aio sub-issue adds aio/test_common.py::TestWithAsyncFetch::test_cancel_without_query.
    • TestAioPandasCursor::test_invalid_arraysize is removed with or after the aio sub-issue's merge of the aio/test_connection.py::TestAioConnection arraysize tests, which must keep AioPandasCursor in the uncapped and not-positive cases.
  4. Rows in this sub-issue whose cover is a REWRITE (parameter trim) in this sub-issue land in the same PR, and the trims keep the cited cases: TestPandasCursor::test_unload_chunks keeps the query with BIGINT '9007199254740993' and check_exact=True; the trims keep TestPandasCursor::test_many_as_pandas[chunksize=1000] and TestPandasCursor::test_as_pandas[unload=False-auto-None].

Findings

1. pandas/test_cursor.py

test change criteria reason covered by
TestPandasCursor::test_binary_as_pandas REMOVE 2 Its binary NULL/empty/value check and its storage-options and chunked paths repeat other tests (2 queries). TestPandasCursor::test_binary_null_vs_empty, TestPandasCursor::test_csv_storage_options (rewritten), TestPandasCursor::test_binary_csv_stream_close (rewritten)
TestPandasCursor::test_binary_csv_stream_close (16 cases), TestPandasCursor::test_binary_csv_stream_close_on_read_error (4 cases) REWRITE 1, 4 They reach through result_set._csv_stream.buffer.raw._reader._file and run 20 queries of about 400 KB each for a close lifecycle that the engine does not change; the rewrite runs the close-method matrix offline over a real source as pandas/test_reader.py::TestBinaryCSVReader::test_close_before_eof does (or keeps one c-engine query) and asserts that the source is closed, StopIteration and fetchone() is None after a close, and, after a failed read, that the read raises the conversion error and later reads and fetches raise ProgrammingError (#1238). pandas/test_reader.py::TestBinaryCSVReader::test_close_before_eof (partial)
TestPandasCursor::test_binary_custom_dialect MERGE 2 Same contract as the target: custom quoting falls back to raw text (2 queries). into TestPandasCursor::test_binary_custom_quoting (a dialect read-options case)
TestPandasCursor::test_binary_duplicate_name_without_converter MERGE 2 Same contract as the target: no varbinary converter gives hex text (1 query and a connection). into TestPandasCursor::test_binary_without_converter (add a duplicate-name column)
TestPandasCursor::test_fetchone, test_fetchmany, test_fetchall, test_iterator, test_arraysize, test_as_pandas, test_many_as_pandas [original] REWRITE 2, 4 Drop the ({"unload": True}, "pyarrow", None) case, which repeats ({"unload": True}, "auto", None); optionally drop chunksize=1_000_000, which is single-chunk for these queries; keep test_many_as_pandas[chunksize=1000] and test_as_pandas[unload=False-auto-None]. the remaining cases of the same test
TestPandasCursor::test_complex_unload_pyarrow, test_complex_unload_as_pandas_pyarrow, test_empty_result_dml_unload [original] REWRITE 2, 4 The auto and pyarrow cases are the same UNLOAD path; keep one. the remaining case of the same test
TestPandasCursor::test_show_columns, test_empty_result_ddl [original] REWRITE 2, 4 DDL is not UNLOAD-wrapped and .txt results always use the C engine, so unload and the engine change nothing; keep chunksize None and 1_000 (test_empty_result_ddl creates 5 external tables). the remaining cases of the same test
TestPandasCursor::test_integer_na_values, test_float_na_values, test_boolean_na_values, test_not_skip_blank_lines, test_null_vs_empty_string, test_null_decimal_value [original] REWRITE 2, 4 Drop the (unload, "pyarrow") case and keep unload False and True. the remaining cases of the same test
TestPandasCursor::test_executemany [original] REWRITE 2, 4 Same (unload, "pyarrow") duplicate (3 INSERTs and 1 SELECT per case); keep unload False and True. the remaining cases of the same test
TestPandasCursor::test_executemany_fetch [original] REWRITE 2, 4 executemany discards result sets, so unload and the engine never matter; keep one case. the remaining case of the same test
TestPandasCursor::test_result_set_file_system REWRITE 1, 4 It patches S3FileSystem.__init__ to count instances and reads the private _csv_stream; the rewrite asserts only the GH-978 contract not cached_file_systems(conn) for CSV and UNLOAD (2 cases instead of 3 queries). —
TestPandasCursor::test_athena_endpoint_url REMOVE 2 GH-576 lives in the connection's s3_client, which pandas result sets read through (1 query and a connection). test_connection.py::TestConnection::test_s3_client_leaves_out_athena_endpoint, TestPandasCursor::test_result_sets_share_s3_client
TestPandasCursor::test_csv_storage_options (4 cases) REWRITE 1 It spies S3FileSystem.open and compares against the private result_set._fs; the rewrite passes storage options whose effect shows in the result (for example invalid credentials make the read raise) and drops the spy and _csv_stream asserts. —
TestPandasCursor::test_pyarrow_engine_string_values (2 cases) REMOVE 1, 2, 4 Same values ('1', NULL, 'nan', '007', '1e3') as the offline case, plus an assert_called_once on the pyarrow helper (4 queries). pandas/test_result_set.py::test_read_csv_with_pyarrow_matches_pandas[numeric_looking_strings-*]
TestPandasCursor::test_pyarrow_engine_multiline_values_across_blocks REMOVE 1, 2, 4 The offline twin parses the same multi-block multiline file; this one also asserts the helper call (1 query, 2.4 MB read). pandas/test_result_set.py::test_read_csv_with_pyarrow_multiline_values_across_blocks
TestPandasCursor::test_unload_chunks (5 cases) REWRITE 4 Chunk sizes 1, 2 and 10 with no options are one path; drop the 1-{} and 10-{} cases and keep 2-{}, 1-numpy_nullable and 2-columns, all with BIGINT '9007199254740993' and check_exact=True (6 UNLOAD queries instead of 10). the remaining cases of the same test
TestPandasCursor::test_unload_chunks_unknown_columns (2 cases) REWRITE 4 ["missing"] and ["k", "missing"] hit the same validation; keep ["k", "missing"] (2 UNLOAD queries instead of 4). the remaining case of the same test
TestPandasCursor::test_auto_determine_chunksize REMOVE 1, 6, 7 It patches AthenaResultSet.__init__ and compares the private _auto_determine_chunksize with class constants. TestPandasCursor::test_pandas_cursor_auto_optimize_chunksize_enabled[chunked] (rewritten)
TestPandasCursor::test_chunksize_configuration_customization REMOVE 1, 6 The same private check with customized class attributes. TestPandasCursor::test_pandas_cursor_auto_optimize_chunksize_enabled[chunked] (rewritten; monkeypatches the same attributes)
TestPandasCursor::test_get_csv_engine_explicit_specification REMOVE 1, 7 It sets eight private attributes and patches _get_available_engine, converters and description to assert an engine name that users see only through results; it failed in CI on a missing private _metadata (#1239). TestPandasCursor::test_duplicate_column_names, TestPandasCursor::test_binary_custom_quoting, pandas/test_result_set.py pyarrow-option test (rewritten from TestPandasCursor::test_reads_csv_with_pyarrow)
TestPandasCursor::test_reads_csv_with_pyarrow (10 cases) REWRITE 1, 7 It asserts a private predicate from the cursor class although its subject is the result set; the rewrite moves it to pandas/test_result_set.py and asserts that engine="pyarrow" honors keep_default_na, na_values=("", "NA") and on_bad_lines in the parsed values (for example "NA" becomes NaN). —
TestPandasCursor::test_pandas_cursor_iter_chunks_with_chunksize REMOVE 2, 5 It sets the private _chunksize, and chunk_count >= 1 passes for any chunking (1 query). TestPandasCursor::test_get_chunk[1000], TestPandasCursor::test_many_as_pandas[chunksize=1000]
TestPandasCursor::test_pandas_cursor_auto_optimize_chunksize_enabled (2 cases) REWRITE 1 It asserts that the private result_set._df_iter._reader is a TextFileReader; the rewrite asserts [len(c) for c in iter_chunks()] == [2, 2, 1], as pandas/test_async_cursor.py::TestAsyncPandasCursor::test_auto_optimize_chunksize does. —
TestPandasCursor::test_pandas_cursor_auto_optimize_chunksize_disabled REMOVE 2, 6 It sets private attributes to their defaults and asserts that as_pandas() returns a DataFrame (1 query). TestPandasCursor::test_as_pandas[unload=False-auto-None]
TestPandasCursor::test_pandas_cursor_explicit_chunksize_overrides_auto_optimize MERGE 1, 2 It sets the private _chunksize and _auto_optimize_chunksize; "an explicit chunksize wins" becomes a case of the target set through cursor_kwargs (1 query). into TestPandasCursor::test_pandas_cursor_auto_optimize_chunksize_enabled
TestPandasCursor::test_pandas_cursor_iter_chunks_without_chunksize REMOVE 2 iter_chunks() without a chunksize yields one whole DataFrame (1 query). TestPandasCursor::test_pandas_cursor_whole_result_reused ([len(chunk) for chunk in iter_chunks()] == [3])
TestPandasCursor::test_pandas_cursor_chunked_vs_regular_same_data REMOVE 1, 2 It sets the private _chunksize and compares chunked and whole reads (2 queries). TestPandasCursor::test_many_as_pandas[chunksize=1000], TestPandasCursor::test_unload_chunks
TestPandasCursor::test_pandas_cursor_actual_chunking_behavior REMOVE 2, 6 It checks pandas' TextFileReader chunk sizes through the private _chunksize (1 query). TestPandasCursor::test_get_chunk[1000]
TestPandasCursor::test_pandas_cursor_iter_chunks_consistency REMOVE 6, 7 iter_chunks() iterates as_pandas(), so the comparison tests the code against itself (2 queries). TestPandasCursor::test_many_as_pandas
TestPandasCursor::test_get_chunk_time MERGE 2, 4 The chunked case of the target already checks get_chunk(1) time truncation on col_time; add a NULL time there (1 query). into TestPandasCursor::test_integer_and_json_with_null[chunked]

2. pandas/test_async_cursor.py

test change criteria reason covered by
TestAsyncPandasCursor::test_binary_null_vs_empty REMOVE 2 The async cursor returns the values of the sync result set (1 query). pandas/test_cursor.py::TestPandasCursor::test_binary_null_vs_empty[c-2]
TestAsyncPandasCursor::test_auto_optimize_chunksize REMOVE 2 Forwarding is checked offline, and the chunking is result-set behavior (1 query). TestAsyncPandasCursor::test_read_options, pandas/test_cursor.py::TestPandasCursor::test_pandas_cursor_auto_optimize_chunksize_enabled[chunked] (rewritten to assert chunk sizes)
TestAsyncPandasCursor::test_many_as_pandas [original] REMOVE 2, 4 Same values as sync (5 queries of 10,000 rows). pandas/test_cursor.py::TestPandasCursor::test_many_as_pandas
TestAsyncPandasCursor::test_show_columns [original] REMOVE 2, 4 A DDL result through the same result set (5 queries). pandas/test_cursor.py::TestPandasCursor::test_show_columns
TestAsyncPandasCursor::test_empty_result_ddl [original] REMOVE 2, 4 Same empty-DDL handling as sync (5 queries and 5 external tables). pandas/test_cursor.py::TestPandasCursor::test_empty_result_ddl
TestAsyncPandasCursor::test_empty_result_dml_unload [original] REMOVE 2, 4 Same as sync (2 UNLOAD queries). pandas/test_cursor.py::TestPandasCursor::test_empty_result_dml_unload
TestAsyncPandasCursor::test_integer_na_values, test_float_na_values, test_boolean_na_values, test_not_skip_blank_lines, test_null_decimal_value [original] REMOVE 2, 4 NA conversion happens in the shared AthenaPandasResultSet (3 queries each). pandas/test_cursor.py::TestPandasCursor::{test_integer_na_values, test_float_na_values, test_boolean_na_values, test_not_skip_blank_lines, test_null_decimal_value}
TestAsyncPandasCursor::test_empty_and_null_string [original] REMOVE 2, 4 Same as sync (6 queries). pandas/test_cursor.py::TestPandasCursor::test_null_vs_empty_string
TestAsyncPandasCursor::test_fetchone, test_fetchmany, test_fetchall, test_iterator, test_arraysize, test_as_pandas, test_description [original] REWRITE 2, 4 Drop the (unload, "pyarrow") case, which repeats (unload, "auto"), and chunksize=1_000_000; unload False and True cover the async plumbing. the remaining cases of the same test
TestAsyncPandasCursor::test_query_execution, test_poll [original] REWRITE 4 The chunk size and the engine cannot change QueryExecution or poll; keep unload False and True, whose query texts differ (5 queries each now). the remaining cases of the same test
TestAsyncPandasCursor::test_bad_query [original] REWRITE 4 The chunk size cannot change a FAILED state; keep unload False and True (4 failing queries now). the remaining cases of the same test

3. aio/pandas/test_cursor.py

test change criteria reason covered by
TestAioPandasCursor::test_binary_null_vs_empty REMOVE 2 The aio cursor returns the values of the sync result set (1 query). pandas/test_cursor.py::TestPandasCursor::test_binary_null_vs_empty[c-2]
TestAioPandasCursor::test_fetchmany REMOVE 2 WithAsyncFetch.fetchmany is a shared to_thread wrapper, and the values come from the sync result set (1 query). aio/test_common.py::TestWithAsyncFetch::test_fetch_runs_result_set_fetch[AioPandasCursor], pandas/test_cursor.py::TestPandasCursor::test_fetchmany
TestAioPandasCursor::test_execute_returns_self MERGE 4 One assert goes into an existing query (1 query). into TestAioPandasCursor::test_fetchone (assert await cursor.execute(...) is cursor)
TestAioPandasCursor::test_description MERGE 2, 4 description comes from the shared WithResultSet; add the assert to an existing query (1 query). into TestAioPandasCursor::test_fetchone
TestAioPandasCursor::test_as_pandas_unload MERGE 4 Same UNLOAD query as the target (1 query). into TestAioPandasCursor::test_fetchone_unload (call as_pandas() there)
TestAioPandasCursor::test_context_manager REMOVE 2 __aenter__/__aexit__ are inherited from WithAsyncFetch (1 query and a connection); the test also has a runtime import. aio/test_common.py::TestWithAsyncFetch::test_context_exit_closes_off_event_loop[AioPandasCursor]
TestAioPandasCursor::test_arraysize_default REMOVE 6 It asserts the shared constant DEFAULT_FETCH_SIZE (1000). —
TestAioPandasCursor::test_invalid_arraysize REMOVE 2 The arraysize setter is inherited. aio/test_connection.py::TestAioConnection::test_cursor_arraysize_uncapped[AioPandasCursor], TestAioConnection::test_cursor_arraysize_not_positive[AioPandasCursor], or their merged sync form (aio sub-issue)
TestAioPandasCursor::test_description_initial REMOVE 2, 6 An inherited property that is None before execute(). test_cursor.py::TestCursor::test_description_initial [original]
TestAioPandasCursor::test_cancel_initial REMOVE 2 WithAsyncFetch.cancel is shared. aio/test_common.py::TestWithAsyncFetch::test_cancel_without_query (added by the aio sub-issue)
TestAioPandasCursor::test_executemany_fetch REMOVE 2, 4 After executemany() the result set is None, so this hits the same "No result set" branches (1 query). TestAioPandasCursor::test_no_result_set_raises (also covers as_pandas())

4. pandas/test_result_set.py

#1238 deletes TestPandasDataFrameIterator::test_close_during_read, TestAthenaPandasResultSet::test_close_during_fetchall, TestAthenaPandasResultSet::test_close_during_fetchall_generator and TestAthenaPandasResultSet::test_iter_parquet_chunks_after_close; they are not listed.

test change criteria reason covered by
TestPandasDataFrameIterator::test_as_pandas_single_dataframe REMOVE 2, 6 An identity return. pandas/test_cursor.py::TestPandasCursor::test_pandas_cursor_whole_result_reused (as_pandas() is df)
TestAthenaPandasResultSet::test_fetchall_across_chunks REMOVE 2, 7 Its only new case is an empty middle chunk, which Athena never produces: UNLOAD never writes empty files, and pandas never yields an empty CSV chunk. pandas/test_cursor.py::TestPandasCursor::test_fetch_across_chunks (CSV and UNLOAD)
TestAthenaPandasResultSet::test_fetch_duplicate_column_names REMOVE 2 Unrenamed duplicate labels; real reads rename them or fall back. pandas/test_cursor.py::TestPandasCursor::test_duplicate_column_names (default and managed)
TestAthenaPandasResultSet::test_get_csv_engine_result_format (36 cases) REWRITE 1, 6, 7 It asserts the private engine choice, and the c/python/auto rows restate the code; the rewrite keeps the two fallbacks with an effect, a .txt DDL result and a result file under 100 bytes with engine="pyarrow", as reads that check the values. TestAthenaPandasResultSet::test_read_csv_ddl_preserves_numeric_looking_names (.txt)
TestAthenaPandasResultSet::test_read_csv_ddl_preserves_numeric_looking_names (12 cases) REWRITE 1 Keep the frame and name asserts; drop the pandas.read_csv spy that asserts engine == "c" and the assert on the exact fs.open kwargs. —
TestAthenaPandasResultSet::test_reads_parquet_in_chunks (15 cases) REWRITE 1, 7 It asserts a private predicate; the rewrite checks the chunk count through _read_unload_result for a few supported and unsupported options. pandas/test_cursor.py::TestPandasCursor::test_unload_chunks (supported options)
TestAthenaPandasResultSet::test_get_csv_open_options (4 cases) REMOVE 1, 6 It asserts that a private method returns its literal constants, which only affect performance. pandas/test_cursor.py::TestPandasCursor::test_read_options (block_size/cache_type forwarding)
TestAthenaPandasResultSet::test_read_parquet_filesystem (5 cases) REMOVE 1, 2 It asserts the exact kwargs of a mocked pandas.read_parquet; this still holds after #1233 adds coerce_int96_timestamp_unit="us" to them. pandas/test_cursor.py::TestPandasCursor::test_unload_with_filesystem_options; INT96: pandas/test_cursor.py::TestPandasCursor::test_unload_int96_unit_option (#1233)
TestAthenaPandasResultSet::test_iter_parquet_chunks_large_integers (2 cases) REMOVE 2 Its Parquet file without statistics is not Athena output, and the exact BIGINT is checked against Athena. pandas/test_cursor.py::TestPandasCursor::test_unload_chunks (BIGINT '9007199254740993', check_exact=True, kept by the trim)
TestAthenaPandasResultSet::test_iter_parquet_chunks_nested_values REMOVE 6, 8 It asserts that pyarrow-to-numpy turns [None] into a float64 array, which is library behavior. pandas/test_cursor.py::TestPandasCursor::test_unload_chunks_complex (array column, chunks equal the whole read)
TestAthenaPandasResultSet::test_iter_parquet_chunks_reader_error REMOVE 7, 8 chunksize=0 only reaches a wrapper branch, and the test matches pyarrow's message. —
TestAthenaPandasResultSet::test_read_csv_open_options (6 cases) REMOVE 1 It patches _get_csv_open_options and asserts the call count and the fs.open kwargs of performance-only options. —
TestAthenaPandasResultSet::test_read_csv_json_null_without_dtype_warning (20 cases) REWRITE 1, 4 Keep the value and no-warning asserts; drop the asserts on internal read_csv kwargs (converters["j"] is _JSONConverter.keep_text, engine, low_memory) and the low_memory axis (20 parses of 400,010 rows). —
TestAthenaPandasResultSet::test_read_csv_json_without_null (30 cases, 10 skipped) REWRITE 1, 8 It asserts the internal low_memory kwarg and checks pandas' dtype inference along a low_memory axis that skips itself for the python engine; collapse that axis and keep the value asserts. pandas/test_cursor.py::TestPandasCursor::test_integer_and_json_with_null (col_json_not_null)
TestAthenaPandasResultSet::test_read_csv_json_with_dtype (8 cases), TestAthenaPandasResultSet::test_read_csv_json_custom_converters (2 cases) REWRITE 1 Keep the dtype and value asserts; drop the identity asserts on internal read_csv kwargs (options["dtype"] is dtype, options["converters"] is converters). —
TestAthenaPandasResultSet::test_read_csv_json_multi_index_level REMOVE 2 A json level in a MultiIndex is converted, with NULL. pandas/test_cursor.py::TestPandasCursor::test_integer_and_json_with_null[multi_index]
TestAthenaPandasResultSet::test_read_csv_keeps_low_memory_default (7 cases) REMOVE 1 It only asserts that the internal read_csv call has no low_memory kwarg. —
TestAthenaPandasResultSet::test_read_csv_json_name_in_multi_index_columns REMOVE 6, 7 header=[0, 1] "left as pandas reads them" tests pandas and a skip branch. —
TestAthenaPandasResultSet::test_read_csv_json_conversion_error MERGE 2, 8 Same contract as the chunk variant, and it matches Python's json message; parametrize whole and chunked reads and assert OperationalError without match. into TestAthenaPandasResultSet::test_read_csv_json_conversion_error_in_chunk
test_read_csv_with_pyarrow_string_dtype_after_dates (2 cases) MERGE 2 Its input parses as a date and is the only check that the string dtype is applied again after parse_dates (pandas/result_set.py:186-193); add it as a parsed_dates case ('"v"\n"2024-01-01"\n\n', expected ["2024-01-01", nan], string dtype). into test_read_csv_with_pyarrow_matches_pandas

5. pandas/test_util.py

test change criteria reason covered by
test_to_sql_with_multiple_partitions [original] REWRITE 5 assert sorted(cursor.fetchall()), [...] asserts a non-empty tuple and always passes, so the partition check never fails; the rewrite changes , to ==. —
test_to_sql_session_credentials_and_s3_config (3 cases) REWRITE 1 len(resources.call_args_list) == expected asserts internal resource counts; the rewrite keeps only all(... max_pool_connections == 37) over a non-empty call list, and the credentials part stays observable. —

6. pandas/test_converter.py

test change criteria reason covered by
TestDefaultPandasUnloadTypeConverter::test_convert_delegates_to_default REMOVE 6, 7 convert("varchar", "hello") == "hello" on a converter with only a default mapping; nothing reaches convert() for UNLOAD results. —

Notes

Borderline KEEPs:

  • pandas/test_cursor.py::TestPandasCursor::test_binary_null_vs_empty (4 cases) is the core binary NULL vs empty regression. It also asserts the private result_set._csv_stream.closed; that one assert can go.
  • TestPandasCursor::test_binary_dataframe_column_names (8 queries) and test_duplicate_column_names_with_newline, _header_options, _read_options keep the c/python engine axis, because the python engine resolves labels on its own path (_read_csv_header_as_labels).
  • TestPandasCursor::test_result_sets_share_s3_client (Share one S3 client per connection across result sets and filesystems #1011) patches session.client and reads the private fs._client, as test_cursor.py::TestCursor::test_builds_no_s3_client does.
  • test_read_options in the sync, async and aio files patch _execute/_poll and assert the constructor kwargs of a mocked result set. Each cursor class forwards options with its own code, and no observable alternative exists offline.
  • TestPandasCursor::test_cancel and TestAsyncPandasCursor::test_cancel [original] sleep 5 to 10 s at random. Cancel tests from another thread stay.
  • pandas/test_result_set.py::TestAthenaPandasResultSet::test_fetch_across_chunks overlaps the cursor test but is the only check that chunks are read lazily during a fetch; its read == [0] assert is close to an implementation detail.
  • TestAthenaPandasResultSet::test_fetch_after_conversion_error triggers the failure with a timestamp of 10**13 s, which Athena cannot return. Break result sets after failed or cancelled fetches and drop the cross-thread guards #1238 rewrites it.
  • TestAthenaPandasResultSet::test_read_parquet_footers_stopped (Retries continue after a result set is closed or an asyncio API call is cancelled #1202) patches the private _read_parquet_footer/_read_data_manifest. It is the only test that a stopped aio cancel stops footer retries.
  • TestPandasDataFrameIterator::test_itertuples_large_chunk and TestAthenaPandasResultSet::test_fetch_after_chunks_run_out depend on the private _ROW_BLOCK_SIZE but check row order and remaining rows across row blocks (Return PandasCursor rows as tuples without a dictionary per row #1167, Take buffered rows in bulk in fetchmany() and fetchall() #1187).
  • The offline CSV and Parquet parser tests in pandas/test_result_set.py (test_read_csv_with_pyarrow_matches_pandas, 26 cases; test_iter_parquet_chunks_null_in_later_file; the json tests) use synthetic files of Athena CSV and multi-file UNLOAD layouts that Athena does not produce on demand.
  • pandas/test_converter.py::TestDefaultPandasTypeConverter::test_convert_delegates_to_mapping is the offline regression for a pass stub (a8df601). The only integration path, managed results, is environment-gated.
  • pandas/test_util.py::test_to_sql_workers_resolve_credentials spies the util.to_parquet arguments but is the only test of "frozen credentials only for session=" (Use the connection's s3_config and session credentials in to_sql() #1068).
  • The [original] chunksize=1_000_000 cases are optional trims; the (unload, "pyarrow") duplicate is the clear one.

Unused after the change:

  • pandas/test_cursor.py: imports PropertyMock, _read_csv_with_pyarrow, PandasDataFrameIterator, and TextFileReader (after the auto-optimize rewrite).
  • pandas/test_async_cursor.py: imports math, random, string, numpy as np, AthenaPandasResultSet. ENV stays (test_query_execution).
  • aio/pandas/test_cursor.py: imports _aio_connect, ENV, AthenaPandasResultSet.
  • pandas/test_result_set.py: imports inspect, collections, ThreadPoolExecutor, sentinel, ANY; module constants _FS, _USER_FS, _READ_MODES. threading stays (test_read_parquet_footers_stopped). _unload_result_set, _read_unload_result, _read_csv_result, _fetching_result_set and _typed_values stay. Re-check these against the file after Break result sets after failed or cancelled fetches and drop the cross-thread guards #1238 and Read UNLOAD timestamps at microseconds #1233.
  • pandas/test_converter.py: the DefaultPandasUnloadTypeConverter import.
  • No conftest fixture becomes unused; pandas_cursor, async_pandas_cursor and aio_pandas_cursor stay.

Each row's cover is re-checked before the row is applied. A row whose cover does not hold becomes MERGE or KEEP.

Activity

  1. laughingman7743
    added
    enhancement
    on Oct 17, 2026
  2. laughingman7743
    added a parent issue on Oct 17, 2026
  3. laughingman7743
    added 4 commits that reference this issue on Oct 17, 2026

    Check the pandas reads against explicit values and close whole binary…

    ecf6f4d

    Share the local result files of the offline pandas tests

    3c543d7

    Check the pandas read options that the independent review found untested

    efbdb7c

    Check the pyarrow CSV dispatch with numeric-looking text

    102e971
  4. laughingman7743
    closed this as completedin #1256on Oct 18, 2026

  5. laughingman7743 commented on Oct 18, 2026

    @laughingman7743
    laughingman7743
    on Oct 18, 2026
    MemberAuthor

    Two rows of section 3 stay after #1256, because their covers come from #1245 (PR #1258, open): aio/pandas/test_cursor.py::TestAioPandasCursor::test_cancel_initial (cover aio/test_common.py::TestWithAsyncFetch::test_cancel_without_query, added by #1245) and TestAioPandasCursor::test_invalid_arraysize (removed with or after #1245's merge of the aio/test_connection.py arraysize tests, which must keep AioPandasCursor in the uncapped and not-positive cases). They are removed in a follow-up PR after #1258 merges.

  6. laughingman7743 commented on Oct 18, 2026

    @laughingman7743
    laughingman7743
    on Oct 18, 2026
    MemberAuthor

    The two rows above are removed by #1265, now that #1258 added their covers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancement

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions