Rename the opt-in S3FS CSV reader and define its protocol

docs/api/s3fs.rst

@@ -30,10 +30,13 @@ S3FS CSV Readers
 S3FSCursor supports pluggable CSV reader implementations to control how NULL values
 and empty strings are handled when parsing Athena's CSV output.
 
+.. autoclass:: pyathena.s3fs.reader.CSVReader
+   :members:
+
 .. autoclass:: pyathena.s3fs.reader.AthenaCSVReader
    :members:
 
-.. autoclass:: pyathena.s3fs.reader.DefaultCSVReader
+.. autoclass:: pyathena.s3fs.reader.EmptyStringAsNullCSVReader
    :members:
 
 S3FS Data Converters

docs/null_handling.md

@@ -327,7 +327,7 @@ df = cursor.execute("SELECT * FROM your_table").as_polars()
 `S3FSCursor` supports two CSV readers with different NULL handling behaviors:
 
 - **AthenaCSVReader** (default): Properly distinguishes NULL from empty strings by respecting CSV quoting rules
-- **DefaultCSVReader**: For backward compatibility; both NULL and empty strings become `None`
+- **EmptyStringAsNullCSVReader**: Opt-in behavior; both NULL and empty strings become `None` in cursor results
 
 ### AthenaCSVReader (Default)
 
@@ -361,26 +361,31 @@ for row in cursor:
 # id=3, value='hello', is_none=False
 ```
 
-### DefaultCSVReader (Backward Compatibility)
+### EmptyStringAsNullCSVReader
 
-If you need backward compatibility where both NULL and empty strings are treated as `None`,
-you can explicitly specify `DefaultCSVReader`:
+Specify `EmptyStringAsNullCSVReader` to treat both NULL and empty strings as `None` when reading query result files.
+
+PyAthena 4.0 renames `DefaultCSVReader` to `EmptyStringAsNullCSVReader`; the old name is removed.
+Update existing imports and `csv_reader` arguments to use the new name, as shown below.
 
 ```python
 from pyathena import connect
 from pyathena.s3fs.cursor import S3FSCursor
-from pyathena.s3fs.reader import DefaultCSVReader
+from pyathena.s3fs.reader import EmptyStringAsNullCSVReader
 
 cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/",
                  region_name="us-west-2",
                  cursor_class=S3FSCursor,
-                 cursor_kwargs={"csv_reader": DefaultCSVReader}).cursor()
+                 cursor_kwargs={"csv_reader": EmptyStringAsNullCSVReader}).cursor()
 
 cursor.execute("SELECT '' AS empty_col, NULL AS null_col")
 row = cursor.fetchone()
 print(row)  # (None, None) - both become None
 ```
 
+For custom reader requirements, see {ref}`s3fs-cursor`.
+The default and opt-in behavior also apply to `AsyncS3FSCursor` and `AioS3FSCursor`, which use the same result set.
+
 ```{note}
 `S3FSCursor` does not support the `unload` option. However, since `AthenaCSVReader` (the default)
 can already distinguish NULL from empty strings when reading CSV files, this is not a limitation for NULL handling.

docs/s3fs.md

@@ -164,7 +164,7 @@ S3FSCursor supports pluggable CSV reader implementations to control how NULL val
 are handled. Two readers are provided:
 
 - `AthenaCSVReader` (default): Custom parser that distinguishes between NULL and empty string
-- `DefaultCSVReader`: Uses Python's built-in `csv` module (treats both NULL and empty string as empty string)
+- `EmptyStringAsNullCSVReader`: Uses Python's built-in `csv` module; both NULL and empty string become `None` in cursor results
 
 **Default behavior (AthenaCSVReader):**
 
@@ -184,21 +184,20 @@ row = cursor.fetchone()
 print(row)  # (None, '')  - NULL is None, empty string is ''
 ```
 
-**Switching to Python's built-in csv module (DefaultCSVReader):**
+**Treating empty strings as NULL:**
 
-If you prefer to use Python's built-in `csv` module, you can switch to `DefaultCSVReader`.
-Note that this reader cannot distinguish between NULL and empty string - both become empty strings
-in the parsed result, which are then converted to `None` by the type converter.
+Use `EmptyStringAsNullCSVReader` when empty strings should be treated as NULL.
+It parses result files with Python's `csv` module and returns `None` for both NULL and quoted empty fields.
 
 ```python
 from pyathena import connect
 from pyathena.s3fs.cursor import S3FSCursor
-from pyathena.s3fs.reader import DefaultCSVReader
+from pyathena.s3fs.reader import EmptyStringAsNullCSVReader
 
 cursor = connect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/",
                  region_name="us-west-2",
                  cursor_class=S3FSCursor,
-                 cursor_kwargs={"csv_reader": DefaultCSVReader}).cursor()
+                 cursor_kwargs={"csv_reader": EmptyStringAsNullCSVReader}).cursor()
 
 cursor.execute("SELECT NULL AS null_col, '' AS empty_col")
 row = cursor.fetchone()
@@ -210,7 +209,7 @@ print(row)  # (None, None)  - Both NULL and empty string become None
 | Reader | Implementation | NULL value | Empty string |
 | --- | --- | --- | --- |
 | AthenaCSVReader (default) | Custom parser | None | '' (empty string) |
-| DefaultCSVReader | Python csv module | None | None |
+| EmptyStringAsNullCSVReader | Python csv module | None | None |
 
 **Why the difference?**
 
@@ -222,6 +221,33 @@ Athena's CSV output format distinguishes between NULL values and empty strings:
 Python's standard `csv` module parses both cases as empty strings, losing this distinction.
 The `AthenaCSVReader` implements a custom parser that preserves the difference.
 
+### CSV reader rename in PyAthena 4.0
+
+PyAthena 4.0 renames `DefaultCSVReader` to `EmptyStringAsNullCSVReader` and removes the old name.
+Update imports and `csv_reader` arguments to use the new name:
+
+```python
+from pyathena.s3fs.reader import EmptyStringAsNullCSVReader
+
+cursor = connection.cursor(S3FSCursor, csv_reader=EmptyStringAsNullCSVReader)
+```
+
+`AthenaCSVReader` remains the default.
+Iterating over `EmptyStringAsNullCSVReader` directly now returns `None` for empty fields instead of empty strings.
+Cursor results are unchanged.
+
+### Custom CSV readers
+
+Custom reader classes must satisfy the `pyathena.s3fs.reader.CSVReader` protocol; inheriting from it is optional.
+Pass the class through `csv_reader` in the cursor constructor or an individual `execute()` call.
+
+- Accept a text stream as `file_obj` and a `delimiter` keyword argument in the constructor.
+- Implement `__iter__()` and `__next__()` to yield sequences of strings or `None`, and raise `StopIteration` at the end of the stream.
+  The cursor converts each value by its column type and treats `None` as NULL.
+- Implement `close()` to close the underlying stream.
+
+The reader is used only for result files; managed query results are read through the Athena API.
+
 ### Limitations
 
 S3FSCursor has some limitations compared to ArrowCursor or PandasCursor:

pyathena/aio/s3fs/cursor.py

@@ -74,7 +74,9 @@ def __init__(
             result_reuse_minutes: Maximum age of a reused query result in minutes.
             csv_reader: CSV reader class for parsing the result files. If None,
                 ``AthenaCSVReader`` is used, which distinguishes NULL from empty
-                strings. ``DefaultCSVReader`` reads both as empty strings.
+                strings. ``EmptyStringAsNullCSVReader`` treats both as NULL in cursor
+                results. Custom readers must satisfy the
+                :class:`~pyathena.s3fs.reader.CSVReader` protocol.
             **kwargs: Other cursor arguments, such as ``connection`` and ``arraysize``,
                 passed to the parent ``__init__``.
         """

pyathena/s3fs/async_cursor.py

@@ -86,8 +86,9 @@ def __init__(
             csv_reader: CSV reader class to use for parsing results.
                 Use AthenaCSVReader (default) to distinguish between NULL
                 (unquoted empty) and empty string (quoted empty "").
-                Use DefaultCSVReader for backward compatibility where empty
-                strings are treated as NULL.
+                Use EmptyStringAsNullCSVReader to treat empty strings as NULL.
+                Custom readers must satisfy the
+                :class:`~pyathena.s3fs.reader.CSVReader` protocol.
             **kwargs: Additional connection parameters.
 
         Example:

pyathena/s3fs/cursor.py

@@ -82,17 +82,18 @@ def __init__(
             csv_reader: CSV reader class to use for parsing results.
                 Use AthenaCSVReader (default) to distinguish between NULL
                 (unquoted empty) and empty string (quoted empty "").
-                Use DefaultCSVReader for backward compatibility where empty
-                strings are treated as NULL.
+                Use EmptyStringAsNullCSVReader to treat empty strings as NULL.
+                Custom readers must satisfy the
+                :class:`~pyathena.s3fs.reader.CSVReader` protocol.
             **kwargs: Additional connection parameters.
 
         Example:
             >>> cursor = connection.cursor(S3FSCursor)
             >>> cursor.execute("SELECT * FROM my_table")
             >>>
-            >>> # Use DefaultCSVReader for backward compatibility
-            >>> from pyathena.s3fs.reader import DefaultCSVReader
-            >>> cursor = connection.cursor(S3FSCursor, csv_reader=DefaultCSVReader)
+            >>> # Treat empty strings as NULL
+            >>> from pyathena.s3fs.reader import EmptyStringAsNullCSVReader
+            >>> cursor = connection.cursor(S3FSCursor, csv_reader=EmptyStringAsNullCSVReader)
         """
         super().__init__(
             s3_staging_dir=s3_staging_dir,

pyathena/s3fs/reader.py

@@ -10,27 +10,49 @@
 from __future__ import annotations
 
 import csv
-from collections.abc import Iterator
-from typing import Any
+from collections.abc import Iterator, Sequence
+from typing import Any, Protocol
 
 from pyathena.util import override
 
 
-class DefaultCSVReader(Iterator[list[str]]):
-    """CSV reader using Python's standard csv module.
+class CSVReader(Protocol):
+    """Contract for readers of Athena CSV result files.
 
-    This reader wraps Python's standard csv.reader and treats empty fields
-    as empty strings. It does not distinguish between NULL and empty strings
-    in Athena's CSV output - both become empty strings.
+    Reader classes accept a text stream and a delimiter. They yield sequences
+    of field values, with None for NULL, and close the underlying stream when
+    closed. Implementations do not need to inherit from this protocol.
+    """
+
+    def __init__(self, file_obj: Any, delimiter: str = ",") -> None:
+        """Initialize the reader from a text stream and field delimiter."""
+        ...
 
-    Use this reader when you need backward compatibility with the behavior
-    where empty strings are treated the same as NULL values.
+    def __iter__(self) -> Iterator[Sequence[str | None]]:
+        """Iterate over rows in the result file."""
+        ...
+
+    def __next__(self) -> Sequence[str | None]:
+        """Read the next row, raising StopIteration at the end of the stream."""
+        ...
+
+    def close(self) -> None:
+        """Close the underlying stream."""
+        ...
+
+
+class EmptyStringAsNullCSVReader(Iterator[list[str | None]]):
+    """CSV reader that reads both NULL and empty string as None.
+
+    This reader wraps Python's standard csv.reader, which does not distinguish
+    between NULL and empty strings in Athena's CSV output. It returns None for
+    every empty field.
 
     Example:
         >>> from io import StringIO
-        >>> reader = DefaultCSVReader(StringIO(',"",text'))
+        >>> reader = EmptyStringAsNullCSVReader(StringIO(',"",text'))
         >>> list(reader)
-        [['', '', 'text']]  # Both NULL and empty string become ''
+        [[None, None, 'text']]  # Both NULL and empty string become None
 
     Note:
         The default reader for S3FSCursor is AthenaCSVReader, which
@@ -48,36 +70,36 @@ def __init__(self, file_obj: Any, delimiter: str = ",") -> None:
         self._reader = csv.reader(file_obj, delimiter=delimiter)
 
     @override
-    def __iter__(self) -> DefaultCSVReader:
+    def __iter__(self) -> EmptyStringAsNullCSVReader:
         """Iterate over rows in the CSV file."""
         return self
 
     @override
-    def __next__(self) -> list[str]:
+    def __next__(self) -> list[str | None]:
         """Read and parse the next line.
 
         Returns:
-            List of field values as strings.
+            List of field values, with None for empty fields.
 
         Raises:
             StopIteration: When end of file is reached or reader is closed.
         """
         if self._file is None:
             raise StopIteration
         row = next(self._reader)
-        # Python's csv.reader returns [] for empty lines; normalize to ['']
+        # Python's csv.reader returns [] for empty lines; normalize to [None]
         # to represent a single empty field (consistent with single-value handling)
         if not row:
-            return [""]
-        return row
+            return [None]
+        return [None if value == "" else value for value in row]
 
     def close(self) -> None:
         """Close the underlying file object."""
         if self._file is not None:
             self._file.close()
             self._file = None
 
-    def __enter__(self) -> DefaultCSVReader:
+    def __enter__(self) -> EmptyStringAsNullCSVReader:
         """Enter context manager."""
         return self
 
@@ -106,8 +128,8 @@ class AthenaCSVReader(Iterator[list[str | None]]):
         [[None, '', 'text']]  # NULL and empty string are distinguished
 
     Note:
-        Use DefaultCSVReader if you need backward compatibility where both
-        NULL and empty string are treated as empty string.
+        Use EmptyStringAsNullCSVReader to treat both NULL and empty string as None
+        in S3FS cursor results.
     """
 
     def __init__(self, file_obj: Any, delimiter: str = ",") -> None:

pyathena/s3fs/result_set.py

@@ -20,13 +20,13 @@
 from pyathena.filesystem.s3 import S3FileSystem
 from pyathena.model import AthenaQueryExecution
 from pyathena.result_set import AthenaResultSet
-from pyathena.s3fs.reader import AthenaCSVReader, DefaultCSVReader
+from pyathena.s3fs.reader import AthenaCSVReader, CSVReader
 from pyathena.util import RetryConfig, override, parse_output_location
 
 if TYPE_CHECKING:
     from pyathena.connection import Connection
 
-CSVReaderType = type[DefaultCSVReader] | type[AthenaCSVReader]
+CSVReaderType = type[CSVReader]
 
 _logger = logging.getLogger(__name__)
 
@@ -88,7 +88,8 @@ def __init__(
             block_size: The default block size in bytes for the filesystem. If not set,
                 ``DEFAULT_BLOCK_SIZE`` is used.
             csv_reader: The CSV reader class for the results. If None,
-                ``AthenaCSVReader`` is used.
+                ``AthenaCSVReader`` is used. Custom readers must satisfy the
+                :class:`~pyathena.s3fs.reader.CSVReader` protocol.
             filesystem_class: The filesystem class for reading the results. If None,
                 PyAthena's ``S3FileSystem`` is used.
             result_set_type_hints: Athena type signatures for complex-type columns,
@@ -115,7 +116,7 @@ def __init__(
         self._csv_reader_class: CSVReaderType = csv_reader or AthenaCSVReader
         self._filesystem_class: type[AbstractFileSystem] = filesystem_class or S3FileSystem
         self._fs = self._create_s3_file_system()
-        self._csv_reader: Any | None = None
+        self._csv_reader: CSVReader | None = None
 
         if self.state == AthenaQueryExecution.STATE_SUCCEEDED and self.output_location:
             self._init_csv_reader()
@@ -199,37 +200,18 @@ def _fetch(self) -> None:
             except StopIteration:
                 break
 
-            # Convert row values using converters
-            # AthenaCSVReader returns None for NULL values directly,
-            # DefaultCSVReader returns empty string which needs conversion
-            if self._csv_reader_class is DefaultCSVReader:
-                if col_hints:
-                    converted_row = tuple(
-                        self._converter.convert(
-                            col_type, value if value != "" else None, type_hint=hint
-                        )
-                        if hint
-                        else self._converter.convert(col_type, value if value != "" else None)
-                        for col_type, value, hint in zip(col_types, row, col_hints, strict=False)
-                    )
-                else:
-                    converted_row = tuple(
-                        self._converter.convert(col_type, value if value != "" else None)
-                        for col_type, value in zip(col_types, row, strict=False)
-                    )
+            if col_hints:
+                converted_row = tuple(
+                    self._converter.convert(col_type, value, type_hint=hint)
+                    if hint
+                    else self._converter.convert(col_type, value)
+                    for col_type, value, hint in zip(col_types, row, col_hints, strict=False)
+                )
             else:
-                if col_hints:
-                    converted_row = tuple(
-                        self._converter.convert(col_type, value, type_hint=hint)
-                        if hint
-                        else self._converter.convert(col_type, value)
-                        for col_type, value, hint in zip(col_types, row, col_hints, strict=False)
-                    )
-                else:
-                    converted_row = tuple(
-                        self._converter.convert(col_type, value)
-                        for col_type, value in zip(col_types, row, strict=False)
-                    )
+                converted_row = tuple(
+                    self._converter.convert(col_type, value)
+                    for col_type, value in zip(col_types, row, strict=False)
+                )
             self._rows.append(converted_row)
             rows_fetched += 1