You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
PandasCursor tuning example passes options read_csv rejects #889
The "Performance tuning options" example in docs/pandas.md passes options that pandas.read_csv does not accept, so two of its calls fail when the results are read:
cursor.execute("SELECT * FROM large_table",
engine="pyarrow",
chunksize=100_000,
use_threads=True)
...
cursor.execute("SELECT * FROM data_table",
engine="c",
chunksize=50_000,
buffer_lines=100_000)
PandasCursor forwards extra keyword arguments to read_csv.
use_threads is not a read_csv parameter. Also, with chunksize set, _get_csv_engine uses the C engine even when engine="pyarrow" is requested.
buffer_lines is not a read_csv parameter either.
Both raise TypeError: read_csv() got an unexpected keyword argument .... The engine="python", low_memory=True example does not raise.
PyAthena master (a86a180); the same section is in v3.36.0.
pandas 3.0.6. Neither option is a read_csv parameter in pandas 2.x either.
Proposed fix (optional)
Replace the invalid examples with options that read_csv accepts. Also state that engine="pyarrow" is not used together with chunksize, as _get_csv_engine documents. Validate with just docs lint and just docs build.
Activity