You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Fix the filesystem's integration with PolarsCursor, the fsspec registry, pickling, and connection= #1339
Four defects where the filesystem meets the rest of PyAthena and fsspec's registry, reproduced offline at 766d79d.
Defects
PolarsCursor's whole-CSV read uses the fsspec registry. It opens the result through fsspec.open(self.output_location, "rb", connection=..., max_workers=..., ...) (polars/result_set.py:608-623, :874). After the re-registration that docs/filesystem.md recommends for using another s3 implementation (fsspec.register_implementation("s3", s3fs.S3FileSystem, clobber=True)), that class receives connection and the other options as unknown storage options, and s3fs ignores them, so the read uses ambient credentials and region instead of the connection's. docs/polars.md says this read "uses PyAthena's S3FileSystem". The pandas result set builds S3FileSystem directly.
fsspec.filesystem("s3", asynchronous=True) returns the synchronous S3FileSystem after import pyathena.pandas or pyathena.polars, which register only that class (filesystem/__init__.py:36-44). It accepts the flag through **kwargs and stores it in storage_options; a caller expecting an async filesystem fails later with AttributeError or blocks.
S3FileSystem(connection=...) and its read files cannot be pickled: fsspec stores every constructor argument in storage_options, including the Connection, whose boto3 clients do not pickle (AttributeError: Can't get local object 'lazy_call.<locals>._handler'). Dask, multiprocessing and Ray ship filesystems and open files this way; a filesystem built from s3fs-style arguments pickles. This is not documented.
connection= silently ignores the client arguments.S3FileSystem(connection=conn, endpoint_url=..., config_kwargs={"max_pool_connections": 64}, anon=True) uses the connection's client, endpoint and pool of 10 (s3.py:217-235); only requester_pays is honored. The docstring hints at this, docs/filesystem.md does not.
Fix direction
Build S3FileSystem directly in the Polars whole-CSV read unless the user gave storage_options, and correct docs/polars.md; decide whether the registration covers only the synchronous class (and document it) or rejects asynchronous=True; make connection= filesystems picklable (for example by reducing to the connection's picklable arguments) or document the limit; raise TypeError when connection= is combined with client-building arguments.
Activity