Repository navigation
Review the filesystem package before a standalone release #1326
Description
Part of #1315.
Summary
Review the structure, implementation, and tests of pyathena.filesystem as a whole before step 5 of #1315 decides on a standalone package, and run fsspec's conformance tests against it.
Background
#1318 found that the filesystem is faster than s3fs in common scenarios (one-object reads, writes, and copies, and AioS3FileSystem batch transfers at high concurrency), with gaps filed as #1322 and #1323. An s3fs alternative is also judged by how maintainable and correct it is, and by the fsspec contract:
- The package is 10,971 lines in 11 modules;
s3.pyhas 3,355 lines ands3_async.py1,721. The typed models (S3Core,S3Object,S3Path,S3PathPairing, the write session of Give multipart writes a single owner of their parts and an explicit state #1211) were introduced step by step, and the package has not been reviewed as a whole since. The audit of 2026-10-03 (S3FileSystem does not route request parameters per S3 operation #969 to S3FileSystem ignores the s3fs profile argument #983) looked for defects, not for structure. - The tests are 15,101 lines with 535 test functions under
tests/pyathena/filesystem/. Remove and rewrite unnecessary tests in the sync S3 filesystem tests #1240 and Remove and rewrite unnecessary tests in the async S3 filesystem and S3 core tests #1241 (Audit the test suite for unnecessary and flaky tests #1239) audited them on 2026-10-17 and 2026-10-18; since that audit merged (59e00fbf), 41 commits have changed them, for Close objects built for cancelled asyncio constructions #1208, Own the AioS3FileSystem pool and keep aio DataFrame reads off the event loop #1209, Move the AioS3FileSystem multipart copy onto the multipart write session #1230, Follow-ups of #1208: duplicated blocking close paths, AsyncCursor work after close, and description() cost #1281, and test_open_async_cancelled fails when another object emits a ResourceWarning during gc.collect() #1309. - PyAthena does not run
fsspec.tests.abstract, which s3fs runs ins3fs/tests/derived.
Sub-issues
- Audit the structure and implementation of the filesystem package #1327: structure and implementation audit: read-only; produces a report and grouped issues.
- Audit the filesystem tests changed since the #1239 audit #1328: test audit of the changes since Audit the test suite for unnecessary and flaky tests #1239, and of contracts without a test.
- Run fsspec's conformance tests against the S3 filesystems #1329: fsspec conformance tests (
fsspec.tests.abstract).
1 and 2 change no code; the issues they file are implemented in their own PRs, and any change of the class or module structure needs the maintainer's approval first. Findings that bear on #1322 or #1323 are relayed there.
Done when
The three sub-issues are closed, and the issues they filed are either resolved or explicitly deferred past step 5.
Activity