Skip to content

Review the filesystem package before a standalone release #1326

Open
1 of 4 issues completed
Open

Description

@laughingman7743

Part of #1315.

Summary

Review the structure, implementation, and tests of pyathena.filesystem as a whole before step 5 of #1315 decides on a standalone package, and run fsspec's conformance tests against it.

Background

#1318 found that the filesystem is faster than s3fs in common scenarios (one-object reads, writes, and copies, and AioS3FileSystem batch transfers at high concurrency), with gaps filed as #1322 and #1323. An s3fs alternative is also judged by how maintainable and correct it is, and by the fsspec contract:

Sub-issues

  1. Audit the structure and implementation of the filesystem package #1327: structure and implementation audit: read-only; produces a report and grouped issues.
  2. Audit the filesystem tests changed since the #1239 audit #1328: test audit of the changes since Audit the test suite for unnecessary and flaky tests #1239, and of contracts without a test.
  3. Run fsspec's conformance tests against the S3 filesystems #1329: fsspec conformance tests (fsspec.tests.abstract).

1 and 2 change no code; the issues they file are implemented in their own PRs, and any change of the class or module structure needs the maintainer's approval first. Findings that bear on #1322 or #1323 are relayed there.

Done when

The three sub-issues are closed, and the issues they filed are either resolved or explicitly deferred past step 5.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions