Skip to content

Run fsspec's conformance tests against the S3 filesystems #1329

Closed

Description

@laughingman7743

Part of #1326.

Summary

Run fsspec's conformance tests, fsspec.tests.abstract, against S3FileSystem and AioS3FileSystem, and classify the failures.

Background

fsspec ships abstract test classes for copy, get, put, mv, open, and pipe (46 test functions in fsspec 2026.9.0, more with parametrization) that a filesystem runs by subclassing AbstractFixtures. s3fs runs them in s3fs/tests/derived against a moto server. PyAthena does not run them, and #1315 lists them as a precondition of step 5.

Plan

  1. Add a fixture class for S3FileSystem and one for AioS3FileSystem, with paths below the test bucket's prefix, and run the classes against S3.
    PyAthena's filesystem tests use real S3 rather than moto; keep the number of objects and the cost bounded, as for the other S3 integration tests.
  2. Classify each failure as a defect of the filesystem, an intended difference from fsspec's local semantics (for example empty directories, which S3 does not have), or a limit of the fixtures, and record the intended differences as skips with their reason.
  3. File the defects as grouped issues.

Done when

The conformance classes run in the filesystem tests, every skip names its reason, and the defects are filed.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions