Skip to content

Remove the filesystem's connection= and pass client= to custom s3fs filesystem classes #1386

Open

Description

@laughingman7743

Use case

#1363 deprecates the connection= argument of S3FileSystem and AioS3FileSystem in favor of client= and retry_config= (#1378), and defers two breaking steps to the next major release.
This issue tracks those steps for 5.0.0.

Proposed change

  1. Remove connection= from S3FileSystem and AioS3FileSystem (step 5 of Replace the filesystem's connection= with client= and retry_config= injection #1363).
    Passing it raises TypeError like any unknown argument.
    Remove the deprecation message, the deprecation warning path in S3FileSystem._resolve_injected_client, the tests for the warning, and the deprecation paragraph in docs/filesystem.md.
  2. Change the keywords passed to the s3fs cursor's filesystem_class (step 6 of Replace the filesystem's connection= with client= and retry_config= injection #1363).
    A custom class is created with client, retry_config, default_block_size, and skip_instance_cache=True instead of connection, through pyathena.filesystem.s3_filesystem(), as S3FileSystem and AioS3FileSystem already are.
    No new factory hook.
    Update the two sentences in docs/s3fs.md that describe the current and next-major behavior.

Both are breaking changes and need release notes:

  • Code that passes connection= must use pyathena.filesystem.s3_filesystem(connection), or client=connection.s3_client and retry_config=connection.retry_config.
  • A custom filesystem_class must accept the client and retry_config keywords.

Validation plan (if implementing)

  • Filesystem unit tests in tests/pyathena/filesystem/ for the removed argument.
  • tests/pyathena/s3fs/test_result_set.py for the keywords a custom filesystem_class receives.
  • The s3fs cursor integration tests for the default and async filesystem classes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions