Skip to content

fix(io): isolate S3FileSystem per signed FsspecFileIO and re-register… - #4418

Closed
SurenderguliaEnginner wants to merge 1 commit into
apache:mainfrom
SurenderguliaEnginner:fix/fsspec-s3-signer-isolation
Closed

SurenderguliaEnginner wants to merge 1 commit into
apache:mainfrom
SurenderguliaEnginner:fix/fsspec-s3-signer-isolation

Conversation

@SurenderguliaEnginner

Copy link
Copy Markdown

Closes #4414

Summary

  • When s3.signer is configured in pyiceberg.io.fsspec._s3, instantiate a private _SignedS3FileSystem(S3FileSystem) subclass with skip_instance_cache=True so signed FsspecFileIO instances never share a cached S3FileSystem or race on fs.s3.meta.events.unregister / register_last.
  • Override async def set_session on _SignedS3FileSystem to re-register the signer events if s3fs recreates the underlying aiobotocore S3 client.
  • Register the signer on the initial fs.s3 client before returning fs.
  • Added unit tests in tests/io/test_fsspec.py:
    • test_s3_signer_isolated_across_file_ios
    • test_s3_signer_re_registered_on_set_session

@SurenderguliaEnginner

Copy link
Copy Markdown
Author

Closing this as #4405 on main already resolves the shared client/signer race by using a per-filesystem AioSession

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FsspecFileIO sends unsigned S3 requests: signer is swapped on a shared, cached S3FileSystem

1 participant