Skip to content

Add a shared mixin for state that cannot be pickled - #4357

Merged
kevinjqliu merged 1 commit into
apache:mainfrom
rambleraptor:transient-state-mixin
Nov 13, 2026
Merged

kevinjqliu merged 1 commit into
apache:mainfrom
rambleraptor:transient-state-mixin

Conversation

@rambleraptor

Copy link
Copy Markdown
Collaborator

Rationale for this change

I've commented on a number of PRs about how some of their changes will cause an object to no longer be pickle-able (https://docs.python.org/3/library/pickle.html) This can potentially be a breaking change for users, so we should avoid it when possible!

I don't love that we have to keep creating __getstate__ and __setstate__ methods throughout the codebase. I've created a mixin that allows developers to define which variables cannot be pickled and then the mixin centralized the __getstate__ and __setstate__ changes for us.

Are these changes tested?

I've added some tests and we're expecting the rest of the suite to work properly.

Are there any user-facing changes?

Locks, thread-locals and caches have to be dropped when pickling and rebuilt
afterwards. Each class has handled this with its own __getstate__ and
__setstate__, and new locks keep landing without it. Add _TransientStateMixin
and port FsspecFileIO and PyArrowFileIO to it.

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @rambleraptor, nice cleanup! Centralizing the transient-state handling removes a whole class of 'added a lock, forgot pickling' bugs, and I verified the motivation on main: a FileIO with an OAuth2AuthManager in its properties fails to pickle today with TypeError: cannot pickle '_thread.lock' object, and pickles fine with this PR. Ran the pickle round-trips locally (all protocols, deepcopy, fresh and used instances) and everything holds.

def _init_transient(self) -> None:
"""Create the attributes listed in `_transient_attrs`."""

def __getstate__(self) -> dict[str, Any]:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: __getstate__/__setstate__ here only round-trip self.__dict__, so a subclass that uses __slots__ (and doesn't mirror its slots into __dict__) would silently lose its slotted state on pickle. The hand-written versions on main had the same behavior, so this is not a regression, but since this is now the central mechanism future adopters might hit it. Maybe worth a line in the docstring that slotted subclasses need their own handling.

@kevinjqliu kevinjqliu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

BTW I dug into the root cause:
issue: RestCatalog stores the live AuthManager object in the FileIO properties (rest/__init__.py#L673-L674), only so the S3 signer can get an auth header (fsspec.py#L135-L136). that makes FileIO pickling depend on every auth manager being picklable. this PR fixes OAuth2, but EntraAuthManager and custom auth managers still fail.

follow up: don't put the AuthManager in FileIO properties. the auth config is already there, so the signer can create its own AuthManager from it with AuthManagerFactory.create and rebuild it after unpickling. the legacy oauth2 path might needs separate handling.

@kevinjqliu
kevinjqliu added this pull request to the merge queue Nov 13, 2026
Merged via the queue into apache:main with commit 141bae4 Nov 13, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants