You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Make files usable after fork with either filesystem #1336
A file opened in a forked child hangs, through either filesystem, when an executor was created or used before the fork. Reproduced offline at 766d79d.
Defects
AioS3FileSystem.open() hangs in a forked child._create_executor uses self._loop (s3_async.py:852-854), which skips the fork check of fsspec's loop property. In the child, the parent's loop still reports is_running(), so S3AioExecutor.submit schedules onto a loop that nothing runs: reading 12 MiB with a 5 MiB block hung (killed by a 5 s alarm), while the same read through S3FileSystem in the child returned the data.
S3ThreadPoolExecutor is not fork-safe, unlike S3AioThreadPool, which resets its state after a fork (s3_executor.py:448-482). After the executor ran one call, work submitted to it in a forked child never ran (s3_executor.py:180-202), so a multi-range read of a file that crossed the fork waits forever.
Fix direction
The smallest fix for 1 is self.loop, which raises after a fork as the other sync methods do; giving S3ThreadPoolExecutor the same after-fork reset fixes 2. #1332 proposes removing S3AioExecutor's loop hop altogether; if it is accepted first, 1 is fixed there and this issue keeps 2.
Activity