Skip to content

Make files usable after fork with either filesystem #1336

Open

Description

@laughingman7743

Part of #1327 (defects 12 and 13 of the audit report).

Summary

A file opened in a forked child hangs, through either filesystem, when an executor was created or used before the fork. Reproduced offline at 766d79d.

Defects

  1. AioS3FileSystem.open() hangs in a forked child. _create_executor uses self._loop (s3_async.py:852-854), which skips the fork check of fsspec's loop property. In the child, the parent's loop still reports is_running(), so S3AioExecutor.submit schedules onto a loop that nothing runs: reading 12 MiB with a 5 MiB block hung (killed by a 5 s alarm), while the same read through S3FileSystem in the child returned the data.
  2. S3ThreadPoolExecutor is not fork-safe, unlike S3AioThreadPool, which resets its state after a fork (s3_executor.py:448-482). After the executor ran one call, work submitted to it in a forked child never ran (s3_executor.py:180-202), so a multi-range read of a file that crossed the fork waits forever.

Fix direction

The smallest fix for 1 is self.loop, which raises after a fork as the other sync methods do; giving S3ThreadPoolExecutor the same after-fork reset fixes 2. #1332 proposes removing S3AioExecutor's loop hop altogether; if it is accepted first, 1 is fixed there and this issue keeps 2.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions