Repository navigation
Consolidate the filesystem executors before parallel batch operations #1332
Copy link
Copy link
Open
Labels
Description
Part of #1327 (design problem 1 of the audit report). Related to #1322 and #1323, which need a decision here.
Summary
Consolidate the filesystem's executors into one thread-pool type with one sizing and ownership policy, and one batch primitive, before #1322 adds concurrency to the batch operations.
Current state (766d79d)
- Four mechanisms.
S3ThreadPoolExecutor(aThreadPoolExecutorwrapper),S3AioThreadPool(a superset: lazy threads,release(), after-fork reset,run/run_shielded),S3AioExecutor(anS3AioThreadPoolreached throughrun_coroutine_threadsafeon fsspec's loop), andReadAheadStream's ownThreadPoolExecutor. - The loop hop of
S3AioExecutoradds failure modes and no function: it needs a running loop, hangs after a fork (Make files usable after fork with either filesystem #1336), a blocking wait on the loop thread never completes, and futures stay pending when the loop stops, while the write session waits on them; every call is wrapped in two_S3WorkItems.AioS3Fileoverrides nothing and exists for this executor. - Pools per call. The synchronous filesystem creates a pool per open file and per
rm, recursivechmod,clear_multipart_uploadsand multipart copy; the asyncio filesystem has one pool plus one peropen()and peropen_async(). Total threads are unbounded across concurrent files. A file shuts down an executor that its caller passed in, so pools cannot be shared. - Sizing.
max_workersdefaults tocpu_count() * 5inS3FileSystem,S3FileandS3AioExecutor, and tomin(32, cpu + 4)inS3AioThreadPoolandAioS3FileSystem; neither is reconciled with botocore'smax_pool_connectionsof 10 (Run the batch operations of S3FileSystem concurrently #1322 question 5). - Nesting.
AioS3FileSystem's delegatedput_file/pipe_file/get_filerun in a filesystem thread and open a file with its own pool: up tomax_workers ** 2parts in flight (16 concurrent UploadPart and 22 threads atmax_workers=4). A call in anS3AioThreadPoolthread that waits for work submitted to the same pool starves once every thread waits (max_workers=1: 0 of 2 done after 2 s). Run the batch operations of S3FileSystem concurrently #1322 must avoid both. - Fan-out by hand. Track-before-schedule, cancel-before-stop and
clear_framesare written at five sites (S3File._fetch_range, recursivechmod,AioS3StreamedFile._fetch_range, the asyncio annotation copies, the write session's_stop_requests), with different failure semantics;rmandclear_multipart_uploadswait for everything and stop nothing.
Proposal (for decision)
- One thread pool type (merge
S3ThreadPoolExecutorintoS3AioThreadPoolunder a neutral name, with aliases), used directly by both file classes; deprecateS3AioExecutor; keepAioS3Fileas a plain subclass. File work would then run to its end when fsspec's loop stops instead of being cancelled with it. - One bounded pool per filesystem, shared by its files and batch operations, with
max_workersas a per-call cap; files stop owning or shutting down executors. Size the pool against the client'smax_pool_connections, or set the latter frommax_workerswhen the filesystem builds the client. - One batch primitive (for example
map(fn, items, *, fail_fast)with an asyncio counterpart) that implements the fan-out rules once, for Run the batch operations of S3FileSystem concurrently #1322's batch operations and the five sites above. - A range reader shared by both file classes and a pipelined
get_file(Download in parallel ranges in get_file() #1323), with the range splitter onS3Core(merging the short tail, aspart_rangesdoes).
Risks: S3AioExecutor and S3ThreadPoolExecutor are public and documented; head-of-line blocking between files sharing a pool; error order of batch operations.
Reactions are currently unavailable
Activity