Download in parallel ranges in get_file() #1323

Part of #1315.

Summary

S3FileSystem.get_file() reads the object through a file one block at a time, so a download runs at one stream's throughput whatever max_workers is. In the s3fs comparison (#1318, recorded by #1321), s3fs's get_file(), which downloads max_concurrency ranges at a time, was 2.2 to 3.7 times faster with 8 MiB chunks and 5 to 9 times faster with its default 50 MiB chunks for objects of 256 MiB and larger.
AioS3FileSystem._get_file() runs the synchronous method in a thread and has the same limit.

Measurements

One object on one c8gn.2xlarge, MiB/s (median of five trials), PyAthena with 8 MiB blocks and 8 or 40 workers:

The same objects read with open().read(), which requests the ranges in parallel, ran at 440 to 600 MiB/s for 1 GiB.

Cause

get_file() opens the remote file and copies it with remote.read(remote.blocksize) in a loop. The file's bytes cache requests the next block with the first read and one block at a time afterwards, so the file's executor never runs more than one range request.

Questions

1. Whether get_file() downloads ranges of block_size with up to max_workers requests at a time, as open().read() does, and writes them to the local file at their offsets or in order.
2. Memory: how many ranges may be held before they are written; s3fs's download of 1 GiB with max_concurrency 40 peaked at 0.9 to 2.1 GiB, against 0.3 GiB for PyAthena's sequential read.
3. Consistency of a multi-request download: keep the IfMatch condition on the ETag that open() sends today, so that an object replaced during a multi-request download fails instead of mixing two versions.
4. Failure handling: no partial local file after a failed download, and callback progress with out-of-order completion.

Validation

Re-run the filesystem suite's get cases at small, medium, and large and compare them with the s3fs cases of the entry above.