Update batch regional download example - #2
Conversation
To use new /all grid endpoint
Co-authored-by: Andrew Byrd <abyrd@conveyal.com>
| "\n", | ||
| " # Save result to a local .zip file\n", | ||
| " with open('results/' + analysisId + '.zip', 'wb') as f:\n", | ||
| " for chunk in zipRequest.iter_content(chunk_size=128):\n", |
There was a problem hiding this comment.
Although the library's syntax implies this is operating on a stream of chunks (and the documentation at https://fr.python-requests.org/en/latest/api.html also seems to imply this is the point of chunk_size), I wonder if this may actually buffer the entire response in memory. While that wouldn't be much of an issue when working with one geotiff at a time, it's probably not ideal to buffer an entire zip containing an unknown number of geotiffs. Commentary on stackoverflow seems to echo this concern: https://stackoverflow.com/q/16694907/778449
But other commentary specifically says iter_chunks exists to solve the buffering problem:
https://stackoverflow.com/questions/46205586/why-to-use-iter-content-and-chunk-size-in-python-requests
There are also several edge cases that could occur when downloading and saving a large file, which might be better handled by a purpose-built "save a file" method instead of a simple loop over the content.
Side note: chunk_size is in bytes according to the Requests docs. 128 bytes is a very small chunk size for file IO. It is probably more efficient to work with larger sizes. But ideally a detail like this will be encapsulated in whatever method we use to save the file.
I will research better ways to handle this.
2 participants
To use new /all grid endpoint