Multiprocessing for cube_rescaling and import VIP in ~1 millisecond

[IainHammond]

As title!

This request adds nproc to cube_rescaling. What's the motivation: rescaling takes quite a long time since it's done in serial (frame-by-frame) by looping over frame_rescaling. This is normally not too much of a problem because a typical cube isn't all that large (e.g. 39 channels for IFS). The problem arises when we have a 4D cube of say 80 temporal frames. Or in a non-coronagraphic dataset when this could be several hundred. For SDI, the cube is scaled not once but twice, so this can lead to several thousand calls to cube_rescaling with no multiprocessing. Now, when nproc > 1, frames are rescaled in parallel via pool_map using a new helper function _frame_rescaling_mp. nproc=1 is still the default and the function behaves as it always has. There are no changes to return values or other parameters. I also use reuse_pool=True so the pool remains open and is not continually closed and re-opened, since we are normally looping over a 4D cube. Pools are cleaned up at the end by atexit which was added recently by Gilles. The function is significantly faster now and mostly limited by read/write. There are still performance gains on 1 CPU (e.g. a laptop) because the jobs are spread across the CPU cores and not run in serial so there are always gains by increasing nproc. Here are the results on a cluster (yes I had a lot of spare time while waiting for a PDS 70 dataset!):

Second part: addressing the impressively slow importing of the VIP package. The current version of VIP will load absolutely everything when any function is imported and all dependencies in the entire package. This can get particularly annoying when just opening a fits file for a quick check or running something simple, and was really noticable on a cluster that was made for storage capacity and not speed. Other packages like numpy and scipy comply with PEP 562, which only import exactly what is needed for a function to run and became standard in python 3.7. So this pull request makes VIP PEP 562 compliant by yoinking what other packages do. Submodules only load when you actually use them and when they do load, they take the same time they always did. The slow loading of preproc (mostly due to torch) only occurs if you actually import vip_hci.preproc. The result is the following:

import vip_hci: ~2.8s -> 0.001s
from vip_hci.fits import open_fits: ~2.8s -> 0.01s

It also saves about 200MB of RAM from not loading every dependency while being crazy fast. In the future we should probably move the torch import to be a lazy load in cube_derotate if the user specifically asks for torch

Enjoy!

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 66.66667% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 32.51%. Comparing base (e492e05) to head (778923a).
⚠️ Report is 15 commits behind head on master.

Files with missing lines	Patch %	Lines
src/vip_hci/__init__.py	62.50%	3 Missing ⚠️
src/vip_hci/preproc/rescaling.py	70.00%	3 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #702      +/-   ##
==========================================
+ Coverage   23.55%   32.51%   +8.96%     
==========================================
  Files          88       88              
  Lines       17010    17048      +38     
==========================================
+ Hits         4006     5543    +1537     
+ Misses      13004    11505    -1499     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[VChristiaens]

Thanks a lot @IainHammond ! These changes look great! Also these are significant speed gains - well done 👍