Process multiple files per HBee #6
Description
Currently the catalog table, when being distributed, sends one file per HBee. If the target files are small, it would be more efficient to process multiple files in a single cloud function.
- Verify that the parquet table works correctly with multiple files (can processing be done in parallel? or is their an other way to schedule downloads that minimizes wait time?)
- Implement the splitting login according to file size
- Find sensible file size thresholds to govern groupings