ParallelQuery batchsize cannot adjust for query size

[bovlb]

Generally it is fastest to make the batchsize as large as possible, and the effective limit is on the total byte size of the encoded query. For example, image uploads need smaller batchsize than descriptor uploads. It would be useful if we could make the batchsize adjust automatically to keep the encoded query under a specific byte limit.

[ad-claw000]

Summary:
The ParallelQuery execution currently relies on a fixed, number-based batchsize, which can be suboptimal because queries vary significantly in byte size (e.g., large images vs. small descriptors). We need a mechanism to dynamically adjust batch sizes so that the total byte size of the encoded commands and blobs remains within a specified limit.

Implementation Approaches:

1. Dynamic byte-size evaluation in ParallelQuery.worker():
   Instead of pre-calculating a fixed number of batches via (end - start) // self.batchsize in aperturedb/ParallelQuery.py (around line 241), we iterate through the generator and dynamically accumulate items. We commit a batch when either the item count reaches batchsize or the combined size of the serialized query JSON and blobs exceeds a new parameter (e.g., max_batchsize_bytes).

   • Pros: Tackles the problem accurately at the execution layer. Keeps changes localized to ParallelQuery where knowledge of 'blobs' and 'commands' exists.
   • Cons: Estimating byte sizes might add slight CPU overhead before sending. Needs care to handle lazy-loaded blobs correctly without loading them prematurely multiple times.

2. Add an estimator to Parallelizer.batched_run():
   Modify aperturedb/Parallelizer.py's worker to accept a generic max_bytes limit and a size_estimator function provided by the child class. The worker advances the batch_end pointer based on the estimator rather than a fixed step.

   • Pros: Provides a generic solution for dynamic chunking that any Parallelizer subclass could theoretically use. Separates parallel scheduling from the query specifics.
   • Cons: More invasive to the generic Parallelizer base class. Requires changes to how total_actions and progress bars are tracked if batches vary drastically.

3. Pre-grouping via a Generator Wrapper:
   Instead of changing the parallel execution logic, we create a wrapper around generator (like ByteLimitedGeneratorWrapper). The wrapper pulls items until reaching the byte limit and yields them as one unified item to ParallelQuery (which would then run with batchsize=1).

   • Pros: Zero changes needed in the complex threading logic of Parallelizer.py or ParallelQuery.py.
   • Cons: Could break the elements_per_thread division logic in Parallelizer if the wrapper hides the true item count, potentially leading to uneven thread distribution.

Recommended Path Forward:
I recommend Approach 1, modifying ParallelQuery.worker().
Because ParallelQuery already overrides the worker() method from Parallelizer, it is the most appropriate place to introduce ApertureDB-specific byte-counting logic (summing JSON command length and blob bytes). We can introduce a max_batchsize_bytes parameter alongside the existing batchsize parameter (which will act as an upper bound on item count), allowing batches to dynamically shrink when queries are large, while maximizing throughput.