Retry failed batches for Loaders

[luisremis]

Sometime, when a batch fails during the loader, many elements may not be inserted that could have been inserted if it wasn't for batching them into a single transaction.

Venkat suggested that failed batches can be split into smaller batches and retried. And this split can happen from 4K -> 2K -> 1K -> ... 1

It would be good to explore this or any other similar idea.

[ad-claw000]

This issue requests a mechanism to retry failed batched operations during loading. Currently, if a single element in a batch causes the entire transaction to fail, valid elements within that same batch are also not inserted. By splitting failed batches into smaller halves and retrying, we can isolate the problematic elements and insert the rest.

Approach 1: Recursive splitting inside ParallelQuery.do_batch

This approach modifies the do_batch method to recursively halve the data payload when it encounters a database failure (result == 1).
Pros:

• Clean and minimal implementation, directly reusing the existing do_batch parsing and statistics gathering logic.
• Operates at the data (generator) level, ensuring that generate_batch correctly re-builds valid database commands for the halved batches.
• Automatically terminates recursion when len(data) == 1, isolating the exact failing record.
  Cons:
• Increases log verbosity if many elements are systematically failing.
• Will hit the database more frequently during a failure scenario, though bounded by O(log N) retries.

Approach 2: Iterative retry queue inside ParallelQuery.worker

This approach modifies the worker loop to maintain a dynamic queue of batch bounds (batch_start, batch_end). On failure, the boundaries are halved and pushed back to the queue.
Pros:

• Avoids Python recursion depth limits completely.
• Keeps the do_batch function strictly focused on a single execution attempt.
  Cons:
• Much more complex to integrate with the existing tqdm progress bar and worker loop structure.
• Requires refactoring how the worker calculates and iterates over the total expected batches.

Approach 3: Command-level retry in CommonLibrary.execute_query

This approach modifies the base execute_query function to split the Commands and Blobs arrays directly when a transaction failure is detected.
Pros:

• Applies the retry logic universally across the entire SDK, not just for Loaders.
• Could theoretically reduce overhead by not regenerating the JSON queries.
  Cons:
• Extremely risky, as execute_query lacks context on how many commands/blobs belong to a single "element" or "record".
• Could inadvertently split a multi-command atomic transaction into invalid partial queries.

Recommendation

Approach 1 is the recommended path forward. It is the safest and most robust method because it relies on the QueryGenerator's own data boundaries to split the batches. By adding a small block inside do_batch (around aperturedb/ParallelQuery.py:205), we can safely say if result == 1 and len(data) > 1:, split the list into two halves, and call self.do_batch recursively on each half. A flag (e.g., retry_failed_batches=True inside ParallelQuery.__init__) could be added to allow users to opt-in or opt-out if needed.