[siliataider]

This Pull request:

Changes or fixes:

In lazy mode, the RDataLoader fills its staging buffer with as many clusters as fit in batch_size * batches_in_memory entries, and loaded each of them with its own ChangeBeginAndEndEntries and event loop. The clusters of a buffer that follow each other in the same dataframe are now merged into a single entry range before loading, so they are read in one event loop.

This is particularly useful when shuffling is disabled because the clusters of a buffer always follow each other, so each buffer is loaded in a single event loop (instead of however many clusters it contained).

Ex.: 10 clusters of 100 entries (1000 entries total), read with batch_size=100 and batches_in_memory=4, so a buffer holds 400 entries:

buffer 1: [c1][c2][c3][c4]  → before: 4 event loops, after: 1 over [c1.start, c4.end)
buffer 2: [c5][c6][c7][c8]  → before: 4 event loops, after: 1 over [c5.start, c8.end)
buffer 3: [c9][c10]         → before: 2 event loops, after: 1 over [c9.start, c10.end)

An epoch goes from 10 event loops to 3 in this example. The rows are loaded in the same order, so the batches are unchanged.

Note

This might help when we decide to enable implicit multi-threading.

Checklist:

• tested changes locally

[vepadulano]

Agreed after offline discussion, thanks!

[github-actions]

Test Results

    24 files      24 suites   3d 22h 2m 27s ⏱️
 3 895 tests  3 869 ✅ 0 💤  26 ❌
83 393 runs  82 981 ✅ 0 💤 412 ❌

For more details on these failures, see this check.

Results for commit fdae629.

♻️ This comment has been updated with latest results.