Repository navigation
Conversation
ReservoirSamplingOpExec draws the replacement index with Random.nextInt(n+1) instead of nextInt(n), where n is the number of tuples seen before the current one. Algorithm R replaces a reservoir slot for the (n+1)-th item with probability count/(n+1); the old nextInt(n) made the first item after the reservoir fills always replace (probability 1.0), biasing the sample toward later rows. A zero-length reservoir (k coerced to 0) now draws nextInt(1) and keeps nothing instead of throwing on nextInt(0). ReservoirSamplingOpExecSpec adds a test that reproduces Algorithm R with the executor's seed and asserts the sampled reservoir matches it, pinning the off-by-one, and updates the k=0 case to expect an empty result rather than an IllegalArgumentException.
Backport auto-label reportThis
|
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9572 +/- ##
============================================
- Coverage 91.57% 89.08% -2.49%
- Complexity 5158 7511 +2353
============================================
Files 1300 1417 +117
Lines 58092 69985 +11893
Branches 7615 9514 +1899
============================================
+ Hits 53195 62344 +9149
- Misses 2959 5426 +2467
- Partials 1938 2215 +277
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 610 | 0.372 | 15,754/24,988/24,988 us | 🔴 +9.8% / 🔴 +80.3% |
| 🟢 | bs=100 sw=10 sl=64 | 1,280 | 0.781 | 78,244/90,230/90,230 us | 🟢 -23.0% / 🟢 -15.6% |
| 🔴 | bs=1000 sw=10 sl=64 | 1,450 | 0.885 | 686,134/788,577/788,577 us | 🔴 +10.3% / 🟢 -11.6% |
Baseline details
Latest main 7f9bdf3 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 610 tuples/sec | 595 tuples/sec | 1,019 tuples/sec | +2.5% | -40.2% |
| bs=10 sw=10 sl=64 | MB/s | 0.372 MB/s | 0.363 MB/s | 0.622 MB/s | +2.5% | -40.2% |
| bs=10 sw=10 sl=64 | p50 | 15,754 us | 14,351 us | 10,136 us | +9.8% | +55.4% |
| bs=10 sw=10 sl=64 | p95 | 24,988 us | 24,571 us | 13,860 us | +1.7% | +80.3% |
| bs=10 sw=10 sl=64 | p99 | 24,988 us | 24,571 us | 16,484 us | +1.7% | +51.6% |
| bs=100 sw=10 sl=64 | throughput | 1,280 tuples/sec | 1,262 tuples/sec | 1,363 tuples/sec | +1.4% | -6.1% |
| bs=100 sw=10 sl=64 | MB/s | 0.781 MB/s | 0.77 MB/s | 0.832 MB/s | +1.4% | -6.1% |
| bs=100 sw=10 sl=64 | p50 | 78,244 us | 75,993 us | 79,194 us | +3.0% | -1.2% |
| bs=100 sw=10 sl=64 | p95 | 90,230 us | 117,151 us | 87,022 us | -23.0% | +3.7% |
| bs=100 sw=10 sl=64 | p99 | 90,230 us | 117,151 us | 106,894 us | -23.0% | -15.6% |
| bs=1000 sw=10 sl=64 | throughput | 1,450 tuples/sec | 1,476 tuples/sec | 1,407 tuples/sec | -1.8% | +3.1% |
| bs=1000 sw=10 sl=64 | MB/s | 0.885 MB/s | 0.901 MB/s | 0.859 MB/s | -1.8% | +3.1% |
| bs=1000 sw=10 sl=64 | p50 | 686,134 us | 678,668 us | 776,422 us | +1.1% | -11.6% |
| bs=1000 sw=10 sl=64 | p95 | 788,577 us | 715,254 us | 814,091 us | +10.3% | -3.1% |
| bs=1000 sw=10 sl=64 | p99 | 788,577 us | 715,254 us | 860,577 us | +10.3% | -8.4% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,327.96,200,128000,610,0.372,15754.30,24988.47,24988.47
1,100,10,64,20,1562.80,2000,1280000,1280,0.781,78243.88,90230.13,90230.13
2,1000,10,64,20,13793.45,20000,12800000,1450,0.885,686134.44,788576.76,788576.76
What changes were proposed in this PR?
Any related issues, documentation, discussions?
Closes: #9569
How was this PR tested?
sbt "WorkflowOperator/testOnly *ReservoirSamplingOpExecSpec"under JDK 17; expect 14 passing, including a new case that reproduces Algorithm R with the executor's seed and asserts the reservoir matches, and the updated k=0 case that now expects an empty result.Was this PR authored or co-authored using generative AI tooling?
Co-authored with Claude Opus 4.8 in compliance with ASF