Skip to content

perf(amber): round-robin partitioner even distribution - #9587

Open
Ma77Ball wants to merge 1 commit into
apache:mainfrom
Ma77Ball:fix/roundrobin-partitioner-skip
Open

Ma77Ball wants to merge 1 commit into
apache:mainfrom
Ma77Ball:fix/roundrobin-partitioner-skip

Conversation

@Ma77Ball

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

  • RoundRobinPartitioner.getBucketIndex emits the current index and then advances, so the cycle starts at receiver 0, and returns an empty iterator when there are no receivers instead of throwing ArithmeticException.

Any related issues, documentation, discussions?

Closes: #9584

How was this PR tested?

  • Run sbt "WorkflowExecutionService/testOnly *PartitionersSpec" under JDK 17; the spec now expects the cycle to start at receiver 0 and adds a case that an empty receiver set yields no bucket index.
  • Could NOT run locally: the amber module needs its jOOQ DAO regenerated from the DDL (a DB-backed build step) to compile, which this environment cannot do, so the spec runs in CI's build / amber job.

Was this PR authored or co-authored using generative AI tooling?

Co-authored with Claude Opus 4.8 in compliance with ASF

RoundRobinPartitioner.getBucketIndex incremented roundRobinIndex before emitting it, so starting from 0 the first tuple went to receiver 1 and receiver 0 was skipped until the second cycle; it also threw ArithmeticException on an empty receiver set. getBucketIndex now emits the current index and then advances, cycling from receiver 0, and returns an empty iterator when there are no receivers.

PartitionersSpec flips the cycle expectation to start at 0 and adds a case asserting an empty receiver set yields no bucket index.
@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @Yicong-Huang, @aglinxinyuan
    You can notify them by mentioning @Yicong-Huang, @aglinxinyuan in a comment.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 1 worse · ⚪ 12 noise (<±5%) · 0 without baseline

Compared against main 7f9bdf3 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 442 0.27 22,735/26,435/26,435 us 🟢 -17.6% / 🔴 +124.3%
⚪ bs=100 sw=10 sl=64 995 0.607 99,436/116,450/116,450 us ⚪ within ±5% / 🔴 +33.8%
⚪ bs=1000 sw=10 sl=64 1,108 0.676 892,782/945,406/945,406 us ⚪ within ±5% / 🔴 -21.3%
Baseline details

Latest main 7f9bdf3 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 442 tuples/sec 463 tuples/sec 1,019 tuples/sec -4.5% -56.6%
bs=10 sw=10 sl=64 MB/s 0.27 MB/s 0.282 MB/s 0.622 MB/s -4.3% -56.6%
bs=10 sw=10 sl=64 p50 22,735 us 20,447 us 10,136 us +11.2% +124.3%
bs=10 sw=10 sl=64 p95 26,435 us 32,076 us 13,860 us -17.6% +90.7%
bs=10 sw=10 sl=64 p99 26,435 us 32,076 us 16,484 us -17.6% +60.4%
bs=100 sw=10 sl=64 throughput 995 tuples/sec 987 tuples/sec 1,363 tuples/sec +0.8% -27.0%
bs=100 sw=10 sl=64 MB/s 0.607 MB/s 0.603 MB/s 0.832 MB/s +0.7% -27.1%
bs=100 sw=10 sl=64 p50 99,436 us 98,507 us 79,194 us +0.9% +25.6%
bs=100 sw=10 sl=64 p95 116,450 us 119,640 us 87,022 us -2.7% +33.8%
bs=100 sw=10 sl=64 p99 116,450 us 119,640 us 106,894 us -2.7% +8.9%
bs=1000 sw=10 sl=64 throughput 1,108 tuples/sec 1,112 tuples/sec 1,407 tuples/sec -0.4% -21.2%
bs=1000 sw=10 sl=64 MB/s 0.676 MB/s 0.679 MB/s 0.859 MB/s -0.4% -21.3%
bs=1000 sw=10 sl=64 p50 892,782 us 895,200 us 776,422 us -0.3% +15.0%
bs=1000 sw=10 sl=64 p95 945,406 us 959,760 us 814,091 us -1.5% +16.1%
bs=1000 sw=10 sl=64 p99 945,406 us 959,760 us 860,577 us -1.5% +9.9%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,452.56,200,128000,442,0.270,22735.29,26435.30,26435.30
1,100,10,64,20,2009.85,2000,1280000,995,0.607,99436.22,116450.37,116450.37
2,1000,10,64,20,18057.75,20000,12800000,1108,0.676,892781.90,945406.48,945406.48

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Round-robin partitioner skips the first receiver and crashes on no receivers

1 participant