Repository navigation
feat(amber): extend RegionPlan for cache-reuse scheduling - #8731
Xiao-zhen-Liu wants to merge 2 commits into
Conversation
Automated Reviewer SuggestionsBased on the
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #8731 +/- ##
============================================
- Coverage 91.57% 91.57% -0.01%
- Complexity 5159 5162 +3
============================================
Files 1300 1301 +1
Lines 58092 58108 +16
Branches 7615 7616 +1
============================================
+ Hits 53198 53211 +13
- Misses 2958 2960 +2
- Partials 1936 1937 +1
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 481 | 0.294 | 20,804/25,743/25,743 us | 🔴 +17.9% / 🔴 +121.4% |
| 🟢 | bs=100 sw=10 sl=64 | 1,028 | 0.627 | 96,278/113,966/113,966 us | 🟢 -32.4% / 🔴 +42.3% |
| 🔴 | bs=1000 sw=10 sl=64 | 1,142 | 0.697 | 871,739/999,557/999,557 us | 🔴 +5.3% / 🔴 +33.4% |
Baseline details
Latest main 7f9bdf3 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 481 tuples/sec | 528 tuples/sec | 1,092 tuples/sec | -8.9% | -55.9% |
| bs=10 sw=10 sl=64 | MB/s | 0.294 MB/s | 0.322 MB/s | 0.666 MB/s | -8.7% | -55.9% |
| bs=10 sw=10 sl=64 | p50 | 20,804 us | 17,645 us | 9,398 us | +17.9% | +121.4% |
| bs=10 sw=10 sl=64 | p95 | 25,743 us | 23,862 us | 12,941 us | +7.9% | +98.9% |
| bs=10 sw=10 sl=64 | p99 | 25,743 us | 23,862 us | 15,496 us | +7.9% | +66.1% |
| bs=100 sw=10 sl=64 | throughput | 1,028 tuples/sec | 977 tuples/sec | 1,486 tuples/sec | +5.2% | -30.8% |
| bs=100 sw=10 sl=64 | MB/s | 0.627 MB/s | 0.596 MB/s | 0.907 MB/s | +5.2% | -30.9% |
| bs=100 sw=10 sl=64 | p50 | 96,278 us | 95,245 us | 72,544 us | +1.1% | +32.7% |
| bs=100 sw=10 sl=64 | p95 | 113,966 us | 168,683 us | 80,069 us | -32.4% | +42.3% |
| bs=100 sw=10 sl=64 | p99 | 113,966 us | 168,683 us | 101,531 us | -32.4% | +12.2% |
| bs=1000 sw=10 sl=64 | throughput | 1,142 tuples/sec | 1,165 tuples/sec | 1,533 tuples/sec | -2.0% | -25.5% |
| bs=1000 sw=10 sl=64 | MB/s | 0.697 MB/s | 0.711 MB/s | 0.936 MB/s | -2.0% | -25.5% |
| bs=1000 sw=10 sl=64 | p50 | 871,739 us | 854,831 us | 711,170 us | +2.0% | +22.6% |
| bs=1000 sw=10 sl=64 | p95 | 999,557 us | 949,399 us | 749,276 us | +5.3% | +33.4% |
| bs=1000 sw=10 sl=64 | p99 | 999,557 us | 949,399 us | 794,786 us | +5.3% | +25.8% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,415.71,200,128000,481,0.294,20803.86,25743.08,25743.08
1,100,10,64,20,1945.61,2000,1280000,1028,0.627,96277.76,113966.44,113966.44
2,1000,10,64,20,17513.96,20000,12800000,1142,0.697,871739.24,999556.58,999556.58|
Sure thing. Let me give the design a look, and I will follow up with any questions that arise from that. |
Add CachedResult (a port's stored base URI and row count) and a matchedResults map on WorkflowContext, empty by default. Only the computing unit builds a WorkflowContext, and no request type carries the map, so the editor cannot supply it. Nothing sets it yet.
324c25a to
3f09487
Compare
What changes were proposed in this PR?
This PR is part of cache reuse (#5881), where a workflow execution reads saved port results from an earlier run instead of recomputing them. It extends
Region,RegionPlan,OutputPortConfig, andScheduleGeneratorso skipped operators sit cleanly ahead of active regions at schedule level0, keeping Amber's four-layer pipeline (LogicalPlan -> PhysicalPlan -> RegionPlan -> Schedule) intact. Nothing populates matched results onmainyet (#8733 wires scheduling and a later PR adds lookup), so execution onmainbehaves as before. Feedback on keeping or renamingRegionPlanonce skip components are included is invited on #5880.WorkflowContext.matchedResultsMap[GlobalPortIdentity, CachedResult](storageUri,tupleCount)WorkflowSettingsand editor requests.Region&OutputPortConfigRegion.skipped: Boolean = falseandOutputPortConfig.cachedTupleCountRegionPlan.ScheduleGenerator.generateScheduleFromRegionPlanRegionLinks on skip regions viarequire; places skip regions at level01..kwhen skip regions exist, and leaves0..kunchanged when none skip.Region.getSourceOperatorsregion.getLinksinstead of operator port link listsPhysicalPlan.getSubPlanstill list links excluded from the region.Any related issues, documentation, discussions?
Part of #5881 (cache reuse). Advances #5884. Design discussion: #5880.
How was this PR tested?
RegionSpec,RegionPlanSpec,ScheduleSpec, andSchedulingConfigsSpecverifyRegion.skipped,Region.getSourceOperators, rejectingRegionLinks attached to skip regions, level0schedule placement, andOutputPortConfig.cachedTupleCount.WorkflowContextSpecchecks the emptymatchedResultsdefault.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude (Claude Code), Gemini (Antigravity)