Repository navigation
feat(amber): schedule a run with cache reuse - #8733
Xiao-zhen-Liu wants to merge 4 commits into
Conversation
Automated Reviewer SuggestionsBased on the
|
47e9a0c to
1f59ab9
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8733 +/- ##
=========================================
Coverage 91.57% 91.57%
+ Complexity 5159 5142 -17
=========================================
Files 1300 1300
Lines 58092 58015 -77
Branches 7615 7605 -10
=========================================
- Hits 53198 53130 -68
+ Misses 2958 2956 -2
+ Partials 1936 1929 -7
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🟢 | bs=10 sw=10 sl=64 | 393 | 0.24 | 23,923/33,515/33,515 us | 🟢 -13.1% / 🔴 +159.0% |
| 🔴 | bs=100 sw=10 sl=64 | 800 | 0.488 | 122,473/175,931/175,931 us | 🔴 +5.7% / 🔴 +119.7% |
| 🟢 | bs=1000 sw=10 sl=64 | 965 | 0.589 | 1,035,645/1,085,253/1,085,253 us | 🟢 -8.0% / 🔴 +45.6% |
Baseline details
Latest main 7f9bdf3 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 393 tuples/sec | 383 tuples/sec | 1,092 tuples/sec | +2.6% | -64.0% |
| bs=10 sw=10 sl=64 | MB/s | 0.24 MB/s | 0.234 MB/s | 0.666 MB/s | +2.6% | -64.0% |
| bs=10 sw=10 sl=64 | p50 | 23,923 us | 24,923 us | 9,398 us | -4.0% | +154.6% |
| bs=10 sw=10 sl=64 | p95 | 33,515 us | 38,569 us | 12,941 us | -13.1% | +159.0% |
| bs=10 sw=10 sl=64 | p99 | 33,515 us | 38,569 us | 15,496 us | -13.1% | +116.3% |
| bs=100 sw=10 sl=64 | throughput | 800 tuples/sec | 817 tuples/sec | 1,486 tuples/sec | -2.1% | -46.2% |
| bs=100 sw=10 sl=64 | MB/s | 0.488 MB/s | 0.499 MB/s | 0.907 MB/s | -2.2% | -46.2% |
| bs=100 sw=10 sl=64 | p50 | 122,473 us | 115,857 us | 72,544 us | +5.7% | +68.8% |
| bs=100 sw=10 sl=64 | p95 | 175,931 us | 172,413 us | 80,069 us | +2.0% | +119.7% |
| bs=100 sw=10 sl=64 | p99 | 175,931 us | 172,413 us | 101,531 us | +2.0% | +73.3% |
| bs=1000 sw=10 sl=64 | throughput | 965 tuples/sec | 951 tuples/sec | 1,533 tuples/sec | +1.5% | -37.1% |
| bs=1000 sw=10 sl=64 | MB/s | 0.589 MB/s | 0.58 MB/s | 0.936 MB/s | +1.6% | -37.1% |
| bs=1000 sw=10 sl=64 | p50 | 1,035,645 us | 1,046,468 us | 711,170 us | -1.0% | +45.6% |
| bs=1000 sw=10 sl=64 | p95 | 1,085,253 us | 1,179,095 us | 749,276 us | -8.0% | +44.8% |
| bs=1000 sw=10 sl=64 | p99 | 1,085,253 us | 1,179,095 us | 794,786 us | -8.0% | +36.5% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,508.61,200,128000,393,0.240,23922.68,33514.57,33514.57
1,100,10,64,20,2501.42,2000,1280000,800,0.488,122473.41,175930.73,175930.73
2,1000,10,64,20,20734.52,20000,12800000,965,0.589,1035644.74,1085252.64,1085252.64|
@parshimers @sshiv012 could you review this one after #8731 and #8732? It sits on both, so until they merge its own change is only the last commit. Design discussion: #5880. Thanks! |
|
This pull request has been automatically marked as stale because it has If this PR is still relevant, please:
To exempt this PR from auto-close, a maintainer can add the |
|
This pull request has been automatically closed due to 37 days of inactivity. |
Add CachedResult (a port's stored base URI and row count) and a matchedResults map on WorkflowContext, empty by default. Only the computing unit builds a WorkflowContext, and no request type carries the map, so the editor cannot supply it. Nothing sets it yet.
1f59ab9 to
aa03117
Compare
What changes were proposed in this PR?
This PR is part of cache reuse (#5881), where an execution reads saved port results from an earlier run instead of recomputing them. It updates
CostBasedScheduleGeneratorandDefaultCostEstimatorto plan schedules directly fromWorkflowContext.matchedResultsusingSearchState(#8732) and skip regions onRegionPlan(#8731).SearchState.requireOutputPortsOfPlanenforces plan-port membership while cache lookup (#5885) excludes loop plans and workflow/warehouse mismatches. Until cache lookup populatesmatchedResultsin a follow-up PR, runs onmainreceive empty matches and schedule identically to before. This PR sits on #8731 and #8732.#8733bottomUpSearch,topDownSearch, baseline)SearchState.eagerSeed(resolveInputRequirements); searches activeMaterializedvs.Pipelinedchoices onretainedPlanCacheReadfixed; adding cost-based reuse later extendsSearchState,CostBasedScheduleGenerator, andCostEstimatorwithout alteringRegion,RegionPlan,ScheduleGenerator, orWorkflowScheduler.createRegions)readerUrisByInputPortintoIntermediateInputPortConfigalongside materialized writer linksUnion) while omittingCacheReadlinks fromRegion.physicalLinks.DefaultCostEstimator.withResourceAllocator)DefaultResourceAllocator(fullPlan)and memo map per(retainedPlan, readerUrisByInputPort)fullPlankeepsUnionpartitioning intact); fast-paths emptymatchedResultsto the default evaluator.RegionPlan(buildRegionPlan)skipRegions(Region.skipped = true) to the winning region DAG beforeScheduleGenerator0ahead of active levels1..k(or level0alone with cost0.0when every operator skips). Executing skip regions comes in a follow-up PR.Any related issues, documentation, discussions?
Part of #5881 (cache reuse). Advances #5884. Design discussion: #5880. Related: #8734 keeps
CacheReadlinks omitted fromRegion.physicalLinksout of a reconfiguration's scope.How was this PR tested?
DefaultCostEstimatorSpec,ResourcePoliciesSpec,CostBasedScheduleGeneratorSpec, andWorkflowSchedulerSpecverify shared statistics without extra queries, Pass 2 saved-result reader URIs (includingUnionports), sub-plan allocator isolation, rejecting invalidmatchedResultsports, 4-edge-state searches in pipelined and materialized modes, level0skip regions, and join/split cache-reuse plans.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude (Claude Code), Gemini (Antigravity)