Repository navigation
feat(amber): add 4-edge-state search state for cache reuse - #8732
Xiao-zhen-Liu wants to merge 3 commits into
Conversation
Automated Reviewer SuggestionsBased on the
|
7e84c92 to
e59b595
Compare
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🟢 | bs=10 sw=10 sl=64 | 558 | 0.34 | 17,084/24,818/24,818 us | 🟢 -62.4% / 🔴 +91.8% |
| 🟢 | bs=100 sw=10 sl=64 | 1,267 | 0.773 | 78,574/98,847/98,847 us | 🟢 -6.6% / 🔴 +23.5% |
| 🔴 | bs=1000 sw=10 sl=64 | 1,435 | 0.876 | 697,443/823,354/823,354 us | 🔴 +11.7% / 🔴 +9.9% |
Baseline details
Latest main 7f9bdf3 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 558 tuples/sec | 475 tuples/sec | 1,092 tuples/sec | +17.5% | -48.9% |
| bs=10 sw=10 sl=64 | MB/s | 0.34 MB/s | 0.29 MB/s | 0.666 MB/s | +17.2% | -49.0% |
| bs=10 sw=10 sl=64 | p50 | 17,084 us | 18,707 us | 9,398 us | -8.7% | +81.8% |
| bs=10 sw=10 sl=64 | p95 | 24,818 us | 66,083 us | 12,941 us | -62.4% | +91.8% |
| bs=10 sw=10 sl=64 | p99 | 24,818 us | 66,083 us | 15,496 us | -62.4% | +60.2% |
| bs=100 sw=10 sl=64 | throughput | 1,267 tuples/sec | 1,216 tuples/sec | 1,486 tuples/sec | +4.2% | -14.7% |
| bs=100 sw=10 sl=64 | MB/s | 0.773 MB/s | 0.742 MB/s | 0.907 MB/s | +4.2% | -14.8% |
| bs=100 sw=10 sl=64 | p50 | 78,574 us | 84,098 us | 72,544 us | -6.6% | +8.3% |
| bs=100 sw=10 sl=64 | p95 | 98,847 us | 102,532 us | 80,069 us | -3.6% | +23.5% |
| bs=100 sw=10 sl=64 | p99 | 98,847 us | 102,532 us | 101,531 us | -3.6% | -2.6% |
| bs=1000 sw=10 sl=64 | throughput | 1,435 tuples/sec | 1,443 tuples/sec | 1,533 tuples/sec | -0.6% | -6.4% |
| bs=1000 sw=10 sl=64 | MB/s | 0.876 MB/s | 0.881 MB/s | 0.936 MB/s | -0.6% | -6.4% |
| bs=1000 sw=10 sl=64 | p50 | 697,443 us | 692,060 us | 711,170 us | +0.8% | -1.9% |
| bs=1000 sw=10 sl=64 | p95 | 823,354 us | 737,278 us | 749,276 us | +11.7% | +9.9% |
| bs=1000 sw=10 sl=64 | p99 | 823,354 us | 737,278 us | 794,786 us | +11.7% | +3.6% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,358.52,200,128000,558,0.340,17084.33,24817.71,24817.71
1,100,10,64,20,1579.12,2000,1280000,1267,0.773,78574.30,98847.38,98847.38
2,1000,10,64,20,13938.27,20000,12800000,1435,0.876,697442.78,823354.15,823354.15
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8732 +/- ##
=========================================
Coverage 91.57% 91.57%
+ Complexity 5159 5154 -5
=========================================
Files 1300 1300
Lines 58092 58084 -8
Branches 7615 7614 -1
=========================================
- Hits 53198 53191 -7
Misses 2958 2958
+ Partials 1936 1935 -1
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
This pull request has been automatically marked as stale because it has If this PR is still relevant, please:
To exempt this PR from auto-close, a maintainer can add the |
|
This pull request has been automatically closed due to 37 days of inactivity. |
Add CachedResult (a port's stored base URI and row count) and a matchedResults map on WorkflowContext, empty by default. Only the computing unit builds a WorkflowContext, and no request type carries the map, so the editor cannot supply it. Nothing sets it yet.
7de696b to
4b4eb24
Compare
What changes were proposed in this PR?
This PR is part of cache reuse (#5881), where an execution reads saved port results from an earlier run instead of recomputing them. It adds package-private
EdgeTypeandSearchState(private[scheduling]) soCostBasedScheduleGeneratorcan represent active links, saved-result reads, and skipped upstream branches in one search state. Nothing callsSearchStateyet (#8733 wires it into the generator), so runs onmainbehave as before. This PR sits on #8731.Important
Assumption: eager reuse. Every running operator reading a port in
matchedResultsreads its saved result, and an operator skips whenever saved results cover every output needed from it. Cost-based reuse choices will be added later (design overview in #5880).SearchStateMaterializedRegionLinkPipelinedRegionPIPELINEDmode.CacheReadRegionLinkUnusedresolveInputRequirements.reusedOutputslinkTypesandrequireOutputPortsOfPlan.Why
SearchState.resolveInputRequirementsis needed (WHAT/WHY/HOW):Because
SearchStatelabels each link individually, setting matched ports or changing one link during search leaves upstream links out of sync with which operators run. In one reverse-topological pass (sinks -> sources),resolveInputRequirementspropagates data-input requirements backward from zero-output sinks and un-reusedrequiredOutputs:opophas no un-reused required output and no active outgoing linkUnused, cascading upstream across dead branches while leaving shared active diamonds untouched.opruns and has anUnusedincoming linkCacheRead(stopping propagation) and unmatched incoming links todefaultActiveType(falling back toMaterialized), requiring upstream producers to run.SearchState.requireOutputPortsOfPlanenforces viarequirethat everymatchedResultskey is an output port ofphysicalPlan; workflow, warehouse, and loop eligibility remain at cache lookup on the computing unit.SearchStateSearchStateretainedPlanCacheReadactive links.readerUrisByInputPortcacheReadPortsCachedResultmetadata read in this run.skipRegionsRegion(skipped = true)components withOutputPortConfig.cachedTupleCount.Any related issues, documentation, discussions?
Part of #5881 (cache reuse). Advances #5884. Design discussion: #5880.
How was this PR tested?
SearchStateSpeccovers 25 test cases verifying port-membershiprequirepreconditions, required output discovery, single-passresolveInputRequirementsagainst an independent backward-reachability oracle (including un-reusingCacheReadlinks, skipped chains, empty plans, fallback modes, and invalid preconditions) across 400 random DAGs (Random(42)),retainedPlan,readerUrisByInputPort, andskipRegions.WorkflowContextSpecverifiesmatchedResults.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude (Claude Code), Gemini (Antigravity)