Skip to content

feat(amber): schedule a run with cache reuse - #8733

Open
Xiao-zhen-Liu wants to merge 4 commits into
apache:mainfrom
Xiao-zhen-Liu:xiaozhen-cache-schedule
Open

Xiao-zhen-Liu wants to merge 4 commits into
apache:mainfrom
Xiao-zhen-Liu:xiaozhen-cache-schedule

Conversation

@Xiao-zhen-Liu

@Xiao-zhen-Liu Xiao-zhen-Liu commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

This PR is part of cache reuse (#5881), where an execution reads saved port results from an earlier run instead of recomputing them. It updates CostBasedScheduleGenerator and DefaultCostEstimator to plan schedules directly from WorkflowContext.matchedResults using SearchState (#8732) and skip regions on RegionPlan (#8731). SearchState.requireOutputPortsOfPlan enforces plan-port membership while cache lookup (#5885) excludes loop plans and workflow/warehouse mismatches. Until cache lookup populates matchedResults in a follow-up PR, runs on main receive empty matches and schedule identically to before. This PR sits on #8731 and #8732.

Cache-aware scheduling flow in CostBasedScheduleGenerator

Stage / Helper Behavior in #8733 Design note
Seed & search (bottomUpSearch, topDownSearch, baseline) Starts at SearchState.eagerSeed (resolveInputRequirements); searches active Materialized vs. Pipelined choices on retainedPlan Holds CacheRead fixed; adding cost-based reuse later extends SearchState, CostBasedScheduleGenerator, and CostEstimator without altering Region, RegionPlan, ScheduleGenerator, or WorkflowScheduler.
Pass 2 (createRegions) Merges readerUrisByInputPort into IntermediateInputPortConfig alongside materialized writer links Supports single-input and multi-writer ports (such as Union) while omitting CacheRead links from Region.physicalLinks.
Allocator & stats isolation (DefaultCostEstimator.withResourceAllocator) Allocates an isolated DefaultResourceAllocator(fullPlan) and memo map per (retainedPlan, readerUrisByInputPort) Shares loaded runtime statistics without extra DB queries (fullPlan keeps Union partitioning intact); fast-paths empty matchedResults to the default evaluator.
Extended RegionPlan (buildRegionPlan) Adds skipRegions (Region.skipped = true) to the winning region DAG before ScheduleGenerator Skip regions sit at level 0 ahead of active levels 1..k (or level 0 alone with cost 0.0 when every operator skips). Executing skip regions comes in a follow-up PR.

Any related issues, documentation, discussions?

Part of #5881 (cache reuse). Advances #5884. Design discussion: #5880. Related: #8734 keeps CacheRead links omitted from Region.physicalLinks out of a reconfiguration's scope.

How was this PR tested?

DefaultCostEstimatorSpec, ResourcePoliciesSpec, CostBasedScheduleGeneratorSpec, and WorkflowSchedulerSpec verify shared statistics without extra queries, Pass 2 saved-result reader URIs (including Union ports), sub-plan allocator isolation, rejecting invalid matchedResults ports, 4-edge-state searches in pipelined and materialized modes, level 0 skip regions, and join/split cache-reuse plans.

sbt -batch scalafmtCheckAll Test/compile "WorkflowCore/testOnly *WorkflowContextSpec" "WorkflowExecutionService/testOnly org.apache.texera.amber.engine.architecture.scheduling.* org.apache.texera.amber.engine.architecture.coordinator.* org.apache.texera.amber.engine.common.*"

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude (Claude Code), Gemini (Antigravity)

@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @aglinxinyuan, @Yicong-Huang, @Ma77Ball
    You can notify them by mentioning @aglinxinyuan, @Yicong-Huang, @Ma77Ball in a comment.

@codecov-commenter

codecov-commenter commented Sep 28, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.57%. Comparing base (4db10cf) to head (aa03117).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff            @@
##               main    #8733   +/-   ##
=========================================
  Coverage     91.57%   91.57%           
+ Complexity     5159     5142   -17     
=========================================
  Files          1300     1300           
  Lines         58092    58015   -77     
  Branches       7615     7605   -10     
=========================================
- Hits          53198    53130   -68     
+ Misses         2958     2956    -2     
+ Partials       1936     1929    -7     
Flag Coverage Δ *Carryforward flag
access-control-service 77.44% <ø> (ø)
agent-service 99.56% <ø> (ø) Carriedforward from 4b4eb24
amber 85.43% <ø> (-0.02%) ⬇️ Carriedforward from 4b4eb24
computing-unit-managing-service 63.88% <ø> (ø)
config-service 87.50% <ø> (ø)
file-service 82.16% <ø> (ø)
frontend 96.45% <ø> (ø) Carriedforward from 4b4eb24
notebook-migration-service 83.77% <ø> (ø)
pyamber 98.38% <ø> (ø) Carriedforward from 4b4eb24
result-service 85.71% <ø> (ø)
workflow-compiling-service 78.30% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 4 better · 🔴 1 worse · ⚪ 10 noise (<±5%) · 0 without baseline

Compared against main 7f9bdf3 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🟢 bs=10 sw=10 sl=64 393 0.24 23,923/33,515/33,515 us 🟢 -13.1% / 🔴 +159.0%
🔴 bs=100 sw=10 sl=64 800 0.488 122,473/175,931/175,931 us 🔴 +5.7% / 🔴 +119.7%
🟢 bs=1000 sw=10 sl=64 965 0.589 1,035,645/1,085,253/1,085,253 us 🟢 -8.0% / 🔴 +45.6%
Baseline details

Latest main 7f9bdf3 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 393 tuples/sec 383 tuples/sec 1,092 tuples/sec +2.6% -64.0%
bs=10 sw=10 sl=64 MB/s 0.24 MB/s 0.234 MB/s 0.666 MB/s +2.6% -64.0%
bs=10 sw=10 sl=64 p50 23,923 us 24,923 us 9,398 us -4.0% +154.6%
bs=10 sw=10 sl=64 p95 33,515 us 38,569 us 12,941 us -13.1% +159.0%
bs=10 sw=10 sl=64 p99 33,515 us 38,569 us 15,496 us -13.1% +116.3%
bs=100 sw=10 sl=64 throughput 800 tuples/sec 817 tuples/sec 1,486 tuples/sec -2.1% -46.2%
bs=100 sw=10 sl=64 MB/s 0.488 MB/s 0.499 MB/s 0.907 MB/s -2.2% -46.2%
bs=100 sw=10 sl=64 p50 122,473 us 115,857 us 72,544 us +5.7% +68.8%
bs=100 sw=10 sl=64 p95 175,931 us 172,413 us 80,069 us +2.0% +119.7%
bs=100 sw=10 sl=64 p99 175,931 us 172,413 us 101,531 us +2.0% +73.3%
bs=1000 sw=10 sl=64 throughput 965 tuples/sec 951 tuples/sec 1,533 tuples/sec +1.5% -37.1%
bs=1000 sw=10 sl=64 MB/s 0.589 MB/s 0.58 MB/s 0.936 MB/s +1.6% -37.1%
bs=1000 sw=10 sl=64 p50 1,035,645 us 1,046,468 us 711,170 us -1.0% +45.6%
bs=1000 sw=10 sl=64 p95 1,085,253 us 1,179,095 us 749,276 us -8.0% +44.8%
bs=1000 sw=10 sl=64 p99 1,085,253 us 1,179,095 us 794,786 us -8.0% +36.5%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,508.61,200,128000,393,0.240,23922.68,33514.57,33514.57
1,100,10,64,20,2501.42,2000,1280000,800,0.488,122473.41,175930.73,175930.73
2,1000,10,64,20,20734.52,20000,12800000,965,0.589,1035644.74,1085252.64,1085252.64

@Xiao-zhen-Liu

Xiao-zhen-Liu commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor Author

@parshimers @sshiv012 could you review this one after #8731 and #8732? It sits on both, so until they merge its own change is only the last commit. Design discussion: #5880. Thanks!

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

This pull request has been automatically marked as stale because it has
not had any activity for 30 days. It will be closed in 7 days if no
further activity occurs.

If this PR is still relevant, please:

  • Push new commits or rebase onto the latest main, or
  • Comment with a status update, or
  • Address any open review feedback.

To exempt this PR from auto-close, a maintainer can add the pinned label.
Thank you for your contribution to Apache Texera (incubating)!

@github-actions github-actions Bot added the stale label Nov 2, 2026
@github-actions

Copy link
Copy Markdown
Contributor

This pull request has been automatically closed due to 37 days of inactivity.
If you would like to continue this work, please re-open the PR or open a new
one referencing this. Thank you for your contribution.

@github-actions github-actions Bot closed this Nov 16, 2026
Add CachedResult (a port's stored base URI and row count) and a
matchedResults map on WorkflowContext, empty by default. Only the
computing unit builds a WorkflowContext, and no request type carries the
map, so the editor cannot supply it. Nothing sets it yet.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants