Skip to content

feat(amber): extend RegionPlan for cache-reuse scheduling - #8731

Open
Xiao-zhen-Liu wants to merge 2 commits into
apache:mainfrom
Xiao-zhen-Liu:xiaozhen-cache-skeleton
Open

Xiao-zhen-Liu wants to merge 2 commits into
apache:mainfrom
Xiao-zhen-Liu:xiaozhen-cache-skeleton

Conversation

@Xiao-zhen-Liu

@Xiao-zhen-Liu Xiao-zhen-Liu commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

This PR is part of cache reuse (#5881), where a workflow execution reads saved port results from an earlier run instead of recomputing them. It extends Region, RegionPlan, OutputPortConfig, and ScheduleGenerator so skipped operators sit cleanly ahead of active regions at schedule level 0, keeping Amber's four-layer pipeline (LogicalPlan -> PhysicalPlan -> RegionPlan -> Schedule) intact. Nothing populates matched results on main yet (#8733 wires scheduling and a later PR adds lookup), so execution on main behaves as before. Feedback on keeping or renaming RegionPlan once skip components are included is invited on #5880.

Extending RegionPlan and Schedule for skip regions

Component Change Why
WorkflowContext.matchedResults Adds Map[GlobalPortIdentity, CachedResult] (storageUri, tupleCount) Set on the computing unit only; omitted from WorkflowSettings and editor requests.
Region & OutputPortConfig Adds Region.skipped: Boolean = false and OutputPortConfig.cachedTupleCount Represents disconnected skip components and their saved row counts inside RegionPlan.
ScheduleGenerator.generateScheduleFromRegionPlan Rejects RegionLinks on skip regions via require; places skip regions at level 0 Shifts active levels to 1..k when skip regions exist, and leaves 0..k unchanged when none skip.
Region.getSourceOperators Reads region.getLinks instead of operator port link lists Operators kept by PhysicalPlan.getSubPlan still list links excluded from the region.

Any related issues, documentation, discussions?

Part of #5881 (cache reuse). Advances #5884. Design discussion: #5880.

How was this PR tested?

RegionSpec, RegionPlanSpec, ScheduleSpec, and SchedulingConfigsSpec verify Region.skipped, Region.getSourceOperators, rejecting RegionLinks attached to skip regions, level 0 schedule placement, and OutputPortConfig.cachedTupleCount. WorkflowContextSpec checks the empty matchedResults default.

sbt -batch scalafmtCheckAll Test/compile "WorkflowCore/testOnly *WorkflowContextSpec" "WorkflowExecutionService/testOnly org.apache.texera.amber.engine.architecture.scheduling.* org.apache.texera.amber.engine.architecture.coordinator.* org.apache.texera.amber.engine.common.*"

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude (Claude Code), Gemini (Antigravity)

@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @aglinxinyuan, @Yicong-Huang, @Ma77Ball
    You can notify them by mentioning @aglinxinyuan, @Yicong-Huang, @Ma77Ball in a comment.

@codecov-commenter

codecov-commenter commented Sep 28, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.66667% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.57%. Comparing base (4db10cf) to head (3f09487).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...ne/architecture/scheduling/config/PortConfig.scala 75.00% 0 Missing and 1 partial ⚠️
...ache/texera/amber/core/workflow/CachedResult.scala 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #8731      +/-   ##
============================================
- Coverage     91.57%   91.57%   -0.01%     
- Complexity     5159     5162       +3     
============================================
  Files          1300     1301       +1     
  Lines         58092    58108      +16     
  Branches       7615     7616       +1     
============================================
+ Hits          53198    53211      +13     
- Misses         2958     2960       +2     
- Partials       1936     1937       +1     
Flag Coverage Δ *Carryforward flag
access-control-service 77.44% <ø> (ø)
agent-service 99.56% <ø> (ø) Carriedforward from 4db10cf
amber 85.43% <91.66%> (-0.01%) ⬇️
computing-unit-managing-service 63.88% <ø> (ø)
config-service 87.50% <ø> (ø)
file-service 82.16% <ø> (ø)
frontend 96.45% <ø> (ø) Carriedforward from 4db10cf
notebook-migration-service 83.77% <ø> (ø)
pyamber 98.38% <ø> (ø) Carriedforward from 4db10cf
result-service 85.71% <ø> (ø)
workflow-compiling-service 78.30% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 4 better · 🔴 7 worse · ⚪ 4 noise (<±5%) · 0 without baseline

Compared against main 7f9bdf3 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 481 0.294 20,804/25,743/25,743 us 🔴 +17.9% / 🔴 +121.4%
🟢 bs=100 sw=10 sl=64 1,028 0.627 96,278/113,966/113,966 us 🟢 -32.4% / 🔴 +42.3%
🔴 bs=1000 sw=10 sl=64 1,142 0.697 871,739/999,557/999,557 us 🔴 +5.3% / 🔴 +33.4%
Baseline details

Latest main 7f9bdf3 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 481 tuples/sec 528 tuples/sec 1,092 tuples/sec -8.9% -55.9%
bs=10 sw=10 sl=64 MB/s 0.294 MB/s 0.322 MB/s 0.666 MB/s -8.7% -55.9%
bs=10 sw=10 sl=64 p50 20,804 us 17,645 us 9,398 us +17.9% +121.4%
bs=10 sw=10 sl=64 p95 25,743 us 23,862 us 12,941 us +7.9% +98.9%
bs=10 sw=10 sl=64 p99 25,743 us 23,862 us 15,496 us +7.9% +66.1%
bs=100 sw=10 sl=64 throughput 1,028 tuples/sec 977 tuples/sec 1,486 tuples/sec +5.2% -30.8%
bs=100 sw=10 sl=64 MB/s 0.627 MB/s 0.596 MB/s 0.907 MB/s +5.2% -30.9%
bs=100 sw=10 sl=64 p50 96,278 us 95,245 us 72,544 us +1.1% +32.7%
bs=100 sw=10 sl=64 p95 113,966 us 168,683 us 80,069 us -32.4% +42.3%
bs=100 sw=10 sl=64 p99 113,966 us 168,683 us 101,531 us -32.4% +12.2%
bs=1000 sw=10 sl=64 throughput 1,142 tuples/sec 1,165 tuples/sec 1,533 tuples/sec -2.0% -25.5%
bs=1000 sw=10 sl=64 MB/s 0.697 MB/s 0.711 MB/s 0.936 MB/s -2.0% -25.5%
bs=1000 sw=10 sl=64 p50 871,739 us 854,831 us 711,170 us +2.0% +22.6%
bs=1000 sw=10 sl=64 p95 999,557 us 949,399 us 749,276 us +5.3% +33.4%
bs=1000 sw=10 sl=64 p99 999,557 us 949,399 us 794,786 us +5.3% +25.8%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,415.71,200,128000,481,0.294,20803.86,25743.08,25743.08
1,100,10,64,20,1945.61,2000,1280000,1028,0.627,96277.76,113966.44,113966.44
2,1000,10,64,20,17513.96,20000,12800000,1142,0.697,871739.24,999556.58,999556.58

@Xiao-zhen-Liu

Xiao-zhen-Liu commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor Author

@parshimers @sshiv012 could you review this one? It is the first of four cache-reuse PRs (#8731 to #8734): #8731 is the base, #8732 builds on #8731, #8733 builds on both, and #8734 is independent. Design discussion: #5880. Thanks!

@parshimers

Copy link
Copy Markdown
Member

Sure thing. Let me give the design a look, and I will follow up with any questions that arise from that.

Add CachedResult (a port's stored base URI and row count) and a
matchedResults map on WorkflowContext, empty by default. Only the
computing unit builds a WorkflowContext, and no request type carries the
map, so the editor cannot supply it. Nothing sets it yet.
@Xiao-zhen-Liu
Xiao-zhen-Liu force-pushed the xiaozhen-cache-skeleton branch from 324c25a to 3f09487 Compare November 20, 2026 00:16
@Xiao-zhen-Liu Xiao-zhen-Liu changed the title feat(amber): decide what a run skips for cache reuse feat(amber): extend RegionPlan for cache-reuse scheduling Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants