Skip to content

refactor(amber): use region links in Fries for cache reuse - #8734

Open
Xiao-zhen-Liu wants to merge 1 commit into
apache:mainfrom
Xiao-zhen-Liu:xiaozhen-cache-fries
Open

Xiao-zhen-Liu wants to merge 1 commit into
apache:mainfrom
Xiao-zhen-Liu:xiaozhen-cache-fries

Conversation

@Xiao-zhen-Liu

@Xiao-zhen-Liu Xiao-zhen-Liu commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

This PR is part of cache reuse (#5881), where an execution reads the saved results of an earlier run instead of computing them again. When FriesReconfigurationAlgorithm computes a reconfiguration's scope, it filters candidate port links with region.getLinks.contains(link) (Region.physicalLinks) instead of following every link listed on an operator's port objects.

Before and after scoping Fries reconfiguration to region.getLinks

Case Operator port link lists region.getLinks behavior
No cache reuse (main) Contains every active link between operators in the region Identical reconfiguration scope.
Cache reuse (R -> U1 -> U2, Split -> U1 / U2, #8733) Split and U1 still list the CacheRead link on Split:0 Omitted from R_1.getLinks, so reconfiguring {Split, U2} skips Split:0 -> U1 and leaves U1 untouched.

Any related issues, documentation, discussions?

Part of #5881 (cache reuse). Advances #5884. Design discussion: #5880. Related: #8733, where CostBasedScheduleGenerator leaves CacheRead links out of Region.physicalLinks while operators keep their full link lists.

How was this PR tested?

FriesReconfigurationAlgorithmSpec adds two cases where a link listed only on operator ports would otherwise pull an unrelated operator into a reconfiguration's scope, while keeping all existing reconfiguration cases green.

sbt -batch scalafmtCheckAll Test/compile "WorkflowExecutionService/testOnly *FriesReconfigurationAlgorithmSpec" "WorkflowExecutionService/testOnly org.apache.texera.amber.engine.architecture.scheduling.* org.apache.texera.amber.engine.architecture.coordinator.* org.apache.texera.amber.engine.common.*"

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude (Claude Code), Gemini (Antigravity)

@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @aglinxinyuan
    You can notify them by mentioning @aglinxinyuan in a comment.

@codecov-commenter

codecov-commenter commented Sep 28, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.57%. Comparing base (4db10cf) to head (5c63d6f).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #8734      +/-   ##
============================================
- Coverage     91.57%   91.57%   -0.01%     
+ Complexity     5159     5158       -1     
============================================
  Files          1300     1300              
  Lines         58092    58093       +1     
  Branches       7615     7615              
============================================
- Hits          53198    53197       -1     
- Misses         2958     2959       +1     
- Partials       1936     1937       +1     
Flag Coverage Δ *Carryforward flag
access-control-service 77.44% <ø> (ø) Carriedforward from 4db10cf
agent-service 99.56% <ø> (ø) Carriedforward from 4db10cf
amber 85.43% <100.00%> (-0.01%) ⬇️
computing-unit-managing-service 63.88% <ø> (ø) Carriedforward from 4db10cf
config-service 87.50% <ø> (ø) Carriedforward from 4db10cf
file-service 82.16% <ø> (ø) Carriedforward from 4db10cf
frontend 96.45% <ø> (ø) Carriedforward from 4db10cf
notebook-migration-service 83.77% <ø> (ø) Carriedforward from 4db10cf
pyamber 98.38% <ø> (ø) Carriedforward from 4db10cf
result-service 85.71% <ø> (ø) Carriedforward from 4db10cf
workflow-compiling-service 78.30% <ø> (ø) Carriedforward from 4db10cf

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 6 worse · ⚪ 9 noise (<±5%) · 0 without baseline

Compared against main 7f9bdf3 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 520 0.318 19,108/23,165/23,165 us 🔴 -6.8% / 🔴 +103.3%
🔴 bs=100 sw=10 sl=64 1,248 0.762 79,497/95,082/95,082 us 🔴 +5.5% / 🔴 +18.7%
🔴 bs=1000 sw=10 sl=64 1,441 0.88 686,623/842,034/842,034 us 🔴 +8.7% / 🔴 +12.4%
Baseline details

Latest main 7f9bdf3 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 520 tuples/sec 558 tuples/sec 1,092 tuples/sec -6.8% -52.4%
bs=10 sw=10 sl=64 MB/s 0.318 MB/s 0.341 MB/s 0.666 MB/s -6.7% -52.3%
bs=10 sw=10 sl=64 p50 19,108 us 18,111 us 9,398 us +5.5% +103.3%
bs=10 sw=10 sl=64 p95 23,165 us 23,124 us 12,941 us +0.2% +79.0%
bs=10 sw=10 sl=64 p99 23,165 us 23,124 us 15,496 us +0.2% +49.5%
bs=100 sw=10 sl=64 throughput 1,248 tuples/sec 1,278 tuples/sec 1,486 tuples/sec -2.3% -16.0%
bs=100 sw=10 sl=64 MB/s 0.762 MB/s 0.78 MB/s 0.907 MB/s -2.3% -16.0%
bs=100 sw=10 sl=64 p50 79,497 us 75,381 us 72,544 us +5.5% +9.6%
bs=100 sw=10 sl=64 p95 95,082 us 99,041 us 80,069 us -4.0% +18.7%
bs=100 sw=10 sl=64 p99 95,082 us 99,041 us 101,531 us -4.0% -6.4%
bs=1000 sw=10 sl=64 throughput 1,441 tuples/sec 1,470 tuples/sec 1,533 tuples/sec -2.0% -6.0%
bs=1000 sw=10 sl=64 MB/s 0.88 MB/s 0.897 MB/s 0.936 MB/s -1.9% -6.0%
bs=1000 sw=10 sl=64 p50 686,623 us 674,334 us 711,170 us +1.8% -3.5%
bs=1000 sw=10 sl=64 p95 842,034 us 774,648 us 749,276 us +8.7% +12.4%
bs=1000 sw=10 sl=64 p99 842,034 us 774,648 us 794,786 us +8.7% +5.9%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,384.40,200,128000,520,0.318,19107.67,23164.69,23164.69
1,100,10,64,20,1601.99,2000,1280000,1248,0.762,79497.40,95081.58,95081.58
2,1000,10,64,20,13878.01,20000,12800000,1441,0.880,686623.48,842034.37,842034.37

@Xiao-zhen-Liu

Xiao-zhen-Liu commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor Author

@parshimers @sshiv012 could you review this one too? It is small and independent of the other three. Design discussion: #5880. Thanks!

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

This pull request has been automatically marked as stale because it has
not had any activity for 30 days. It will be closed in 7 days if no
further activity occurs.

If this PR is still relevant, please:

  • Push new commits or rebase onto the latest main, or
  • Comment with a status update, or
  • Address any open review feedback.

To exempt this PR from auto-close, a maintainer can add the pinned label.
Thank you for your contribution to Apache Texera (incubating)!

@github-actions github-actions Bot added the stale label Nov 2, 2026
@github-actions

Copy link
Copy Markdown
Contributor

This pull request has been automatically closed due to 37 days of inactivity.
If you would like to continue this work, please re-open the PR or open a new
one referencing this. Thank you for your contribution.

@github-actions github-actions Bot closed this Nov 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants