Skip to content

feat(amber): add 4-edge-state search state for cache reuse - #8732

Open
Xiao-zhen-Liu wants to merge 3 commits into
apache:mainfrom
Xiao-zhen-Liu:xiaozhen-cache-read-inputs
Open

Xiao-zhen-Liu wants to merge 3 commits into
apache:mainfrom
Xiao-zhen-Liu:xiaozhen-cache-read-inputs

Conversation

@Xiao-zhen-Liu

@Xiao-zhen-Liu Xiao-zhen-Liu commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

This PR is part of cache reuse (#5881), where an execution reads saved port results from an earlier run instead of recomputing them. It adds package-private EdgeType and SearchState (private[scheduling]) so CostBasedScheduleGenerator can represent active links, saved-result reads, and skipped upstream branches in one search state. Nothing calls SearchState yet (#8733 wires it into the generator), so runs on main behave as before. This PR sits on #8731.

Important

Assumption: eager reuse. Every running operator reading a port in matchedResults reads its saved result, and an operator skips whenever saved results cover every output needed from it. Cost-based reuse choices will be added later (design overview in #5880).

Four-state SearchState, resolveInputRequirements, and derived scheduler inputs

State / Entry Meaning in SearchState Notes
Materialized Producer writes output storage URI across a RegionLink Used for blocking links and materialized active links.
Pipelined Producer and consumer run concurrently in one Region Allowed on non-blocking links in PIPELINED mode.
CacheRead Consumer reads saved output URI instead of a RegionLink Producer may skip or still run for another branch.
Unused No data crosses the link because its consumer skips Resolved automatically by resolveInputRequirements.
reusedOutputs Required end-of-run outputs satisfied from storage Checked alongside linkTypes and requireOutputPortsOfPlan.

Why SearchState.resolveInputRequirements is needed (WHAT / WHY / HOW):
Because SearchState labels each link individually, setting matched ports or changing one link during search leaves upstream links out of sync with which operators run. In one reverse-topological pass (sinks -> sources), resolveInputRequirements propagates data-input requirements backward from zero-output sinks and un-reused requiredOutputs:

Direction Condition at operator op Effect on incoming links
Skip unneeded work op has no un-reused required output and no active outgoing link Marks incoming links Unused, cascading upstream across dead branches while leaving shared active diamonds untouched.
Supply required inputs op runs and has an Unused incoming link Resolves matched incoming links to CacheRead (stopping propagation) and unmatched incoming links to defaultActiveType (falling back to Materialized), requiring upstream producers to run.

SearchState.requireOutputPortsOfPlan enforces via require that every matchedResults key is an output port of physicalPlan; workflow, warehouse, and loop eligibility remain at cache lookup on the computing unit.

Scheduler-input helper on SearchState Output derived from a valid SearchState
retainedPlan Active sub-plan of running operators and non-CacheRead active links.
readerUrisByInputPort Deterministic per-input-port list of saved URIs for Pass 2 reader config.
cacheReadPorts Map of saved output ports and CachedResult metadata read in this run.
skipRegions Disconnected Region(skipped = true) components with OutputPortConfig.cachedTupleCount.

Any related issues, documentation, discussions?

Part of #5881 (cache reuse). Advances #5884. Design discussion: #5880.

How was this PR tested?

SearchStateSpec covers 25 test cases verifying port-membership require preconditions, required output discovery, single-pass resolveInputRequirements against an independent backward-reachability oracle (including un-reusing CacheRead links, skipped chains, empty plans, fallback modes, and invalid preconditions) across 400 random DAGs (Random(42)), retainedPlan, readerUrisByInputPort, and skipRegions. WorkflowContextSpec verifies matchedResults.

sbt -batch scalafmtCheckAll Test/compile "WorkflowCore/testOnly *WorkflowContextSpec" "WorkflowExecutionService/testOnly org.apache.texera.amber.engine.architecture.scheduling.* org.apache.texera.amber.engine.architecture.coordinator.* org.apache.texera.amber.engine.common.*"

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude (Claude Code), Gemini (Antigravity)

@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @aglinxinyuan, @Yicong-Huang, @Ma77Ball
    You can notify them by mentioning @aglinxinyuan, @Yicong-Huang, @Ma77Ball in a comment.

@Xiao-zhen-Liu
Xiao-zhen-Liu force-pushed the xiaozhen-cache-read-inputs branch from 7e84c92 to e59b595 Compare September 28, 2026 19:48
@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 6 better · 🔴 2 worse · ⚪ 7 noise (<±5%) · 0 without baseline

Compared against main 7f9bdf3 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🟢 bs=10 sw=10 sl=64 558 0.34 17,084/24,818/24,818 us 🟢 -62.4% / 🔴 +91.8%
🟢 bs=100 sw=10 sl=64 1,267 0.773 78,574/98,847/98,847 us 🟢 -6.6% / 🔴 +23.5%
🔴 bs=1000 sw=10 sl=64 1,435 0.876 697,443/823,354/823,354 us 🔴 +11.7% / 🔴 +9.9%
Baseline details

Latest main 7f9bdf3 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 558 tuples/sec 475 tuples/sec 1,092 tuples/sec +17.5% -48.9%
bs=10 sw=10 sl=64 MB/s 0.34 MB/s 0.29 MB/s 0.666 MB/s +17.2% -49.0%
bs=10 sw=10 sl=64 p50 17,084 us 18,707 us 9,398 us -8.7% +81.8%
bs=10 sw=10 sl=64 p95 24,818 us 66,083 us 12,941 us -62.4% +91.8%
bs=10 sw=10 sl=64 p99 24,818 us 66,083 us 15,496 us -62.4% +60.2%
bs=100 sw=10 sl=64 throughput 1,267 tuples/sec 1,216 tuples/sec 1,486 tuples/sec +4.2% -14.7%
bs=100 sw=10 sl=64 MB/s 0.773 MB/s 0.742 MB/s 0.907 MB/s +4.2% -14.8%
bs=100 sw=10 sl=64 p50 78,574 us 84,098 us 72,544 us -6.6% +8.3%
bs=100 sw=10 sl=64 p95 98,847 us 102,532 us 80,069 us -3.6% +23.5%
bs=100 sw=10 sl=64 p99 98,847 us 102,532 us 101,531 us -3.6% -2.6%
bs=1000 sw=10 sl=64 throughput 1,435 tuples/sec 1,443 tuples/sec 1,533 tuples/sec -0.6% -6.4%
bs=1000 sw=10 sl=64 MB/s 0.876 MB/s 0.881 MB/s 0.936 MB/s -0.6% -6.4%
bs=1000 sw=10 sl=64 p50 697,443 us 692,060 us 711,170 us +0.8% -1.9%
bs=1000 sw=10 sl=64 p95 823,354 us 737,278 us 749,276 us +11.7% +9.9%
bs=1000 sw=10 sl=64 p99 823,354 us 737,278 us 794,786 us +11.7% +3.6%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,358.52,200,128000,558,0.340,17084.33,24817.71,24817.71
1,100,10,64,20,1579.12,2000,1280000,1267,0.773,78574.30,98847.38,98847.38
2,1000,10,64,20,13938.27,20000,12800000,1435,0.876,697442.78,823354.15,823354.15

@codecov-commenter

codecov-commenter commented Sep 28, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.57%. Comparing base (4db10cf) to head (4b4eb24).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff            @@
##               main    #8732   +/-   ##
=========================================
  Coverage     91.57%   91.57%           
+ Complexity     5159     5154    -5     
=========================================
  Files          1300     1300           
  Lines         58092    58084    -8     
  Branches       7615     7614    -1     
=========================================
- Hits          53198    53191    -7     
  Misses         2958     2958           
+ Partials       1936     1935    -1     
Flag Coverage Δ *Carryforward flag
access-control-service 77.44% <ø> (ø)
agent-service 99.56% <ø> (ø) Carriedforward from 3f09487
amber 85.44% <ø> (-0.01%) ⬇️ Carriedforward from 3f09487
computing-unit-managing-service 63.88% <ø> (ø)
config-service 87.50% <ø> (ø)
file-service 82.16% <ø> (ø)
frontend 96.45% <ø> (ø) Carriedforward from 3f09487
notebook-migration-service 83.77% <ø> (ø)
pyamber 98.38% <ø> (ø) Carriedforward from 3f09487
result-service 85.71% <ø> (ø)
workflow-compiling-service 78.30% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Xiao-zhen-Liu

Xiao-zhen-Liu commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor Author

@parshimers @sshiv012 could you review this one after #8731? It sits on #8731, and #8733 sits on both. Design discussion: #5880. Thanks!

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

This pull request has been automatically marked as stale because it has
not had any activity for 30 days. It will be closed in 7 days if no
further activity occurs.

If this PR is still relevant, please:

  • Push new commits or rebase onto the latest main, or
  • Comment with a status update, or
  • Address any open review feedback.

To exempt this PR from auto-close, a maintainer can add the pinned label.
Thank you for your contribution to Apache Texera (incubating)!

@github-actions github-actions Bot added the stale label Nov 9, 2026
@github-actions

Copy link
Copy Markdown
Contributor

This pull request has been automatically closed due to 37 days of inactivity.
If you would like to continue this work, please re-open the PR or open a new
one referencing this. Thank you for your contribution.

@github-actions github-actions Bot closed this Nov 16, 2026
Add CachedResult (a port's stored base URI and row count) and a
matchedResults map on WorkflowContext, empty by default. Only the
computing unit builds a WorkflowContext, and no request type carries the
map, so the editor cannot supply it. Nothing sets it yet.
@Xiao-zhen-Liu Xiao-zhen-Liu reopened this Nov 20, 2026
@Xiao-zhen-Liu
Xiao-zhen-Liu force-pushed the xiaozhen-cache-read-inputs branch from 7de696b to 4b4eb24 Compare November 20, 2026 00:16
@Xiao-zhen-Liu Xiao-zhen-Liu changed the title feat(amber): schedule saved-result reads for cache reuse feat(amber): add 4-edge-state search state for cache reuse Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants