Skip to content

feat: add reusable, versioned workflow macros - #9541

Draft
carloea2 wants to merge 12 commits into
apache:mainfrom
carloea2:feat/workflow-macros
Draft

carloea2 wants to merge 12 commits into
apache:mainfrom
carloea2:feat/workflow-macros

Conversation

@carloea2

@carloea2 carloea2 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Add versioned workflow macros. Users can select operators on the canvas, expose inputs, outputs and Form View fields, and reuse the resulting macro in another workflow. Breadcrumbs open a nested occurrence on the same canvas.

Each occurrence pins a saved revision. Saving an edit through a parent updates that path. Other usages and earlier workflow snapshots keep their pins. Export includes the transitive macro definitions and revisions needed to reopen the graph on another deployment.

The Scala compiler resolves the pins, validates exposed properties, and removes macro boundaries before submitting a native graph to Amber. It retains the occurrence paths used to display errors, schemas, results and live or historical statistics. Agent edits use the same compiler validation.

Draft status. Merged and tested against apache:main at f020980c3e. The PR remains open for design feedback.

The nested canvas shows explicit graph boundaries. These screenshots compare an earlier iteration and the current terminal icons.

Earlier macro editor Current macro editor
Earlier shared icon for a macro terminal Distinct Inputs and Outputs operators on the nested canvas

Any related issues, documentation, discussions?

Closes #8913. The draft moves expansion into the compiler, updating the frontend-expansion proposal recorded in that issue. The lifecycle and portability questions are open for community feedback. Design discussion · Xiaozhen’s original UI demo

How was this PR tested?

On merge commit aa6bee1c38, 419 Scala tests and 363 agent tests passed. The full Angular suite passed 7,659 tests with one skip. Chromium passed 28 standard browser tests and all 187 tests in the focused macro and result suites. All 22 compiler cases matched their original input, expected output and error trees.

A real Amber execution ran five occurrences of the same macro at three depths, processing 10, 20, 30, 40 and 50 rows. The test reopened the saved execution from Iceberg and checked the macro, inner Split, and both terminals against their respective live per-port counts. Other regressions cover nested saves, Undo/Redo and Canvas/Form View import/export.

sbt "WorkflowCore/testOnly *WorkflowGraphIDsSpec" \
    "WorkflowCompiler/testOnly *Macro* *LogicalPlanSpec *WorkflowCompilerSpec" \
    "WorkflowCompilingService/testOnly *WorkflowCompilationResourceSpec" \
    "WorkflowExecutionService/testOnly *ExecutionStatsServiceSpec *SyncMacroExecutionSpec *ExecutionResultServiceSpec *WorkflowServiceSpec *WorkflowResourceSpec *WorkflowMacroCloneSpec *WorkflowGraphIDsResourceSpec *SyncExecutionResourceSpec *WorkflowVersionResourceSpec *SessionStateSpec"
(cd frontend && node node_modules/@angular/cli/bin/ng.js test --watch=false --progress=false)
(cd frontend && node node_modules/@angular/cli/bin/ng.js run gui:test-browser --watch=false --progress=false)
(cd frontend && node node_modules/@angular/cli/bin/ng.js run gui:test-browser --watch=false --progress=false \
  --include='**/workflow-macro.spec.ts' \
  --include='**/macro.service.spec.ts' \
  --include='**/workflow-compiling.service.spec.ts' \
  --include='**/macro-navigation.service.spec.ts' \
  --include='**/workflow-editor.component.browser.spec.ts' \
  --include='**/workflow-result.service.spec.ts')
(cd agent-service && bun test)

To try it in the UI, select connected operators, create a macro, then drag a second instance onto the canvas. Run the workflow and open each occurrence using its breadcrumb path. Reopen the saved execution to compare the same per-port counts.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-6)

Create reusable nested macros from selected operators, pin their definition
versions, expose ports and properties, and expand them before compilation,
execution, and paused reconfiguration. Bundle dependencies for portable
JSON exports and keep their graphs intact when cloning workflows.

Closes apache#8913
Discussion: apache#5248 (comment)

Validation:
- node node_modules/@angular/cli/bin/ng.js test --watch=false --progress=false
  (6608 passed, 1 skipped)
- sbt "WorkflowExecutionService/testOnly org.apache.texera.web.resource.dashboard.user.workflow.WorkflowMacroCloneSpec"
  (2 passed)
- node .yarn/releases/yarn-4.14.1.cjs build
- node .yarn/releases/yarn-4.14.1.cjs format:ci
- sbt "scalafixAll --check"
- sbt scalafmtCheckAll
- Playwright browser checks with mocked API: selection collapse, exposed
  properties, nested inspection/back navigation, and independent editable copy.
Preserve pinned definitions through agent edits and reuse graph expansion, validation and logical plans. Normalize portable bundles and reuse palette controls and search, keeping deep macro inspection responsive.

Related to apache#8913
Resolve each occurrence in the compiler, preserve nested revision snapshots and portable imports, and expose run statistics consistently in the canvas, Form View and agent workflows.

Closes apache#8913
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Committers with relevant context: @parshimers
    You can request their reviews formally with /request-review @parshimers.

  • Contributors with relevant context: @aglinxinyuan, @yangzhang75, @mengw15
    You can notify them by mentioning @aglinxinyuan, @yangzhang75, @mengw15 in a comment.

@github-actions github-actions Bot added feature engine dependencies Pull requests that update a dependency file ddl-change Changes to the TexeraDB DDL frontend Changes related to the frontend GUI docs Changes related to documentations infra common platform Non-amber Scala service paths agent-service labels Nov 19, 2026
@codecov-commenter

codecov-commenter commented Nov 19, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.88158% with 49 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.48%. Comparing base (f020980) to head (aa6bee1).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...-core/src/main/typescript/workflow-macros/index.ts 70.65% 49 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #9541      +/-   ##
============================================
- Coverage     91.56%   91.48%   -0.09%     
+ Complexity     5157     5152       -5     
============================================
  Files          1300     1301       +1     
  Lines         58092    57883     -209     
  Branches       7615     7549      -66     
============================================
- Hits          53194    52953     -241     
- Misses         2959     3004      +45     
+ Partials       1939     1926      -13     
Flag Coverage Δ *Carryforward flag
access-control-service 77.44% <ø> (ø)
agent-service 97.99% <83.88%> (-1.57%) ⬇️
amber 85.41% <ø> (-0.01%) ⬇️ Carriedforward from f020980
computing-unit-managing-service 63.88% <ø> (ø)
config-service 87.50% <ø> (ø)
file-service 82.16% <ø> (ø)
frontend 96.46% <ø> (+<0.01%) ⬆️ Carriedforward from f020980
notebook-migration-service 83.77% <ø> (ø)
pyamber 98.38% <ø> (ø) Carriedforward from f020980
result-service 85.71% <ø> (ø)
workflow-compiling-service 77.51% <ø> (-0.80%) ⬇️

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 4 better · 🔴 3 worse · ⚪ 8 noise (<±5%) · 0 without baseline

Compared against main f020980 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 490 0.299 19,987/24,682/24,682 us 🟢 -22.6% / 🔴 +109.3%
🟢 bs=100 sw=10 sl=64 1,019 0.622 97,012/106,127/106,127 us 🟢 -12.8% / 🔴 -30.5%
⚪ bs=1000 sw=10 sl=64 1,130 0.689 879,954/979,140/979,140 us ⚪ within ±5% / 🔴 +28.1%
Baseline details

Latest main f020980 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 490 tuples/sec 520 tuples/sec 1,084 tuples/sec -5.8% -54.8%
bs=10 sw=10 sl=64 MB/s 0.299 MB/s 0.317 MB/s 0.662 MB/s -5.7% -54.8%
bs=10 sw=10 sl=64 p50 19,987 us 17,682 us 9,548 us +13.0% +109.3%
bs=10 sw=10 sl=64 p95 24,682 us 31,901 us 12,811 us -22.6% +92.7%
bs=10 sw=10 sl=64 p99 24,682 us 31,901 us 15,432 us -22.6% +59.9%
bs=100 sw=10 sl=64 throughput 1,019 tuples/sec 1,036 tuples/sec 1,467 tuples/sec -1.6% -30.5%
bs=100 sw=10 sl=64 MB/s 0.622 MB/s 0.632 MB/s 0.895 MB/s -1.6% -30.5%
bs=100 sw=10 sl=64 p50 97,012 us 93,453 us 74,441 us +3.8% +30.3%
bs=100 sw=10 sl=64 p95 106,127 us 121,763 us 81,800 us -12.8% +29.7%
bs=100 sw=10 sl=64 p99 106,127 us 121,763 us 103,712 us -12.8% +2.3%
bs=1000 sw=10 sl=64 throughput 1,130 tuples/sec 1,145 tuples/sec 1,517 tuples/sec -1.3% -25.5%
bs=1000 sw=10 sl=64 MB/s 0.689 MB/s 0.699 MB/s 0.926 MB/s -1.4% -25.6%
bs=1000 sw=10 sl=64 p50 879,954 us 858,710 us 725,911 us +2.5% +21.2%
bs=1000 sw=10 sl=64 p95 979,140 us 1,006,526 us 764,152 us -2.7% +28.1%
bs=1000 sw=10 sl=64 p99 979,140 us 1,006,526 us 808,901 us -2.7% +21.0%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,407.98,200,128000,490,0.299,19987.22,24681.65,24681.65
1,100,10,64,20,1962.86,2000,1280000,1019,0.622,97012.19,106126.86,106126.86
2,1000,10,64,20,17706.28,20000,12800000,1130,0.689,879953.57,979139.93,979139.93

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-service common ddl-change Changes to the TexeraDB DDL dependencies Pull requests that update a dependency file docs Changes related to documentations engine feature frontend Changes related to the frontend GUI infra platform Non-amber Scala service paths

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add reusable and independently versioned workflow macros

2 participants