Skip to content

refactor(workflow-operator): share operator and test code - #9518

Open
aglinxinyuan wants to merge 1 commit into
apache:mainfrom
aglinxinyuan:refactor/workflow-operator-cleanup
Open

aglinxinyuan wants to merge 1 commit into
apache:mainfrom
aglinxinyuan:refactor/workflow-operator-cleanup

Conversation

@aglinxinyuan

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Consolidate repeated operator codegen, physical-op/schema wiring, parameter specs, execution helpers, and test scaffolding across common/workflow-operator/ (+3,218/−5,302 lines across 109 files: 104 modified, 5 added) without altering user-facing behavior or test assertions:

  • Plotly/Python visualization helpers (PythonCodeFragment, PythonVisualizationOpDesc, HierarchyChartSupport): Share indented template rendering (render), render_error blocks (runtimeError), empty-table guards (emptyTableGuard), and Plotly HTML yields (plotlyHtmlYield) across PythonVisualizationOpDesc descriptors (BandConfig, LineConfig, FigureFactoryTableOpDesc, HierarchyChartOpDesc, IcicleChartOpDesc, and 37 chart *OpDesc descriptors), plus shared NumericValueRuleJson / ColorPatternJson schema rules.
  • CSV, file, and text scan sources (CsvScanSupport, TextSourceOpDesc): Extract shared physical-op wiring, delimiter/URI validation, schema construction, line slicing/casting, and standalone Python codegen across CSVScanSourceOpDesc, ParallelCSVScanSourceOpDesc, CSVOldScanSourceOpDesc, FileScanOpDesc, FileScanSourceOpDesc, TextSourceOpDesc, and TextInputSourceOpDesc.
  • Execution, codegen, and descriptor helpers: Share tweetSourcePhysicalOp and TweetSchema.create() in TwitterSourceOpDesc; hyperparameter ParamSpec / ParamClass defaults and GammaPattern across KNN/SVC/SVR Sklearn advanced descriptors and parameter classes; registeredCodegens and rankingPayload / audioDataUrl dispatch across HuggingFace codegen/inference operators; requireNumericOrTimestamp, sumAgg, and extremumAgg in AggregationOperation; key/comparison helpers in StableMergeSortOpExec; and time-window probing helpers in IntervalJoinOpExec.
  • Shared test fixtures and suite helpers: Extract BinarySetOpExecBehaviors (for difference/intersect/symmetric-difference specs), SklearnVectorizerCodegenBehaviors (for classifier/training codegen specs), and ChartConfigSpecSupport (for 5 chart configuration/step/section specs); consolidate repeated helpers across DescriptorChecker, ArrowUtilsSpec, AggregateOpSpec / AggregationOperationSpec (AggregationTestSupport), SklearnMLOperatorDescriptorSpec and the 4 Sklearn advanced trainer specs, ProjectionOpDescSpec, ProjectionOpExecSpec, KeywordSearchOpExecSpec, DictionaryMatcherOpExecSpec, IntervalOpExecSpec, StableMergeSortOpDescSpec, StableMergeSortOpExecSpec, ArrowSourceOpDescSpec, ArrowSourceOpExecSpec, and Twitter/HuggingFace specs.

Any related issues, documentation, discussions?

Closes #9517.

How was this PR tested?

Verified on JDK 17 with UDF_PYTHON_PATH configured after final commit f6d40a8652:

sbt "WorkflowOperator/scalafmtCheckAll" "WorkflowOperator/scalafixAll --check" "WorkflowCompiler/Test/compile" "WorkflowExecutionService/Test/compile"

Observed result: all four checks passed cleanly (WorkflowOperator/scalafmtCheckAll checked 37 main + 86 test Scala sources; WorkflowOperator/scalafixAll --check and cross-module test compilations exited 0).

sbt "WorkflowOperator/test"

Observed result: Suites: completed 299, aborted 0; Total number of tests run: 3011 (Tests: succeeded 3009, failed 2, canceled 1, ignored 0, pending 2). Both failures (FileScanSourceOpDescSpec UTF-16 encoding test, UrlVizOpDescSpec URL export test), the 1 canceled case (FileScanSourceOpDescSpec LARGE_BINARY test), and the 2 pending cases (BulletChartOpDescSpec, VolcanoPlotOpDescSpec) are pre-existing on the clean origin/main Windows baseline (43c398229d), and JUnit XML comparison against that baseline matched across all 299 suites / 3,014 XML testcases (Missing suites: [], Extra suites: [], Testcase diffs: 0).

sbt "WorkflowCompilingService/testOnly org.apache.texera.amber.translator.*"

Observed result: Suites: completed 2, aborted 0, Total number of tests run: 47 (Tests: succeeded 47, failed 0, canceled 0, ignored 0, pending 0, All tests passed.).

Ten negative mutation controls across production helpers (PythonCodeFragment, HierarchyChartSupport, TwitterSourceOpDesc, IntervalJoinOpExec, StableMergeSortOpExec, HuggingFaceCodegenBase, AggregationOperation, SklearnAdvancedBaseDesc, TextSourceOpDesc, CsvScanSupport) also failed their targeted test suites as expected before restoring the final clean tree.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Antigravity

Copilot AI balanced review requested due to automatic review settings November 17, 2026 06:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @kz930, @PG1204, @bobbai00
    You can notify them by mentioning @kz930, @PG1204, @bobbai00 in a comment.

@codecov-commenter

codecov-commenter commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.67%. Comparing base (0aea3df) to head (0eec4c4).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #9518      +/-   ##
============================================
+ Coverage     91.57%   91.67%   +0.10%     
+ Complexity     5157     5046     -111     
============================================
  Files          1300     1300              
  Lines         58092    57499     -593     
  Branches       7615     7517      -98     
============================================
- Hits          53195    52712     -483     
+ Misses         2959     2890      -69     
+ Partials       1938     1897      -41     
Flag Coverage Δ *Carryforward flag
access-control-service 77.44% <ø> (ø)
agent-service 99.56% <ø> (ø) Carriedforward from 0aea3df
amber 85.54% <ø> (+0.11%) ⬆️ Carriedforward from 0aea3df
computing-unit-managing-service 63.88% <ø> (ø) Carriedforward from 0aea3df
config-service 87.50% <ø> (ø)
file-service 82.16% <ø> (ø) Carriedforward from 0aea3df
frontend 96.45% <ø> (ø) Carriedforward from 0aea3df
notebook-migration-service 83.77% <ø> (ø)
pyamber 98.38% <ø> (ø) Carriedforward from 0aea3df
result-service 85.71% <ø> (ø)
workflow-compiling-service 78.30% <ø> (ø) Carriedforward from 0aea3df

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 1 worse · ⚪ 12 noise (<±5%) · 0 without baseline

Compared against main f020980 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 593 0.362 16,697/22,069/22,069 us 🔴 +14.3% / 🔴 +74.9%
⚪ bs=100 sw=10 sl=64 1,301 0.794 76,575/87,687/87,687 us ⚪ within ±5% / 🟢 -15.5%
⚪ bs=1000 sw=10 sl=64 1,458 0.89 683,424/725,913/725,913 us ⚪ within ±5% / 🟢 -10.3%
Baseline details

Latest main f020980 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 593 tuples/sec 605 tuples/sec 1,084 tuples/sec -2.0% -45.3%
bs=10 sw=10 sl=64 MB/s 0.362 MB/s 0.369 MB/s 0.662 MB/s -1.9% -45.3%
bs=10 sw=10 sl=64 p50 16,697 us 14,606 us 9,548 us +14.3% +74.9%
bs=10 sw=10 sl=64 p95 22,069 us 24,003 us 12,811 us -8.1% +72.3%
bs=10 sw=10 sl=64 p99 22,069 us 24,003 us 15,432 us -8.1% +43.0%
bs=100 sw=10 sl=64 throughput 1,301 tuples/sec 1,313 tuples/sec 1,467 tuples/sec -0.9% -11.3%
bs=100 sw=10 sl=64 MB/s 0.794 MB/s 0.801 MB/s 0.895 MB/s -0.9% -11.3%
bs=100 sw=10 sl=64 p50 76,575 us 75,618 us 74,441 us +1.3% +2.9%
bs=100 sw=10 sl=64 p95 87,687 us 91,734 us 81,800 us -4.4% +7.2%
bs=100 sw=10 sl=64 p99 87,687 us 91,734 us 103,712 us -4.4% -15.5%
bs=1000 sw=10 sl=64 throughput 1,458 tuples/sec 1,470 tuples/sec 1,517 tuples/sec -0.8% -3.9%
bs=1000 sw=10 sl=64 MB/s 0.89 MB/s 0.897 MB/s 0.926 MB/s -0.8% -3.9%
bs=1000 sw=10 sl=64 p50 683,424 us 676,257 us 725,911 us +1.1% -5.9%
bs=1000 sw=10 sl=64 p95 725,913 us 739,610 us 764,152 us -1.9% -5.0%
bs=1000 sw=10 sl=64 p99 725,913 us 739,610 us 808,901 us -1.9% -10.3%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,337.41,200,128000,593,0.362,16696.93,22068.56,22068.56
1,100,10,64,20,1537.58,2000,1280000,1301,0.794,76575.07,87687.43,87687.43
2,1000,10,64,20,13717.50,20000,12800000,1458,0.890,683424.40,725913.11,725913.11

Consolidate duplicated codegen, schema/parameter definitions, source scan/Twitter helpers, and operator/visualization/pybuilder test fixtures across 109 files in common/workflow-operator with zero behavior changes.
@aglinxinyuan
aglinxinyuan force-pushed the refactor/workflow-operator-cleanup branch from f6d40a8 to 0eec4c4 Compare November 19, 2026 05:43

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common refactor Refactor the code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Share workflow-operator code and test fixtures

3 participants