Repository navigation
refactor(workflow-operator): share operator and test code - #9518
Open
aglinxinyuan wants to merge 1 commit into
Open
aglinxinyuan wants to merge 1 commit into
aglinxinyuan wants to merge 1 commit into
Conversation
Contributor
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9518 +/- ##
============================================
+ Coverage 91.57% 91.67% +0.10%
+ Complexity 5157 5046 -111
============================================
Files 1300 1300
Lines 58092 57499 -593
Branches 7615 7517 -98
============================================
- Hits 53195 52712 -483
+ Misses 2959 2890 -69
+ Partials 1938 1897 -41
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 593 | 0.362 | 16,697/22,069/22,069 us | 🔴 +14.3% / 🔴 +74.9% |
| ⚪ | bs=100 sw=10 sl=64 | 1,301 | 0.794 | 76,575/87,687/87,687 us | ⚪ within ±5% / 🟢 -15.5% |
| ⚪ | bs=1000 sw=10 sl=64 | 1,458 | 0.89 | 683,424/725,913/725,913 us | ⚪ within ±5% / 🟢 -10.3% |
Baseline details
Latest main f020980 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 593 tuples/sec | 605 tuples/sec | 1,084 tuples/sec | -2.0% | -45.3% |
| bs=10 sw=10 sl=64 | MB/s | 0.362 MB/s | 0.369 MB/s | 0.662 MB/s | -1.9% | -45.3% |
| bs=10 sw=10 sl=64 | p50 | 16,697 us | 14,606 us | 9,548 us | +14.3% | +74.9% |
| bs=10 sw=10 sl=64 | p95 | 22,069 us | 24,003 us | 12,811 us | -8.1% | +72.3% |
| bs=10 sw=10 sl=64 | p99 | 22,069 us | 24,003 us | 15,432 us | -8.1% | +43.0% |
| bs=100 sw=10 sl=64 | throughput | 1,301 tuples/sec | 1,313 tuples/sec | 1,467 tuples/sec | -0.9% | -11.3% |
| bs=100 sw=10 sl=64 | MB/s | 0.794 MB/s | 0.801 MB/s | 0.895 MB/s | -0.9% | -11.3% |
| bs=100 sw=10 sl=64 | p50 | 76,575 us | 75,618 us | 74,441 us | +1.3% | +2.9% |
| bs=100 sw=10 sl=64 | p95 | 87,687 us | 91,734 us | 81,800 us | -4.4% | +7.2% |
| bs=100 sw=10 sl=64 | p99 | 87,687 us | 91,734 us | 103,712 us | -4.4% | -15.5% |
| bs=1000 sw=10 sl=64 | throughput | 1,458 tuples/sec | 1,470 tuples/sec | 1,517 tuples/sec | -0.8% | -3.9% |
| bs=1000 sw=10 sl=64 | MB/s | 0.89 MB/s | 0.897 MB/s | 0.926 MB/s | -0.8% | -3.9% |
| bs=1000 sw=10 sl=64 | p50 | 683,424 us | 676,257 us | 725,911 us | +1.1% | -5.9% |
| bs=1000 sw=10 sl=64 | p95 | 725,913 us | 739,610 us | 764,152 us | -1.9% | -5.0% |
| bs=1000 sw=10 sl=64 | p99 | 725,913 us | 739,610 us | 808,901 us | -1.9% | -10.3% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,337.41,200,128000,593,0.362,16696.93,22068.56,22068.56
1,100,10,64,20,1537.58,2000,1280000,1301,0.794,76575.07,87687.43,87687.43
2,1000,10,64,20,13717.50,20000,12800000,1458,0.890,683424.40,725913.11,725913.11Consolidate duplicated codegen, schema/parameter definitions, source scan/Twitter helpers, and operator/visualization/pybuilder test fixtures across 109 files in common/workflow-operator with zero behavior changes.
aglinxinyuan
force-pushed
the
refactor/workflow-operator-cleanup
branch
from
November 19, 2026 05:43
f6d40a8 to
0eec4c4
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
Consolidate repeated operator codegen, physical-op/schema wiring, parameter specs, execution helpers, and test scaffolding across
common/workflow-operator/(+3,218/−5,302 lines across 109 files: 104 modified, 5 added) without altering user-facing behavior or test assertions:PythonCodeFragment,PythonVisualizationOpDesc,HierarchyChartSupport): Share indented template rendering (render),render_errorblocks (runtimeError), empty-table guards (emptyTableGuard), and Plotly HTML yields (plotlyHtmlYield) acrossPythonVisualizationOpDescdescriptors (BandConfig,LineConfig,FigureFactoryTableOpDesc,HierarchyChartOpDesc,IcicleChartOpDesc, and 37 chart*OpDescdescriptors), plus sharedNumericValueRuleJson/ColorPatternJsonschema rules.CsvScanSupport,TextSourceOpDesc): Extract shared physical-op wiring, delimiter/URI validation, schema construction, line slicing/casting, and standalone Python codegen acrossCSVScanSourceOpDesc,ParallelCSVScanSourceOpDesc,CSVOldScanSourceOpDesc,FileScanOpDesc,FileScanSourceOpDesc,TextSourceOpDesc, andTextInputSourceOpDesc.tweetSourcePhysicalOpandTweetSchema.create()inTwitterSourceOpDesc; hyperparameterParamSpec/ParamClassdefaults andGammaPatternacross KNN/SVC/SVR Sklearn advanced descriptors and parameter classes;registeredCodegensandrankingPayload/audioDataUrldispatch across HuggingFace codegen/inference operators;requireNumericOrTimestamp,sumAgg, andextremumAgginAggregationOperation; key/comparison helpers inStableMergeSortOpExec; and time-window probing helpers inIntervalJoinOpExec.BinarySetOpExecBehaviors(for difference/intersect/symmetric-difference specs),SklearnVectorizerCodegenBehaviors(for classifier/training codegen specs), andChartConfigSpecSupport(for 5 chart configuration/step/section specs); consolidate repeated helpers acrossDescriptorChecker,ArrowUtilsSpec,AggregateOpSpec/AggregationOperationSpec(AggregationTestSupport),SklearnMLOperatorDescriptorSpecand the 4 Sklearn advanced trainer specs,ProjectionOpDescSpec,ProjectionOpExecSpec,KeywordSearchOpExecSpec,DictionaryMatcherOpExecSpec,IntervalOpExecSpec,StableMergeSortOpDescSpec,StableMergeSortOpExecSpec,ArrowSourceOpDescSpec,ArrowSourceOpExecSpec, and Twitter/HuggingFace specs.Any related issues, documentation, discussions?
Closes #9517.
How was this PR tested?
Verified on JDK 17 with
UDF_PYTHON_PATHconfigured after final commitf6d40a8652:Observed result: all four checks passed cleanly (
WorkflowOperator/scalafmtCheckAllchecked 37 main + 86 test Scala sources;WorkflowOperator/scalafixAll --checkand cross-module test compilations exited 0).sbt "WorkflowOperator/test"Observed result:
Suites: completed 299, aborted 0;Total number of tests run: 3011(Tests: succeeded 3009, failed 2, canceled 1, ignored 0, pending 2). Both failures (FileScanSourceOpDescSpecUTF-16 encoding test,UrlVizOpDescSpecURL export test), the 1 canceled case (FileScanSourceOpDescSpecLARGE_BINARYtest), and the 2 pending cases (BulletChartOpDescSpec,VolcanoPlotOpDescSpec) are pre-existing on the cleanorigin/mainWindows baseline (43c398229d), and JUnit XML comparison against that baseline matched across all 299 suites / 3,014 XML testcases (Missing suites: [],Extra suites: [],Testcase diffs: 0).sbt "WorkflowCompilingService/testOnly org.apache.texera.amber.translator.*"Observed result:
Suites: completed 2, aborted 0,Total number of tests run: 47(Tests: succeeded 47, failed 0, canceled 0, ignored 0, pending 0,All tests passed.).Ten negative mutation controls across production helpers (
PythonCodeFragment,HierarchyChartSupport,TwitterSourceOpDesc,IntervalJoinOpExec,StableMergeSortOpExec,HuggingFaceCodegenBase,AggregationOperation,SklearnAdvancedBaseDesc,TextSourceOpDesc,CsvScanSupport) also failed their targeted test suites as expected before restoring the final clean tree.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Antigravity