Skip to content

fix(amber): keep the failing worker when an error crosses ControlError - #9015

Open
sshiv012 wants to merge 5 commits into
apache:mainfrom
sshiv012:sshiv012/tex-86-control-error-worker
Open

sshiv012 wants to merge 5 commits into
apache:mainfrom
sshiv012:sshiv012/tex-86-control-error-worker

Conversation

@sshiv012

@sshiv012 sshiv012 commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

#8668 tags a failed worker RPC with the worker it targeted (WorkflowRuntimeException with relatedWorkerId), so the surfaced error can point at that worker. But the worker is dropped one hop later: the coordinator returns the error to the web service as a ControlError, which has no field for it, and ErrorUtils.reconstructThrowable rebuilds a plain Throwable. WorkflowService's error handler then finds no worker and records the error as "unknown operator".

You can see this with a Python UDF that fails while starting (e.g. the default template, which has no class). The coordinator log shows WorkflowRuntimeException … Worker:WF23-PythonUDFV2-operator-udfdemo-main-0, but the UI gets operatorId: "unknown operator". So clicking the failed UDF shows only an empty Console and no error.

This PR carries the worker the rest of the way:

  • ControlError gets an optional relatedWorkerId (core.ActorVirtualIdentity).
  • mkControlError fills it from a WorkflowRuntimeException.
  • reconstructThrowable rebuilds a WorkflowRuntimeException with that worker when the field is set, so WorkflowService maps the error to the failing operator.

Errors without a worker are unchanged.

Any related issues, documentation, discussions?

Follow-up to #8668 (issue #8667). Part of #8736. With this, #8771 no longer needs a frontend workaround to show a failed UDF's startup error when the UDF is clicked.

How was this PR tested?

  • New specs in ErrorUtilsSpec:
    • mkControlError carries the worker of a WorkflowRuntimeException, and leaves it empty otherwise;
    • reconstructThrowable rebuilds a WorkflowRuntimeException with the worker, message and cause;
    • the rebuilt error maps to the right operator id via getOperatorFromActorIdOpt;
    • errors without a worker still rebuild a plain Throwable.
  • The two existing specs that build a ControlError directly pass the new field (None).
  • New e2e test in DataProcessingSpec: a source feeding a Java UDF that doesn't compile. The UDF is in the first region, so its error comes back as the reply to startWorkflow, through ControlError. The test expects a WorkflowRuntimeException naming the UDF worker. It fails on main and passes with this PR.
sbt "WorkflowExecutionService/testOnly org.apache.texera.amber.engine.e2e.DataProcessingSpec -- -z \"fails to initialize\""
sbt "WorkflowExecutionService/testOnly org.apache.texera.amber.error.ErrorUtilsSpec org.apache.texera.amber.engine.architecture.scheduling.RegionExecutionManagerSpec org.apache.texera.amber.engine.architecture.scheduling.WorkflowExecutionManagerSpec"
sbt "WorkflowExecutionService/scalafmtCheck" "WorkflowExecutionService/Test/scalafmtCheck" "WorkflowExecutionService/scalafixAll --check"
bash bin/python-proto-gen.sh && cd amber && pytest -m "not integration"

End to end, on a local Texera built from current main plus this PR and #8771 (which no longer has a frontend filter), I ran a Python UDF that fails while starting:

operatorId the UI receives Clicking the failed UDF
main "unknown operator" (workerId empty) only an empty Console
this PR "PythonUDFV2-operator-udfdemo" (workerId Worker:WF23-PythonUDFV2-operator-udfdemo-main-0) a Static Error tab next to the Console, with the error

The message also loses its java.lang.Throwable: prefix, since the rebuilt WorkflowRuntimeException prints just its message.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5)

apache#8668 tags a failed worker RPC with the worker (WorkflowRuntimeException
with relatedWorkerId), but ControlError had no field for it. When the
coordinator returned the error to the web service, reconstructThrowable
rebuilt a plain Throwable, so WorkflowService recorded "unknown
operator" and the UI could not show the error under the failed
operator (e.g. a Python UDF that fails while starting).

- Add relatedWorkerId to ControlError.
- mkControlError fills it from a WorkflowRuntimeException.
- reconstructThrowable rebuilds a WorkflowRuntimeException when it is
  set, so the worker reaches WorkflowService's error handler.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Yicong-Huang Yicong-Huang added release/v1.3 back porting to release/v1.3 release/v1.2 back porting to release/v1.2 labels Oct 16, 2026
@github-actions
github-actions Bot requested a review from mengw15 October 16, 2026 22:30
@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @eugenegujing, @VuMartin, @aglinxinyuan
    You can notify them by mentioning @eugenegujing, @VuMartin, @aglinxinyuan in a comment.

@github-actions
github-actions Bot requested a review from xuang7 October 16, 2026 22:30
@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Backport auto-label report

This fix: PR was checked against each actively-supported release branch. A release/* label nominates a backport target; the branch's release manager approving this PR is what sends the fix there. The required Backport Approvals check stays red until every label below is approved, so each manager either approves or removes their own label — which is why the labels left on a merged PR are exactly the branches it reached.

Release branch Analysis
🚫 release/v1.3 Label was removed earlier (opt-out); not re-added. Re-add it by hand if this fix should be backported here after all — the branch's release manager then has to approve before this PR can merge.
🚫 release/v1.2 Label was removed earlier (opt-out); not re-added. Re-add it by hand if this fix should be backported here after all — the branch's release manager then has to approve before this PR can merge.

Auto-label run.

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 6 worse · ⚪ 9 noise (<±5%) · 0 without baseline

Compared against main f020980 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 519 0.317 17,707/29,442/29,442 us 🔴 +18.4% / 🔴 +129.8%
🔴 bs=100 sw=10 sl=64 1,083 0.661 90,093/120,188/120,188 us 🔴 +14.2% / 🔴 +46.9%
⚪ bs=1000 sw=10 sl=64 1,281 0.782 779,283/832,083/832,083 us ⚪ within ±5% / 🔴 -15.6%
Baseline details

Latest main f020980 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 519 tuples/sec 577 tuples/sec 1,084 tuples/sec -10.1% -52.1%
bs=10 sw=10 sl=64 MB/s 0.317 MB/s 0.352 MB/s 0.662 MB/s -9.9% -52.1%
bs=10 sw=10 sl=64 p50 17,707 us 16,891 us 9,548 us +4.8% +85.5%
bs=10 sw=10 sl=64 p95 29,442 us 24,863 us 12,811 us +18.4% +129.8%
bs=10 sw=10 sl=64 p99 29,442 us 24,863 us 15,432 us +18.4% +90.8%
bs=100 sw=10 sl=64 throughput 1,083 tuples/sec 1,136 tuples/sec 1,467 tuples/sec -4.7% -26.2%
bs=100 sw=10 sl=64 MB/s 0.661 MB/s 0.693 MB/s 0.895 MB/s -4.6% -26.2%
bs=100 sw=10 sl=64 p50 90,093 us 87,160 us 74,441 us +3.4% +21.0%
bs=100 sw=10 sl=64 p95 120,188 us 105,197 us 81,800 us +14.2% +46.9%
bs=100 sw=10 sl=64 p99 120,188 us 105,197 us 103,712 us +14.2% +15.9%
bs=1000 sw=10 sl=64 throughput 1,281 tuples/sec 1,272 tuples/sec 1,517 tuples/sec +0.7% -15.6%
bs=1000 sw=10 sl=64 MB/s 0.782 MB/s 0.776 MB/s 0.926 MB/s +0.8% -15.5%
bs=1000 sw=10 sl=64 p50 779,283 us 778,791 us 725,911 us +0.1% +7.4%
bs=1000 sw=10 sl=64 p95 832,083 us 838,839 us 764,152 us -0.8% +8.9%
bs=1000 sw=10 sl=64 p99 832,083 us 838,839 us 808,901 us -0.8% +2.9%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,384.99,200,128000,519,0.317,17707.17,29441.52,29441.52
1,100,10,64,20,1847.32,2000,1280000,1083,0.661,90093.08,120187.63,120187.63
2,1000,10,64,20,15606.86,20000,12800000,1281,0.782,779283.38,832083.23,832083.23

@codecov-commenter

codecov-commenter commented Oct 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.50000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 91.54%. Comparing base (f020980) to head (b2e14c4).

Files with missing lines Patch % Lines
...ala/org/apache/texera/amber/error/ErrorUtils.scala 87.50% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #9015      +/-   ##
============================================
- Coverage     91.56%   91.54%   -0.02%     
+ Complexity     5157     5135      -22     
============================================
  Files          1300     1296       -4     
  Lines         58092    57781     -311     
  Branches       7615     7572      -43     
============================================
- Hits          53194    52898     -296     
+ Misses         2959     2940      -19     
- Partials       1939     1943       +4     
Flag Coverage Δ *Carryforward flag
access-control-service 77.44% <ø> (ø) Carriedforward from 103a0eb
agent-service 99.55% <ø> (-0.01%) ⬇️ Carriedforward from 103a0eb
amber 85.42% <87.50%> (-0.01%) ⬇️
computing-unit-managing-service 64.05% <ø> (+0.17%) ⬆️ Carriedforward from 103a0eb
config-service 87.73% <ø> (+0.23%) ⬆️ Carriedforward from 103a0eb
file-service 82.16% <ø> (ø) Carriedforward from 103a0eb
frontend 96.41% <ø> (-0.05%) ⬇️ Carriedforward from 103a0eb
notebook-migration-service 83.77% <ø> (ø) Carriedforward from 103a0eb
pyamber 98.50% <ø> (+0.11%) ⬆️ Carriedforward from 103a0eb
result-service ?
workflow-compiling-service 78.30% <ø> (ø) Carriedforward from 103a0eb

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@sshiv012

sshiv012 commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

When a Python UDF fails while starting, its error loses track of which operator failed on the way back to the UI, so it gets recorded as "unknown operator" and clicking the failed UDF shows an empty Console. This PR carries the failing worker through ControlError so the error lands on the right operator, and since v1.2 already has #8668 but not this fix, v1.2 and v1.3 users would hit the same confusing empty Console. @mengw15 @yangzhang75 please check if this needs to be backported (it is a small fix).

@mengw15 mengw15 removed the release/v1.3 back porting to release/v1.3 label Nov 9, 2026
@xuang7

xuang7 commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Removing release/v1.2: diagnostics only, and it changes the control protocol.

@xuang7 xuang7 removed the release/v1.2 back porting to release/v1.2 label Nov 16, 2026
@sshiv012

Copy link
Copy Markdown
Contributor Author

/request-review @Xiao-zhen-Liu

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants