Skip to content

feat(pyamber): clear errors for return-instead-of-yield and output mismatches - #8769

Merged
Xiao-zhen-Liu merged 6 commits into
apache:mainfrom
sshiv012:sshiv012/tex-86-udf-runtime-errors
Oct 15, 2026
Merged

Xiao-zhen-Liu merged 6 commits into
apache:mainfrom
sshiv012:sshiv012/tex-86-udf-runtime-errors

Conversation

@sshiv012

@sshiv012 sshiv012 commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Clear messages for the common Python UDF mistakes that show up while the workflow runs:

  • return instead of yield in process_tuple, process_batch, process_table or produce now says process_table must yield results, not return them. Replace return x with yield x. Before, users saw pampy.MatchError: '_' not provided or 'NoneType' object is not iterable. Returning a list still works.
  • __init__ without super().__init__() on a table or batch UDF no longer crashes with '_TableOperator__table_data'. The buffers are created on first use, the same way Operator already creates its template decoder.
  • A DataFrame in a non-binary column now says which column, and to yield the DataFrame itself or make the column binary. A DataFrame in a BINARY column is pickled as before.
  • Output columns that don't match keep the original message and point to "Retain input columns" and "Extra output column(s)" (or "Columns" for a 1-out Python UDF).

Before and after, in the Console:

Before After
runtime-before runtime-after

Any related issues, documentation, discussions?

Part of #8736. Split out of #8737.

How was this PR tested?

New tests in test_operator.py (each return case, a returned list still working, table and batch UDFs with and without super().__init__()) and test_data_processor.py (process_tuple and on_finish returning a value, a DataFrame in BINARY, LARGE_BINARY and STRING columns, a missing or misspelled column next to a DataFrame, a wrong column type).

cd amber
pytest -m "not integration" src/test/python/core/models/test_operator.py src/test/python/core/runnables/test_data_processor.py
pytest -m "not integration"
ruff check src/main/python src/test/python && ruff format --check src/main/python src/test/python

The screenshots are from a local Texera built from this branch.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5)

…smatches

- `return` instead of `yield` in process_tuple, process_batch,
  process_table or produce now raises "<method> must `yield` results,
  not `return` them" instead of MatchError / 'NoneType' not iterable.
  Returning a list still works.
- UDFTableOperator / UDFBatchOperator create their buffers lazily, so
  an __init__ that skips super().__init__() no longer crashes with
  '_TableOperator__table_data' (same approach as the lazy template
  decoder in Operator).
- Yielding a dict of DataFrames says to yield the DataFrame itself.
- Output column mismatches point to 'Retain input columns' and
  'Extra output column(s)'.

Refs apache#8736

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @carloea2, @eugenegujing, @aglinxinyuan
    You can notify them by mentioning @carloea2, @eugenegujing, @aglinxinyuan in a comment.

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 12 worse · ⚪ 3 noise (<±5%) · 0 without baseline

Compared against main cf75462 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 526 0.321 18,163/25,058/25,058 us 🔴 -11.6% / 🔴 +62.0%
🔴 bs=100 sw=10 sl=64 1,173 0.716 83,878/103,772/103,772 us 🔴 +7.9% / 🟢 -13.9%
🔴 bs=1000 sw=10 sl=64 1,402 0.856 708,072/843,124/843,124 us 🔴 +14.7% / 🟢 +27.6%
Baseline details

Latest main cf75462 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 526 tuples/sec 595 tuples/sec 822.11 tuples/sec -11.6% -36.0%
bs=10 sw=10 sl=64 MB/s 0.321 MB/s 0.363 MB/s 0.502 MB/s -11.6% -36.0%
bs=10 sw=10 sl=64 p50 18,163 us 16,913 us 12,129 us +7.4% +49.7%
bs=10 sw=10 sl=64 p95 25,058 us 22,757 us 15,469 us +10.1% +62.0%
bs=10 sw=10 sl=64 p99 25,058 us 22,757 us 18,357 us +10.1% +36.5%
bs=100 sw=10 sl=64 throughput 1,173 tuples/sec 1,249 tuples/sec 1,058 tuples/sec -6.1% +10.9%
bs=100 sw=10 sl=64 MB/s 0.716 MB/s 0.762 MB/s 0.646 MB/s -6.0% +10.9%
bs=100 sw=10 sl=64 p50 83,878 us 77,767 us 97,399 us +7.9% -13.9%
bs=100 sw=10 sl=64 p95 103,772 us 97,744 us 104,184 us +6.2% -0.4%
bs=100 sw=10 sl=64 p99 103,772 us 97,744 us 117,615 us +6.2% -11.8%
bs=1000 sw=10 sl=64 throughput 1,402 tuples/sec 1,433 tuples/sec 1,099 tuples/sec -2.2% +27.6%
bs=1000 sw=10 sl=64 MB/s 0.856 MB/s 0.875 MB/s 0.671 MB/s -2.2% +27.6%
bs=1000 sw=10 sl=64 p50 708,072 us 701,145 us 946,743 us +1.0% -25.2%
bs=1000 sw=10 sl=64 p95 843,124 us 734,933 us 985,437 us +14.7% -14.4%
bs=1000 sw=10 sl=64 p99 843,124 us 734,933 us 1,012,622 us +14.7% -16.7%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,380.20,200,128000,526,0.321,18162.86,25057.62,25057.62
1,100,10,64,20,1704.55,2000,1280000,1173,0.716,83878.23,103772.24,103772.24
2,1000,10,64,20,14265.08,20000,12800000,1402,0.856,708071.50,843124.29,843124.29

@sshiv012

sshiv012 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

/request-review @Xiao-zhen-Liu

@github-actions
github-actions Bot requested a review from Xiao-zhen-Liu October 2, 2026 20:28
- Only explain a schema failure after finalize fails, catching KeyError
  and TypeError. A dict holding a DataFrame into a BINARY column is
  pickled on main and works again; valid rows are no longer inspected.
- A wrong column type now gets the same output-columns hint.
- on_finish goes through require_yielded like process_tuple.
- require_yielded's docstring says which returned values it catches.
- Table and batch buffers are created only by their getters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov-commenter

codecov-commenter commented Oct 6, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.33%. Comparing base (cf75462) to head (aa96279).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #8769      +/-   ##
============================================
+ Coverage     92.30%   92.33%   +0.03%     
  Complexity     5045     5045              
============================================
  Files          1259     1258       -1     
  Lines         53917    53884      -33     
  Branches       6721     6716       -5     
============================================
- Hits          49766    49752      -14     
+ Misses         2488     2476      -12     
+ Partials       1663     1656       -7     
Flag Coverage Δ *Carryforward flag
access-control-service 77.38% <ø> (ø) Carriedforward from 6a101ad
agent-service 99.16% <ø> (ø) Carriedforward from 6a101ad
amber 87.31% <ø> (+<0.01%) ⬆️ Carriedforward from 6a101ad
computing-unit-managing-service 60.48% <ø> (ø) Carriedforward from 6a101ad
config-service 87.37% <ø> (ø) Carriedforward from 6a101ad
file-service 82.00% <ø> (ø) Carriedforward from 6a101ad
frontend 96.60% <ø> (+0.05%) ⬆️ Carriedforward from 6a101ad
notebook-migration-service 83.73% <ø> (ø) Carriedforward from 6a101ad
pyamber 98.51% <100.00%> (+0.07%) ⬆️
workflow-compiling-service 72.43% <ø> (ø) Carriedforward from 6a101ad

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

A local symlink at amber/src/main/python/proto slipped into the previous
commit. It blocked CI's proto generation (mkdir: File exists).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
renovate-bot pushed a commit to renovate-bot/apache-_-texera that referenced this pull request Oct 7, 2026
…pache#8887)

### What changes were proposed in this PR?

Two `.gitignore` patterns end in `/`, which only matches directories:

```
amber/src/main/python/proto/
**/node_modules/
```

A symlink at either path isn't ignored. This is easy to hit when you run
tests in a second worktree and symlink the generated protos or
`node_modules` from your main checkout instead of regenerating them. A
`git add -A` then commits the symlink.

That happened to me in apache#8769: the symlink got committed, and CI's
`bin/python-proto-gen.sh` failed with `mkdir: cannot create directory
'.../amber/src/main/python/proto': File exists`, so every job stopped
before running tests.

This drops the trailing slash on both lines, so files, symlinks and
directories at those paths are all ignored.

### Any related issues, documentation, discussions?

Found while working on apache#8769.

### How was this PR tested?

In a scratch repo with a symlink at `amber/src/main/python/proto` and at
`agent-service/node_modules`:

- old patterns: both symlinks show up in `git status` as untracked
- new patterns: both are ignored

Real directories at those paths are still ignored, since a pattern
without a trailing slash matches directories too.

### Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5)

Co-authored-by: suryaacharan <suryaacharan@suryaacharans-MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
suryaacharan and others added 3 commits October 8, 2026 11:36
- The DataFrame hint now fires only for a TypeError (a wrong type in a
  column), never for a missing or extra column, and only when a
  DataFrame sits in an output column that isn't BINARY. A misspelled or
  missing column next to a DataFrame gets the output-columns hint.
- The hint keeps the original message and names the column.
- The output-columns hint also mentions 'Columns' for 1-out Python UDFs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Xiao-zhen-Liu
Xiao-zhen-Liu added this pull request to the merge queue Oct 15, 2026
sshiv012 pushed a commit to sshiv012/texera that referenced this pull request Oct 15, 2026
- Result panel: the empty states now say "No operator is selected" or
  "This operator has no results yet"; clicking a failed box shows its
  Static Error tab (apache#8771).
- UDF guide: new rows for "does not implement <method>" and "does not
  subclass a pytexera base class", @OVERRIDES on a helper, the
  "(NameError at line 2 of the UDF code)" format, and "Columns" for
  1-out UDFs (apache#8768, apache#8769).
- Retake errors-all-operators.png with the final message, on a Texera
  built from all five code PRs on current main.

Refs apache#8736

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merged via the queue into apache:main with commit 91f4753 Oct 15, 2026
24 checks passed
aglinxinyuan added a commit to aglinxinyuan/texera that referenced this pull request Oct 21, 2026
…tter

Main (apache#8769) creates TableOperator's table data on first use, so the
attribute is missing until a tuple arrives; _collected_table now goes
through _get_table_data like the rest of the class.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants