Repository navigation
feat(pyamber): clear errors for return-instead-of-yield and output mismatches - #8769
Conversation
…smatches - `return` instead of `yield` in process_tuple, process_batch, process_table or produce now raises "<method> must `yield` results, not `return` them" instead of MatchError / 'NoneType' not iterable. Returning a list still works. - UDFTableOperator / UDFBatchOperator create their buffers lazily, so an __init__ that skips super().__init__() no longer crashes with '_TableOperator__table_data' (same approach as the lazy template decoder in Operator). - Yielding a dict of DataFrames says to yield the DataFrame itself. - Output column mismatches point to 'Retain input columns' and 'Extra output column(s)'. Refs apache#8736 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Automated Reviewer SuggestionsBased on the
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 526 | 0.321 | 18,163/25,058/25,058 us | 🔴 -11.6% / 🔴 +62.0% |
| 🔴 | bs=100 sw=10 sl=64 | 1,173 | 0.716 | 83,878/103,772/103,772 us | 🔴 +7.9% / 🟢 -13.9% |
| 🔴 | bs=1000 sw=10 sl=64 | 1,402 | 0.856 | 708,072/843,124/843,124 us | 🔴 +14.7% / 🟢 +27.6% |
Baseline details
Latest main cf75462 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 526 tuples/sec | 595 tuples/sec | 822.11 tuples/sec | -11.6% | -36.0% |
| bs=10 sw=10 sl=64 | MB/s | 0.321 MB/s | 0.363 MB/s | 0.502 MB/s | -11.6% | -36.0% |
| bs=10 sw=10 sl=64 | p50 | 18,163 us | 16,913 us | 12,129 us | +7.4% | +49.7% |
| bs=10 sw=10 sl=64 | p95 | 25,058 us | 22,757 us | 15,469 us | +10.1% | +62.0% |
| bs=10 sw=10 sl=64 | p99 | 25,058 us | 22,757 us | 18,357 us | +10.1% | +36.5% |
| bs=100 sw=10 sl=64 | throughput | 1,173 tuples/sec | 1,249 tuples/sec | 1,058 tuples/sec | -6.1% | +10.9% |
| bs=100 sw=10 sl=64 | MB/s | 0.716 MB/s | 0.762 MB/s | 0.646 MB/s | -6.0% | +10.9% |
| bs=100 sw=10 sl=64 | p50 | 83,878 us | 77,767 us | 97,399 us | +7.9% | -13.9% |
| bs=100 sw=10 sl=64 | p95 | 103,772 us | 97,744 us | 104,184 us | +6.2% | -0.4% |
| bs=100 sw=10 sl=64 | p99 | 103,772 us | 97,744 us | 117,615 us | +6.2% | -11.8% |
| bs=1000 sw=10 sl=64 | throughput | 1,402 tuples/sec | 1,433 tuples/sec | 1,099 tuples/sec | -2.2% | +27.6% |
| bs=1000 sw=10 sl=64 | MB/s | 0.856 MB/s | 0.875 MB/s | 0.671 MB/s | -2.2% | +27.6% |
| bs=1000 sw=10 sl=64 | p50 | 708,072 us | 701,145 us | 946,743 us | +1.0% | -25.2% |
| bs=1000 sw=10 sl=64 | p95 | 843,124 us | 734,933 us | 985,437 us | +14.7% | -14.4% |
| bs=1000 sw=10 sl=64 | p99 | 843,124 us | 734,933 us | 1,012,622 us | +14.7% | -16.7% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,380.20,200,128000,526,0.321,18162.86,25057.62,25057.62
1,100,10,64,20,1704.55,2000,1280000,1173,0.716,83878.23,103772.24,103772.24
2,1000,10,64,20,14265.08,20000,12800000,1402,0.856,708071.50,843124.29,843124.29|
/request-review @Xiao-zhen-Liu |
- Only explain a schema failure after finalize fails, catching KeyError and TypeError. A dict holding a DataFrame into a BINARY column is pickled on main and works again; valid rows are no longer inspected. - A wrong column type now gets the same output-columns hint. - on_finish goes through require_yielded like process_tuple. - require_yielded's docstring says which returned values it catches. - Table and batch buffers are created only by their getters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8769 +/- ##
============================================
+ Coverage 92.30% 92.33% +0.03%
Complexity 5045 5045
============================================
Files 1259 1258 -1
Lines 53917 53884 -33
Branches 6721 6716 -5
============================================
- Hits 49766 49752 -14
+ Misses 2488 2476 -12
+ Partials 1663 1656 -7
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
A local symlink at amber/src/main/python/proto slipped into the previous commit. It blocked CI's proto generation (mkdir: File exists). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pache#8887) ### What changes were proposed in this PR? Two `.gitignore` patterns end in `/`, which only matches directories: ``` amber/src/main/python/proto/ **/node_modules/ ``` A symlink at either path isn't ignored. This is easy to hit when you run tests in a second worktree and symlink the generated protos or `node_modules` from your main checkout instead of regenerating them. A `git add -A` then commits the symlink. That happened to me in apache#8769: the symlink got committed, and CI's `bin/python-proto-gen.sh` failed with `mkdir: cannot create directory '.../amber/src/main/python/proto': File exists`, so every job stopped before running tests. This drops the trailing slash on both lines, so files, symlinks and directories at those paths are all ignored. ### Any related issues, documentation, discussions? Found while working on apache#8769. ### How was this PR tested? In a scratch repo with a symlink at `amber/src/main/python/proto` and at `agent-service/node_modules`: - old patterns: both symlinks show up in `git status` as untracked - new patterns: both are ignored Real directories at those paths are still ignored, since a pattern without a trailing slash matches directories too. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) Co-authored-by: suryaacharan <suryaacharan@suryaacharans-MacBook-Pro.local> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
- The DataFrame hint now fires only for a TypeError (a wrong type in a column), never for a missing or extra column, and only when a DataFrame sits in an output column that isn't BINARY. A misspelled or missing column next to a DataFrame gets the output-columns hint. - The hint keeps the original message and names the column. - The output-columns hint also mentions 'Columns' for 1-out Python UDFs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Result panel: the empty states now say "No operator is selected" or "This operator has no results yet"; clicking a failed box shows its Static Error tab (apache#8771). - UDF guide: new rows for "does not implement <method>" and "does not subclass a pytexera base class", @OVERRIDES on a helper, the "(NameError at line 2 of the UDF code)" format, and "Columns" for 1-out UDFs (apache#8768, apache#8769). - Retake errors-all-operators.png with the final message, on a Texera built from all five code PRs on current main. Refs apache#8736 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tter Main (apache#8769) creates TableOperator's table data on first use, so the attribute is missing until a tuple arrives; _collected_table now goes through _get_table_data like the rest of the class.
What changes were proposed in this PR?
Clear messages for the common Python UDF mistakes that show up while the workflow runs:
returninstead ofyieldinprocess_tuple,process_batch,process_tableorproducenow saysprocess_table must yield results, not return them. Replace return x with yield x.Before, users sawpampy.MatchError: '_' not providedor'NoneType' object is not iterable. Returning a list still works.__init__withoutsuper().__init__()on a table or batch UDF no longer crashes with'_TableOperator__table_data'. The buffers are created on first use, the same wayOperatoralready creates its template decoder.Before and after, in the Console:
Any related issues, documentation, discussions?
Part of #8736. Split out of #8737.
How was this PR tested?
New tests in
test_operator.py(eachreturncase, a returned list still working, table and batch UDFs with and withoutsuper().__init__()) andtest_data_processor.py(process_tupleandon_finishreturning a value, a DataFrame in BINARY, LARGE_BINARY and STRING columns, a missing or misspelled column next to a DataFrame, a wrong column type).The screenshots are from a local Texera built from this branch.
Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)