Skip to content

refactor(pyamber, amber): remove redundant Python code - #9550

Open
aglinxinyuan wants to merge 1 commit into
apache:mainfrom
aglinxinyuan:refactor/pyamber-redundant-code-cleanup
Open

aglinxinyuan wants to merge 1 commit into
apache:mainfrom
aglinxinyuan:refactor/pyamber-redundant-code-cleanup

Conversation

@aglinxinyuan

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Remove redundant statements, unused imports, unnecessary control-flow branches, and legacy idioms across amber/src/main/python/ and amber/src/main/scala/org/apache/texera/web/resource/aiassistant/type_annotation_visitor.py with no runtime behavior change:

  • Redundant statements and unused imports: Remove redundant pass statements in docstring-only abstract methods and marker classes (InternalMarker, BufferedItemWriter, ReadonlyVirtualDocument, and UDFOperatorV2/UDFSourceOperator/UDFTableOperator/UDFBatchOperator), remove trailing return None in BackpressureHandler, drop unused import core, unnecessary dict unpacking, and intermediate return variables in iceberg_utils and timestamp, and remove the unused enumerate index in TypeAnnotationVisitor.
  • Unnecessary control-flow branches: Remove redundant else/elif blocks following return in detect_attribute_type_from_arrow_field, Table.__eq__, DocumentFactory, and IcebergCatalogInstance.get_instance, and merge identical staticmethod decorator checks in TypeAnnotationVisitor.
  • Idiomatic constructors, formatting, and stdlib equivalents: Replace list()/dict()/bytes() with literal forms ([]/{}/b""), replace super(ProxyServer, self) with super(), replace map(lambda ...) with a generator expression and simplify the payload conditional in ProxyServer, adopt datetime.UTC and @functools.cache in attribute_type, and use !r/!s f-strings in Schema, ProxyServer, Heartbeat, and ExpressionEvaluator.

Any related issues, documentation, discussions?

Closes #9543.

How was this PR tested?

Ran in amber/ on 2a319878af (Python 3.12.10, ruff 0.14.7, pytest 7.4.0):

ruff check src/main/python src/test/python
ruff format --check src/main/python src/test/python
ruff check src/main/scala/org/apache/texera/web/resource/aiassistant/type_annotation_visitor.py
PYTHONPATH=src/main/scala/org/apache/texera/web/resource/aiassistant \
  pytest src/main/scala/org/apache/texera/web/resource/aiassistant/test_type_annotation_visitor.py -q
pytest \
  src/test/python/core/architecture/handlers/actorcommand/test_actor_handler_base.py \
  src/test/python/core/architecture/managers/test_exception_manager.py \
  src/test/python/core/architecture/rpc/test_async_rpc_client.py \
  src/test/python/core/architecture/sendsemantics/test_partitioners.py \
  src/test/python/core/models/schema/test_arrow_schema_utils.py \
  src/test/python/core/models/schema/test_attribute_type.py \
  src/test/python/core/models/schema/test_schema.py \
  src/test/python/core/models/test_table.py \
  src/test/python/core/proxy/test_proxy_client.py \
  src/test/python/core/proxy/test_proxy_server.py \
  src/test/python/core/runnables/test_console_message.py \
  src/test/python/core/runnables/test_heartbeat.py \
  src/test/python/core/runnables/test_main_loop.py \
  src/test/python/core/storage/iceberg/test_iceberg_catalog_instance.py \
  src/test/python/core/storage/iceberg/test_iceberg_utils_large_binary.py \
  src/test/python/core/storage/model/test_virtual_document.py \
  src/test/python/core/storage/test_document_factory.py \
  src/test/python/pytexera/udf/test_udf_operator.py -q
pytest src/test/python/core/storage/iceberg/test_iceberg_utils_catalog.py \
  -k "not test_signed_filesystems_are_isolated_between_file_ios" -q
pytest src/test/python/core/util/test_expression_evaluator.py \
  -k "not object and not container" -q
  • ruff check and ruff format --check (218 files): passed.
  • test_type_annotation_visitor.py: 5 passed.
  • 18 targeted amber Python unit-test suites: 320 passed, 9 skipped (test_attribute_type.py Windows time.tzset skips).
  • test_iceberg_utils_catalog.py (excluding the 3 s3fs-dependent cases absent locally): 19 passed.
  • test_expression_evaluator.py (excluding the 2 --import-mode=importlib repr cases): 3 passed.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Antigravity (Gemini 4 Neon)

Copilot AI balanced review requested due to automatic review settings November 19, 2026 09:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @Yicong-Huang, @carloea2, @kz930
    You can notify them by mentioning @Yicong-Huang, @carloea2, @kz930 in a comment.

@codecov-commenter

codecov-commenter commented Nov 19, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.45455% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 91.58%. Comparing base (4db10cf) to head (2a31987).

Files with missing lines Patch % Lines
amber/src/main/python/core/proxy/proxy_server.py 83.33% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #9550      +/-   ##
============================================
+ Coverage     91.57%   91.58%   +0.01%     
+ Complexity     5159     5158       -1     
============================================
  Files          1300     1300              
  Lines         58092    58067      -25     
  Branches       7615     7614       -1     
============================================
- Hits          53198    53182      -16     
+ Misses         2958     2948      -10     
- Partials       1936     1937       +1     
Flag Coverage Δ *Carryforward flag
access-control-service 77.44% <ø> (ø) Carriedforward from 4db10cf
agent-service 99.56% <ø> (ø) Carriedforward from 4db10cf
amber 85.43% <ø> (-0.01%) ⬇️
computing-unit-managing-service 63.88% <ø> (ø) Carriedforward from 4db10cf
config-service 87.50% <ø> (ø) Carriedforward from 4db10cf
file-service 82.16% <ø> (ø) Carriedforward from 4db10cf
frontend 96.45% <ø> (ø) Carriedforward from 4db10cf
notebook-migration-service 83.77% <ø> (ø) Carriedforward from 4db10cf
pyamber 98.58% <95.45%> (+0.19%) ⬆️
result-service 85.71% <ø> (ø) Carriedforward from 4db10cf
workflow-compiling-service 78.30% <ø> (ø) Carriedforward from 4db10cf

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 7 worse · ⚪ 6 noise (<±5%) · 0 without baseline

Compared against main 4db10cf benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 490 0.299 18,830/39,338/39,338 us 🔴 +54.9% / 🔴 +207.1%
🟢 bs=100 sw=10 sl=64 1,247 0.761 82,141/87,858/87,858 us 🟢 -11.9% / 🟢 -15.3%
🔴 bs=1000 sw=10 sl=64 1,429 0.872 700,032/825,342/825,342 us 🔴 +10.8% / 🔴 +8.0%
Baseline details

Latest main 4db10cf from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 490 tuples/sec 586 tuples/sec 1,084 tuples/sec -16.4% -54.8%
bs=10 sw=10 sl=64 MB/s 0.299 MB/s 0.357 MB/s 0.662 MB/s -16.2% -54.8%
bs=10 sw=10 sl=64 p50 18,830 us 16,817 us 9,548 us +12.0% +97.2%
bs=10 sw=10 sl=64 p95 39,338 us 25,403 us 12,811 us +54.9% +207.1%
bs=10 sw=10 sl=64 p99 39,338 us 25,403 us 15,432 us +54.9% +154.9%
bs=100 sw=10 sl=64 throughput 1,247 tuples/sec 1,274 tuples/sec 1,467 tuples/sec -2.1% -15.0%
bs=100 sw=10 sl=64 MB/s 0.761 MB/s 0.777 MB/s 0.895 MB/s -2.1% -15.0%
bs=100 sw=10 sl=64 p50 82,141 us 79,151 us 74,441 us +3.8% +10.3%
bs=100 sw=10 sl=64 p95 87,858 us 99,697 us 81,800 us -11.9% +7.4%
bs=100 sw=10 sl=64 p99 87,858 us 99,697 us 103,712 us -11.9% -15.3%
bs=1000 sw=10 sl=64 throughput 1,429 tuples/sec 1,425 tuples/sec 1,517 tuples/sec +0.3% -5.8%
bs=1000 sw=10 sl=64 MB/s 0.872 MB/s 0.87 MB/s 0.926 MB/s +0.2% -5.8%
bs=1000 sw=10 sl=64 p50 700,032 us 707,138 us 725,911 us -1.0% -3.6%
bs=1000 sw=10 sl=64 p95 825,342 us 744,895 us 764,152 us +10.8% +8.0%
bs=1000 sw=10 sl=64 p99 825,342 us 744,895 us 808,901 us +10.8% +2.0%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,408.04,200,128000,490,0.299,18829.71,39338.04,39338.04
1,100,10,64,20,1604.03,2000,1280000,1247,0.761,82141.28,87858.36,87858.36
2,1000,10,64,20,13999.70,20000,12800000,1429,0.872,700031.65,825341.88,825341.88

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remove redundant Python statements, imports, and branches in Amber

3 participants