Repository navigation
Conversation
|
This replaces stale-closed #8214. The problem still reproduces on current main. The patch passes its regression, 352 broader tests and Ruff checks. /request-review @aglinxinyuan |
|
/request-review @aglinxinyuan |
Automated Reviewer SuggestionsBased on the
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #8958 +/- ##
============================================
- Coverage 91.80% 91.80% -0.01%
+ Complexity 5025 5024 -1
============================================
Files 1279 1279
Lines 56557 56575 +18
Branches 7419 7427 +8
============================================
+ Hits 51924 51936 +12
- Misses 2768 2770 +2
- Partials 1865 1869 +4
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 773 | 0.472 | 12,441/16,558/16,558 us | 🔴 -6.5% / 🔴 +16.7% |
| 🔴 | bs=100 sw=10 sl=64 | 1,775 | 1.083 | 53,890/77,472/77,472 us | 🔴 +29.7% / 🟢 +54.2% |
| 🔴 | bs=1000 sw=10 sl=64 | 2,098 | 1.281 | 470,817/551,632/551,632 us | 🔴 +6.7% / 🟢 +76.5% |
Baseline details
Latest main 6db4a64 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 773 tuples/sec | 827 tuples/sec | 880.16 tuples/sec | -6.5% | -12.2% |
| bs=10 sw=10 sl=64 | MB/s | 0.472 MB/s | 0.505 MB/s | 0.537 MB/s | -6.5% | -12.1% |
| bs=10 sw=10 sl=64 | p50 | 12,441 us | 12,429 us | 11,544 us | +0.1% | +7.8% |
| bs=10 sw=10 sl=64 | p95 | 16,558 us | 17,702 us | 14,186 us | -6.5% | +16.7% |
| bs=10 sw=10 sl=64 | p99 | 16,558 us | 17,702 us | 17,122 us | -6.5% | -3.3% |
| bs=100 sw=10 sl=64 | throughput | 1,775 tuples/sec | 1,931 tuples/sec | 1,151 tuples/sec | -8.1% | +54.2% |
| bs=100 sw=10 sl=64 | MB/s | 1.083 MB/s | 1.179 MB/s | 0.703 MB/s | -8.1% | +54.1% |
| bs=100 sw=10 sl=64 | p50 | 53,890 us | 51,366 us | 89,900 us | +4.9% | -40.1% |
| bs=100 sw=10 sl=64 | p95 | 77,472 us | 59,747 us | 96,346 us | +29.7% | -19.6% |
| bs=100 sw=10 sl=64 | p99 | 77,472 us | 59,747 us | 110,344 us | +29.7% | -29.8% |
| bs=1000 sw=10 sl=64 | throughput | 2,098 tuples/sec | 2,164 tuples/sec | 1,189 tuples/sec | -3.0% | +76.4% |
| bs=1000 sw=10 sl=64 | MB/s | 1.281 MB/s | 1.321 MB/s | 0.726 MB/s | -3.0% | +76.5% |
| bs=1000 sw=10 sl=64 | p50 | 470,817 us | 456,887 us | 873,675 us | +3.0% | -46.1% |
| bs=1000 sw=10 sl=64 | p95 | 551,632 us | 517,130 us | 918,440 us | +6.7% | -39.9% |
| bs=1000 sw=10 sl=64 | p99 | 551,632 us | 517,130 us | 953,075 us | +6.7% | -42.1% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,258.78,200,128000,773,0.472,12440.73,16557.68,16557.68
1,100,10,64,20,1126.91,2000,1280000,1775,1.083,53890.08,77471.52,77471.52
2,1000,10,64,20,9532.86,20000,12800000,2098,1.281,470816.54,551632.00,551632.001e9079e to
834d3bb
Compare
|
/request-review @sshiv012 @parshimers |
834d3bb to
46a24eb
Compare
What changes were proposed in this PR?
Preserve user dictionaries containing
__texera_type__when state crosses Python and the JVM. Both codecs escape these dictionaries and only decode exact internal envelopes. The JVM also rejects malformed envelope payloads and non-object state content.Any related issues, documentation, discussions?
Closes #8213. Replaces #8214.
How was this PR tested?
Six new JVM regression tests failed before the JVM change. After the change:
sbt "WorkflowCore/testOnly *StateSpec"python -m pytest src/test/python/core/models src/test/python/core/architecture/sendsemantics/test_partitioners.py --deselect src/test/python/core/models/test_tuple.py::TestTuple::test_hash -q --tb=short -p no:cacheproviderfromamberpython -m ruff check src/main/python src/test/pythonfromamberpython -m ruff format --check src/main/python src/test/pythonfromambersbt "WorkflowCore/scalafix --check" "WorkflowCore/Test/scalafix --check" scalafmtCheckAllTested in a detached worktree from main at
6db4a642d0. Local sbt runs reused ignored generated DAO sources withset DAO / Compile / sourceGenerators := Seq.emptyinstead of regenerating the database schema.Was this PR authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex, GPT-6