Skip to content

feat: preserve state marker dictionaries across runtimes - #8958

Open
carloea2 wants to merge 1 commit into
apache:mainfrom
carloea2:codex/feature-state-marker-escape
Open

carloea2 wants to merge 1 commit into
apache:mainfrom
carloea2:codex/feature-state-marker-escape

Conversation

@carloea2

@carloea2 carloea2 commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Preserve user dictionaries containing __texera_type__ when state crosses Python and the JVM. Both codecs escape these dictionaries and only decode exact internal envelopes. The JVM also rejects malformed envelope payloads and non-object state content.

Any related issues, documentation, discussions?

Closes #8213. Replaces #8214.

How was this PR tested?

Six new JVM regression tests failed before the JVM change. After the change:

Check Result
sbt "WorkflowCore/testOnly *StateSpec" 18 passed
Python to JVM tuple to Python round trips 13 passed, including marker dictionaries, nested bytes, Unicode, empty state and loop metadata
python -m pytest src/test/python/core/models src/test/python/core/architecture/sendsemantics/test_partitioners.py --deselect src/test/python/core/models/test_tuple.py::TestTuple::test_hash -q --tb=short -p no:cacheprovider from amber 381 passed, 9 skipped, 1 xfailed; 2 known Windows timestamp-hash cases excluded, tracked separately in #8957
python -m ruff check src/main/python src/test/python from amber Passed
python -m ruff format --check src/main/python src/test/python from amber Passed
sbt "WorkflowCore/scalafix --check" "WorkflowCore/Test/scalafix --check" scalafmtCheckAll Passed

Tested in a detached worktree from main at 6db4a642d0. Local sbt runs reused ignored generated DAO sources with set DAO / Compile / sourceGenerators := Seq.empty instead of regenerating the database schema.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex, GPT-6

@carloea2

Copy link
Copy Markdown
Contributor Author

This replaces stale-closed #8214. The problem still reproduces on current main. The patch passes its regression, 352 broader tests and Ruff checks.

/request-review @aglinxinyuan

@carloea2

Copy link
Copy Markdown
Contributor Author

/request-review @aglinxinyuan

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @aglinxinyuan
    You can notify them by mentioning @aglinxinyuan in a comment.

@codecov-commenter

codecov-commenter commented Oct 12, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.28571% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.80%. Comparing base (6db4a64) to head (46a24eb).

Files with missing lines Patch % Lines
...ala/org/apache/texera/amber/core/state/State.scala 84.21% 0 Missing and 3 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #8958      +/-   ##
============================================
- Coverage     91.80%   91.80%   -0.01%     
+ Complexity     5025     5024       -1     
============================================
  Files          1279     1279              
  Lines         56557    56575      +18     
  Branches       7419     7427       +8     
============================================
+ Hits          51924    51936      +12     
- Misses         2768     2770       +2     
- Partials       1865     1869       +4     
Flag Coverage Δ *Carryforward flag
access-control-service 77.44% <ø> (ø)
agent-service 99.53% <ø> (ø) Carriedforward from 6db4a64
amber 85.78% <84.21%> (-0.01%) ⬇️
computing-unit-managing-service 64.05% <ø> (ø)
config-service 87.73% <ø> (ø)
file-service 82.31% <ø> (ø)
frontend 96.51% <ø> (ø) Carriedforward from 6db4a64
notebook-migration-service 83.77% <ø> (ø)
pyamber 98.44% <100.00%> (-0.07%) ⬇️
workflow-compiling-service 78.30% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 8 worse · ⚪ 5 noise (<±5%) · 0 without baseline

Compared against main 6db4a64 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 773 0.472 12,441/16,558/16,558 us 🔴 -6.5% / 🔴 +16.7%
🔴 bs=100 sw=10 sl=64 1,775 1.083 53,890/77,472/77,472 us 🔴 +29.7% / 🟢 +54.2%
🔴 bs=1000 sw=10 sl=64 2,098 1.281 470,817/551,632/551,632 us 🔴 +6.7% / 🟢 +76.5%
Baseline details

Latest main 6db4a64 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 773 tuples/sec 827 tuples/sec 880.16 tuples/sec -6.5% -12.2%
bs=10 sw=10 sl=64 MB/s 0.472 MB/s 0.505 MB/s 0.537 MB/s -6.5% -12.1%
bs=10 sw=10 sl=64 p50 12,441 us 12,429 us 11,544 us +0.1% +7.8%
bs=10 sw=10 sl=64 p95 16,558 us 17,702 us 14,186 us -6.5% +16.7%
bs=10 sw=10 sl=64 p99 16,558 us 17,702 us 17,122 us -6.5% -3.3%
bs=100 sw=10 sl=64 throughput 1,775 tuples/sec 1,931 tuples/sec 1,151 tuples/sec -8.1% +54.2%
bs=100 sw=10 sl=64 MB/s 1.083 MB/s 1.179 MB/s 0.703 MB/s -8.1% +54.1%
bs=100 sw=10 sl=64 p50 53,890 us 51,366 us 89,900 us +4.9% -40.1%
bs=100 sw=10 sl=64 p95 77,472 us 59,747 us 96,346 us +29.7% -19.6%
bs=100 sw=10 sl=64 p99 77,472 us 59,747 us 110,344 us +29.7% -29.8%
bs=1000 sw=10 sl=64 throughput 2,098 tuples/sec 2,164 tuples/sec 1,189 tuples/sec -3.0% +76.4%
bs=1000 sw=10 sl=64 MB/s 1.281 MB/s 1.321 MB/s 0.726 MB/s -3.0% +76.5%
bs=1000 sw=10 sl=64 p50 470,817 us 456,887 us 873,675 us +3.0% -46.1%
bs=1000 sw=10 sl=64 p95 551,632 us 517,130 us 918,440 us +6.7% -39.9%
bs=1000 sw=10 sl=64 p99 551,632 us 517,130 us 953,075 us +6.7% -42.1%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,258.78,200,128000,773,0.472,12440.73,16557.68,16557.68
1,100,10,64,20,1126.91,2000,1280000,1775,1.083,53890.08,77471.52,77471.52
2,1000,10,64,20,9532.86,20000,12800000,2098,1.281,470816.54,551632.00,551632.00

@carloea2
carloea2 force-pushed the codex/feature-state-marker-escape branch from 1e9079e to 834d3bb Compare October 15, 2026 06:43
@carloea2

Copy link
Copy Markdown
Contributor Author

/request-review @sshiv012 @parshimers

@carloea2
carloea2 force-pushed the codex/feature-state-marker-escape branch from 834d3bb to 46a24eb Compare November 11, 2026 02:35
@carloea2 carloea2 changed the title feat(pyamber): escape state marker collisions feat: preserve state marker dictionaries across runtimes Nov 11, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

State byte markers can overwrite user dictionaries

3 participants