Skip to content

Fix PickleCoder.as_deterministic_coder() raising TypeError - #39943

Open
AmirF194 wants to merge 1 commit into
apache:masterfrom
AmirF194:fix/28558-pickle-coder-as-deterministic-coder-typeerror
Open

Fix PickleCoder.as_deterministic_coder() raising TypeError#39943
AmirF194 wants to merge 1 commit into
apache:masterfrom
AmirF194:fix/28558-pickle-coder-as-deterministic-coder-typeerror

Conversation

@AmirF194

Copy link
Copy Markdown

PickleCoder.as_deterministic_coder() and _MemoizingPickleCoder.as_deterministic_coder() both construct FastPrimitivesCoder with a requires_deterministic keyword argument:

def as_deterministic_coder(self, step_label, error_message=None):
  return FastPrimitivesCoder(self, requires_deterministic=step_label)

FastPrimitivesCoder.__init__(self, fallback_coder=PickleCoder()) has never accepted that argument, so both raise TypeError unconditionally. The bug goes back to commit 9be70c9, which changed PickleCoder.as_deterministic_coder from the working DeterministicFastPrimitivesCoder(self, step_label) to this broken call, in the same diff that added a requires_deterministic kwarg to a different class (coder_impl.FastPrimitivesCoderImpl, the Cython impl, not the coders.FastPrimitivesCoder wrapper referenced here). _MemoizingPickleCoder later copied the already-broken line.

The invariant as_deterministic_coder() is supposed to hold, that it always returns a working deterministic coder rather than raising, is exactly what FastPrimitivesCoder's own as_deterministic_coder already does by delegating to _update_compatible_deterministic_fast_primitives_coder. This PR routes both call sites through that same helper instead.

A reachable consumer: GroupByEncryptedKey.expand() calls coder.as_deterministic_coder(...) inside except ValueError, expecting a non-deterministic key to log a warning. When the coder resolves to PickleCoder/_MemoizingPickleCoder, the uncaught TypeError crashes pipeline construction instead.

Verified:

  • PickleCoder().as_deterministic_coder('x') and _MemoizingPickleCoder().as_deterministic_coder('x') raise TypeError on master, confirmed live against the installed apache-beam package (byte-identical coders.py) and against this branch.
  • New PickleCoderTest.test_as_deterministic_coder fails on master, passes on this branch, both runs in the same container.
  • coders_test.py, typecoders_test.py, and fast_coders_test.py (the standard-coder correctness suite, all classes including FastPrimitivesCoder/DeterministicFastPrimitivesCoder) pass unchanged.
  • ruff check and yapf --diff are clean on both changed files.
  • Not checked: the Cython (coder_impl) fast paths, since this fix is confined to the plain-Python coders.py wrapper layer and does not touch coder_impl.

Fixes #39942

PickleCoder and _MemoizingPickleCoder construct FastPrimitivesCoder with a
requires_deterministic kwarg the constructor has never accepted, raising
TypeError on every call instead of returning a deterministic coder. Route
both through the existing _update_compatible_deterministic_fast_primitives_coder
helper, matching FastPrimitivesCoder's own as_deterministic_coder.

Fixes apache#39942
@github-actions

Copy link
Copy Markdown
Contributor

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: PickleCoder/_MemoizingPickleCoder.as_deterministic_coder() raises TypeError

1 participant