Fix PickleCoder.as_deterministic_coder() raising TypeError - #39943
Open
AmirF194 wants to merge 1 commit into
Open
Fix PickleCoder.as_deterministic_coder() raising TypeError#39943AmirF194 wants to merge 1 commit into
AmirF194 wants to merge 1 commit into
Conversation
PickleCoder and _MemoizingPickleCoder construct FastPrimitivesCoder with a requires_deterministic kwarg the constructor has never accepted, raising TypeError on every call instead of returning a deterministic coder. Route both through the existing _update_compatible_deterministic_fast_primitives_coder helper, matching FastPrimitivesCoder's own as_deterministic_coder. Fixes apache#39942
Contributor
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PickleCoder.as_deterministic_coder()and_MemoizingPickleCoder.as_deterministic_coder()both constructFastPrimitivesCoderwith arequires_deterministickeyword argument:FastPrimitivesCoder.__init__(self, fallback_coder=PickleCoder())has never accepted that argument, so both raiseTypeErrorunconditionally. The bug goes back to commit 9be70c9, which changedPickleCoder.as_deterministic_coderfrom the workingDeterministicFastPrimitivesCoder(self, step_label)to this broken call, in the same diff that added arequires_deterministickwarg to a different class (coder_impl.FastPrimitivesCoderImpl, the Cython impl, not thecoders.FastPrimitivesCoderwrapper referenced here)._MemoizingPickleCoderlater copied the already-broken line.The invariant
as_deterministic_coder()is supposed to hold, that it always returns a working deterministic coder rather than raising, is exactly whatFastPrimitivesCoder's ownas_deterministic_coderalready does by delegating to_update_compatible_deterministic_fast_primitives_coder. This PR routes both call sites through that same helper instead.A reachable consumer:
GroupByEncryptedKey.expand()callscoder.as_deterministic_coder(...)insideexcept ValueError, expecting a non-deterministic key to log a warning. When the coder resolves toPickleCoder/_MemoizingPickleCoder, the uncaughtTypeErrorcrashes pipeline construction instead.Verified:
PickleCoder().as_deterministic_coder('x')and_MemoizingPickleCoder().as_deterministic_coder('x')raiseTypeErroronmaster, confirmed live against the installedapache-beampackage (byte-identicalcoders.py) and against this branch.PickleCoderTest.test_as_deterministic_coderfails onmaster, passes on this branch, both runs in the same container.coders_test.py,typecoders_test.py, andfast_coders_test.py(the standard-coder correctness suite, all classes includingFastPrimitivesCoder/DeterministicFastPrimitivesCoder) pass unchanged.ruff checkandyapf --diffare clean on both changed files.coder_impl) fast paths, since this fix is confined to the plain-Pythoncoders.pywrapper layer and does not touchcoder_impl.Fixes #39942