Repository navigation
[BP-2.4][FLINK-41180][core] Sort VARIANT object fields by UTF-8 bytes - #29622
Merged
Merged
Conversation
Variant.getField decoded each key it compared into a new String. Finding a field in a 13-key object could create 13 throwaway strings.
For objects below BINARY_SEARCH_THRESHOLD fields, getField now encodes the wanted key once and compares it with the UTF-8 bytes of the metadata dictionary. Larger objects still decode each key and compare Strings, because the binary search depends on the order in which the builder sorts the keys.
The wanted key is encoded the same way the builder encodes the keys it writes. So below the threshold, getField("a\uD800") now finds the field stored as "a?". Larger objects still miss such a key.
Metadata keys that are not valid UTF-8 no longer match the U+FFFD name that getFieldNames lists for them, in objects below the threshold. Flink does not write such keys, and distinct malformed keys already decoded to the same name.
JMH on JDK 17, 4 lookups per row of 13 keys: 285 ns to 132 ns per row, 1368 to 296 bytes allocated.
Generated-by: Claude Code 2.1.309 (Claude Opus 5.5)
(cherry picked from commit 82834bb)
The zh page keeps the VARIANT section in English, but it missed later changes: the TIME bullet, the cast of a whole ARRAY, MAP, ROW or STRUCTURED value into a VARIANT, the matching cells of the cast matrix, and note 8. Copy them from the en page. Generated-by: Claude Code 2.1.310 (Claude Opus 5.5) (cherry picked from commit e9939d5)
The variant spec orders the fields of an object by the unsigned UTF-8 bytes of their names. The builder sorted them with String.compareTo, which is UTF-16 order, and getField binary-searched objects of 32 or more fields in the same order. The two orders differ when, at the first differing character, one name has a code point from U+E000 to U+FFFF and the other one from U+10000 up. In a spec-ordered object of 32 fields with the keys U+FF61 and U+1F600, getField(U+1F600) returned null.
The builder now sorts fields by their stored key bytes, and getField binary-searches in the same order. Objects written in UTF-16 order stay readable: after a miss for a key with a code point from U+E000 up, getField scans the fields linearly. For any other key both orders agree, so the miss stands.
Keys that differ as Strings but are stored as the same bytes, such as "a\uD800" and "a?", are now duplicates. Such an object held two fields with the same name before. getField("a\uD800") now finds the field in objects of any size.
Objects with keys from U+E000 up can have a different field order now, which shows in getFieldNames and in CAST to STRING. A Flink version without this change can miss such keys in objects of 32 or more fields. Such an object stored before the upgrade is not byte-equal to the same object built after it.
Generated-by: Claude Code 2.1.310 (Claude Opus 5.5)
(cherry picked from commit dab40d3)
Collaborator
twalthr
approved these changes
Oct 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
Backport of #29607 to release-2.4. Without it, 2.4.0 writes VARIANT objects in UTF-16 order instead of the spec's UTF-8 byte order, and
getField("😀")misses the field in a spec-ordered object of 32 or more fields, such as one another engine wrote. 2.4.0 is not released yet, so taking this now avoids changing the field order in a 2.4.x patch release.Brief change log
Verifying this change
BinaryVariantTest,BinaryVariantInternalBuilderTest,JsonVariantFormatterTest,FromVariantConverterTest,ToVariantConverterTest,VariantCastUtilsTestandXmlToVariantParserTestpass on this branch, 428 tests.Does this pull request potentially affect one of the following parts:
@Public(Evolving): yes, behavior only.BinaryVariantBuildertreats keys with the same UTF-8 bytes as duplicates. No signature changes.getFieldand object building.Documentation
data-types.mdnow name the key order.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code 2.1.310 (Claude Opus 5.5)