[Coral-Schema] Reconstruct multi-branch unions from Iceberg union-structs#610
[Coral-Schema] Reconstruct multi-branch unions from Iceberg union-structs#610yyy1000 wants to merge 1 commit into
Conversation
…ucts
Iceberg has no union type, so a Hive uniontype<A,B,C> persisted into an
Iceberg table surfaces as the struct {tag:INT, field0:A, field1:B, field2:C}
(the Trino union representation, trinodb/trino#3483 -- the same encoding
HiveToCoralTypeConverter.convertUnion produces), while the partner Avro still
describes the column as a real union [null,A,B,C]. MergeCoralSchemaWithAvro
previously could not match the union partner, so it synthesized a record:
multi-branch unions collapsed and union-typed field defaults failed outright.
Add a union-struct reconstruction path:
- unionPartnerOrNull detects the case when all of (1) the Coral struct is
union-shaped (tag:INT then field0..fieldN-1), (2) the partner is an Avro
union, and (3) the partner's non-null branch count equals the member count.
A memberCount >= 2 guard makes detection collision-free: a genuine nullable
struct yields [null, record] (one non-null branch), which can never equal a
member count of two or more. Single-member union-structs fall back to the
struct path.
- mergeUnionStruct merges each fieldN member against the partner union branch
by ordinal, strips per-member null wrappers (the union's own NULL branch
carries nullability), and emits the NULL branch first when the partner union
has one -- matching MergeHiveSchemaWithAvro.union so the Iceberg path stays
faithful to the legacy Hive Avro baseline. Emitting a real union also lets
the partner's field default apply exactly as on the Hive path.
Adds six tests covering multi-branch reconstruction, the no-null-branch case,
arrays of unions, the union-default regression, and a negative case proving a
genuine record partner is not mistaken for a union.
| * struct (the pre-existing behavior). | ||
| */ | ||
| @Nullable | ||
| private Schema unionPartnerOrNull(StructType structType, @Nullable Schema partner) { |
There was a problem hiding this comment.
Suggestion: return a boolean and rename to isMultiBranchUnionStruct
This method only ever returns partner unchanged or null — it never constructs or transforms the schema it hands back. That makes the Schema-or-null return effectively a boolean in disguise, and the caller has to re-derive the meaning from the null check:
Schema unionPartner = unionPartnerOrNull(structType, partner);
if (unionPartner != null) {
return mergeUnionStruct(structType, unionPartner);
}I'd return a boolean named for what's being asserted, and let the caller use partner directly:
if (isMultiBranchUnionStruct(structType, partner)) {
return mergeUnionStruct(structType, partner);
}
return mergeStruct(structType, partner);private boolean isMultiBranchUnionStruct(StructType structType, @Nullable Schema partner) {
if (partner == null || partner.getType() != Schema.Type.UNION || !isUnionStruct(structType)) {
return false;
}
int memberCount = structType.getFields().size() - 1; // exclude the leading "tag" field
int nonNullBranchCount = SchemaUtilities.discardNullFromUnionIfExist(partner).getTypes().size();
return memberCount >= 2 && memberCount == nonNullBranchCount;
}A predicate name documents the decision at the call site, drops the @Nullable return, and removes the unionPartner local that's just an alias for an input. (The @link references in isUnionStruct and mergeUnionStruct would need updating to the new name.)
| } | ||
|
|
||
| @Test | ||
| public void shouldReconstructUnionWithoutNullBranch() { |
There was a problem hiding this comment.
Suggestion: add coverage for non-primitive union members
Both union tests here use primitive members (int/string/boolean). The reconstruction path does more than copy primitives across — each member is recursively merged against its partner branch and then unwrapped via extractIfOption, so I'd add cases where a fieldN member is itself:
- a record (
uniontype<int, struct<x:int>>) — verify the struct member becomes a record branch with its fields merged from the partner, and is not double-wrapped in its own[null, record]option (the union's own NULL branch already carries nullability) - an array (
uniontype<int, array<string>>) — verify the member stays an array branch rather than being collapsed - a member whose partner branch is an enum (
uniontype<string, int>with partner[null, enum, int]) — verify the STRING member promotes to the enum, confirming branches go through the normal promotion path rather than emitting a bare string
These are the cases most likely to regress if the per-branch merge or the extractIfOption unwrap is ever changed, and they aren't exercised today.
What
Adds a multi-branch union reconstruction path to
MergeCoralSchemaWithAvro(the Iceberg-first Avro merge engine introduced in #600/#604).Why
Iceberg has no union type. A Hive
uniontype<A,B,C>that has been persisted into an Iceberg table surfaces as the struct{tag:INT, field0:A, field1:B, field2:C}— the Trino union representation (trinodb/trino#3483), the same encodingHiveToCoralTypeConverter.convertUnionproduces — while the partner Avro still describes the column as a real union[null,A,B,C].Before this change, the engine could not match the union partner, so it synthesized a record: multi-branch unions collapsed, and union-typed field defaults failed outright. This diverged from the legacy Hive Avro path (
MergeHiveSchemaWithAvro.union).How
unionPartnerOrNulldetects the case, requiring all of: (1) the Coral struct is union-shaped (tag:INTthenfield0..fieldN-1in order), (2) the partner is an Avro union, and (3) the partner's non-null branch count equals the member count. AmemberCount >= 2guard makes detection collision-free — a genuine nullable struct yields[null, record](one non-null branch), which can never equal a member count of two or more. Single-member union-structs fall back to the struct path.mergeUnionStructmerges eachfieldNmember against the partner union branch by ordinal, strips per-member null wrappers (the union's own NULL branch carries nullability), and emits the NULL branch first when the partner union has one — matchingMergeHiveSchemaWithAvro.union. Emitting a real union also lets the partner's field default apply exactly as on the Hive path.Testing
Six new tests (full
coral-schemasuite green, spotless clean): multi-branch reconstruction, the no-null-branch case, arrays of unions, the union-default regression, and a negative case proving a genuine record partner is not mistaken for a union.