Skip to content

[Coral-Schema] Reconstruct multi-branch unions from Iceberg union-structs#610

Open
yyy1000 wants to merge 1 commit into
linkedin:masterfrom
yyy1000:iceberg-first-avro/pr4-multibranch-union
Open

[Coral-Schema] Reconstruct multi-branch unions from Iceberg union-structs#610
yyy1000 wants to merge 1 commit into
linkedin:masterfrom
yyy1000:iceberg-first-avro/pr4-multibranch-union

Conversation

@yyy1000

@yyy1000 yyy1000 commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

What

Adds a multi-branch union reconstruction path to MergeCoralSchemaWithAvro (the Iceberg-first Avro merge engine introduced in #600/#604).

Why

Iceberg has no union type. A Hive uniontype<A,B,C> that has been persisted into an Iceberg table surfaces as the struct {tag:INT, field0:A, field1:B, field2:C} — the Trino union representation (trinodb/trino#3483), the same encoding HiveToCoralTypeConverter.convertUnion produces — while the partner Avro still describes the column as a real union [null,A,B,C].

Before this change, the engine could not match the union partner, so it synthesized a record: multi-branch unions collapsed, and union-typed field defaults failed outright. This diverged from the legacy Hive Avro path (MergeHiveSchemaWithAvro.union).

How

  • unionPartnerOrNull detects the case, requiring all of: (1) the Coral struct is union-shaped (tag:INT then field0..fieldN-1 in order), (2) the partner is an Avro union, and (3) the partner's non-null branch count equals the member count. A memberCount >= 2 guard makes detection collision-free — a genuine nullable struct yields [null, record] (one non-null branch), which can never equal a member count of two or more. Single-member union-structs fall back to the struct path.
  • mergeUnionStruct merges each fieldN member against the partner union branch by ordinal, strips per-member null wrappers (the union's own NULL branch carries nullability), and emits the NULL branch first when the partner union has one — matching MergeHiveSchemaWithAvro.union. Emitting a real union also lets the partner's field default apply exactly as on the Hive path.

Testing

Six new tests (full coral-schema suite green, spotless clean): multi-branch reconstruction, the no-null-branch case, arrays of unions, the union-default regression, and a negative case proving a genuine record partner is not mistaken for a union.

…ucts

Iceberg has no union type, so a Hive uniontype<A,B,C> persisted into an
Iceberg table surfaces as the struct {tag:INT, field0:A, field1:B, field2:C}
(the Trino union representation, trinodb/trino#3483 -- the same encoding
HiveToCoralTypeConverter.convertUnion produces), while the partner Avro still
describes the column as a real union [null,A,B,C]. MergeCoralSchemaWithAvro
previously could not match the union partner, so it synthesized a record:
multi-branch unions collapsed and union-typed field defaults failed outright.

Add a union-struct reconstruction path:

- unionPartnerOrNull detects the case when all of (1) the Coral struct is
  union-shaped (tag:INT then field0..fieldN-1), (2) the partner is an Avro
  union, and (3) the partner's non-null branch count equals the member count.
  A memberCount >= 2 guard makes detection collision-free: a genuine nullable
  struct yields [null, record] (one non-null branch), which can never equal a
  member count of two or more. Single-member union-structs fall back to the
  struct path.

- mergeUnionStruct merges each fieldN member against the partner union branch
  by ordinal, strips per-member null wrappers (the union's own NULL branch
  carries nullability), and emits the NULL branch first when the partner union
  has one -- matching MergeHiveSchemaWithAvro.union so the Iceberg path stays
  faithful to the legacy Hive Avro baseline. Emitting a real union also lets
  the partner's field default apply exactly as on the Hive path.

Adds six tests covering multi-branch reconstruction, the no-null-branch case,
arrays of unions, the union-default regression, and a negative case proving a
genuine record partner is not mistaken for a union.
* struct (the pre-existing behavior).
*/
@Nullable
private Schema unionPartnerOrNull(StructType structType, @Nullable Schema partner) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: return a boolean and rename to isMultiBranchUnionStruct

This method only ever returns partner unchanged or null — it never constructs or transforms the schema it hands back. That makes the Schema-or-null return effectively a boolean in disguise, and the caller has to re-derive the meaning from the null check:

Schema unionPartner = unionPartnerOrNull(structType, partner);
if (unionPartner != null) {
  return mergeUnionStruct(structType, unionPartner);
}

I'd return a boolean named for what's being asserted, and let the caller use partner directly:

if (isMultiBranchUnionStruct(structType, partner)) {
  return mergeUnionStruct(structType, partner);
}
return mergeStruct(structType, partner);
private boolean isMultiBranchUnionStruct(StructType structType, @Nullable Schema partner) {
  if (partner == null || partner.getType() != Schema.Type.UNION || !isUnionStruct(structType)) {
    return false;
  }
  int memberCount = structType.getFields().size() - 1; // exclude the leading "tag" field
  int nonNullBranchCount = SchemaUtilities.discardNullFromUnionIfExist(partner).getTypes().size();
  return memberCount >= 2 && memberCount == nonNullBranchCount;
}

A predicate name documents the decision at the call site, drops the @Nullable return, and removes the unionPartner local that's just an alias for an input. (The @link references in isUnionStruct and mergeUnionStruct would need updating to the new name.)

}

@Test
public void shouldReconstructUnionWithoutNullBranch() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: add coverage for non-primitive union members

Both union tests here use primitive members (int/string/boolean). The reconstruction path does more than copy primitives across — each member is recursively merged against its partner branch and then unwrapped via extractIfOption, so I'd add cases where a fieldN member is itself:

  • a record (uniontype<int, struct<x:int>>) — verify the struct member becomes a record branch with its fields merged from the partner, and is not double-wrapped in its own [null, record] option (the union's own NULL branch already carries nullability)
  • an array (uniontype<int, array<string>>) — verify the member stays an array branch rather than being collapsed
  • a member whose partner branch is an enum (uniontype<string, int> with partner [null, enum, int]) — verify the STRING member promotes to the enum, confirming branches go through the normal promotion path rather than emitting a bare string

These are the cases most likely to regress if the per-branch merge or the extractIfOption unwrap is ever changed, and they aren't exercised today.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants