Kafka cross-partition ordering causes dropped Glue entries on concurrent HMS operations

[jamespfaulkner]

Two bugs cause broken Glue state after Iceberg table rename

This issue covers two independent bugs that both manifest when an Iceberg table is renamed to a
name freed by a prior drop. Each bug produces a different broken Glue entry. They require
separate fixes.

Before

Table	HMS	Glue
db.table	exists (Hive)	exists
db.table_iceberg	exists (Iceberg)	exists

Steps to reproduce

Perform these two HMS operations in order, within the same second:

1. DROP TABLE db.table (freeing the name)
2. RENAME db.table_iceberg → db.table

Expected after

Table	HMS	Glue
db.table	exists (Iceberg, renamed from table_iceberg)	exists
db.table_iceberg	does not exist	does not exist

Actual after

Table	HMS	Glue
db.table	exists (Iceberg)	does not exist
db.table_iceberg	does not exist	still exists (orphaned)

---

Bug 1 — Kafka cross-partition ordering deletes the renamed table

What happens

Each HMS operation produces a Kafka message keyed on database.tableName (ALTER events use the
old table name per KafkaMetaStoreEventListener.withPayload):

HMS operation	Partition key	Glue operation
DROP table	db.table	DeleteTable(table)
RENAME table_iceberg → table	db.table_iceberg	UpdateTable(table)

Two different partition keys → two different Kafka partitions → two different consumer pods,
with no ordering guarantee between them.

HMS (sequential)                   Kafka partition    Consumer pod
────────────────────────────────────────────────────────────────────
T+0  DROP   table               →  partition A    →  Pod 1
T+1  RENAME table_iceberg→table →  partition B    →  Pod 2

                    Pod 2 (partition B)     Pod 1 (partition A)
                    ───────────────────     ───────────────────
T+n                 UpdateTable(table) ✓
T+n+1                                       DeleteTable(table) ✗  ← undoes Pod 2

Pod 1 and Pod 2 each see a valid, in-order stream on their own partition. GlueSync records both
calls as result:success. The DROP was issued before the RENAME in HMS but arrived at Glue one
second after it, deleting db.table.

Root cause

KafkaMessageSender hashes qualifiedTableName to select a partition:

int partition = Math.abs(kafkaMessage.getQualifiedTableName().hashCode() % numberOfPartitions);

The class-level comment acknowledges this: "Note that in order to preserve the order of the
events the topic must have only single partition." Cross-table operations that depend on ordering
are not safe when the topic has more than one partition.

Possible mitigations

None of these are good options. They all involve significant trade-offs with no clean solution:

Option	Trade-off
Single-partition topic	Fully serialized — guaranteed ordering, no parallelism
Partition by database instead of database.table	All tables in a DB serialize; fixes cross-table ordering within a DB
Idempotent consumer with version checks	Complex; requires GlueSync to compare event timestamps before destructive ops

---

Bug 2 — Iceberg rename path never deletes the old Glue entry

What happens

ApiaryGlueSync.onAlterTable routes Iceberg renames through updateOrCreateTable(newTable),
which calls UpdateTable (or CreateTable on EntityNotFoundException). It never deletes the
old Glue entry, so db.table_iceberg is left as an orphan regardless of whether Bug 1 occurs.

Root cause

// Only Hive renames go through doRenameOperation, which creates new + deletes old.
// Iceberg renames skip it and fall through to updateOrCreateTable — no delete of old name.
if (isTableRename(oldTable, newTable) && !isIcebergPredicate.test(oldTable.getParameters())) {
    doRenameOperation(oldTable, newTable);
    return;
}
updateOrCreateTable(newTable);

Fix

Remove the !isIcebergPredicate guard so all renames go through doRenameOperation:

if (isTableRename(oldTable, newTable)) {
    doRenameOperation(oldTable, newTable);
    return;
}
updateOrCreateTable(newTable);