Repository navigation
Description
Apache Iceberg version
main (development)
Query engine
None
Please describe the bug 🐞
After #17578, Parquet types are validated against the Iceberg type. This is great!
In an attempt to reduce the number of readers that had to consider CRSes or PROJJSON or axis ordering, arrow-rs and Arrow C++ (i.e., the arrow interfaces to Parquet writing) make an attempt to write a defaulted CRS. Unfortunately, this means those interfaces can't currently write Parquet files with "EPSG:4326" as the Geometry or Geography CRS.
Notably, iceberg-rust is using this interface. This means that when it writes a table in "EPSG:4326", the Parquet files have Geometry or Geography fields with a defaulted CRS, which iceberg-java reads back as "ogc:crs84", considers un equal, and fails validation. EPSG:4326 is probably the most common thing somebody might type themselves here.
Would it be possible to have the Equals implementations be updated to consider EPSG:4326 and OGC:CRS84 equal to each other?
iceberg/api/src/main/java/org/apache/iceberg/types/Types.java
Lines 685 to 699 in 3239fc6
| /** | |
| * Two geography types are equal when their edge algorithms are equal and their CRS match | |
| * case-insensitively, so {@code OGC:CRS84} and {@code ogc:crs84} are equal. | |
| */ | |
| @Override | |
| public boolean equals(Object o) { | |
| if (this == o) { | |
| return true; | |
| } else if (!(o instanceof GeographyType)) { | |
| return false; | |
| } | |
| GeographyType that = (GeographyType) o; | |
| return crs.equalsIgnoreCase(that.crs) && Objects.equals(algorithm, that.algorithm); | |
| } |
iceberg/api/src/main/java/org/apache/iceberg/types/Types.java
Lines 608 to 622 in 3239fc6
| /** | |
| * Two geometry types are equal when their CRS match case-insensitively, so {@code OGC:CRS84} | |
| * and {@code ogc:crs84} are equal. | |
| */ | |
| @Override | |
| public boolean equals(Object o) { | |
| if (this == o) { | |
| return true; | |
| } else if (!(o instanceof GeometryType)) { | |
| return false; | |
| } | |
| GeometryType that = (GeometryType) o; | |
| return crs.equalsIgnoreCase(that.crs); | |
| } |
..or could this be considered "type promotion" to allow the two concepts to coexist more readily?
iceberg/api/src/main/java/org/apache/iceberg/types/TypeUtil.java
Lines 471 to 497 in 3239fc6
| public static boolean isPromotionAllowed(Type from, Type.PrimitiveType to) { | |
| // Warning! Before changing this function, make sure that the type change doesn't introduce | |
| // compatibility problems in partitioning. | |
| if (from.equals(to)) { | |
| return true; | |
| } | |
| switch (from.typeId()) { | |
| case INTEGER: | |
| return to.typeId() == Type.TypeID.LONG; | |
| case FLOAT: | |
| return to.typeId() == Type.TypeID.DOUBLE; | |
| case DECIMAL: | |
| Types.DecimalType fromDecimal = (Types.DecimalType) from; | |
| if (to.typeId() != Type.TypeID.DECIMAL) { | |
| return false; | |
| } | |
| Types.DecimalType toDecimal = (Types.DecimalType) to; | |
| return fromDecimal.scale() == toDecimal.scale() | |
| && fromDecimal.precision() <= toDecimal.precision(); | |
| } | |
| return false; | |
| } |
cc @dannycjones who spotted this!
Willingness to contribute
- I can contribute a fix for this bug independently
- I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- I cannot contribute a fix for this bug at this time
Activity
RussellSpitzer commentedon Oct 28, 2026
I don't know enough to understand the "EPSG:4326" ~ "ogc:crs84" - relationship. If these are actually equivelent strings then I don't see an issue with changing the equality method?
dannycjones commentedon Oct 28, 2026
I'm hoping to share something to the dev list soon because its an ecosystem problem, but I'm aligned that we need to recommend Iceberg implementations consider CRS equal here for this latlng case ("EPSG:4326" ~ "ogc:crs84") which the Parquet and Iceberg spec kind of hint are considered equivalent. Additionally, we need to ensure each language implementation is comparing case insensitively.
Thanks for writing this up @paleolimbot!
paleolimbot commentedon Oct 28, 2026
Briefly, ogc:crs84 is officially defined as longitude, latitude. epsg:4326 is officially defined as latitude, longitude; however, for convenience, GIS practitioners ignored the order for several decades and stored longitude, latitude anyway because it looked better on a map. Like other standards (e.g., GeoArrow, GeoParquet, GeoPackage), we codified this in Parquet. There are a few pairs where this can happen but epsg:4326 and ogc:crs84 is the problematic one here because it intersects with the "default" CRS in Parquet.
For what it's worth, this is mostly my fault (I think using OGC:CRS84 for the default CRS originates from one of my early comments in GeoParquet, and, while academically correct, has resulted in much confusion).