You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Support Iceberg tables on HDFS in the native scan #5894
Comet's native Iceberg scan cannot read a table whose data lives on HDFS. hdfs:// is not in the scheme allowlist that mirrors storage_factory_for, so any Iceberg table on HDFS falls back to the JVM reader and gets no native acceleration.
This is a gap relative to the plain-Parquet native scan, which does read HDFS (through libhdfs/JNI, via fs.comet.libhdfs.schemes). On-premise Iceberg deployments are commonly HDFS-backed, so the Iceberg acceleration Comet offers is unavailable to them today.
Describe the potential solution
Route hdfs:// to iceberg-rust's hdfs-native OpenDAL backend, a pure-Rust HDFS RPC client that needs no JNI or libhdfs. That backend is being added upstream in apache/iceberg-rust#3111; this issue tracks the Comet-side work once it lands.
Comet-side work:
Add an hdfs arm to storage_factory_for and admit the scheme in the JVM gates (CometScanRule.icebergReadableSchemes, and CometIcebergNativeWrite.SupportedStorageSchemes for the write path).
Forward the hdfs./hadoop. property prefixes to the native FileIO.
Resolve the NameNode endpoints for HA clusters. opendal's HdfsNativeBuilder never dials the authority written in the path: it builds one client against a synthetic authority and synthesizes the HA config from the comma-separated hdfs.name-node value. iceberg-rust falls back to the path authority only when that property is absent, which is correct just for a real host:port. An HA location reads hdfs://<nameservice>/..., and a nameservice is not a routable host, so the endpoints have to be derived from dfs.ha.namenodes.<ns> / dfs.namenode.rpc-address.<ns>.<nn> in the session Hadoop configuration.
Decline a scan whose data/delete files span more than one HDFS authority: one hdfs.name-node per scan overrides the authority of every path the FileIO opens, so a second nameservice would be read from the first one's NameNode at the same relative path.
Additional context
Note that this is a second, independent HDFS client in the same process: the plain-Parquet path reaches HDFS through libhdfs/JNI while an Iceberg table would be opened over pure-Rust RPC. The two read the same $HADOOP_CONF_DIR XML but hold separate connections and separate Kerberos state.
Activity