feat: first attempt at item table logic

[hrodmn]

This is the initial attempt at icestac: the STAC extension for Apache Iceberg.

It defers almost all Iceberg operations to pyiceberg but includes some helper functions for generating Iceberg table schemas from STAC items and a function for loading items into an existing table.

At first I had a special IcebergCatalog class that handled some partitioning defaults and other table mechanics, but after some deliberation I decided that it would be better to keep the icestac layer as thin as possible for now and provide some working examples on how to stuff STAC items into Iceberg tables.

The docker-compose.yml file will spin up a local S3 storage backend and an Iceberg REST service that you can interact with via main.py which loads some HLS STAC items into a table in the local Iceberg catalog.

[codecov-commenter]

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️

[gadomski]

Sorry about getting nitpicky on the API design, but I think this is a critical PR ... if we get it right now, it'll be smooth later.

[gadomski]

I find it quite interesting that we get to share the Catalog name between STAC and iceberg. This can be convenient, and it has the potential to introduce "shadowing" to our namespace, such that we get confused about which "catalog" we're referring to.

No action required, just an observation.

[hrodmn]

I have not thought about it this way. In Iceberg the Catalog is like a database in PostgreSQL.

[gadomski]

Which makes sense, if we think of a single pgstac instance as being a Catalog?

[gadomski]

The fact that there's a partition_spec in Iceberg is encouraging to me. Are there other partition specs out there that we could coalesce into a generalized "STAC partition spec"?

[hrodmn]

I think a STAC Iceberg spec should include some kind of spatial structure but I'm not sure if we would partition on a spatial key necessarily.

The pyiceberg docs show this example on how to apply a sort_key which could be used for sorting records spatially (with a Hilbert curve or something):

# Sort on the symbol
sort_order = SortOrder(SortField(source_id=2, transform='identity'))

catalog.create_table(
    identifier="docs_example.bids",
    schema=schema,
    partition_spec=partition_spec,
    sort_order=sort_order,
)

[hrodmn]

I still want to explore ideas around the STAC PartitionSpec, but the rest of the library is looking pretty good to me. While working on the config changes I realized that most of the settings were just being passed onto pyiceberg and should therefore be delegated to pyiceberg's config system, so I ripped it out. @gadomski does that make sense to you?

[gadomski]

@gadomski does that make sense to you?

Yup!

[hrodmn]

After putting this on ice for a few months I came back to finally address @gadomski's comments. The package is set up to create a catalog structure like this:

• create one table per collection
  • collection id must match the table id which places some limits on the characters in a table id (can't have - 😢)
• items can be loaded into a collection table directly from arrow tables or STAC json dicts
• items are partitioned temporally by month for now
• items are not partitioned or ordered spatially (but we should look into that next)

I haven't thought much about schema migration yet but that is something that icestac is theoretically good for since Iceberg has methods for this.

I need to revisit how we are representing the spatial columns and/or spatial metadata to ensure the tables are making best use of the geoparquet format and downstream capabilities. I am a bit lost with the state of geometry in parquet/geoparquet/arrow so if anyone has any ideas on the right way to set that up in 2026 let me know! Maybe that's not necessary to sort out in this PR but definitely before we do any kind of release.

[gadomski]

collection id must match the table id which places some limits on the characters in a table id (can't have - 😢)

This is a bummer, - in collection IDs is quite common. Are there any workarounds we could explore?

Other than that, 👍🏼 to picking reasonable defaults for now, leaving the customization knobs for later.

I am a bit lost with the state of geometry in parquet/geoparquet/arrow so if anyone has any ideas on the right way to set that up in 2026 let me know! Maybe that's not necessary to sort out in this PR but definitely before we do any kind of release.

Agreed, it's messy, and (in my experience) often abstracted away into upstream tooling (but not always).

[hrodmn]

This is a bummer, - in collection IDs is quite common. Are there any workarounds we could explore?

Actually, I am mistaken. Hyphens are allowed in Iceberg table names but some catalog backends don't allow them (i.e. AWS Glue). The only character is that is universally not allowed is . which is a delimiter between the namespace and the table. FWIW duckdb is not crazy about - in table names - you have to quote the table name when you query it.

SELECT id, datetime, collection, geometry
FROM catalog.icestac."icesat2-boreal-v3-1-agb"
LIMIT 10;

The table name should match collection ids exactly so clients can confidently do queries by collection without any guesswork. I suppose we could specify a slugification system for translating any collection id into a valid table name, but then client applications would need to be aware of the translation function. e.g. collection id icesat2-boreal-v3.1-agb -> table icesat2_boreal_v3_1_agb. There are tradeoffs either way (limiting collection ids to valid table names vs allowing table names to drift from the actual collection ids) and I'm not sure what is best.

[gadomski]

My instinct is that it's better to keep the table name === collection id constraint, with some documentation guidance on how to "fix" datasets that don't work on (e.g.) AWS Glue.

[hrodmn]

I am back again with a few updates, this time I think the item handling API might be ready to ship:

1. allow users to pass PartitionSpec and SortOrder objects directly in IcestacCatalog.create_item_table. Rather than creating some abstraction or custom API around the pyiceberg API, I refactored the API so that the user can explicitly define the partitioning and sorting scheme. The default is still a monthly partitioning scheme on the datetime property, but now that is fully configurable.

   I like the idea of defining a more well-defined StacPartitionSpec, but with spatial support in Iceberg being so limited today, it would probably be better to just demonstrate strategies for non-native spatial partitioning with derived fields for now.

   from pyiceberg.catalog import load_catalog
   from pyiceberg.partitioning import PartitionField, PartitionSpec
   from pyiceberg.table.sorting import SortField, SortOrder
   from pyiceberg.transforms import YearTransform
   
   from icestac.catalog import IcestacCatalog
   from icestac.schema import get_schema_from_items
   
   catalog = IcestacCatalog(catalog=load_catalog())
   
   items = ...
   
   iceberg_schema = get_schema_from_items(items)
   
   datetime_id = iceberg_schema.find_field("datetime").field_id
   sort_id = iceberg_schema.find_field("sortme").field_id
   catalog.create_item_table(
       collection_id=collection_id,
       iceberg_schema=iceberg_schema,
       partition_spec=PartitionSpec(
           PartitionField(
               source_id=datetime_id,
               field_id=1000,
               transform=YearTransform(),
               name="datetime_year",
           )
       ),
       sort_order=SortOrder(SortField(source_id=sort_id)),
   )

2. add the evolve_schema boolean arg to IcestacCatalog.load_items (default False) - this makes it possible to load items where the schema may have drifted from the existing table

   I actually hit this in my demo script (main.py) because the schema in the HLS items changed in June 2026 😅

3. since users will need the actual Iceberg schema to get field IDs for custom partitioning/sorting schemes the get_schema_from_items returns an Iceberg schema (instead of an arrow schema).