Skip to content

feat(csharp): define geospatial Arrow types - #699

Open
cathleeny wants to merge 2 commits into
mainfrom
feat/csharp-geospatial-arrow-types
Open

cathleeny wants to merge 2 commits into
mainfrom
feat/csharp-geospatial-arrow-types

Conversation

@cathleeny

@cathleeny cathleeny commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

What's Changed

  • Recognize Reyden's internal GEOMETRY and GEOGRAPHY transport as struct<srid:int32,wkb:binary>.
  • Define the public GeoArrow WKB field contract using Arrow Binary, the geoarrow.wkb extension name, and CRS/edge metadata.
  • Parse default, fixed, and ANY SRIDs and validate malformed or conflicting transport tags.
  • Add focused SQL type and transport-schema tests.

The Databricks tagged struct is an internal server transport contract. PR #700 converts it before exposing results to ADBC consumers.

Stack: 1 of 3.

Testing

  • Stack-wide C# unit suite: 981 passed.
  • CI is running on the rewritten stack.

@cathleeny
cathleeny added this pull request to stack #702 October 8, 2026 22:46

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium · 1 Low

Looks solid overall — clean canonical-type definition with good parse/validation test coverage. One medium concern: the GEOMETRY/GEOGRAPHY → StructType mapping is wired into the live SEA schema path (StatementExecutionStatement.cs:712) with no serializing stream rewriting geospatial batches to that struct, so it relies on Reyden already emitting struct<srid,wkb> natively — worth confirming given the PR defers client representation to a later stack. One low note on TryGetTag throwing despite the Try* name. Nit (summary-only): the SRID regex -?\\d+ accepts a literal GEOMETRY(-1), which maps to the same tag as the ANY sentinel (srid = -1) — benign today since real SRIDs are non-negative, but the two become indistinguishable on round-trip.

Comment thread csharp/src/ArrowTypeParser.cs Outdated
Comment thread csharp/src/GeospatialArrowType.cs Outdated

@lidavidm lidavidm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't seem to use GeoArrow types? Shouldn't the driver use the standard types instead of defining its own keys?

@cathleeny
cathleeny marked this pull request as draft October 8, 2026 23:01
@cathleeny
cathleeny force-pushed the feat/csharp-geospatial-arrow-types branch 2 times, most recently from 775826c to 5cb1652 Compare October 14, 2026 18:27
@cathleeny
cathleeny marked this pull request as ready for review October 14, 2026 19:01

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 2 Low

Looks good — a clean, well-tested new file defining the geospatial Arrow type contract; 2 low-severity notes only. The parsing, tag validation, SRID defaults (GEOMETRY=0 / GEOGRAPHY=4326 / ANY=-1), and GeoArrow metadata generation are internally consistent and match the Databricks geometry(0) base-name convention noted in the learning log. The notes concern the Try-prefixed method that throws and a missing test for the documented untagged-struct-returns-false path.

Comment thread csharp/src/GeospatialArrowType.cs Outdated
Comment thread csharp/src/GeospatialArrowType.cs
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
@cathleeny
cathleeny force-pushed the feat/csharp-geospatial-arrow-types branch from 5cb1652 to 992b767 Compare October 14, 2026 22:47

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

Comment thread csharp/src/GeospatialArrowType.cs Outdated
/// Fixed positive SRIDs are advertised as opaque database SRIDs. Databricks
/// supports CRS authorities other than EPSG, and the manifest exposes only the
/// integer SRID. ANY has no column CRS because its SRID varies by row; those
/// rows carry EWKB instead.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For GEOMETRY(ANY) the comment says rows carry EWKB, but the field is still labeled geoarrow.wkb. My understanding is the GeoArrow spec expects ISO WKB here, so a strict reader might reject the EWKB SRID flag or silently drop the SRID. Can you confirm against the spec what readers (e.g. geoarrow-c, GDAL, geopandas) do with EWKB in a geoarrow.wkb column? If it isn't supported, options include ISO WKB plus a separate per-row SRID, or documenting ANY as a known deviation.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GeoArrow recommends ISO WKB where possible, but allows consumers to accept EWKB. For a fixed SRID geometry, we use ISO WKB. But for GEOMETRY(ANY), ISO WKB cannot preserve each row’s different SRID, so EWKB is the only lossless single column representation.

geoarrow-c and geopandas accept EWKB, but GDAL rejects it (this is only for GEOGRAPHY(ANY))

Comment thread csharp/src/GeospatialArrowType.cs Outdated
Comment thread csharp/test/Unit/GeospatialArrowTypeTests.cs Outdated
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants