Skip to content

feat(osm): Store each feature's country in the OSM index - #934

Merged
brawer merged 1 commit into
mainfrom
osm-index-country
Nov 11, 2026
Merged

brawer merged 1 commit into
mainfrom
osm-index-country

Conversation

@brawer

@brawer brawer commented Nov 11, 2026

Copy link
Copy Markdown
Owner

Requested by Sascha · project thread

Follow-up to #928, part of #924. OsmFeatureIndex now keeps the country of each feature's centroid, so later steps needn't decode geometry to find it.

Format. One byte per feature: 0 for none, else 1 + the country's position in a table of two-letter codes. The table is stored in the same file (2 bytes per code), not taken from the running binary, so an index stays readable after the boundary data changes; a code the binary doesn't know reads as no country. The feature storage signature becomes featstg1. OsmFeatureIndex::exists now checks it, so an index in the older format counts as missing and gets built again. If the planet file was deleted, fetch_inputs downloads it again for that.

country-boundaries 1.2.0 knows exactly 255 two-letter codes, which just fits into a byte with 0 for none. countries::tests::test_all fails if a newer version has more; we'd then need two bytes per feature.

Cost. The lookup runs in pass 1 of OsmFeatureIndex::create, from the centroid cell that's already the sort key. In a release build, a lookup takes about 0.16 µs (2 million points over Europe in 0.32 s), so about 30 seconds for the 163 million features of the planet. The file grows by one byte per feature, about 160 MB.

Users.

  • collect_wikidata_ids (feat(wikidata): Count OSM features and names per brand #928) takes the country for its per-country brand totals from the index, and no longer decodes geometry.
  • Deletion candidates: deletion_reason now takes the country as an argument. A branded unmatched OSM feature gets its geometry decoded only once it turns out to be a candidate; before, every one was decoded to find its country.
  • Phone matching doesn't change: it looks up the country at the ATP position (for conflation) or from conflated.parquet (for edits), not from the index.

Testing. New tests: countries stored and read back (Zürich, the Atlantic, no centroid, Taipei), an index in the older format counting as missing, and the code table (sorted, fits a byte, covers every country on a 1° grid). Tests that stood in an empty file for an index now write one with the current signature. cargo test --lib, cargo clippy --all-targets -- -D warnings and cargo fmt pass. CI runs the integration tests, since they need tippecanoe.



Generated by Claude Code

OsmFeatureIndex now keeps the country of each feature's centroid, one
byte per feature: 0 for none, else 1 + the country's position in a
table of two-letter codes stored in the same file. Storing the table
keeps an index readable after the boundary data changes. The lookup
runs while the index is written, at about 0.16 µs per feature, or
about 30 seconds for the planet's 163 million features.

The feature storage format becomes "featstg1". An index in the older
format counts as missing, so it is built again, and an OSM planet
whose index is outdated gets downloaded again if it was deleted.

Two steps use the stored country instead of decoding geometry:
collect_wikidata_ids for its per-country brand totals, and the
deletion candidates, which now decode the geometry of a branded OSM
feature only if it turns out to be a candidate.

country-boundaries 1.2.0 knows exactly 255 two-letter codes, which
just fits; a test fails if a newer version has more.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ttfb2LhDRExioXkdHbxLno
@brawer brawer self-assigned this Nov 11, 2026
@github-actions github-actions Bot added the enhancement New feature or request label Nov 11, 2026
@brawer
brawer merged commit 567e197 into main Nov 11, 2026
6 checks passed
@brawer
brawer deleted the osm-index-country branch November 11, 2026 13:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants