Repository navigation
feat(osm): Store each feature's country in the OSM index - #934
Merged
Merged
Conversation
OsmFeatureIndex now keeps the country of each feature's centroid, one byte per feature: 0 for none, else 1 + the country's position in a table of two-letter codes stored in the same file. Storing the table keeps an index readable after the boundary data changes. The lookup runs while the index is written, at about 0.16 µs per feature, or about 30 seconds for the planet's 163 million features. The feature storage format becomes "featstg1". An index in the older format counts as missing, so it is built again, and an OSM planet whose index is outdated gets downloaded again if it was deleted. Two steps use the stored country instead of decoding geometry: collect_wikidata_ids for its per-country brand totals, and the deletion candidates, which now decode the geometry of a branded OSM feature only if it turns out to be a candidate. country-boundaries 1.2.0 knows exactly 255 two-letter codes, which just fits; a test fails if a newer version has more. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ttfb2LhDRExioXkdHbxLno
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Sascha · project thread
Follow-up to #928, part of #924.
OsmFeatureIndexnow keeps the country of each feature's centroid, so later steps needn't decode geometry to find it.Format. One byte per feature: 0 for none, else 1 + the country's position in a table of two-letter codes. The table is stored in the same file (2 bytes per code), not taken from the running binary, so an index stays readable after the boundary data changes; a code the binary doesn't know reads as no country. The feature storage signature becomes
featstg1.OsmFeatureIndex::existsnow checks it, so an index in the older format counts as missing and gets built again. If the planet file was deleted,fetch_inputsdownloads it again for that.country-boundaries1.2.0 knows exactly 255 two-letter codes, which just fits into a byte with 0 for none.countries::tests::test_allfails if a newer version has more; we'd then need two bytes per feature.Cost. The lookup runs in pass 1 of
OsmFeatureIndex::create, from the centroid cell that's already the sort key. In a release build, a lookup takes about 0.16 µs (2 million points over Europe in 0.32 s), so about 30 seconds for the 163 million features of the planet. The file grows by one byte per feature, about 160 MB.Users.
collect_wikidata_ids(feat(wikidata): Count OSM features and names per brand #928) takes the country for its per-country brand totals from the index, and no longer decodes geometry.deletion_reasonnow takes the country as an argument. A branded unmatched OSM feature gets its geometry decoded only once it turns out to be a candidate; before, every one was decoded to find its country.conflated.parquet(for edits), not from the index.Testing. New tests: countries stored and read back (Zürich, the Atlantic, no centroid, Taipei), an index in the older format counting as missing, and the code table (sorted, fits a byte, covers every country on a 1° grid). Tests that stood in an empty file for an index now write one with the current signature.
cargo test --lib,cargo clippy --all-targets -- -D warningsandcargo fmtpass. CI runs the integration tests, since they needtippecanoe.Generated by Claude Code