feat(tiles): show the most important places at coarse zooms

docs/TECHNICAL_DESIGN.md

@@ -398,8 +398,9 @@ fetch away.
   `suggest_edits` below) exactly once, writing all four layers both of
   `conflated.pmtiles`' passes need:
   - the coarse **overview** `matched`/`unmatched` layers — one
-    deliberately minimal feature per row: `spider`, `matched`, and
-    `osm:type`/`osm:id` when matched, but **no tags and no `fid`**. At
+    deliberately minimal feature per row: `spider`, `matched`,
+    `importance` (rounded to two decimals), and `osm:type`/`osm:id`
+    when matched, but **no tags and no `fid`**. At
     z0 a single tile holds the whole planet, and every per-feature byte
     counts: tags push the tile so far past tippecanoe’s size limit that
     `--drop-densest-as-needed` guts it to a few hundred features
@@ -477,7 +478,16 @@ fetch away.
   — runs tippecanoe over those layers to build `diffed-places.pmtiles`
   for visual review. **~2m48s** at full-planet scale (cpx42 run). The
   same function now builds both this and `conflated.pmtiles` above,
-  parameterized by output filename.
+  parameterized by output filename. For this archive and the
+  `conflated.pmtiles` overview, a tippecanoe feature filter (`-j`) on
+  the rounded `importance` decides what coarse zooms show: the top 1%
+  at z0, 4% at z1, 16% at z2, 64% at z3 and everything from z4 on
+  (four times as many per zoom, since each tile splits into four),
+  with no random dot-dropping (`-r 1`) and `--drop-densest-as-needed`
+  only as the safety net for oversized tiles. A per-feature
+  `"tippecanoe": {"minzoom": N}` would be the obvious way, but with
+  tippecanoe v2.49 it made tippecanoe drop nearly all features at
+  every zoom.
 - **`upload_conflated` / `upload_conflated_tiles` / `upload_tiles`**
   ([`src/pipeline/upload.rs`](../src/pipeline/upload.rs)) — push the
   data output and both sets of tiles to the public (`PUBLIC_S3_*`)
@@ -652,7 +662,7 @@ until the full pipeline was proven out:
   conflating municipal tree datasets against OSM by spatial distance
   and species looks particularly tractable). See
   [#708](https://github.com/brawer/osmdiffs/issues/708).
-- There’s no per-row match-confidence or cartographic-importance
-  signal yet, so both PMTiles archives rely entirely on tippecanoe’s
-  own density-based dropping at low zoom — see
-  [#713](https://github.com/brawer/osmdiffs/issues/713).
+- There’s no per-row match-confidence signal yet. Low-zoom tiles
+  select features by `importance` (see
+  [#713](https://github.com/brawer/osmdiffs/issues/713)), but not by
+  how sure we are of a match.

docs/outputs/CONFLATED_TILES.md

@@ -48,8 +48,17 @@ against a full planet: full tags → ~370 features visible at z0; `+fid`
 
 Geometry is `osm_geometry` (the matched OpenStreetMap shape — point,
 line, or polygon) for `matched` rows, `atp_geometry` for `unmatched`.
-Properties: `spider`, `matched`, and — `matched` rows only — `osm:type`
-/ `osm:id`. To see a row's tags, click through to the detail layer
+Properties: `spider`, `matched`, `importance`, and — `matched` rows
+only — `osm:type` / `osm:id`.
+
+Coarse zooms show the most important rows, by their
+[`importance`](CONFLATED_PARQUET.md#importance): the top 1% at z0
+(`importance >= 0.99`), 4% at z1 (`>= 0.96`), 16% at z2 (`>= 0.84`),
+64% at z3 (`>= 0.36`), and every row from z4 on. Each zoom shows four
+times as many as the one before, since a tile splits into four, so a
+tile holds about as many features at every zoom. Where a tile would
+still be too large, tippecanoe's `--drop-densest-as-needed` thins it
+out further. To see a row's tags, click through to the detail layer
 (below): `matched` rows join on `osm:type`+`osm:id`, `unmatched` rows
 on nearest `part: "atp"` feature of the same `spider`.
 
@@ -95,6 +104,8 @@ for the full investigation.
   `atp.fetched.spider` in `conflated.parquet`). On every feature.
 - `matched` (`true` | `false`) — overview features only; the detail
   layer conveys the same thing through its layer name and `part`.
+- `importance` — the row's `importance` from `conflated.parquet`,
+  rounded to two decimals. Overview features only.
 - `osm:type`, `osm:id` — the matched OpenStreetMap object. On the
   `matched` overview feature (where they double as the join key to the
   detail layer) and the z13–16 `osm`/`link` detail features.

src/pipeline/conflated_tiles.rs

@@ -44,7 +44,8 @@
 //! [#775](https://github.com/brawer/osmdiffs/issues/775) for
 //! the full design discussion.
 
-use crate::utils::parquet::{get_binary, get_string, get_struct, get_tags, get_u64};
+use super::tiles::round_importance;
+use crate::utils::parquet::{get_binary, get_f32, get_string, get_struct, get_tags, get_u64};
 use crate::{TileLayer, make_progress_bar};
 use anyhow::{Context, Result};
 use arrow::array::{Array, RecordBatch};
@@ -153,6 +154,8 @@ pub struct ConflatedLayers {
 /// skipped entirely, the same defensive handling `edits.rs` applies.
 struct ConflatedTileRow {
     spider: String,
+    /// See `pipeline::importance`.
+    importance: f32,
     atp_tags: Vec<(String, String)>,
     atp_geometry_wkb: Vec<u8>,
     osm: Option<OsmSide>,
@@ -179,12 +182,10 @@ impl ConflatedTileRow {
     /// the z13+ detail layer ([`Self::to_detail_geojson_lines`] /
     /// [`Self::to_unmatched_detail_geojson_line`]).
     ///
-    /// TODO(#713): once conflated.parquet carries a per-row rank/
-    /// importance signal, this is the natural place to also emit it
-    /// as a feature property (or otherwise feed it to tippecanoe),
-    /// so --drop-densest-as-needed can prefer important places over
-    /// low-priority ones at low zoom instead of today's
-    /// density-only heuristic.
+    /// Also carries the row's `importance`, rounded to two decimals so
+    /// that it costs the tile at most 101 distinct values, which the
+    /// overview's tippecanoe filter uses to show the most important
+    /// places at low zooms (see `tiles::importance_filter`).
     fn to_overview_geojson_line(&self) -> Result<String> {
         let geometry_wkb: &[u8] = match &self.osm {
             Some(osm) => &osm.osm_geometry_wkb,
@@ -199,6 +200,10 @@ impl ConflatedTileRow {
             properties.insert("osm:type".to_string(), Value::String(osm.osm_type.clone()));
             properties.insert("osm:id".to_string(), Value::from(osm.osm_id));
         }
+        properties.insert(
+            "importance".to_string(),
+            Value::from(round_importance(self.importance)),
+        );
         geometry_to_geojson_line(&geometry, properties)
     }
 
@@ -561,6 +566,7 @@ fn extract_conflated_tile_row(batch: &RecordBatch, row: usize) -> Result<Option<
 
     Ok(Some(ConflatedTileRow {
         spider: get_string(fetched, "spider", row)?,
+        importance: get_f32(batch, "importance", row)?,
         atp_tags: get_tags(atp, "tags", row)?,
         atp_geometry_wkb: get_binary(batch, "atp_geometry", row)?,
         osm,
@@ -629,6 +635,7 @@ mod tests {
     fn overview_unmatched_row_is_the_atp_point_with_no_tags() {
         let row = ConflatedTileRow {
             spider: "acme".to_string(),
+            importance: 0.5,
             atp_tags: vec![("shop".to_string(), "bakery".to_string())],
             atp_geometry_wkb: encode_point(8.5, 47.2),
             osm: None,
@@ -654,6 +661,7 @@ mod tests {
     fn overview_matched_row_is_the_osm_shape_with_only_the_osm_id() {
         let row = ConflatedTileRow {
             spider: "acme".to_string(),
+            importance: 0.5,
             atp_tags: vec![("shop".to_string(), "bakery".to_string())],
             atp_geometry_wkb: encode_point(8.5, 47.2),
             osm: Some(OsmSide {
@@ -683,10 +691,31 @@ mod tests {
         assert!(props.get("atp:shop").is_none());
     }
 
+    #[test]
+    fn overview_carries_rounded_importance() {
+        let row = ConflatedTileRow {
+            spider: "acme".to_string(),
+            importance: 0.98765,
+            atp_tags: vec![],
+            atp_geometry_wkb: encode_point(8.5, 47.2),
+            osm: None,
+        };
+        let value = parse(&row.to_overview_geojson_line().expect("overview line"));
+        assert_eq!(value["properties"]["importance"], 0.99);
+
+        // The detail features don't: at z13+ every row is shown.
+        let detail = parse(
+            &row.to_unmatched_detail_geojson_line(7)
+                .expect("detail line"),
+        );
+        assert!(detail["properties"].get("importance").is_none());
+    }
+
     #[test]
     fn overview_geometry_coordinates_are_rounded_to_seven_decimal_places() {
         let row = ConflatedTileRow {
             spider: "acme".to_string(),
+            importance: 0.5,
             atp_tags: vec![],
             atp_geometry_wkb: encode_point(8.123_456_789, 47.987_654_321),
             osm: None,
@@ -713,6 +742,7 @@ mod tests {
     fn matched_detail_features_carry_their_own_tags_and_a_shared_fid() {
         let row = ConflatedTileRow {
             spider: "acme".to_string(),
+            importance: 0.5,
             atp_tags: vec![("name".to_string(), "Acme ATP".to_string())],
             atp_geometry_wkb: encode_point(8.5, 47.2),
             osm: None, // unused; osm side passed explicitly below
@@ -763,6 +793,7 @@ mod tests {
     fn matched_detail_omits_the_connector_when_offset_is_below_threshold() {
         let row = ConflatedTileRow {
             spider: "acme".to_string(),
+            importance: 0.5,
             atp_tags: vec![],
             atp_geometry_wkb: encode_point(8.5, 47.2),
             osm: None,
@@ -790,6 +821,7 @@ mod tests {
     fn unmatched_detail_is_a_single_full_tag_atp_point() {
         let row = ConflatedTileRow {
             spider: "acme".to_string(),
+            importance: 0.5,
             atp_tags: vec![
                 ("name".to_string(), "Acme Foods".to_string()),
                 ("addr:city".to_string(), "Zürich".to_string()),

src/pipeline/edits.rs

@@ -11,9 +11,12 @@
 //! `crate::edit_suggesters::EditSuggester` what to change for each
 //! already-matched pair.
 
+use super::tiles::round_importance;
 use crate::edit_suggesters::create_edit_suggester;
 use crate::pipeline::EXTERNAL_SORT_CHUNK_BYTES;
-use crate::utils::parquet::{get_binary, get_string, get_struct, get_tags, get_u32, get_u64};
+use crate::utils::parquet::{
+    get_binary, get_f32, get_string, get_struct, get_tags, get_u32, get_u64,
+};
 use crate::{TileLayer, make_progress_bar};
 use anyhow::{Context, Result};
 use arrow::array::{Array, RecordBatch};
@@ -55,6 +58,8 @@ pub struct SuggestedEdit {
     /// down; revisit once tile layers are known to handle non-point
     /// geometry the way this pipeline needs.
     centroid: (f64, f64),
+    /// The conflated row's `importance`; see `pipeline::importance`.
+    importance: f32,
 }
 
 impl PartialEq for SuggestedEdit {
@@ -90,6 +95,12 @@ impl SuggestedEdit {
             String::from("@osm_version"),
             serde_json::Value::from(self.osm_version),
         );
+        // Lets tippecanoe show the most important edits at coarse
+        // zooms; see `tiles::importance_filter`.
+        properties.insert(
+            String::from("@importance"),
+            serde_json::Value::from(round_importance(self.importance)),
+        );
 
         geojson::Feature {
             bbox: None,
@@ -182,6 +193,7 @@ struct ConflatedRow {
     osm_tags: Vec<(String, String)>,
     osm_version: u32,
     osm_geometry_wkb: Vec<u8>,
+    importance: f32,
 }
 
 fn produce_edits(
@@ -211,6 +223,7 @@ fn produce_edits(
                 osm_version: row.osm_version,
                 tags,
                 centroid: decode_centroid(&row.osm_geometry_wkb)?,
+                importance: row.importance,
             };
             num_edits.fetch_add(1, Ordering::Relaxed);
             // TODO: Dispatch to one of {shops, infrastructure, trees}
@@ -283,6 +296,7 @@ fn extract_conflated_row(batch: &RecordBatch, row: usize) -> Result<Option<Confl
         // geometry columns to live at the schema root (see
         // `pipeline::conflate::writer::GEO_METADATA_KEY`'s doc comment).
         osm_geometry_wkb: get_binary(batch, "osm_geometry", row)?,
+        importance: get_f32(batch, "importance", row)?,
     }))
 }
 
@@ -328,3 +342,23 @@ fn write_edits(edits: Receiver<SuggestedEdit>, path: &Path, workdir: &Path) -> R
 
     Ok(num_edits.load(Ordering::SeqCst))
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn to_geojson_carries_rounded_importance() {
+        let edit = SuggestedEdit {
+            osm_id: 42,
+            osm_type: "node".to_string(),
+            osm_version: 3,
+            tags: vec![("opening_hours".to_string(), "Mo-Fr 08:00-18:00".to_string())],
+            centroid: (8.5, 47.2),
+            importance: 0.4567,
+        };
+        let value = serde_json::to_value(edit.to_geojson()).expect("json");
+        assert_eq!(value["properties"]["@importance"], 0.46);
+        assert_eq!(value["properties"]["@osm_version"], 3);
+    }
+}

src/pipeline/mod.rs

@@ -311,6 +311,7 @@ fn run_pipeline_steps(
                 min: 0,
                 max: conflated_tiles::DETAIL_MIN_ZOOM - 1,
             },
+            Some("importance"),
         )
     })?;
     let conflated_detail = run_step("render_conflated_detail", || {
@@ -326,6 +327,7 @@ fn run_pipeline_steps(
                 min: conflated_tiles::DETAIL_MIN_ZOOM,
                 max: conflated_tiles::DETAIL_MAX_ZOOM,
             },
+            None,
         )
     })?;
     let conflated_tiles_out = run_step("join_conflated_tiles", || {
@@ -349,6 +351,7 @@ fn run_pipeline_steps(
             workdir,
             "diffed-places.pmtiles",
             tiles::ZoomRange::Auto,
+            Some("@importance"),
         )
     })?;
     published.push(run_step("upload_tiles", || {