feat(pipeline): write and publish spider-stats.parquet

docs/TECHNICAL_DESIGN.md

@@ -239,6 +239,14 @@ graph TD
     JOIN_CONFLATED_TILES --> CONFLATED_PMTILES[conflated.pmtiles]
     CONFLATED_PMTILES --> UPLOAD_CONFLATED_TILES(upload_conflated_tiles) --> S3C[("public S3")]
 
+    CONFLATED --> SPIDER_STATS(compute_spider_stats)
+    SPIDER_COUNTS --> SPIDER_STATS
+    ATP_STATS --> SPIDER_STATS
+    OSM_BRANDS --> SPIDER_STATS
+    BRAND_NAMES --> SPIDER_STATS
+    SPIDER_STATS --> SPIDER_STATS_PARQUET[spider-stats.parquet]
+    SPIDER_STATS_PARQUET --> UPLOAD_SPIDER_STATS(upload_spider_stats) --> S3A
+
     CONFLATED --> SUGGEST_EDITS(suggest_edits) --> LAYERS["*.jsonl (for visualization)"]
     LAYERS --> RENDER_TILES("render_tiles<br/>(tippecanoe, auto zoom)") --> PMTILES[diffed-places.pmtiles]
     PMTILES --> UPLOAD_TILES(upload_tiles) --> S3B[("public S3")]
@@ -247,10 +255,11 @@ graph TD
     UPLOAD_CONFLATED --> UPLOAD_DATAPACKAGE
     UPLOAD_CONFLATED_TILES --> UPLOAD_DATAPACKAGE
     UPLOAD_TILES --> UPLOAD_DATAPACKAGE
+    UPLOAD_SPIDER_STATS --> UPLOAD_DATAPACKAGE
     UPLOAD_BOM --> UPLOAD_DATAPACKAGE(upload_datapackage<br/>data/datapackage.json) --> S3A
 
     classDef process fill:#fce4ec,stroke:#ad1457,stroke-width:2px,color:#4a0e28,font-weight:bold;
-    class FETCH_INPUTS,IMPORT_ATP,OSMVIEWS_RANKS,COLLECT_WIKI,QRANK_QUANTILES,IMPORT_OSM,CONFLATE,UPLOAD_CONFLATED,EXTRACT_CONFLATED_LAYERS,RENDER_CONFLATED_OVERVIEW,RENDER_CONFLATED_DETAIL,JOIN_CONFLATED_TILES,UPLOAD_CONFLATED_TILES,SUGGEST_EDITS,RENDER_TILES,UPLOAD_TILES,UPLOAD_BOM,UPLOAD_DATAPACKAGE process;
+    class FETCH_INPUTS,IMPORT_ATP,OSMVIEWS_RANKS,COLLECT_WIKI,QRANK_QUANTILES,IMPORT_OSM,CONFLATE,UPLOAD_CONFLATED,EXTRACT_CONFLATED_LAYERS,RENDER_CONFLATED_OVERVIEW,RENDER_CONFLATED_DETAIL,JOIN_CONFLATED_TILES,UPLOAD_CONFLATED_TILES,SPIDER_STATS,UPLOAD_SPIDER_STATS,SUGGEST_EDITS,RENDER_TILES,UPLOAD_TILES,UPLOAD_BOM,UPLOAD_DATAPACKAGE process;
 ```
 
 (Pink boxes are processing steps; plain rectangles are the files they
@@ -520,6 +529,16 @@ fetch away.
   higher zoom levels” example). **~2m41s**, **2.16 GB peak RSS**,
   **~1.8 GB output** (z0–16) on the same run — total added cost over a
   single-pass `conflated.pmtiles` build: **~4m47s**, **+~1.05 GB**.
+- **`compute_spider_stats`**
+  ([`src/pipeline/spider_stats.rs`](../src/pipeline/spider_stats.rs))
+  — scans `conflated.parquet` once and writes `spider-stats.parquet`,
+  one row per AllThePlaces spider: what became of its features, how
+  they matched, how far their tags agree, and how its brands compare
+  with OpenStreetMap by country. Adds the counts that `import_atp`
+  and `collect_wikidata_ids` set aside, and AllThePlaces' own run
+  statistics. See
+  [`docs/outputs/SPIDER_STATS.md`](outputs/SPIDER_STATS.md) for the
+  schema. Uploaded by `upload_spider_stats`, like the files below.
 - **`suggest_edits`** ([`src/pipeline/edits.rs`](../src/pipeline/edits.rs))
   — scans `conflated.parquet` for matched rows and asks an
   [`edit_suggesters`](../src/edit_suggesters/) implementation what

docs/outputs/README.md

@@ -12,6 +12,9 @@ pipeline’s code, not for people using its output.)
 - [`CONFLATED_TILES.md`](CONFLATED_TILES.md) — `conflated.pmtiles`,
   visualizing `conflated.parquet`, matched or not, useful for
   debugging.
+- [`SPIDER_STATS.md`](SPIDER_STATS.md) — `spider-stats.parquet`, what
+  became of each AllThePlaces spider's features, for a dashboard;
+  useful for debugging.
 
 ## Downloading the current output
 
@@ -62,7 +65,9 @@ have `jq` installed there.
 
 The `conflated` resource is [`conflated.parquet`](CONFLATED_PARQUET.md);
 `bom` is its [CycloneDX](CONFLATED_PARQUET.md#data-provenance)
-provenance BOM as a standalone file. The two `*-tiles` resources are
+provenance BOM as a standalone file. `spider-stats` is
+[`spider-stats.parquet`](SPIDER_STATS.md), a debugging aid like the
+tiles. The two `*-tiles` resources are
 PMTiles archives you can open right in the browser via
 [`pmtiles.io`](https://pmtiles.io) (append
 `#url=<the-url>&inspectFeatures=true`) — but they’re a debugging aid,

docs/outputs/SPIDER_STATS.md

@@ -0,0 +1,166 @@
+# `spider-stats.parquet`
+
+`spider-stats.parquet` says, for each
+[AllThePlaces](https://www.alltheplaces.xyz/) spider, what became of
+its features: how many it wrote, how many we left out and why, how
+they matched OpenStreetMap, how far their tags agree, and how its
+brands compare with OpenStreetMap. It is meant for a dashboard that
+shows where conflation goes badly and why (see
+[brawer/osmdiffs#924](https://github.com/brawer/osmdiffs/issues/924)).
+
+This file is a debugging aid, not a data product. Its structure may
+change, or it may stop being published, without notice. To build
+something on top of this project, use
+[`conflated.parquet`](CONFLATED_PARQUET.md) instead.
+
+## Layout
+
+There is one row per spider, sorted by spider name. Row groups hold 64
+spiders each, so that a web page can fetch one spider with HTTP range
+requests, for example with
+[hyparquet](https://github.com/hyparam/hyparquet), without
+downloading the whole file.
+
+All numbers are counts, never shares (with one exception,
+`opening_hours_top_share`). That way, you can add them up across
+spiders, countries or brands before you divide.
+
+The list includes every spider that AllThePlaces lists in its run
+statistics, has a file in its dump, or has a row in `conflated.parquet`.
+A spider that AllThePlaces lists but that has no file in the dump has
+nulls for the dump's counts.
+
+## Schema
+
+```
+spider                   UTF8     non-null
+atp_features             UINT64   nullable — features AllThePlaces says the spider wrote
+atp_errors               UINT64   nullable — errors AllThePlaces says the spider logged
+dump_features            UINT64   nullable — features in the spider's file of the dump
+not_for_osm              UINT64   nullable — left out: license doesn't allow OpenStreetMap
+invalid                  UINT64   nullable — left out: no usable geometry, or no spider name
+no_matcher               UINT64   nullable — kept, but no matcher handles their kind
+in_china                 UINT64   nullable — features in mainland China
+datum                    UTF8     nullable — "wgs84", "gcj02" or "bd09", for in_china
+rows                     UINT64   non-null — rows in conflated.parquet
+same                     UINT64   non-null — rows with match.relation = "same"
+located_in               UINT64   non-null — rows with match.relation = "located_in"
+missing_in_osm           UINT64   non-null — rows with match.relation = "missing_in_osm"
+shared_osm_features      UINT64   non-null — same rows whose OSM feature another feature of the spider also matched
+distance_m               LIST<UINT64> non-null — histogram of match distances, see below
+opening_hours_top_share  FLOAT    nullable — share of features with the most common opening_hours
+rules                    LIST<STRUCT> non-null
+  rule                   UTF8     non-null — match.evidence.rule
+  rows                   UINT64   non-null
+tags                     LIST<STRUCT> non-null
+  tag                    UTF8     non-null — "name", "phone", "opening_hours", ...
+  atp                    UINT64   non-null — same rows whose ATP feature has the tag
+  agreeing               UINT64   non-null — ... and OSM agrees
+  differing              UINT64   non-null — ... and OSM has a different value
+  missing_in_osm         UINT64   non-null — ... and OSM has none
+brands                   LIST<STRUCT> non-null
+  country                UTF8     nullable — ISO 3166-1 alpha-2 code
+  wikidata_id            UTF8     nullable — "Q379911"; null for features without one
+  name                   UTF8     nullable — a display name for wikidata_id
+  atp_features           UINT64   non-null
+  same                   UINT64   non-null
+  located_in             UINT64   non-null
+  missing_in_osm         UINT64   non-null
+  osm_features           UINT64   nullable — OSM features with this ID in this country
+  deletion_candidates    UINT64   nullable — deletion candidates with this ID in this country
+```
+
+### What became of the features
+
+The counts add up like this:
+
+```
+dump_features = not_for_osm + invalid + no_matcher + rows
+rows          = same + located_in + missing_in_osm
+```
+
+`atp_features` and `atp_errors` come from AllThePlaces' own statistics
+for the run. `atp_features` normally equals `dump_features`. Where it
+doesn't, something went wrong between the spider and the dump. A
+spider that crashed often has errors and no features at all.
+
+`in_china` and `datum` concern features in mainland China. Many
+Chinese sources give coordinates in the GCJ-02 or BD-09 datum, which
+are hundreds of meters off. We find out which datum a spider uses by
+comparing its features with OpenStreetMap, and convert them to WGS-84
+before matching.
+
+### How they matched
+
+`rules` counts rows by `match.evidence.rule`; see
+[`CONFLATED_PARQUET.md`](CONFLATED_PARQUET.md#match)
+for what each rule means. Rows without a rule, such as
+`missing_in_osm`, are not listed.
+
+`distance_m` is a histogram of `match.evidence.distance_m` over the
+`same` rows. Its bins are bounded by 1, 10, 25, 50, 100, 250 and 1000
+meters: the first counts distances below 1 m, the second from 1 m up
+to 10 m, and the last 1000 m or more. The file's key-value metadata
+`osmdiffs:distance_bins_m` lists the bounds, in case they change. A
+spider whose matches pile up far away likely has wrong coordinates;
+one whose matches are all at exactly the same distance may place every
+feature at the center of its town.
+
+`shared_osm_features` counts `same` rows whose OpenStreetMap feature
+also matched another feature of the same spider. A high count hints at
+duplicates in the spider, or at OpenStreetMap having one feature where
+there are several.
+
+`opening_hours_top_share` is the share of the spider's features with
+`opening_hours` that have its most common value. Near 1 for a large
+spider, it suggests that the spider writes the same hours everywhere,
+such as those of its head office.
+
+### How far tags agree
+
+`tags` compares the tags that `match.evidence.agreeing` compares, for
+the `same` rows. `atp` counts rows whose AllThePlaces feature has the
+tag; these then split into `agreeing`, `differing` and `missing_in_osm`.
+Tags no feature of the spider has are not listed. Phone numbers, email
+addresses and websites count under `phone`, `email` and `website`
+whichever key they're under, such as `contact:phone`.
+
+### Brands
+
+`brands` splits the spider's rows by the country of its features and
+by their `brand:wikidata`, `operator:wikidata` and `network:wikidata`.
+A feature with several IDs counts for each of them; a feature without
+any counts under a null `wikidata_id`. So the brand counts of a spider
+can add up to more than its rows.
+
+`name` is the most common `brand`, `operator` or `network` name for
+the ID in OpenStreetMap and AllThePlaces, preferring one in Latin
+script; it is there so that readers needn't look up Wikidata IDs.
+
+`osm_features` counts the OpenStreetMap features in the country with
+the same ID, of the same kinds as the spider's features (such as
+`shop=supermarket`), whether they matched or not. If a spider has 300
+supermarkets in a country and OpenStreetMap 500, either the spider
+misses some or OpenStreetMap has some that closed.
+`deletion_candidates` counts the `deletion_candidate` rows with the
+same ID in the country. Both are null for features without an ID.
+
+## Example
+
+With [DuckDB](https://duckdb.org/), the spiders with the most features
+that found no match:
+
+```sql
+SELECT spider, missing_in_osm, rows
+FROM 'spider-stats.parquet'
+ORDER BY missing_in_osm DESC
+LIMIT 10;
+```
+
+And the brands of one spider:
+
+```sql
+SELECT UNNEST(brands, recursive := true)
+FROM 'spider-stats.parquet'
+WHERE spider = 'denner_ch';
+```

src/pipeline/atp/mod.rs

@@ -27,8 +27,8 @@ use crate::{PROGRESS_BAR_STYLE, matchers::MatchMask};
 mod datum;
 mod fetch;
 pub(super) mod nondescript;
-mod run_stats;
-mod spider_counts;
+pub(super) mod run_stats;
+pub(super) mod spider_counts;
 
 // Only these two re-exported crate-wide (rather than making all of
 // `fetch` pub(crate)): provenance needs them to assemble this

src/pipeline/atp/run_stats.rs

@@ -10,7 +10,7 @@ use std::path::Path;
 
 /// What AllThePlaces reports for one spider.
 #[derive(Clone, Debug, Default, Deserialize, Eq, PartialEq)]
-pub(super) struct SpiderRunStats {
+pub(in crate::pipeline) struct SpiderRunStats {
     pub spider: String,
 
     /// Features the spider wrote.
@@ -35,7 +35,7 @@ pub(super) fn parse(json: &[u8]) -> Result<Vec<SpiderRunStats>> {
 
 /// Reads the statistics that `fetch_atp` stored in `workdir`, by
 /// spider, or `None` if there are none.
-pub(super) fn read(workdir: &Path) -> Result<Option<HashMap<String, SpiderRunStats>>> {
+pub(in crate::pipeline) fn read(workdir: &Path) -> Result<Option<HashMap<String, SpiderRunStats>>> {
     let path = workdir.join(super::fetch::STATS_JSON_FILENAME);
     if !path.exists() {
         return Ok(None);

src/pipeline/atp/spider_counts.rs

@@ -16,12 +16,12 @@ const FILENAME: &str = "alltheplaces.spider-counts.tsv";
 const HEADER: &str = "spider\tfeatures\tnot_for_osm\tinvalid\tin_china\tdatum";
 
 /// Path of the counts that `import_atp` writes in `workdir`.
-pub(super) fn path(workdir: &Path) -> PathBuf {
+pub(in crate::pipeline) fn path(workdir: &Path) -> PathBuf {
     workdir.join(FILENAME)
 }
 
 #[derive(Clone, Debug, Default, Eq, PartialEq)]
-pub(super) struct SpiderCounts {
+pub(in crate::pipeline) struct SpiderCounts {
     pub spider: String,
 
     /// Features in the spider's file of the dump. A spider that crashed
@@ -75,8 +75,7 @@ pub(super) fn write(counts: &mut [SpiderCounts], out: &Path) -> Result<()> {
 }
 
 /// Reads counts that [write] wrote.
-#[allow(dead_code)] // Used by tests until the spider stats read it.
-pub(super) fn read(path: &Path) -> Result<Vec<SpiderCounts>> {
+pub(in crate::pipeline) fn read(path: &Path) -> Result<Vec<SpiderCounts>> {
     let file = std::fs::File::open(path).with_context(|| format!("could not open {path:?}"))?;
     let mut lines = BufReader::new(file).lines();
     if lines.next().transpose()?.as_deref() != Some(HEADER) {

src/pipeline/brand_totals.rs

@@ -191,6 +191,72 @@ fn clean_name(name: &str) -> String {
         .collect()
 }
 
+/// OSM features per Wikidata ID, country and main category.
+pub(super) type OsmFeatureCounts = HashMap<(u64, Option<Country>, String), u64>;
+
+/// Reads [features_path].
+pub(super) fn read_features(workdir: &Path) -> Result<OsmFeatureCounts> {
+    let mut features = HashMap::new();
+    for cols in read_tsv(&features_path(workdir), 4)? {
+        let n = cols[3]
+            .parse()
+            .with_context(|| format!("bad count {:?}", cols[3]))?;
+        features.insert(
+            (
+                parse_id(&cols[0])?,
+                parse_country(&cols[1]),
+                cols[2].clone(),
+            ),
+            n,
+        );
+    }
+    Ok(features)
+}
+
+/// Reads [names_path] into one name per Wikidata ID: its `brand` name,
+/// or else its `operator` name, or else its `network` name.
+pub(super) fn read_names(workdir: &Path) -> Result<HashMap<u64, String>> {
+    let rank = |key: &str| {
+        ["brand", "operator", "network"]
+            .iter()
+            .position(|k| *k == key)
+    };
+    let mut best: HashMap<u64, (usize, String)> = HashMap::new();
+    for cols in read_tsv(&names_path(workdir), 5)? {
+        let Some(r) = rank(&cols[1]) else { continue };
+        let id = parse_id(&cols[0])?;
+        if best.get(&id).is_none_or(|(old, _)| r < *old) {
+            best.insert(id, (r, cols[2].clone()));
+        }
+    }
+    Ok(best.into_iter().map(|(id, (_, name))| (id, name)).collect())
+}
+
+fn parse_id(id: &str) -> Result<u64> {
+    id.strip_prefix('Q')
+        .and_then(|n| n.parse().ok())
+        .with_context(|| format!("bad Wikidata ID {id:?}"))
+}
+
+fn parse_country(code: &str) -> Option<Country> {
+    <[u8; 2]>::try_from(code.as_bytes())
+        .ok()
+        .and_then(Country::from_code)
+}
+
+/// The lines of a TSV file after its header, split into `columns`.
+fn read_tsv(path: &Path, columns: usize) -> Result<Vec<Vec<String>>> {
+    let data = std::fs::read_to_string(path).with_context(|| format!("could not read {path:?}"))?;
+    data.lines()
+        .skip(1)
+        .map(|line| {
+            let cols: Vec<String> = line.split('\t').map(String::from).collect();
+            anyhow::ensure!(cols.len() == columns, "bad line in {path:?}: {line:?}");
+            Ok(cols)
+        })
+        .collect()
+}
+
 fn write_tsv(out: &Path, header: &str, lines: impl Iterator<Item = String>) -> Result<()> {
     let mut tmp = out.to_path_buf();
     tmp.add_extension("tmp");
@@ -275,6 +341,27 @@ mod tests {
         Ok(())
     }
 
+    #[test]
+    fn test_read() -> Result<()> {
+        let mut totals = BrandTotals::default();
+        let tags = [
+            ("shop", "supermarket"),
+            ("brand", "Migros"),
+            ("brand:wikidata", "Q680727"),
+            ("operator", "Migros Zürich"),
+            ("operator:wikidata", "Q680727"),
+        ];
+        totals.add(Source::Osm, tags.iter().copied(), CH);
+        totals.add(Source::Osm, tags.iter().copied(), None);
+        let dir = tempfile::tempdir()?;
+        totals.write(dir.path())?;
+        let features = read_features(dir.path())?;
+        assert_eq!(features[&(680727, CH, "shop=supermarket".to_string())], 1);
+        assert_eq!(features[&(680727, None, "shop=supermarket".to_string())], 1);
+        assert_eq!(read_names(dir.path())?[&680727], "Migros");
+        Ok(())
+    }
+
     #[test]
     fn test_display_name() {
         let shown = |list: &[(&str, u64)]| {