[brawer]

Requested by Sascha · project thread

Part 5 of #924: the file the dashboard reads.

A new step, compute_spider_stats, runs after conflate, scans conflated.parquet once, and writes spider-stats.parquet: one row per spider, 64 spiders per row group, so a web page can fetch one spider with range requests. All numbers are counts, so they can be summed across spiders, countries or brands before dividing.

Per spider:

• What became of the features: AllThePlaces' own feature and error counts, then the dump's features, not_for_osm, invalid, no_matcher, and the China datum. dump_features = not_for_osm + invalid + no_matcher + rows.
• How they matched: same / located_in / missing_in_osm, rows per rule, and a histogram of match distances (bins at 1, 10, 25, 50, 100, 250 and 1000 m, also in the key-value metadata).
• Suspicious patterns: shared_osm_features (one OSM feature matched by several features of the spider) and opening_hours_top_share (the same hours everywhere).
• Tags: for each tag that match.evidence.agreeing compares, how many matched features have it, and whether OSM agrees, differs or lacks it.
• Brands: by country and Wikidata ID, with a display name, the ATP counts, the OSM total for the same ID, country and kinds of features, and deletion candidates.

The file is uploaded as spider-stats and listed in datapackage.json as a debugging aid. The schema is in docs/outputs/SPIDER_STATS.md.

Testing

• Unit tests for the aggregation (including distance bin edges, shared OSM features, opening hours share) and for writing and reading back the file.
• The integration test now checks spider-stats.parquet against the fixture, and the manifest's fifth resource. Ran it locally with tippecanoe.
• cargo fmt, cargo clippy --all-targets -D warnings, cargo test.

Not in this PR

• Matcher diagnostics (best rejected candidate, offsets, chance pairs): PR 6.
• A pointer to the previous week's file, for trends: a follow-up.

---

Generated by Claude Code