[brawer]

Requested by Sascha · project thread

Part 6 of #924: matcher diagnostics in spider-stats.parquet.

From conflate

Conflate writes conflate.spider-diagnostics.tsv on the side (new conflate/diagnostics.rs), because conflated.parquet doesn't keep rejected candidates:

• Why features are missing. Each missing_in_osm row counts as missing_no_candidate (no OSM feature of the kind within the matcher's distance), missing_rejected (there are some, but none scored above 0), or missing_taken (accepted candidates went to other features of the spider in assign_exclusive). These three add up to missing_in_osm.
• Nearest rejected candidate. For missing_rejected, a histogram of the distance to the nearest rejected candidate, with the bins of distance_m. To bound the cost, it looks at up to 64 rejected candidates per feature and decodes their geometry only for features that end up missing.
• Chance pairs. After the real matching, about 100 features per spider (all of a smaller spider) are moved 2 km in a random direction and matched again, ignoring exclusivity. chance_matched / chance_sampled overstates the share of wrong matches, as in the infrastructure notes. The sample comes from hashing spider and position in the file, so it's the same on every run.

Conflate's cache now also requires the side file to exist.

From conflated.parquet

• offset_east_m / offset_north_m: the medians of how far matched ATP features lie east and north of their OSM feature, centroid to centroid. Given per spider, and per brand and country in brands, so a shift in one country shows (like hema in the Netherlands).
• shared_positions: features at exactly the same position as another feature of the spider.

Cost

Not yet measured on the planet. The extra work is the chance pairs (about 100 matchings per spider, at most ~576k in all) and the nearest-candidate distances for missing features. I'd check the conflate.chance_pairs: done log line after the next weekly run.

Testing

• Unit tests for the TSV round trip, the 2 km shift, offset medians and shared positions.
• The integration test checks that the missing reasons add up, that one Misenso shop has a rejected same-kind OSM shop under 1 m away, the chance sample sizes, and that offsets are present exactly for spiders with matches. Ran locally with tippecanoe; the reproducibility test passes too.
• cargo fmt, cargo clippy --all-targets -D warnings, cargo test.

Left for later

From the "trust" list in #924: classifying tag differences (such as opening hours off by more than 60 minutes) and counting values that don't parse.

---

Generated by Claude Code