[brawer]

Requested by Sascha · project thread

Fixes #910.

Keeps ATP's per-spider dataset_attributes (license, attribution, use:openstreetmap, spider:lineage, …) through the pipeline, as a new dataset_attributes MAP<UTF8, UTF8> column of spider-stats.parquet, right after datum. Downstream tools join on atp.fetched.spider rather than having the attributes repeated on every row of conflated.parquet.

• import_atp reads the attributes from each FeatureCollection's first line (already parsed there) and stores them, sorted by key, as a JSON object in a new last column of alltheplaces.spider-counts.tsv. Non-string values are kept as their JSON text; nulls are dropped. Attributes are kept for spiders whose features are not_for_osm too, since the license explains why.
• compute_spider_stats writes them as a map: empty for a crashed spider (empty file), null for a spider without a file in the dump, like the other dump columns.
• Counts files from an earlier version still read fine (no attributes), and import_atp treats them as missing so it counts again when the zip is there.
• docs/outputs/SPIDER_STATS.md documents the column, with a DuckDB example.

The map type is unsorted like the tags of conflated.parquet, since MapBuilder can't produce a sorted one, but the entries are sorted.

Testing

cargo test (unit + integration) and cargo clippy --all-targets -- -D warnings pass. New unit tests cover the TSV round trip with tabs and newlines in values, reading the old header, and value conversion; the integration test checks license and attribution:name:en of winterthur_ch from the fixture dump.

---

Generated by Claude Code