feat: classify endpoint locality and filter by traffic direction

[flamboh]

Note

🤖 Claude Opus 5.5 on behalf of Oliver

ELI5

Instead of guessing "inside vs outside" from a network-specific anonymization flag, each dataset now lists rules for what counts as internal, and the dashboard can filter traffic by direction: incoming, outgoing, inside-to-inside, or neither.

Why

Endpoint "visibility" came from the anonymizer's ToS bits, which only works on our network and misclassifies known literal internal hosts. Direction (ingress/egress/lateral) is what analyses actually need.

Implementation

• Per-dataset locality rules: tos_anonymized (the old behavior), prefixes (CIDRs), addresses (inline list or file). An endpoint is internal if any rule matches. Validation is strict; IPv4-mapped IPv6 is canonicalized.
• Direction from (src, dst): egress, ingress, lateral, transit (external→external, kept so bad records stay visible).
• Storage keeps its shape: src_locality/dst_locality (internal/external/all) replace src_visibility/dst_visibility.
• Product identity records rule counts and SHA-256 digests (never the addresses), so editing rules yields a new product.
• Address files are read only for the dataset being run, so one unreadable file doesn't break other datasets.
• PR feat(netflow-db): add coordinated active-source subsets #105's daily_active_sources selection migrated to locality.
• Web: shared direction filter (all/ingress/egress/lateral/transit) on every chart; hidden for datasets without rules (datasets.has_locality).

Review path

1. Registry mode. Add to a dataset in datasets.json:

   "locality": [
     { "type": "tos_anonymized" },
     { "type": "addresses", "path": "data/private/internal.txt" }
   ]

   Run pipeline --dataset <id> for one day. Expected: rows use src_locality/dst_locality; the four directions sum exactly to all; datasets.has_locality = 1. Relative paths resolve from the repository root.
2. Config mode. Put "locality": [...] at the top level of a pipeline config (a per-dataset locality inside a config is rejected), with a relative address file next to the config. Run pipeline --config <path> from another directory. Expected: the file resolves from the config's directory, and every dataset in the product gets has_locality = 1.
3. Dashboard. The direction filter changes every chart; a dataset without rules shows no filter.
4. Obsolete local DBs. Leave a pre-locality netflow.sqlite (with src_visibility columns and no has_locality) under data/, including one that reuses a current dataset ID. Expected: the dashboard lists only current products; the old DB is skipped and doesn't shadow the current one or break the listing.

Edge cases and decisions

• Local discovery skips any SQLite file that lacks bucket_coverage or datasets.has_locality, or that has a stats table without both locality columns. It doesn't error on them.
• A locality filter (--src-locality/--dst-locality, daily_active_sources) without rules is rejected, because every endpoint would be external.
• Private address lists belong under the gitignored data/.
• ⚠️ Drizzle migration 0003 keeps only all/all rows: literal/anonymized can't be mapped to internal/external. Migration 0004 adds datasets.has_locality.
• Production D1 validation and cutover are pending. Production will be cut over to a fresh database in the infra stack rather than migrated in place. This PR does not validate against production D1.

Verification

• Automated: bun run format, bun run lint, bun run typecheck, bun run test:web (includes the pre-locality/duplicate-ID discovery regression and the migrations for all five stats tables), bun run test:db (rule evaluation, direction mapping, fingerprint changes, registry errors, rollup parity, config-mode has_locality and config-relative address files), bun run test:e2e (passes; the fixture now uses the shared local schema and checks the served dataset metadata), cargo fmt --check, cargo clippy -D warnings.
• Manual: run a real one-day dataset through the dashboard direction filter; production D1 cutover (pending, see above).

---

Made by Claude Opus 5.5 (with Opus 5.5 subagents) via Claude Code.