feat: store packet- and byte-weighted MAAD measures

[flamboh]

Note

🤖 Claude Opus 5.5 on behalf of Oliver

What changes

The pipeline stores packet- and byte-weighted MAAD next to the existing address-count MAAD for every bucket, scope, and address side. Flows that report 0 packets are dropped before aggregation. For now you switch measures on the file detail page. A dashboard view of the weighted measures will follow in a PR on top of #119.

Flows to exercise

Setup. Build a small CSV fixture in an empty directory $FIX (all addresses are documentation ranges):

cat > $FIX/flows.csv <<'EOF'
received,src,dst,packets,bytes,protocol
0,192.0.2.1,198.51.100.1,4,400,TCP
10,192.0.2.1,198.51.100.9,0,0,TCP
20,192.0.2.2,198.51.100.1,1,60,UDP
300,192.0.2.3,198.51.100.3,0,40,UDP
3600,192.0.2.1,198.51.100.1,2,120,TCP
EOF
cat > $FIX/mapping.json <<'EOF'
{"timestamp_format":"unix","timestamp_timezone":"UTC","columns":{"time_received":"received","src_ip":"src","dst_ip":"dst","packets":"packets","bytes":"bytes","protocol":"protocol"},"protocol_map":{"TCP":6,"UDP":17},"source_id":{"value":"edge"}}
EOF
cat > $FIX/pipeline.json <<EOF
{"database_path":"$FIX/fixture.sqlite","timezone":"UTC","locality":[{"type":"prefixes","prefixes":["192.0.2.0/24"]}],"inputs":[{"input_kind":"csv","path":"$FIX/flows.csv","mapping_path":"$FIX/mapping.json"}]}
EOF

1. Pipeline and the zero-packet drop. Run RUST_LOG=info ./scripts/netflow-db.sh pipeline --config $FIX/pipeline.json --maad-workers 2.
   • Expect: 3 complete, 0 partial, 10 unknown coverage. The logs show dropped zero-packet flows before aggregation with dropped_zero_packet_flows=1 for buckets 0 and 300.
   • SELECT bucket_start, flows, packets FROM traffic_stats WHERE granularity='5m' AND ip_version=4 AND src_locality='all' AND dst_locality='all' AND bucket_start <= 300 returns 0|2|5 and 300|0|0. Bucket 300 stays observed (complete coverage, zero traffic) even though its only flow was dropped.
   • SELECT measure, COUNT(*) FROM address_structure_stats WHERE structure_kind='structure' GROUP BY measure returns equal counts for addresses, packets, and bytes.
2. Verify. On a dataset-mode database (datasets registry with datasets.json, per docs/user/setup-pipeline.md), run ./scripts/netflow-db.sh verify <db> --dataset-id <id> --require-data --require-maad-data --require-processed --require-rollup-parity --require-no-raw-ip. Expect OK. To see the per-key check fail, delete a single bytes structure row for one bucket and side. Expect a nonzero exit and address_structure_stats is missing bytes structure rows for source … bucket … <side>.
3. --maad-workers. --maad-workers 0 (or -1 or many) is rejected by argument parsing. Omit the option to keep the default of available parallelism capped at 8. The worker count is not part of the product identity, so changing it on an existing database does not force a rebuild.
4. File page. Run bun run dev against the database. Open a file detail page and switch the measure between addresses, packets, and bytes. Spectrum is hidden for packets and bytes.
5. API. /api/netflow/structure-stats?measure=packets returns rows. /api/netflow/spectrum-stats?measure=packets returns 400.

Edge cases and decisions

• Zero-packet flows are dropped, not rounded up to 1. This was the owner's decision. The rule applies to native nfcapd and CSV input (sorted and unsorted CSV) and to daily-active-source qualification. A dropped flow adds nothing to traffic, protocols, ports, or address sets. It also adds nothing to the addresses measure, so an address seen only in zero-packet flows disappears from every product. Bucket coverage is unchanged. The pipeline logs the dropped flow count for each bucket, counted as flow_count rather than records.
• Missing packet count ≠ reported zero. Some CSV sources have no packets column, or a row has an empty cell. Those rows keep contributing. Their addresses become zero-weight for packets and are counted in zeroWeightAddrs. Without this rule, a CSV dataset with no packets column would publish as empty.
• Weighted measures: Addresses with a summed bytes value of 0 are left out of bytes, as are packets-unknown CSV addresses from packets. Their count is in metadata_json.zeroWeightAddrs. Prefix validity still uses distinct-address counts. Weighted measures have no spectrum. The owner judged it degenerate, and a DB CHECK enforces its absence.
• Product identity: The pipeline contract goes to v7, and the measure set is part of the identity, so this needs a fresh database. The zero-packet rule ships inside the same version bump. Drizzle 0005 marks existing D1 rows as addresses.
• Verify checks measure presence per full scope key (source, granularity, bucket, IP version, locality pair, side), not global counts.
• Cost (accepted by owner): On a real day with the full stack, wall time was 24:37 vs 16:41 on main, and peak RSS was 5.9 GB vs 3.3 GB. Storage is 658 MB/day vs 125 MB. feat: store MAAD results as compact f32 rows #119 compacts MAAD storage before the reprocess.

Verification

• Automated: cargo fmt --all --check, cargo clippy --workspace --all-targets --all-features --locked -- -D warnings, bun run format, bun run lint, bun run typecheck, bun run test:db. bun run test:web covers the web layer from the original PR; the review fixes don't touch web code.
• New regressions:
  • native zero-only and mixed flows sharing a source address
  • CSV sorted and unsorted, zero-only and mixed, plus a CSV with no packets column
  • an end-to-end CSV pipeline CLI run that asserts the drop logs and the observed zero bucket
  • publish profile drop count
  • verify with equal global counts but mismatched keys
  • --maad-workers parsing and pool-resize rejection
• Remaining manual checks:
  • flows 4–5 in a browser
  • one real-day pipeline + verify --require-maad-data on a fresh database, checking the zero-packet drop log volume