Add a DuckDB-queryable edge-list Parquet for the citation graph

[ak2k]

Summary

Propose emitting the citation graph (and the other CSR relationship types) as a sorted, deduplicated edge-list Parquet, alongside the existing .npz CSR matrices, so the graph is queryable directly in DuckDB/Arrow — in the dataset's own format — without loading the whole matrix into Python.

This is additive: the .npz files and their consumers are untouched.

Motivation

• The pipeline already computes it. build_csr deduplicates and sorts the full edge set in DuckDB mid-build (the tmp_edges intermediate), then deletes it. Emitting it is near-zero marginal compute — it reuses the dedup/sort the CSR build already performs.
• The .npz isn't queryable without a full load. scipy.sparse.load_npz pulls the entire matrix into RAM (the citation graph is tens of GB) and is Python/scipy-specific. Everything else in the dataset is Parquet; an edge-list Parquet lets DuckDB/Arrow/Polars answer citation queries in place.
• Citation queries are the common case. "What does work X cite" and "who cites work X" are the staple graph queries; on a sorted edge list with row-group pruning they're millisecond range scans.

Proposed shape

Per directed relationship (starting with work_referenced_works):

• columns (src, tgt) in original OpenAlex IDs, so it joins work_main and the relationship tables directly — no dense-index / id-map indirection
• deduplicated, zstd-compressed, written with sorted row groups so DuckDB zonemaps prune

A Parquet file has a single sort order, so the two query directions want two sorted copies:

file	sort	serves
…__by_src.parquet	(src, tgt)	WHERE src = X → what X cites
…__by_tgt.parquet	(tgt, src)	WHERE tgt = X → who cites X

Measurements

On a 60M-edge / 3M-node sample (≈1/50th of work_referenced_works):

• edge-list Parquet (original IDs, zstd-3, 1M-row row groups): 288 MB (~4.8 bytes/edge)
• DuckDB straight off it, no full load: out-neighbours of a node ~5 ms, COUNT(DISTINCT src) ~36 ms
• the existing .npz for the same graph: ~203 MB, but requires a full in-RAM scipy load

Extrapolated to full work_referenced_works (~3B edges): ~14 GB per sorted copy.

The one trade-off: one direction or two

by_src is the natural canonical "graph as Parquet" — minimal, non-redundant. by_tgt answers the (arguably more common) "who cites X", but roughly doubles the graph's storage in a downloaded dataset (~14 GB → ~28 GB at full scale). The reverse copy is also a one-line re-sort for anyone who needs it locally.

So the open question: ship by_src only (lean, canonical) and document the reverse re-sort, or both out of the box?

Suggested rollout

If you're open to this, two small independent PRs so you can accept incrementally:

1. by_src — the emit mechanism + the canonical artifact (low-controversy).
2. by_tgt — the reverse sort, its own storage cost, take-or-leave.

Happy to open them. This pairs naturally with the build_csr perf PRs (#1–#3), which already touch this path.

Minor related notes (not blocking)

• The CSR .npz stores a data array of all-ones (an unweighted graph has no weights) — ~12 GB at full scale, and ~80% of the serialize time, since it's DEFLATE'd away. The edge list sidesteps it; within the .npz format there's no clean fix (a scipy CSR requires a data array), so noting only as context.
• build_csr's dense node index is int32, i.e. it assumes < ~2.1B unique nodes (work_referenced_works is ~250M today — comfortable, but worth a guard someday).
• The per-batch np.bincount(..., minlength=n_nodes) in the CSR build is O(batches × n_nodes); deriving indptr from a DuckDB GROUP BY src would remove it. A reasonable future micro-optimisation, out of scope here.