PR 6: Cross-source KG merge - #10
Merged
Merged
Conversation
Build the KG merger in graph/ that combines per-study subgraphs from different sources into one knowledge graph, deduped by ontology entity ID. - graph/merge.py: merge_subgraphs() dedups biological entities by ontology_id (back-filling unknown names, logging type clashes), keeps all study/dataset/sample nodes (deduped by their own ID -> idempotent), and preserves all edges (only exact duplicates collapse). - Provenance is DERIVED from the retained edges, not stored on nodes: entity_provenance()/cross_source_entities() attribute each entity edge to its owning study (via EXTRACTED_FROM/HAS_SAMPLE) and that study's source. No schema change (respects the frozen extra="forbid" models and PR 2's edge-only principle). - Wire merge_subgraphs into main.py (both paths) and drop the stale single-subgraph TODO. - tests/test_graph_merge.py (20): cross-source case runs the real CELLxGENE + GEO normalizers offline on two studies sharing lung (UBERON:0002048) and human (NCBITaxon:9606); plus dedup/idempotency/edge-preservation/provenance unit tests. Gates green incl. hermetic no-.env run: ruff check, ruff format --check, mypy (29 files), 188 unit tests. No dep changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mengerj
marked this pull request as ready for review
June 30, 2026 06:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
PR 6 from the roadmap: merge per-study subgraphs from different sources (CELLxGENE + GEO) into one knowledge graph, deduped by ontology entity ID, so studies link through shared entity nodes. Builds the merger in
graph/(reserved for exactly this since PR 3). Offline-only.What changed
src/parce/graph/merge.py(new, fully mypy-checked):merge_subgraphs(Iterable[KnowledgeGraphOutput]) -> KnowledgeGraphOutput— biological entities dedup byontology_id; study/dataset/sample nodes deduped by their own ID (cross-source IDs never collide, so this just makes the merge idempotent); all edges preserved, only exact(source, target, relation)duplicates collapse. Order-preserving (first-seen wins)._merge_entityback-fills anunknown/empty entity name from another source; logs a WARNING on anentity_typeclash (keeps first).entity_provenance(graph)/cross_source_entities(graph)— derive which studies/sources touch each shared entity.graph/__init__.pyexports the new API.main.py— bothrunandrun_geonow persistmerge_subgraphs(subgraphs)(wassubgraphs[0]with a stale "PR 6" TODO).tests/test_graph_merge.py(20 tests).Key design decision — provenance is derived from edges, not stored on nodes
The roadmap said "provenance on edges." I read that as keep the edges intact — only entity nodes dedup, so every edge that pointed at a shared entity survives and still records who touched it.
entity_provenancetraces each entity edge to its owning study (the study itself for GEO study-level edges; viaEXTRACTED_FROMfor CELLxGENE dataset edges; viaHAS_SAMPLEfor sample edges) →StudyNode.sourcegives the repo.Storing a study-list on
BiologicalEntityNodeinstead would denormalize a relationship the edges already own (drift trap PR 2's "containment is edge-only" rejected) and mutate the frozenextra="forbid"schema. No schema change. Recorded in ARCHITECTURE §4.Verification
All four CI gates pass locally, including the hermetic run with the repo-root
.envmoved aside (reproduces the CI runner):ruff check .— cleanruff format --check .— 49 files formattedmypy src/parce— 29 source files, no issues, no new exemptionspytest -m "not integration"— 188 passed, 13 integration deselectedThe cross-source test runs the real CELLxGENE + GEO normalizers (fake resolvers/extractor, fully offline) on two studies sharing lung
UBERON:0002048and humanNCBITaxon:9606, then asserts the shared entity is a single node touched by both sources while blood (CELLxGENE-only) stays single-source.No dependency changes →
uv.lockuntouched.Note
The CLI still runs one source per invocation; a single command that fetches multiple sources and merges across them is later orchestration (logical fit: PR 8 export). The merger + tests are the PR 6 deliverable. The PR 5 blocker stands: the live Azure extraction round-trip is still unverified (no
AZURE_AI_PROJECT_ENDPOINTin the headless env).🤖 Generated with Claude Code