diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 179fd89..9edffd2 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -93,16 +93,19 @@ Source-agnostic nodes (Pydantic v2, `extra="forbid"`). Implemented in PR 2 in `models/graph_schema.py`: - `StudyNode` — `study_id` (DOI/accession), `title`, `source` (provenance), - `modality`. *(No `experimental_narrative`; that field is removed.)* Raw free - text (abstracts, full descriptions) is **not** stored on the node — it belongs - to the per-source `RawRecord`; the canonical node holds only normalized, - design-describing fields. *(Target, §5: this free-text `modality` is refined - into an EFO `assay` term plus a derived coarse `molecular_layer` enum — PR 4.)* -- `DatasetNode` — `dataset_id`, `data_uri`, `assay` (to be grounded to an EFO - term ID, see §5), `cell_count`/size. Its parent study is a typed + `assay`, `molecular_layer`. *(No `experimental_narrative`; that field is + removed.)* Raw free text (abstracts, full descriptions) is **not** stored on + the node — it belongs to the per-source `RawRecord`; the canonical node holds + only normalized, design-describing fields. *(PR 4b: the free-text `modality` + string is gone — replaced by the EFO `assay` term ID of the study's + representative/dominant assay plus the derived coarse `molecular_layer` enum, + per §5.)* +- `DatasetNode` — `dataset_id`, `data_uri`, `assay` (an **EFO term ID**, §5), + `molecular_layer`, `cell_count`/size. Its parent study is a typed `EXTRACTED_FROM` **edge**, not a stored foreign-key field. *(Decision, PR 2: containment/relationships live on edges only; duplicating them as node fields - invites drift and gives two sources of truth.)* + invites drift and gives two sources of truth.)* *(PR 4b: `assay` changed from a + free-text name to the grounded EFO ID, and `molecular_layer` was added.)* - `SampleNode` — `sample_id`, `data_uri`, and **design covariates**: `condition`, `perturbation`, `timepoint`, `subject`, `organism`. (Reintroduced; the prior schema was dataset-level only.) All covariates are optional — different @@ -156,16 +159,33 @@ Avoid a free-text `modality`. Instead store, per dataset/study: term's `is-a` ancestors** to a small set of anchor classes. The lineage does the classification; we never re-string it. -> **PR 4 — anchors & default.** `MolecularLayer` lives in `models/graph_schema.py` -> (canonical-vocabulary home); the derivation is `parce.ontology.layers. -> derive_molecular_layer`. The anchor set is keyed by EFO ancestor **label**, not -> term ID — honouring "pin ontologies/anchors, not IDs" (OLS returns canonical -> labels for every ancestor), and matched most-specific-first. The **no-anchor -> default is `MolecularLayer.UNKNOWN`**. The current anchor labels are -> *provisional* (an informed first cut not yet checked against live EFO); the -> marked `tests/test_ontology_integration.py` is the validation harness, and PR 4b -> (which adds the stored field) must tighten them. The derivation logic + default -> are decided; only the exact label strings remain to be confirmed. +> **PR 4 / 4b — anchors, default & stored fields.** `MolecularLayer` lives in +> `models/graph_schema.py` (canonical-vocabulary home); the derivation is +> `parce.ontology.layers.derive_molecular_layer`. The anchor set is keyed by EFO +> ancestor **labels/keywords**, not term IDs — honouring "pin ontologies/anchors, +> not IDs" (OLS returns canonical labels for every ancestor) — and matched +> most-specific-first. The **no-anchor default is `MolecularLayer.UNKNOWN`**. +> +> **Decision (PR 4b): match keywords by case-insensitive *substring*, not exact +> whole-label.** Validating the PR 4 anchors against live EFO showed the assay +> branch is too inconsistent for exact matching: the 10x family — the bulk of +> CELLxGENE — never reaches `RNA assay`, announcing itself only via a parent +> `…transcription profiling` label; ATAC-seq/ChIP-seq/methylation/WGS all collapse +> to a generic `DNA assay` ancestor whose distinguishing signal survives only in +> the term's own label. Ordered substring keywords (transcriptome/epigenome/ +> proteome/metabolome before the broad GENOME `DNA` signals) classify all of these; +> exact full-label matching derived `UNKNOWN` for 10x. The keyword set is validated +> against live EFO by the marked `tests/test_ontology_integration.py`. Genuinely +> ambiguous lineages stay `UNKNOWN` on purpose — bare "mass spectrometry" (proteome +> vs metabolome) and multi-omic EFO terms (e.g. mCT-seq carries both `RNA assay` +> and `DNA assay` ancestors). +> +> **PR 4b stores the result:** `StudyNode`/`DatasetNode` now carry the EFO `assay` +> term ID + `molecular_layer`. For CELLxGENE the assay ID is *taken from Census's +> already-grounded payload* (not re-resolved via OLS — Census is authoritative for +> its own data and re-resolving could drift); the resolver is used only to walk the +> EFO lineage for the layer. Other (free-text) sources resolve the assay string via +> `OntologyResolver` first, then derive the layer from the resulting ID. The model sees a clean cross-modality categorical (`molecular_layer`) plus a precise term (`assay`) — both controlled, no free text on either. @@ -228,11 +248,13 @@ that already ship SDRF. - **Sample granularity for CELLxGENE.** Census is per-cell/dataset, not per-sample in the GEO sense. Defer mapping cxg to `SampleNode` until needed; keep it dataset-level for now. -- **`molecular_layer` anchor set.** *Partly resolved (PR 4):* the derivation - mechanism and the no-anchor default (`UNKNOWN`) are pinned, and the anchors are - keyed by EFO **label**. **Still open:** the exact label strings are provisional - and unvalidated against live EFO — confirm via the marked integration test and - tighten in PR 4b. +- **`molecular_layer` anchor set.** *Resolved (PR 4b):* anchors are ordered, + case-insensitive **substring keywords** over EFO ancestor (and own-) labels, + validated against live EFO via the marked integration test — the 10x family, + scRNA/Smart-seq, spatial, ATAC/ChIP/methylation, WGS and metabolomics all + classify correctly; bare mass-spec and multi-omic terms stay `UNKNOWN` by + design. **Still open:** extend the keyword set as GEO/PRIDE bring bulk RNA-seq, + proteomics and other modalities with their own lineages. - **Ontology resolver dependency.** *Resolved (PR 4):* **OLS4 REST only** for now (it covers organism grounding + the lineage walk). text2term/Zooma are deferred to PR 5, where GEO's messy `characteristics_ch1` strings need fuzzy diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 0930d70..0ffcb7e 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -8,18 +8,22 @@ protocol and [ARCHITECTURE.md](ARCHITECTURE.md) for the design. ## ▶ Next up -**PR 4b — Schema refinement: EFO `assay` term + stored `molecular_layer`.** PR 4 -shipped the ontology *stage* (resolver, registry, OLS4 client, cache, -`molecular_layer` **derivation**) but did not change the canonical schema. This -PR consumes it: on `StudyNode`/`DatasetNode`, replace the free-text `modality` -string with an EFO `assay` **term ID** (resolved via `OntologyResolver`) plus a -stored `molecular_layer` enum field (derived via `OntologyResolver.molecular_ -layer`). Wire both into `CellxgeneNormalizer` (it already resolves organism; -extend to ground the assay string and derive the layer). Migrate the schema + -all tests. `MolecularLayer` already lives in `models/graph_schema.py`. **Gotcha:** -the provisional anchor labels in `ontology/layers.py` are unvalidated against live -EFO — run `tests/test_ontology_integration.py` and tighten them first, else every -assay derives `UNKNOWN`. +**PR 5 — GEO extraction agent (vertical slice).** First source whose metadata is +*unstructured* free text, so the first to use the LLM. Add a GEO adapter +(NCBI E-utilities / GEOparse) emitting a `RawRecord`, and an **Azure extraction +normalizer** that fills the canonical schema via `response_format` (structured +output only — never prose). Extract sample-level **design** covariates from +`characteristics_ch1` (`condition`, `perturbation`, `timepoint`, `subject`, +`organism`) into `SampleNode`s — never data-inferred annotations. Ground the +extracted free-text facets through the **existing `OntologyResolver`** (organism +→ NCBITaxon, assay → EFO + `molecular_layer`, tissue → UBERON, disease → MONDO), +and **supply the agent as the resolver's LLM-fallback callback** for strings OLS +can't map (the hook already exists, default off). Mark live tests +`@pytest.mark.integration`; keep unit tests offline by mocking the Azure client. +**Remove the `parce.agent.*` mypy exemption** in `pyproject.toml` once the agent +moves to the normalizer interface. **Blocker risk:** needs Azure creds + an +`az login` session; if absent, build/unit-test the deterministic scaffolding and +log the integration boundary as a blocker rather than working around it. --- @@ -52,15 +56,19 @@ Each PR is one branch, one focused scope, green CI, and a roadmap update. — CELLxGENE ships IDs, only organism free-text needed grounding). IDs resolved at runtime; none hardcoded. *Split from the original PR 4: the schema change (store the EFO assay term + `molecular_layer`) became PR 4b.* -- [ ] **PR 4b — Schema refinement.** Replace free-text `modality` with an EFO - `assay` term ID + a stored `molecular_layer` enum on the study/dataset nodes; - wire the resolver's assay grounding + layer derivation into the normalizer; - migrate the schema and tests. *(Next up — see top of file.)* +- [x] **PR 4b — Schema refinement.** Replaced free-text `modality` with an EFO + `assay` term ID + a stored `molecular_layer` enum on `StudyNode`/`DatasetNode`; + wired assay grounding (taken from Census's already-grounded payload) + layer + derivation into `CellxgeneNormalizer` via a new `OntologyService` contract; + migrated the schema and all tests. **Switched `molecular_layer` matching from + exact EFO labels to ordered substring keywords** after validating against live + EFO (the 10x family never reaches `RNA assay`); ambiguous lineages (bare + mass-spec, multi-omic terms) stay `UNKNOWN` by design. - [ ] **PR 5 — GEO extraction agent (vertical slice).** GEO adapter (E-utilities/GEOparse) + Azure extraction normalizer emitting the canonical schema via `response_format`; extract sample covariates from `characteristics_ch1`. Integration test (marked). This is the agent's real - job; remove the `parce.agent.*` mypy exemption. + job; remove the `parce.agent.*` mypy exemption. *(Next up — see top of file.)* - [ ] **PR 6 — Cross-source KG merge.** Merge CELLxGENE + GEO into one graph linked through shared ontology entities; dedup; provenance on edges. Assert a cross-source edge exists in tests. @@ -85,6 +93,47 @@ Each PR is one branch, one focused scope, green CI, and a roadmap update. Newest first. One entry per working session: what changed, decisions made, and what the next session should know. Keep entries short and factual. +### 2026-06-26 — PR 4b: Schema refinement (EFO assay term + stored molecular_layer) + +- Branch `pr4b-schema-refinement` off `main` (35a28ce, the PR 4 merge). Note: the + scheduled session started on a stale worktree whose `main` predated the PR 4 + merge; `gh` showed PR #7 already MERGED, so PR 4 was done and **PR 4b was the + real ▶ Next up**. Re-based onto `origin/main` before starting. +- **Schema (`models/graph_schema.py`).** Dropped `StudyNode.modality`. Both + `StudyNode` and `DatasetNode` now carry `assay` (EFO term ID, required) + + `molecular_layer` (`MolecularLayer`, default `UNKNOWN`). `DatasetNode.assay` + changed meaning from free-text name → grounded EFO ID. +- **Normalizer (`normalize/cellxgene.py`).** Per dataset: pick the grounded EFO + assay ID from Census's payload (`ontology_summary.assays`, matched to the + dominant `modality` *name*) and derive `molecular_layer` from it; `StudyNode` + built at the end with the **most-frequent** dataset assay as its representative + + that assay's layer. Layer derivation memoised per assay ID; only `EFO:` IDs + are walked (an ungrounded `unknown` assay → `UNKNOWN`, no network call). +- **Decision — assay taken from payload, not re-resolved.** CELLxGENE ships the + EFO assay ID; Census is authoritative for its own data and re-resolving via OLS + could drift, so the resolver is used *only* for the lineage walk. Free-text + sources (GEO/PRIDE) will resolve the assay string first. (ARCHITECTURE §5.) +- **Decision — `molecular_layer` matching: exact labels → substring keywords.** + Probing live EFO exposed the PR 4 gotcha for real: `EFO:0009922` (10x 3' v3, the + bulk of CELLxGENE) never reaches `RNA assay` — it sits under `…transcription + profiling`/`library preparation`; ATAC/ChIP/methylation/WGS all collapse to a + generic `DNA assay` ancestor. Rewrote `ontology/layers.py` to ordered, + case-insensitive **substring keywords** (transcriptome/epigenome/proteome/ + metabolome before the broad GENOME `DNA` signals), validated against live EFO. + Bare mass-spec and multi-omic terms stay `UNKNOWN` by design (genuinely + ambiguous). The marked integration test now asserts exact layers (10x + scRNA → + TRANSCRIPTOME, ATAC → EPIGENOME) and **passed live** this session. +- **New contract `OntologyService`** (`ontology/base.py`, exported) = `TermResolver` + + `molecular_layer`; the normalizer depends on it so offline fakes inject both + methods. `base.py` now imports `MolecularLayer` (models is a leaf; no cycle). +- **Gates green incl. hermetic no-`.env` run:** ruff check, ruff format --check + (40 files), mypy (24 files), **133 unit tests**. Live OLS integration suite (6) + also passed. No dep changes; no new mypy exemptions. +- **Next session:** PR 5 (GEO extraction agent) — first LLM/Azure source; needs + credentials. Wire the GEO free-text facets through the existing resolver and + supply the agent as its LLM-fallback callback. If Azure creds are missing, stop + at the integration boundary and log the blocker. + ### 2026-06-25 — PR 4: Ontology resolver (stage + organism wiring) - Branch `pr4-ontology-resolver` off `main` (64ed4f4). diff --git a/src/parce/models/graph_schema.py b/src/parce/models/graph_schema.py index aab0f41..2d46614 100644 --- a/src/parce/models/graph_schema.py +++ b/src/parce/models/graph_schema.py @@ -77,9 +77,21 @@ class StudyNode(BaseModel): ..., description="Provenance of the study record (e.g. 'CELLxGENE', 'GEO', 'PRIDE').", ) - modality: str = Field( + assay: str = Field( ..., - description="High-level assay modality of the study (e.g. 'scRNA-seq', 'proteomics').", + description=( + "EFO term ID of the study's representative (dominant) assay " + "(e.g. 'EFO:0009922' for 10x 3' v3); 'unknown' if ungrounded. Replaces " + "the former free-text 'modality' string — see docs/ARCHITECTURE.md §5." + ), + ) + molecular_layer: MolecularLayer = Field( + default=MolecularLayer.UNKNOWN, + description=( + "Coarse molecular readout, derived from the assay's EFO 'is-a' " + "lineage (never re-strung from text). 'unknown' when the lineage " + "reaches no anchor." + ), ) @@ -95,7 +107,18 @@ class DatasetNode(BaseModel): ) assay: str = Field( ..., - description="Specific assay/technology (e.g. \"10x 3' v3\", 'Smart-seq2').", + description=( + "EFO term ID of the dataset's assay/technology (e.g. 'EFO:0009922' for " + "10x 3' v3); 'unknown' if ungrounded. Fine-grained: every 10x chemistry " + "is its own term." + ), + ) + molecular_layer: MolecularLayer = Field( + default=MolecularLayer.UNKNOWN, + description=( + "Coarse molecular readout derived from the assay's EFO 'is-a' lineage " + "(see docs/ARCHITECTURE.md §5). 'unknown' when no anchor is reached." + ), ) cell_count: int = Field(..., description="Number of cells (or rows) in the dataset.") diff --git a/src/parce/normalize/cellxgene.py b/src/parce/normalize/cellxgene.py index 9f151c9..754dcdd 100644 --- a/src/parce/normalize/cellxgene.py +++ b/src/parce/normalize/cellxgene.py @@ -1,7 +1,7 @@ """Deterministic normalizer: a CELLxGENE ``RawRecord`` → canonical KG nodes. No LLM is involved: CELLxGENE Census already ships ontology-grounded terms, so -this is a pure structural mapping. Three design rules show up directly here: +this is a pure structural mapping. The design rules show up directly here: * **Cell type is never consumed** — the adapter does not even read it (data-inferred → leakage; see docs/ARCHITECTURE.md §1). @@ -12,11 +12,17 @@ Census, so only the bare organism string needs runtime resolution (to NCBITaxon, via OLS). An organism that fails to resolve is skipped rather than emitted ungrounded. +* **Assay is stored as its EFO term ID plus a derived ``molecular_layer``**, not + a free-text modality string (ARCHITECTURE §5). Census already grounds the + assay (it ships the EFO ID alongside the name), so the normalizer *picks* that + ID rather than re-resolving it; the resolver is used only to walk the EFO + lineage for the coarse ``molecular_layer``. """ from __future__ import annotations import logging +from collections import Counter from typing import Any from parce.models.graph_schema import ( @@ -25,17 +31,14 @@ EntityType, GraphEdge, KnowledgeGraphOutput, + MolecularLayer, StudyNode, ) from parce.models.raw_record import RawRecord -from parce.ontology import Facet, OntologyResolver, ResolvedTerm, TermResolver +from parce.ontology import Facet, OntologyResolver, OntologyService, ResolvedTerm logger = logging.getLogger(__name__) -# High-level study modality for everything CELLxGENE ingests. (Refined into an -# EFO ``assay`` term + derived ``molecular_layer`` in the schema-refinement PR.) -_STUDY_MODALITY = "scRNA-seq" - # Ontology categories that become design-context entities. Cell types are # deliberately absent (data-inferred → leakage). _CATEGORY_TO_ENTITY_TYPE: dict[str, EntityType] = { @@ -50,29 +53,25 @@ "assays": "MEASURED_WITH", } +_UNKNOWN_ASSAY = "unknown" + class CellxgeneNormalizer: """:class:`~parce.normalize.base.Normalizer` for CELLxGENE ``RawRecord``s. - Takes a :class:`~parce.ontology.base.TermResolver` (default: a real + Takes an :class:`~parce.ontology.base.OntologyService` (default: a real :class:`~parce.ontology.resolver.OntologyResolver`) used to ground organism - strings. Inject a deterministic fake to keep unit tests offline. + strings and to derive each assay's ``molecular_layer``. Inject a + deterministic fake to keep unit tests offline. """ - def __init__(self, resolver: TermResolver | None = None) -> None: - self._resolver: TermResolver = resolver if resolver is not None else OntologyResolver() + def __init__(self, resolver: OntologyService | None = None) -> None: + self._resolver: OntologyService = resolver if resolver is not None else OntologyResolver() def normalize(self, record: RawRecord) -> KnowledgeGraphOutput: """Assemble the canonical single-study subgraph for one CELLxGENE study.""" study_id = record.study_id - study = StudyNode( - study_id=study_id, - title=record.title, - source=record.source, - modality=_STUDY_MODALITY, - ) - datasets: list[DatasetNode] = [] edges: list[GraphEdge] = [] entity_registry: dict[str, BiologicalEntityNode] = {} @@ -80,15 +79,26 @@ def normalize(self, record: RawRecord) -> KnowledgeGraphOutput: # string → resolution so the same string is grounded at most once. species_ids: set[str] = set() organism_cache: dict[str, ResolvedTerm | None] = {} + # Per-record memos so each distinct assay term is layer-derived once, and + # the study's representative assay can be chosen by frequency. + layer_cache: dict[str, MolecularLayer] = {} + assay_labels: dict[str, str] = {} + assay_counts: Counter[str] = Counter() for ds in record.payload.get("datasets", []): dataset_id = ds["dataset_id"] + ontology: dict[str, Any] = ds.get("ontology_summary", {}) + + assay_id, assay_label = _select_assay(ontology) + assay_counts[assay_id] += 1 + assay_labels.setdefault(assay_id, assay_label) datasets.append( DatasetNode( dataset_id=dataset_id, data_uri=ds["h5ad_uri"], - assay=ds.get("modality", "unknown"), + assay=assay_id, + molecular_layer=self._layer_for(assay_id, assay_label, layer_cache), cell_count=ds["cell_count"], ) ) @@ -101,8 +111,6 @@ def normalize(self, record: RawRecord) -> KnowledgeGraphOutput: ) ) - ontology: dict[str, Any] = ds.get("ontology_summary", {}) - # Ground the organism string to a NCBITaxon term via the resolver. organism_text = ontology.get("organism", "unknown") species = self._resolve_organism(organism_text, organism_cache) @@ -146,6 +154,20 @@ def normalize(self, record: RawRecord) -> KnowledgeGraphOutput: ) ) + # The study's representative assay is the most frequent across its + # datasets (CELLxGENE collections are single-modality in practice); its + # layer is reused from the per-dataset memo. + study_assay = assay_counts.most_common(1)[0][0] if assay_counts else _UNKNOWN_ASSAY + study = StudyNode( + study_id=study_id, + title=record.title, + source=record.source, + assay=study_assay, + molecular_layer=self._layer_for( + study_assay, assay_labels.get(study_assay, study_assay), layer_cache + ), + ) + kg = KnowledgeGraphOutput( studies=[study], datasets=datasets, @@ -155,14 +177,31 @@ def normalize(self, record: RawRecord) -> KnowledgeGraphOutput: ) logger.info( - "Normalized CELLxGENE study=%s: datasets=%d entities=%d edges=%d", + "Normalized CELLxGENE study=%s: datasets=%d entities=%d edges=%d layer=%s", study_id, len(kg.datasets), len(kg.biological_entities), len(kg.edges), + study.molecular_layer, ) return kg + def _layer_for( + self, assay_id: str, assay_label: str, memo: dict[str, MolecularLayer] + ) -> MolecularLayer: + """Derive (and memoise) the molecular layer for one assay term. + + Only EFO assay IDs are walked — the lineage walk is EFO-specific and an + ungrounded ``'unknown'`` assay has no lineage — so anything else maps to + :attr:`MolecularLayer.UNKNOWN` without a network call. + """ + if assay_id not in memo: + if assay_id.startswith("EFO:"): + memo[assay_id] = self._resolver.molecular_layer(assay_id, assay_label=assay_label) + else: + memo[assay_id] = MolecularLayer.UNKNOWN + return memo[assay_id] + def _resolve_organism( self, organism_text: str, memo: dict[str, ResolvedTerm | None] ) -> ResolvedTerm | None: @@ -170,3 +209,22 @@ def _resolve_organism( if organism_text not in memo: memo[organism_text] = self._resolver.resolve_term(organism_text, Facet.ORGANISM) return memo[organism_text] + + +def _select_assay(ontology: dict[str, Any]) -> tuple[str, str]: + """Pick the dataset's grounded (EFO) assay ID and label from the payload. + + Census ships the assay already grounded: ``ontology['assays']`` is a list of + ``{name, ontology_id}`` and ``ontology['modality']`` is the dominant assay's + *name*. We match the dominant name to its grounded ID; failing that, take the + first grounded assay; failing that (no assays), return ``('unknown', name)``. + """ + assays: list[dict[str, str]] = ontology.get("assays", []) + dominant = ontology.get("modality", _UNKNOWN_ASSAY) + for term in assays: + if term.get("name") == dominant: + return term.get("ontology_id", _UNKNOWN_ASSAY), term.get("name", dominant) + if assays: + first = assays[0] + return first.get("ontology_id", _UNKNOWN_ASSAY), first.get("name", dominant) + return _UNKNOWN_ASSAY, dominant diff --git a/src/parce/ontology/__init__.py b/src/parce/ontology/__init__.py index f1f4be7..94dc5dc 100644 --- a/src/parce/ontology/__init__.py +++ b/src/parce/ontology/__init__.py @@ -7,13 +7,13 @@ * :class:`Facet` / :data:`FACET_ONTOLOGY` — which ontology grounds which facet; * :class:`OntologyResolver` — OLS-first resolver with on-disk cache and an optional LLM fallback hook; -* :class:`ResolvedTerm` / :class:`TermResolver` — the value type and the narrow - contract normalizers depend on. +* :class:`ResolvedTerm` / :class:`TermResolver` / :class:`OntologyService` — the + value type and the contracts normalizers depend on. """ from __future__ import annotations -from parce.ontology.base import ResolvedTerm, TermResolver +from parce.ontology.base import OntologyService, ResolvedTerm, TermResolver from parce.ontology.cache import ResolutionCache from parce.ontology.layers import derive_molecular_layer from parce.ontology.ols import OlsClient, OlsTerm @@ -28,6 +28,7 @@ "OlsTerm", "Ontology", "OntologyResolver", + "OntologyService", "ResolutionCache", "ResolvedTerm", "TermResolver", diff --git a/src/parce/ontology/base.py b/src/parce/ontology/base.py index 6ebba2e..e7d1d0a 100644 --- a/src/parce/ontology/base.py +++ b/src/parce/ontology/base.py @@ -1,8 +1,9 @@ -"""Value types and the resolver contract for the shared ontology stage. +"""Value types and the resolver contracts for the shared ontology stage. -Kept dependency-light (only the registry) so both the cache and the normalizers -can import :class:`ResolvedTerm` / :class:`TermResolver` without pulling in the -OLS client or ``requests``. +Kept dependency-light (only the registry and the canonical ``MolecularLayer`` +enum) so both the cache and the normalizers can import :class:`ResolvedTerm` / +:class:`TermResolver` / :class:`OntologyService` without pulling in the OLS +client or ``requests``. """ from __future__ import annotations @@ -10,6 +11,7 @@ from dataclasses import dataclass from typing import Protocol, runtime_checkable +from parce.models.graph_schema import MolecularLayer from parce.ontology.registry import Facet @@ -37,3 +39,19 @@ class TermResolver(Protocol): def resolve_term(self, text: str, facet: Facet) -> ResolvedTerm | None: """Return the grounded term for ``text`` under ``facet``, else ``None``.""" ... + + +@runtime_checkable +class OntologyService(TermResolver, Protocol): + """A :class:`TermResolver` that also derives an assay's molecular layer. + + The fuller contract the CELLxGENE normalizer depends on: ground free text + (organism) *and* classify an EFO ``assay`` term into a coarse + :class:`~parce.models.graph_schema.MolecularLayer`. The concrete + :class:`~parce.ontology.resolver.OntologyResolver` satisfies it; unit tests + inject an offline fake implementing both methods. + """ + + def molecular_layer(self, assay_id: str, *, assay_label: str | None = None) -> MolecularLayer: + """Return the coarse molecular layer for EFO ``assay_id``.""" + ... diff --git a/src/parce/ontology/layers.py b/src/parce/ontology/layers.py index 9123ca0..5fa811e 100644 --- a/src/parce/ontology/layers.py +++ b/src/parce/ontology/layers.py @@ -1,17 +1,28 @@ """Derive a coarse :class:`~parce.models.graph_schema.MolecularLayer` from an EFO assay term's ``is-a`` ancestry. -The classification is done by the *lineage*, not by re-parsing strings: we walk -the assay term's ancestors (via OLS) and match their labels against a small set -of pinned **anchor classes**. The anchors are keyed by their canonical EFO -**label**, not by term ID — the registry pins ontologies/anchors, not IDs -(docs/ARCHITECTURE.md §5), and OLS returns canonical labels for every ancestor. - -PROVISIONAL anchor set: the exact EFO ancestor labels below are an informed -first cut. They are validated against the live EFO assay branch by the marked -integration test ``tests/test_ontology_integration.py``; correct/extend them -there as real lineages are observed. An assay whose ancestry reaches no anchor -falls back to :attr:`MolecularLayer.UNKNOWN` (the pinned no-anchor default). +The classification is done by the *lineage*, not by re-parsing the assay's free +text: we walk the assay term's ancestors (via OLS) and match their labels — plus +the assay's own label — against a small set of pinned **anchor keywords**. + +Why keywords (substring), not exact full labels: the live EFO assay branch is +inconsistent. Some readouts announce themselves in an ancestor's label +("RNA assay"), others only in a *parent process* label ("10x 3' transcription +profiling" — the 10x family never reaches "RNA assay"), and several (ATAC-seq, +ChIP-seq, methylation, WGS) collapse to a generic "DNA assay" ancestor whose +distinguishing signal survives only in the term's own label. Ordered +case-insensitive substring matching against distinctive phrases handles all of +these; exact full-label matching cannot. + +The keyword set is validated against the live EFO branch by the marked +integration test ``tests/test_ontology_integration.py``; extend it there as new +modalities (GEO bulk assays, PRIDE proteomics) bring real lineages. Order +matters: the first keyword (top-to-bottom) present in the lineage wins, so the +more specific epigenome DNA assays beat the broad GENOME ``DNA``/``whole genome`` +signals. An assay whose lineage matches no keyword falls back to +:attr:`MolecularLayer.UNKNOWN` (the pinned no-anchor default) — e.g. bare +"mass spectrometry", which is genuinely ambiguous between proteome and +metabolome until a more specific term is known. """ from __future__ import annotations @@ -20,43 +31,57 @@ from parce.models.graph_schema import MolecularLayer -# Anchor EFO ancestor label (lower-cased) → molecular layer. Ordered -# most-specific first: when an assay's ancestry hits several anchors, the first -# match in this mapping wins, so narrower readouts beat broader ones. -_ANCHOR_LABELS: dict[str, MolecularLayer] = { - # Transcriptome - "rna assay": MolecularLayer.TRANSCRIPTOME, - "transcription profiling assay": MolecularLayer.TRANSCRIPTOME, - "transcription profiling by high throughput sequencing": MolecularLayer.TRANSCRIPTOME, - # Epigenome (chromatin accessibility, methylation, histone marks) - "atac-seq": MolecularLayer.EPIGENOME, - "dna methylation profiling assay": MolecularLayer.EPIGENOME, - "methylation profiling assay": MolecularLayer.EPIGENOME, - "chromatin immunoprecipitation assay": MolecularLayer.EPIGENOME, - # Proteome - "proteomic profiling assay": MolecularLayer.PROTEOME, - "protein assay": MolecularLayer.PROTEOME, - "mass spectrometry assay": MolecularLayer.PROTEOME, - # Metabolome - "metabolite profiling assay": MolecularLayer.METABOLOME, - "metabolomics assay": MolecularLayer.METABOLOME, - # Genome (sequence/variation, kept last as the broadest DNA readout) - "whole genome sequencing assay": MolecularLayer.GENOME, - "genotyping assay": MolecularLayer.GENOME, - "dna sequencing": MolecularLayer.GENOME, -} +# Ordered (substring keyword, molecular layer). The first keyword present in the +# lineage (case-insensitive) wins, so this list runs most-specific → broadest: +# the transcriptome/epigenome/proteome/metabolome readouts are matched before the +# broad GENOME DNA signals, which would otherwise swallow ATAC/ChIP/methylation +# (all of which carry a generic "DNA assay" ancestor). +_ANCHOR_KEYWORDS: tuple[tuple[str, MolecularLayer], ...] = ( + # Transcriptome — covers every CELLxGENE assay: the 10x family via + # "transcription profiling", scRNA/Smart-seq/spatial via "RNA assay"/"RNA + # sequencing"/"transcriptom". + ("transcription profiling", MolecularLayer.TRANSCRIPTOME), + ("rna sequencing", MolecularLayer.TRANSCRIPTOME), + ("rna-seq", MolecularLayer.TRANSCRIPTOME), + ("rna assay", MolecularLayer.TRANSCRIPTOME), + ("transcriptom", MolecularLayer.TRANSCRIPTOME), + # Epigenome — chromatin accessibility, methylation, histone marks. Listed + # before GENOME so these DNA-branch assays don't fall through to "DNA assay". + ("atac", MolecularLayer.EPIGENOME), + ("methylation", MolecularLayer.EPIGENOME), + ("bisulfite", MolecularLayer.EPIGENOME), + ("chip-seq", MolecularLayer.EPIGENOME), + ("immunoprecipitation", MolecularLayer.EPIGENOME), + ("chromatin", MolecularLayer.EPIGENOME), + ("histone", MolecularLayer.EPIGENOME), + # Proteome. + ("proteom", MolecularLayer.PROTEOME), + ("protein assay", MolecularLayer.PROTEOME), + # Metabolome. + ("metabolom", MolecularLayer.METABOLOME), + ("metabolite", MolecularLayer.METABOLOME), + # Genome — DNA sequence/variation, the broadest DNA readout (kept last). + ("whole genome", MolecularLayer.GENOME), + ("genome sequencing", MolecularLayer.GENOME), + ("genome shotgun", MolecularLayer.GENOME), + ("genotyping", MolecularLayer.GENOME), + ("dna sequencing", MolecularLayer.GENOME), + ("exome", MolecularLayer.GENOME), + ("dna-seq", MolecularLayer.GENOME), +) def derive_molecular_layer(ancestor_labels: Iterable[str]) -> MolecularLayer: """Classify an assay from the labels of its ``is-a`` ancestors. ``ancestor_labels`` should include the assay term's own label plus its - ancestors' labels. Matching is case-insensitive and exact against the pinned - anchor labels; the first anchor (in :data:`_ANCHOR_LABELS` order) present in - the lineage wins. Returns :attr:`MolecularLayer.UNKNOWN` when none match. + ancestors' labels (the resolver appends the former). Matching is + case-insensitive substring against the pinned anchor keywords; the first + keyword (in :data:`_ANCHOR_KEYWORDS` order) found anywhere in the lineage + wins. Returns :attr:`MolecularLayer.UNKNOWN` when none match. """ - present = {label.strip().lower() for label in ancestor_labels if label} - for anchor, layer in _ANCHOR_LABELS.items(): - if anchor in present: + present = [label.strip().lower() for label in ancestor_labels if label and label.strip()] + for keyword, layer in _ANCHOR_KEYWORDS: + if any(keyword in label for label in present): return layer return MolecularLayer.UNKNOWN diff --git a/tests/test_graph_schema.py b/tests/test_graph_schema.py index 809e6cc..b57e5f0 100644 --- a/tests/test_graph_schema.py +++ b/tests/test_graph_schema.py @@ -11,6 +11,7 @@ EntityType, GraphEdge, KnowledgeGraphOutput, + MolecularLayer, SampleNode, StudyNode, ) @@ -40,10 +41,23 @@ def test_valid(self): study_id="10.1038/s41586-023-05869-0", title="A study", source="CELLxGENE", - modality="scRNA-seq", + assay="EFO:0009922", + molecular_layer=MolecularLayer.TRANSCRIPTOME, ) assert study.study_id == "10.1038/s41586-023-05869-0" assert study.source == "CELLxGENE" + assert study.assay == "EFO:0009922" + assert study.molecular_layer is MolecularLayer.TRANSCRIPTOME + + def test_modality_replaced_by_assay_and_layer(self): + """The free-text 'modality' string is gone; assay (EFO ID) + layer replace it.""" + assert "modality" not in StudyNode.model_fields + assert "assay" in StudyNode.model_fields + assert "molecular_layer" in StudyNode.model_fields + + def test_molecular_layer_defaults_to_unknown(self): + study = StudyNode(study_id="10.1234/test", title="T", source="CELLxGENE", assay="unknown") + assert study.molecular_layer is MolecularLayer.UNKNOWN def test_no_narrative_field(self): """The narrative field was removed from the canonical schema.""" @@ -55,7 +69,7 @@ def test_extra_forbidden(self): study_id="10.1234/test", title="T", source="CELLxGENE", - modality="scRNA-seq", + assay="EFO:0009922", abstract="leftover", ) @@ -65,11 +79,17 @@ def test_valid(self): ds = DatasetNode( dataset_id="abc-123", data_uri="s3://cellxgene-data-public/cell-census/h5ads/abc-123.h5ad", - assay="10x 3' v3", + assay="EFO:0009922", + molecular_layer=MolecularLayer.TRANSCRIPTOME, cell_count=50000, ) assert ds.cell_count == 50000 - assert ds.assay == "10x 3' v3" + assert ds.assay == "EFO:0009922" + assert ds.molecular_layer is MolecularLayer.TRANSCRIPTOME + + def test_molecular_layer_defaults_to_unknown(self): + ds = DatasetNode(dataset_id="x", data_uri="s3://x", assay="EFO:0009922", cell_count=1) + assert ds.molecular_layer is MolecularLayer.UNKNOWN def test_extra_forbidden(self): with pytest.raises(ValidationError): @@ -169,14 +189,16 @@ def test_roundtrip_json(self): study_id="10.1234/test", title="Test", source="CELLxGENE", - modality="scRNA-seq", + assay="EFO:0009922", + molecular_layer=MolecularLayer.TRANSCRIPTOME, ) ], datasets=[ DatasetNode( dataset_id="ds-1", data_uri="s3://bucket/ds-1.h5ad", - assay="10x 3' v3", + assay="EFO:0009922", + molecular_layer=MolecularLayer.TRANSCRIPTOME, cell_count=1000, ) ], diff --git a/tests/test_normalize.py b/tests/test_normalize.py index cb1ce9f..1edaba1 100644 --- a/tests/test_normalize.py +++ b/tests/test_normalize.py @@ -6,7 +6,7 @@ from __future__ import annotations -from parce.models.graph_schema import EntityType, KnowledgeGraphOutput +from parce.models.graph_schema import EntityType, KnowledgeGraphOutput, MolecularLayer from parce.models.raw_record import RawRecord from parce.normalize.cellxgene import CellxgeneNormalizer from parce.ontology import Facet, ResolvedTerm @@ -16,15 +16,28 @@ "Mus musculus": ResolvedTerm("NCBITaxon:10090", "Mus musculus"), } +# Both CELLxGENE assays in _RECORD are scRNA-seq → transcriptome. +_ASSAY_LAYERS = { + "EFO:0009922": MolecularLayer.TRANSCRIPTOME, + "EFO:0008931": MolecularLayer.TRANSCRIPTOME, +} + class _FakeResolver: """Deterministic, offline stand-in for the OLS-backed OntologyResolver.""" + def __init__(self) -> None: + self.layer_calls: list[tuple[str, str | None]] = [] + def resolve_term(self, text: str, facet: Facet) -> ResolvedTerm | None: if facet is Facet.ORGANISM: return _ORGANISMS.get(text) return None + def molecular_layer(self, assay_id: str, *, assay_label: str | None = None) -> MolecularLayer: + self.layer_calls.append((assay_id, assay_label)) + return _ASSAY_LAYERS.get(assay_id, MolecularLayer.UNKNOWN) + def _normalizer() -> CellxgeneNormalizer: return CellxgeneNormalizer(resolver=_FakeResolver()) @@ -107,14 +120,41 @@ def test_basic_structure(self): assert kg.studies[0].study_id == "10.1234/test" assert kg.studies[0].title == "Test Study" assert kg.studies[0].source == "CELLxGENE" - assert kg.studies[0].modality == "scRNA-seq" assert len(kg.datasets) == 2 assert kg.datasets[0].dataset_id == "ds-001" assert kg.datasets[0].data_uri == "s3://bucket/ds-001.h5ad" - assert kg.datasets[0].assay == "10x 3' v3" + assert kg.datasets[0].assay == "EFO:0009922" assert kg.datasets[1].dataset_id == "ds-002" + def test_dataset_assay_grounded_to_efo_id(self): + """Each dataset's assay is the EFO term ID, not the free-text name.""" + kg = _normalizer().normalize(_RECORD) + assert kg.datasets[0].assay == "EFO:0009922" # 10x 3' v3 + assert kg.datasets[1].assay == "EFO:0008931" # Smart-seq2 + + def test_dataset_molecular_layer_derived(self): + """molecular_layer is derived (via the resolver) for each EFO assay.""" + kg = _normalizer().normalize(_RECORD) + assert kg.datasets[0].molecular_layer is MolecularLayer.TRANSCRIPTOME + assert kg.datasets[1].molecular_layer is MolecularLayer.TRANSCRIPTOME + + def test_study_assay_is_dominant_and_layer_derived(self): + """The study's assay is its datasets' most frequent assay, with its layer.""" + kg = _normalizer().normalize(_RECORD) + # ds-001 (EFO:0009922) and ds-002 (EFO:0008931) tie 1-1; the first seen wins. + assert kg.studies[0].assay == "EFO:0009922" + assert kg.studies[0].molecular_layer is MolecularLayer.TRANSCRIPTOME + + def test_layer_derivation_memoised_per_assay(self): + """Each distinct EFO assay is layer-derived at most once (study reuses it).""" + resolver = _FakeResolver() + CellxgeneNormalizer(resolver=resolver).normalize(_RECORD) + derived_ids = [call[0] for call in resolver.layer_calls] + assert sorted(set(derived_ids)) == ["EFO:0008931", "EFO:0009922"] + # No id derived twice despite the study reusing the dominant dataset's assay. + assert len(derived_ids) == len(set(derived_ids)) + def test_study_source_from_record(self): """StudyNode.source is taken from the record, not hardcoded.""" record = _empty_record() @@ -216,6 +256,11 @@ class _NoOpResolver: def resolve_term(self, text: str, facet: Facet) -> ResolvedTerm | None: return None + def molecular_layer( + self, assay_id: str, *, assay_label: str | None = None + ) -> MolecularLayer: + return MolecularLayer.TRANSCRIPTOME + kg = CellxgeneNormalizer(resolver=_NoOpResolver()).normalize(_RECORD) species = [e for e in kg.biological_entities if e.entity_type == EntityType.SPECIES] @@ -223,3 +268,32 @@ def resolve_term(self, text: str, facet: Facet) -> ResolvedTerm | None: assert [e for e in kg.edges if e.relation_type == "STUDIES"] == [] # Non-organism entities are unaffected (they arrive pre-grounded). assert any(e.ontology_id == "UBERON:0000178" for e in kg.biological_entities) + + def test_ungrounded_assay_is_unknown_without_resolver_call(self): + """A dataset with no grounded assay gets assay='unknown', layer=UNKNOWN, + and the lineage walk is skipped entirely (no EFO id to walk).""" + record = RawRecord( + source="CELLxGENE", + study_id="10.1234/test", + title="Test Study", + payload={ + "datasets": [ + { + "dataset_id": "ds-x", + "dataset_title": "No assay", + "h5ad_uri": "s3://bucket/ds-x.h5ad", + "modality": "unknown", + "cell_count": 10, + "ontology_summary": {"organism": "Homo sapiens", "assays": []}, + } + ] + }, + ) + resolver = _FakeResolver() + kg = CellxgeneNormalizer(resolver=resolver).normalize(record) + + assert kg.datasets[0].assay == "unknown" + assert kg.datasets[0].molecular_layer is MolecularLayer.UNKNOWN + assert kg.studies[0].assay == "unknown" + assert kg.studies[0].molecular_layer is MolecularLayer.UNKNOWN + assert resolver.layer_calls == [] # non-EFO assay → no lineage walk diff --git a/tests/test_ontology_integration.py b/tests/test_ontology_integration.py index 53fc8b0..6954a14 100644 --- a/tests/test_ontology_integration.py +++ b/tests/test_ontology_integration.py @@ -7,9 +7,10 @@ They validate the network plumbing the unit tests mock: CURIE→IRI construction, double-encoding, search field parsing, and the ancestor walk. The organism -assertions are exact and reliable. The molecular_layer assertions are softer -because the anchor labels in ``parce.ontology.layers`` are provisional — see the -note there; tighten these to exact ``==`` once a real lineage is observed. +assertions are exact and reliable. The molecular_layer assertions are exact too: +the anchor keywords in ``parce.ontology.layers`` were validated against this live +EFO branch (the 10x family resolves via the 'transcription profiling' parent; +scRNA/Smart-seq via 'RNA assay'). Extend the cases here as new modalities land. """ from __future__ import annotations @@ -51,11 +52,22 @@ def test_disease_resolves_to_mondo(self, resolver): class TestLiveMolecularLayer: - def test_scrna_seq_ancestor_walk_runs(self, resolver): - # EFO:0008913 = "single cell RNA sequencing". Assert the walk produces a - # valid layer (plumbing works); log it so anchor labels can be confirmed. + def test_scrna_seq_is_transcriptome(self, resolver): + # EFO:0008913 = "single cell RNA sequencing"; reaches the 'RNA assay' anchor. layer = resolver.molecular_layer("EFO:0008913", assay_label="single cell RNA sequencing") logger.info("Derived molecular_layer for scRNA-seq: %s", layer) - assert isinstance(layer, MolecularLayer) - # Target once anchors are validated: this should be TRANSCRIPTOME. - assert layer in {MolecularLayer.TRANSCRIPTOME, MolecularLayer.UNKNOWN} + assert layer is MolecularLayer.TRANSCRIPTOME + + def test_10x_chemistry_is_transcriptome(self, resolver): + # EFO:0009922 = "10x 3' v3". The 10x family never reaches 'RNA assay'; it + # must be caught by the 'transcription profiling' parent-process anchor. + layer = resolver.molecular_layer("EFO:0009922", assay_label="10x 3' v3") + logger.info("Derived molecular_layer for 10x 3' v3: %s", layer) + assert layer is MolecularLayer.TRANSCRIPTOME + + def test_atac_seq_is_epigenome(self, resolver): + # EFO:0007045 = "ATAC-seq"; the epigenome signal lives in the term's own + # label (its ancestors collapse to a generic 'DNA assay'). + layer = resolver.molecular_layer("EFO:0007045", assay_label="ATAC-seq") + logger.info("Derived molecular_layer for ATAC-seq: %s", layer) + assert layer is MolecularLayer.EPIGENOME diff --git a/tests/test_ontology_layers.py b/tests/test_ontology_layers.py index f1a9fa1..874e32c 100644 --- a/tests/test_ontology_layers.py +++ b/tests/test_ontology_layers.py @@ -37,6 +37,39 @@ def test_blank_labels_ignored(self): def test_specific_anchor_wins_over_broad(self): """When transcriptome and genome anchors co-occur, the earlier-listed - (more specific) anchor wins deterministically.""" + (more specific) keyword wins deterministically.""" labels = ["RNA assay", "whole genome sequencing assay"] assert derive_molecular_layer(labels) is MolecularLayer.TRANSCRIPTOME + + def test_10x_transcription_profiling_branch(self): + """The 10x family never reaches 'RNA assay'; it is caught by the + 'transcription profiling' keyword on a parent-process label. This is the + real live-EFO lineage exact-label matching used to miss (→ UNKNOWN).""" + labels = [ + "10x 3' transcription profiling", + "10x transcription profiling", + "single cell library construction", + "library preparation", + ] + assert derive_molecular_layer(labels) is MolecularLayer.TRANSCRIPTOME + + def test_substring_keyword_matches_within_label(self): + """Matching is substring, not whole-label: ChIP's 'immunoprecipitation + assay' / 'chromatin' ancestor labels classify it as epigenome.""" + assert ( + derive_molecular_layer(["immunoprecipitation assay", "DNA assay"]) + is MolecularLayer.EPIGENOME + ) + + def test_epigenome_dna_assay_beats_genome(self): + """ATAC/ChIP/methylation carry a generic 'DNA assay' ancestor; the + epigenome keyword must win over the broad GENOME 'dna' signals.""" + assert ( + derive_molecular_layer(["ATAC-seq", "DNA assay", "DNA-seq"]) is MolecularLayer.EPIGENOME + ) + + def test_bare_mass_spectrometry_is_unknown(self): + """Mass spectrometry alone is ambiguous (proteome vs metabolome), so its + lineage intentionally reaches no anchor.""" + labels = ["assay by mass spectrometry", "assay by instrument", "assay"] + assert derive_molecular_layer(labels) is MolecularLayer.UNKNOWN diff --git a/tests/test_ontology_resolver.py b/tests/test_ontology_resolver.py index 6ad0d44..c194edb 100644 --- a/tests/test_ontology_resolver.py +++ b/tests/test_ontology_resolver.py @@ -155,11 +155,12 @@ def test_derives_from_ancestors(self, tmp_path): assert client.ancestors_calls == [("EFO:0009922", "efo")] def test_uses_assay_label_when_term_is_its_own_anchor(self, tmp_path): + # Some assays (e.g. ATAC-seq) carry the layer signal only in their own + # label; their EFO ancestors collapse to a generic "DNA assay". The label + # must be consulted even when no ancestor matches. client = _FakeClient(ancestors_terms=[]) # no ancestors returned - layer = _resolver(tmp_path, client).molecular_layer( - "EFO:0008931", assay_label="mass spectrometry assay" - ) - assert layer is MolecularLayer.PROTEOME + layer = _resolver(tmp_path, client).molecular_layer("EFO:0007045", assay_label="ATAC-seq") + assert layer is MolecularLayer.EPIGENOME def test_unmatched_lineage_is_unknown(self, tmp_path): client = _FakeClient(ancestors_terms=[_term("EFO:0000001", "experimental factor")]) diff --git a/tests/test_orchestration.py b/tests/test_orchestration.py index 2538abd..beecc06 100644 --- a/tests/test_orchestration.py +++ b/tests/test_orchestration.py @@ -11,17 +11,23 @@ from unittest.mock import patch from parce.main import run +from parce.models.graph_schema import MolecularLayer from parce.ontology import Facet, ResolvedTerm class _FakeResolver: - """Offline organism resolver so the real normalizer never calls OLS.""" + """Offline resolver so the real normalizer never calls OLS (organism + layer).""" def resolve_term(self, text: str, facet: Facet) -> ResolvedTerm | None: if facet is Facet.ORGANISM and text == "Homo sapiens": return ResolvedTerm("NCBITaxon:9606", "Homo sapiens") return None + def molecular_layer(self, assay_id: str, *, assay_label: str | None = None) -> MolecularLayer: + if assay_id.startswith("EFO:"): + return MolecularLayer.TRANSCRIPTOME + return MolecularLayer.UNKNOWN + _MOCK_PAPER = { "doi": "10.1234/mock", @@ -77,7 +83,13 @@ def test_full_pipeline(self, tmp_path): assert kg["studies"][0]["source"] == "CELLxGENE" # The canonical KG never stores a narrative. assert "experimental_narrative" not in kg["studies"][0] + # The free-text 'modality' is gone; assay (EFO ID) + molecular_layer replace it. + assert "modality" not in kg["studies"][0] + assert kg["studies"][0]["assay"] == "EFO:0009922" + assert kg["studies"][0]["molecular_layer"] == "transcriptome" assert len(kg["datasets"]) == 1 + assert kg["datasets"][0]["assay"] == "EFO:0009922" + assert kg["datasets"][0]["molecular_layer"] == "transcriptome" assert len(kg["biological_entities"]) > 0 assert len(kg["edges"]) > 0