PR 4: Ontology resolution stage + CELLxGENE organism wiring

docs/ARCHITECTURE.md

@@ -156,6 +156,17 @@ Avoid a free-text `modality`. Instead store, per dataset/study:
    term's `is-a` ancestors** to a small set of anchor classes. The lineage does
    the classification; we never re-string it.
 
+> **PR 4 — anchors & default.** `MolecularLayer` lives in `models/graph_schema.py`
+> (canonical-vocabulary home); the derivation is `parce.ontology.layers.
+> derive_molecular_layer`. The anchor set is keyed by EFO ancestor **label**, not
+> term ID — honouring "pin ontologies/anchors, not IDs" (OLS returns canonical
+> labels for every ancestor), and matched most-specific-first. The **no-anchor
+> default is `MolecularLayer.UNKNOWN`**. The current anchor labels are
+> *provisional* (an informed first cut not yet checked against live EFO); the
+> marked `tests/test_ontology_integration.py` is the validation harness, and PR 4b
+> (which adds the stored field) must tighten them. The derivation logic + default
+> are decided; only the exact label strings remain to be confirmed.
+
 The model sees a clean cross-modality categorical (`molecular_layer`) plus a
 precise term (`assay`) — both controlled, no free text on either.
 
@@ -170,6 +181,18 @@ precise term (`assay`) — both controlled, no free text on either.
 - **LLM** — fallback only, for strings the deterministic resolvers can't
   confidently map.
 
+> **PR 4 — what shipped, and why OLS4-only (for now).** The stage is implemented
+> in `parce/ontology/` as `OntologyResolver` (cache → OLS4 exact-then-fuzzy →
+> optional LLM fallback) over the registry above. **Only OLS4 is wired**;
+> text2term/Zooma are deferred to PR 5, where GEO's messy `characteristics_ch1`
+> strings actually need fuzzy batch mapping. CELLxGENE already ships ontology IDs
+> for tissue/disease/assay, so the only free-text grounding needed today is the
+> organism string → NCBITaxon. The **LLM fallback is a pluggable `Callable`
+> (default `None`)**, not a hard dependency — this keeps `ontology` free of any
+> `parce.agent`/Azure import (dependency direction holds); PR 5 supplies the
+> extraction agent as the callback. Resolutions are memoised to an on-disk cache
+> (negative results included) so runs are reproducible and don't re-query.
+
 Template to follow rather than reinvent: **SDRF / MAGE-TAB** (and
 **SDRF-Proteomics**) already specify per-sample, ontology-annotated experiment
 description and *which ontology per column* — almost exactly the `SampleNode` +
@@ -205,12 +228,15 @@ that already ship SDRF.
 - **Sample granularity for CELLxGENE.** Census is per-cell/dataset, not
   per-sample in the GEO sense. Defer mapping cxg to `SampleNode` until needed;
   keep it dataset-level for now.
-- **`molecular_layer` anchor set.** The exact EFO ancestor classes that define
-  each coarse layer need pinning during PR4 (and a default for assays whose
-  lineage doesn't reach an anchor).
-- **Ontology resolver dependency.** OLS4 REST client (needed anyway for the
-  lineage walk) vs. adding text2term/Zooma — prefer the lightest combination
-  that covers messy GEO strings; decide in PR4.
+- **`molecular_layer` anchor set.** *Partly resolved (PR 4):* the derivation
+  mechanism and the no-anchor default (`UNKNOWN`) are pinned, and the anchors are
+  keyed by EFO **label**. **Still open:** the exact label strings are provisional
+  and unvalidated against live EFO — confirm via the marked integration test and
+  tighten in PR 4b.
+- **Ontology resolver dependency.** *Resolved (PR 4):* **OLS4 REST only** for
+  now (it covers organism grounding + the lineage walk). text2term/Zooma are
+  deferred to PR 5, where GEO's messy `characteristics_ch1` strings need fuzzy
+  batch mapping — add the lightest option that covers them then.
 - **Graph persistence/export format** for the modeling step (per-study context +
   sample manifest + URIs). Specified in a later PR.
 - **Multi-omics is core, not optional.** CELLxGENE alone cannot carry the

docs/ROADMAP.md

@@ -8,14 +8,18 @@ protocol and [ARCHITECTURE.md](ARCHITECTURE.md) for the design.
 
 ## ▶ Next up
 
-**PR 4 — Ontology resolver.** Shared `ontology/` stage (see ARCHITECTURE §5). Pin
-the **facet → ontology registry** as a constant (EFO, UBERON, MONDO, NCBITaxon,
-ChEBI, PSI-MS, EDAM). Implement free-text → term resolution (OLS4 REST +
-text2term/Zooma, on-disk cache; LLM fallback) and the **`molecular_layer`
-derivation** (walk EFO `is-a` ancestors to anchor classes). Decide the anchor set
-+ the no-anchor default. Wire into normalizers — start by replacing the hardcoded
-`_ORGANISM_ONTOLOGY` map in `normalize/cellxgene.py`. Resolve IDs at runtime via
-OLS — do not hardcode term IDs.
+**PR 4b — Schema refinement: EFO `assay` term + stored `molecular_layer`.** PR 4
+shipped the ontology *stage* (resolver, registry, OLS4 client, cache,
+`molecular_layer` **derivation**) but did not change the canonical schema. This
+PR consumes it: on `StudyNode`/`DatasetNode`, replace the free-text `modality`
+string with an EFO `assay` **term ID** (resolved via `OntologyResolver`) plus a
+stored `molecular_layer` enum field (derived via `OntologyResolver.molecular_
+layer`). Wire both into `CellxgeneNormalizer` (it already resolves organism;
+extend to ground the assay string and derive the layer). Migrate the schema +
+all tests. `MolecularLayer` already lives in `models/graph_schema.py`. **Gotcha:**
+the provisional anchor labels in `ontology/layers.py` are unvalidated against live
+EFO — run `tests/test_ontology_integration.py` and tighten them first, else every
+assay derives `UNKNOWN`.
 
 ---
 
@@ -36,14 +40,22 @@ Each PR is one branch, one focused scope, green CI, and a roadmap update.
   `models/narrative.py`, the `NarrativeOutput` schema, and the step-2 block in
   `main.py`). Dropped cell-type extraction. Removed the `parce.tools.*` mypy
   exemption as those modules moved under `sources/`.
-- [ ] **PR 4 — Ontology resolver.** Shared `ontology/` stage (see ARCHITECTURE
-  §5). Pin the **facet → ontology registry** as a constant (EFO, UBERON, MONDO,
-  NCBITaxon, ChEBI, PSI-MS, EDAM). Implement free-text → term resolution
-  (OLS4 REST + text2term/Zooma, on-disk cache; LLM fallback) and the
-  **`molecular_layer` derivation** (walk EFO `is-a` ancestors to anchor classes).
-  Decide the anchor set + the no-anchor default. Wire into normalizers (replace
-  the hardcoded `_ORGANISM_ONTOLOGY` map in `normalize/cellxgene.py`). Resolve
-  IDs at runtime via OLS — do not hardcode term IDs. *(Next up.)*
+- [x] **PR 4 — Ontology resolver (stage + organism wiring).** Shared `ontology/`
+  stage (see ARCHITECTURE §5): pinned the **facet → ontology registry** constant
+  (EFO/OBI/PSI-MS, UBERON, MONDO, NCBITaxon, ChEBI, EDAM); `OlsClient` (OLS4 REST,
+  retry-wrapped, injectable); on-disk `ResolutionCache` (negative results cached);
+  `OntologyResolver` (cache → OLS exact-then-fuzzy → pluggable LLM-fallback hook,
+  default off); and the **`molecular_layer` derivation** (EFO `is-a` ancestor walk
+  → pinned anchor labels; no-anchor default `UNKNOWN`). Replaced the hardcoded
+  `_ORGANISM_ONTOLOGY` map in `normalize/cellxgene.py` with runtime NCBITaxon
+  resolution. **Decided: OLS4-only** for now (text2term/Zooma deferred to GEO/PR 5
+  — CELLxGENE ships IDs, only organism free-text needed grounding). IDs resolved
+  at runtime; none hardcoded. *Split from the original PR 4: the schema change
+  (store the EFO assay term + `molecular_layer`) became PR 4b.*
+- [ ] **PR 4b — Schema refinement.** Replace free-text `modality` with an EFO
+  `assay` term ID + a stored `molecular_layer` enum on the study/dataset nodes;
+  wire the resolver's assay grounding + layer derivation into the normalizer;
+  migrate the schema and tests. *(Next up — see top of file.)*
 - [ ] **PR 5 — GEO extraction agent (vertical slice).** GEO adapter
   (E-utilities/GEOparse) + Azure extraction normalizer emitting the canonical
   schema via `response_format`; extract sample covariates from
@@ -73,6 +85,69 @@ Each PR is one branch, one focused scope, green CI, and a roadmap update.
 Newest first. One entry per working session: what changed, decisions made, and
 what the next session should know. Keep entries short and factual.
 
+### 2026-06-25 — PR 4: Ontology resolver (stage + organism wiring)
+
+- Branch `pr4-ontology-resolver` off `main` (64ed4f4).
+- **Split decision.** The roadmap's PR 4 bundled the resolver stage *and* the
+  schema change (free-text `modality` → EFO `assay` term + stored
+  `molecular_layer`). Shipped the stage + organism wiring here; the schema
+  migration is now **PR 4b** (new ▶ Next up). Rationale: the stage is a coherent,
+  fully-tested unit; the schema swap touches the canonical models + every
+  CELLxGENE test and is cleaner as its own focused PR.
+- **New package `src/parce/ontology/`** (stable core, fully mypy-checked):
+  - `registry.py` — `Facet` enum + `FACET_ONTOLOGY` constant. Pins *ontologies,
+    not IDs*: EFO (assay; fallbacks OBI then PSI-MS), UBERON, MONDO, NCBITaxon,
+    ChEBI, EDAM. `FacetBinding.ontologies()` gives primary→fallback order.
+  - `ols.py` — `OlsClient` for OLS4 REST: `search` (free text → class hits) and
+    `ancestors` (hierarchical/`is-a`). `http` getter is injectable (offline
+    tests); requests wrapped in `sources._retry.with_retries`. `obo_id_to_iri`
+    builds EFO + generic OBO-PURL IRIs and the path is double-URL-encoded.
+  - `cache.py` — `ResolutionCache`, JSON-on-disk, atomic writes, lock-guarded.
+    **Caches negative results** (a *miss* vs a cached-`None` are distinguished via
+    `get`'s first return value) so unresolvable strings aren't re-queried.
+  - `layers.py` — `derive_molecular_layer(ancestor_labels)`: pure function
+    matching EFO ancestor **labels** (not IDs) against a pinned anchor set;
+    most-specific-first; no-anchor default `MolecularLayer.UNKNOWN`.
+  - `resolver.py` — `OntologyResolver`: `resolve_term(text, facet)` (cache → OLS
+    exact-then-fuzzy across the facet's ontologies → optional LLM fallback) and
+    `molecular_layer(assay_id, assay_label=None)`. All collaborators injectable;
+    default cache is lazy so construction touches neither disk nor network.
+  - `base.py` — `ResolvedTerm` value type + the narrow `TermResolver` Protocol
+    normalizers depend on.
+- **`MolecularLayer` StrEnum added to `models/graph_schema.py`** (its home, since
+  `models` depends on nothing and it becomes a node field in PR 4b). Not yet a
+  stored field anywhere.
+- **Normalizer wired:** `CellxgeneNormalizer(resolver=...)` now grounds the bare
+  organism string to NCBITaxon via the resolver (replacing `_ORGANISM_ONTOLOGY`).
+  Tissue/disease/assay still use the IDs Census already ships — only organism was
+  free text. An organism that fails to resolve is **skipped** (no ungrounded node
+  emitted), logged at WARNING.
+- **Decisions / rationale:**
+  - **OLS4-only** (open question §7 resolved): text2term/Zooma deferred to GEO
+    (PR 5), where the messy `characteristics_ch1` strings actually appear.
+  - **LLM fallback is a pluggable `Callable`, default `None`** — keeps `ontology`
+    free of any `parce.agent`/Azure import (dependency direction preserved); PR 5
+    wires the extraction agent in as the callback.
+  - **Anchor set is keyed by EFO label, not term ID** (honours "pin
+    ontologies/anchors, not IDs"). **PROVISIONAL** — unvalidated against live EFO;
+    `tests/test_ontology_integration.py` (marked, live OLS) is the validation
+    harness. PR 4b must run it and correct labels, else assays derive `UNKNOWN`.
+  - **`ontology` reuses `sources._retry`** (leaf util, no cycle) per CLAUDE.md.
+  - Resolver config (OLS base URL, cache dir) is constructor params, **not**
+    `Settings` — avoids the unit-test `Settings()` hermeticity trap; could move to
+    Settings later.
+- **Tests:** offline unit suites for registry / cache / OLS (fake HTTP getter) /
+  layers / resolver (fake client + real cache on tmp_path); `test_normalize.py`
+  and `test_orchestration.py` inject a fake resolver so they stay offline. New
+  marked integration test for live OLS (organism exact; molecular_layer plumbing).
+- **mypy:** no new exemptions; `parce.agent.*` exemption stays (PR 5). 24 source
+  files checked, clean.
+- **Gates green, incl. hermetic run (worktree has no `.env`):** ruff check,
+  ruff format --check (40 files), mypy, **121 unit tests** (was 47+retry). No dep
+  changes (stdlib + `requests`), so no `uv.lock` change.
+- **Next session:** PR 4b (schema refinement) — and validate the provisional
+  `molecular_layer` anchors against live EFO first.
+
 ### 2026-06-24 — Generic retry helper (resilience follow-up to PR 3)
 
 - Branch `add-generic-retry-helper`. Not a roadmap PR; **closes the retry

src/parce/models/graph_schema.py

@@ -37,6 +37,28 @@ class EntityType(StrEnum):
     ASSAY = "Assay"
 
 
+class MolecularLayer(StrEnum):
+    """Coarse molecular readout an assay measures — a controlled, cross-modality
+    categorical the downstream model can condition on.
+
+    Derived deterministically by walking an EFO ``assay`` term's ``is-a``
+    ancestors to a pinned anchor class (see :mod:`parce.ontology.layers`); never
+    re-strung from free text. ``UNKNOWN`` is the no-anchor default for assays
+    whose lineage reaches none of the anchors.
+
+    Defined here, the canonical-vocabulary home, so the ontology stage can target
+    it now. It becomes a stored field on the study/dataset nodes in the
+    schema-refinement PR (see docs/ROADMAP.md).
+    """
+
+    GENOME = "genome"
+    EPIGENOME = "epigenome"
+    TRANSCRIPTOME = "transcriptome"
+    PROTEOME = "proteome"
+    METABOLOME = "metabolome"
+    UNKNOWN = "unknown"
+
+
 class StudyNode(BaseModel):
     """A study (publication or repository accession), source-agnostic.
 

src/parce/normalize/cellxgene.py

@@ -1,15 +1,17 @@
 """Deterministic normalizer: a CELLxGENE ``RawRecord`` → canonical KG nodes.
 
 No LLM is involved: CELLxGENE Census already ships ontology-grounded terms, so
-this is a pure structural mapping. Two design rules show up directly here:
+this is a pure structural mapping. Three design rules show up directly here:
 
 * **Cell type is never consumed** — the adapter does not even read it
   (data-inferred → leakage; see docs/ARCHITECTURE.md §1).
 * **Census is dataset-level**, not per-sample in the GEO sense, so no
   ``SampleNode`` records are emitted yet (open question in ARCHITECTURE §7).
-
-The free-text → ontology-ID step (here, organism string → NCBITaxon) is a
-hardcoded map for now; it becomes the shared OntologyResolver stage in PR 4.
+* **Organism strings are grounded via the shared OntologyResolver**, not a
+  hardcoded map. Tissue/disease/assay already arrive as ontology IDs from
+  Census, so only the bare organism string needs runtime resolution (to
+  NCBITaxon, via OLS). An organism that fails to resolve is skipped rather than
+  emitted ungrounded.
 """
 
 from __future__ import annotations
@@ -26,22 +28,14 @@
     StudyNode,
 )
 from parce.models.raw_record import RawRecord
+from parce.ontology import Facet, OntologyResolver, ResolvedTerm, TermResolver
 
 logger = logging.getLogger(__name__)
 
 # High-level study modality for everything CELLxGENE ingests. (Refined into an
-# EFO ``assay`` term + derived ``molecular_layer`` in PR 4.)
+# EFO ``assay`` term + derived ``molecular_layer`` in the schema-refinement PR.)
 _STUDY_MODALITY = "scRNA-seq"
 
-# Organism free-text (as Census returns it) → (NCBITaxon ID, canonical name).
-# Stand-in for the PR 4 OntologyResolver.
-_ORGANISM_ONTOLOGY: dict[str, tuple[str, str]] = {
-    "Homo sapiens": ("NCBITaxon:9606", "Homo sapiens"),
-    "Mus musculus": ("NCBITaxon:10090", "Mus musculus"),
-    "homo_sapiens": ("NCBITaxon:9606", "Homo sapiens"),
-    "mus_musculus": ("NCBITaxon:10090", "Mus musculus"),
-}
-
 # Ontology categories that become design-context entities. Cell types are
 # deliberately absent (data-inferred → leakage).
 _CATEGORY_TO_ENTITY_TYPE: dict[str, EntityType] = {
@@ -58,7 +52,15 @@
 
 
 class CellxgeneNormalizer:
-    """:class:`~parce.normalize.base.Normalizer` for CELLxGENE ``RawRecord``s."""
+    """:class:`~parce.normalize.base.Normalizer` for CELLxGENE ``RawRecord``s.
+
+    Takes a :class:`~parce.ontology.base.TermResolver` (default: a real
+    :class:`~parce.ontology.resolver.OntologyResolver`) used to ground organism
+    strings. Inject a deterministic fake to keep unit tests offline.
+    """
+
+    def __init__(self, resolver: TermResolver | None = None) -> None:
+        self._resolver: TermResolver = resolver if resolver is not None else OntologyResolver()
 
     def normalize(self, record: RawRecord) -> KnowledgeGraphOutput:
         """Assemble the canonical single-study subgraph for one CELLxGENE study."""
@@ -74,7 +76,10 @@ def normalize(self, record: RawRecord) -> KnowledgeGraphOutput:
         datasets: list[DatasetNode] = []
         edges: list[GraphEdge] = []
         entity_registry: dict[str, BiologicalEntityNode] = {}
-        species_seen: set[str] = set()
+        # Resolved species (by NCBITaxon ID), and a per-record memo of organism
+        # string → resolution so the same string is grounded at most once.
+        species_ids: set[str] = set()
+        organism_cache: dict[str, ResolvedTerm | None] = {}
 
         for ds in record.payload.get("datasets", []):
             dataset_id = ds["dataset_id"]
@@ -98,15 +103,15 @@ def normalize(self, record: RawRecord) -> KnowledgeGraphOutput:
 
             ontology: dict[str, Any] = ds.get("ontology_summary", {})
 
-            # Register species from the organism field.
-            organism_key = ontology.get("organism", "unknown")
-            if organism_key in _ORGANISM_ONTOLOGY and organism_key not in species_seen:
-                ont_id, name = _ORGANISM_ONTOLOGY[organism_key]
-                species_seen.add(organism_key)
-                entity_registry[ont_id] = BiologicalEntityNode(
+            # Ground the organism string to a NCBITaxon term via the resolver.
+            organism_text = ontology.get("organism", "unknown")
+            species = self._resolve_organism(organism_text, organism_cache)
+            if species is not None and species.ontology_id not in species_ids:
+                species_ids.add(species.ontology_id)
+                entity_registry[species.ontology_id] = BiologicalEntityNode(
                     entity_type=EntityType.SPECIES,
-                    ontology_id=ont_id,
-                    name=name,
+                    ontology_id=species.ontology_id,
+                    name=species.name,
                 )
 
             # Register entities and create edges per design-context category.
@@ -132,8 +137,7 @@ def normalize(self, record: RawRecord) -> KnowledgeGraphOutput:
                     )
 
         # Study → Species edges.
-        for organism_key in species_seen:
-            species_id = _ORGANISM_ONTOLOGY[organism_key][0]
+        for species_id in species_ids:
             edges.append(
                 GraphEdge(
                     source_id=study_id,
@@ -158,3 +162,11 @@ def normalize(self, record: RawRecord) -> KnowledgeGraphOutput:
             len(kg.edges),
         )
         return kg
+
+    def _resolve_organism(
+        self, organism_text: str, memo: dict[str, ResolvedTerm | None]
+    ) -> ResolvedTerm | None:
+        """Resolve an organism string to a NCBITaxon term, memoised per record."""
+        if organism_text not in memo:
+            memo[organism_text] = self._resolver.resolve_term(organism_text, Facet.ORGANISM)
+        return memo[organism_text]