PR 4b: schema refinement — EFO assay term + stored molecular_layer

docs/ARCHITECTURE.md

@@ -93,16 +93,19 @@ Source-agnostic nodes (Pydantic v2, `extra="forbid"`). Implemented in PR 2 in
 `models/graph_schema.py`:
 
 - `StudyNode` — `study_id` (DOI/accession), `title`, `source` (provenance),
-  `modality`. *(No `experimental_narrative`; that field is removed.)* Raw free
-  text (abstracts, full descriptions) is **not** stored on the node — it belongs
-  to the per-source `RawRecord`; the canonical node holds only normalized,
-  design-describing fields. *(Target, §5: this free-text `modality` is refined
-  into an EFO `assay` term plus a derived coarse `molecular_layer` enum — PR 4.)*
-- `DatasetNode` — `dataset_id`, `data_uri`, `assay` (to be grounded to an EFO
-  term ID, see §5), `cell_count`/size. Its parent study is a typed
+  `assay`, `molecular_layer`. *(No `experimental_narrative`; that field is
+  removed.)* Raw free text (abstracts, full descriptions) is **not** stored on
+  the node — it belongs to the per-source `RawRecord`; the canonical node holds
+  only normalized, design-describing fields. *(PR 4b: the free-text `modality`
+  string is gone — replaced by the EFO `assay` term ID of the study's
+  representative/dominant assay plus the derived coarse `molecular_layer` enum,
+  per §5.)*
+- `DatasetNode` — `dataset_id`, `data_uri`, `assay` (an **EFO term ID**, §5),
+  `molecular_layer`, `cell_count`/size. Its parent study is a typed
   `EXTRACTED_FROM` **edge**, not a stored foreign-key field. *(Decision, PR 2:
   containment/relationships live on edges only; duplicating them as node fields
-  invites drift and gives two sources of truth.)*
+  invites drift and gives two sources of truth.)* *(PR 4b: `assay` changed from a
+  free-text name to the grounded EFO ID, and `molecular_layer` was added.)*
 - `SampleNode` — `sample_id`, `data_uri`, and **design covariates**: `condition`,
   `perturbation`, `timepoint`, `subject`, `organism`. (Reintroduced; the prior
   schema was dataset-level only.) All covariates are optional — different
@@ -156,16 +159,33 @@ Avoid a free-text `modality`. Instead store, per dataset/study:
    term's `is-a` ancestors** to a small set of anchor classes. The lineage does
    the classification; we never re-string it.
 
-> **PR 4 — anchors & default.** `MolecularLayer` lives in `models/graph_schema.py`
-> (canonical-vocabulary home); the derivation is `parce.ontology.layers.
-> derive_molecular_layer`. The anchor set is keyed by EFO ancestor **label**, not
-> term ID — honouring "pin ontologies/anchors, not IDs" (OLS returns canonical
-> labels for every ancestor), and matched most-specific-first. The **no-anchor
-> default is `MolecularLayer.UNKNOWN`**. The current anchor labels are
-> *provisional* (an informed first cut not yet checked against live EFO); the
-> marked `tests/test_ontology_integration.py` is the validation harness, and PR 4b
-> (which adds the stored field) must tighten them. The derivation logic + default
-> are decided; only the exact label strings remain to be confirmed.
+> **PR 4 / 4b — anchors, default & stored fields.** `MolecularLayer` lives in
+> `models/graph_schema.py` (canonical-vocabulary home); the derivation is
+> `parce.ontology.layers.derive_molecular_layer`. The anchor set is keyed by EFO
+> ancestor **labels/keywords**, not term IDs — honouring "pin ontologies/anchors,
+> not IDs" (OLS returns canonical labels for every ancestor) — and matched
+> most-specific-first. The **no-anchor default is `MolecularLayer.UNKNOWN`**.
+>
+> **Decision (PR 4b): match keywords by case-insensitive *substring*, not exact
+> whole-label.** Validating the PR 4 anchors against live EFO showed the assay
+> branch is too inconsistent for exact matching: the 10x family — the bulk of
+> CELLxGENE — never reaches `RNA assay`, announcing itself only via a parent
+> `…transcription profiling` label; ATAC-seq/ChIP-seq/methylation/WGS all collapse
+> to a generic `DNA assay` ancestor whose distinguishing signal survives only in
+> the term's own label. Ordered substring keywords (transcriptome/epigenome/
+> proteome/metabolome before the broad GENOME `DNA` signals) classify all of these;
+> exact full-label matching derived `UNKNOWN` for 10x. The keyword set is validated
+> against live EFO by the marked `tests/test_ontology_integration.py`. Genuinely
+> ambiguous lineages stay `UNKNOWN` on purpose — bare "mass spectrometry" (proteome
+> vs metabolome) and multi-omic EFO terms (e.g. mCT-seq carries both `RNA assay`
+> and `DNA assay` ancestors).
+>
+> **PR 4b stores the result:** `StudyNode`/`DatasetNode` now carry the EFO `assay`
+> term ID + `molecular_layer`. For CELLxGENE the assay ID is *taken from Census's
+> already-grounded payload* (not re-resolved via OLS — Census is authoritative for
+> its own data and re-resolving could drift); the resolver is used only to walk the
+> EFO lineage for the layer. Other (free-text) sources resolve the assay string via
+> `OntologyResolver` first, then derive the layer from the resulting ID.
 
 The model sees a clean cross-modality categorical (`molecular_layer`) plus a
 precise term (`assay`) — both controlled, no free text on either.
@@ -228,11 +248,13 @@ that already ship SDRF.
 - **Sample granularity for CELLxGENE.** Census is per-cell/dataset, not
   per-sample in the GEO sense. Defer mapping cxg to `SampleNode` until needed;
   keep it dataset-level for now.
-- **`molecular_layer` anchor set.** *Partly resolved (PR 4):* the derivation
-  mechanism and the no-anchor default (`UNKNOWN`) are pinned, and the anchors are
-  keyed by EFO **label**. **Still open:** the exact label strings are provisional
-  and unvalidated against live EFO — confirm via the marked integration test and
-  tighten in PR 4b.
+- **`molecular_layer` anchor set.** *Resolved (PR 4b):* anchors are ordered,
+  case-insensitive **substring keywords** over EFO ancestor (and own-) labels,
+  validated against live EFO via the marked integration test — the 10x family,
+  scRNA/Smart-seq, spatial, ATAC/ChIP/methylation, WGS and metabolomics all
+  classify correctly; bare mass-spec and multi-omic terms stay `UNKNOWN` by
+  design. **Still open:** extend the keyword set as GEO/PRIDE bring bulk RNA-seq,
+  proteomics and other modalities with their own lineages.
 - **Ontology resolver dependency.** *Resolved (PR 4):* **OLS4 REST only** for
   now (it covers organism grounding + the lineage walk). text2term/Zooma are
   deferred to PR 5, where GEO's messy `characteristics_ch1` strings need fuzzy

docs/ROADMAP.md

@@ -8,18 +8,22 @@ protocol and [ARCHITECTURE.md](ARCHITECTURE.md) for the design.
 
 ## ▶ Next up
 
-**PR 4b — Schema refinement: EFO `assay` term + stored `molecular_layer`.** PR 4
-shipped the ontology *stage* (resolver, registry, OLS4 client, cache,
-`molecular_layer` **derivation**) but did not change the canonical schema. This
-PR consumes it: on `StudyNode`/`DatasetNode`, replace the free-text `modality`
-string with an EFO `assay` **term ID** (resolved via `OntologyResolver`) plus a
-stored `molecular_layer` enum field (derived via `OntologyResolver.molecular_
-layer`). Wire both into `CellxgeneNormalizer` (it already resolves organism;
-extend to ground the assay string and derive the layer). Migrate the schema +
-all tests. `MolecularLayer` already lives in `models/graph_schema.py`. **Gotcha:**
-the provisional anchor labels in `ontology/layers.py` are unvalidated against live
-EFO — run `tests/test_ontology_integration.py` and tighten them first, else every
-assay derives `UNKNOWN`.
+**PR 5 — GEO extraction agent (vertical slice).** First source whose metadata is
+*unstructured* free text, so the first to use the LLM. Add a GEO adapter
+(NCBI E-utilities / GEOparse) emitting a `RawRecord`, and an **Azure extraction
+normalizer** that fills the canonical schema via `response_format` (structured
+output only — never prose). Extract sample-level **design** covariates from
+`characteristics_ch1` (`condition`, `perturbation`, `timepoint`, `subject`,
+`organism`) into `SampleNode`s — never data-inferred annotations. Ground the
+extracted free-text facets through the **existing `OntologyResolver`** (organism
+→ NCBITaxon, assay → EFO + `molecular_layer`, tissue → UBERON, disease → MONDO),
+and **supply the agent as the resolver's LLM-fallback callback** for strings OLS
+can't map (the hook already exists, default off). Mark live tests
+`@pytest.mark.integration`; keep unit tests offline by mocking the Azure client.
+**Remove the `parce.agent.*` mypy exemption** in `pyproject.toml` once the agent
+moves to the normalizer interface. **Blocker risk:** needs Azure creds + an
+`az login` session; if absent, build/unit-test the deterministic scaffolding and
+log the integration boundary as a blocker rather than working around it.
 
 ---
 
@@ -52,15 +56,19 @@ Each PR is one branch, one focused scope, green CI, and a roadmap update.
   — CELLxGENE ships IDs, only organism free-text needed grounding). IDs resolved
   at runtime; none hardcoded. *Split from the original PR 4: the schema change
   (store the EFO assay term + `molecular_layer`) became PR 4b.*
-- [ ] **PR 4b — Schema refinement.** Replace free-text `modality` with an EFO
-  `assay` term ID + a stored `molecular_layer` enum on the study/dataset nodes;
-  wire the resolver's assay grounding + layer derivation into the normalizer;
-  migrate the schema and tests. *(Next up — see top of file.)*
+- [x] **PR 4b — Schema refinement.** Replaced free-text `modality` with an EFO
+  `assay` term ID + a stored `molecular_layer` enum on `StudyNode`/`DatasetNode`;
+  wired assay grounding (taken from Census's already-grounded payload) + layer
+  derivation into `CellxgeneNormalizer` via a new `OntologyService` contract;
+  migrated the schema and all tests. **Switched `molecular_layer` matching from
+  exact EFO labels to ordered substring keywords** after validating against live
+  EFO (the 10x family never reaches `RNA assay`); ambiguous lineages (bare
+  mass-spec, multi-omic terms) stay `UNKNOWN` by design.
 - [ ] **PR 5 — GEO extraction agent (vertical slice).** GEO adapter
   (E-utilities/GEOparse) + Azure extraction normalizer emitting the canonical
   schema via `response_format`; extract sample covariates from
   `characteristics_ch1`. Integration test (marked). This is the agent's real
-  job; remove the `parce.agent.*` mypy exemption.
+  job; remove the `parce.agent.*` mypy exemption. *(Next up — see top of file.)*
 - [ ] **PR 6 — Cross-source KG merge.** Merge CELLxGENE + GEO into one graph
   linked through shared ontology entities; dedup; provenance on edges. Assert a
   cross-source edge exists in tests.
@@ -85,6 +93,47 @@ Each PR is one branch, one focused scope, green CI, and a roadmap update.
 Newest first. One entry per working session: what changed, decisions made, and
 what the next session should know. Keep entries short and factual.
 
+### 2026-06-26 — PR 4b: Schema refinement (EFO assay term + stored molecular_layer)
+
+- Branch `pr4b-schema-refinement` off `main` (35a28ce, the PR 4 merge). Note: the
+  scheduled session started on a stale worktree whose `main` predated the PR 4
+  merge; `gh` showed PR #7 already MERGED, so PR 4 was done and **PR 4b was the
+  real ▶ Next up**. Re-based onto `origin/main` before starting.
+- **Schema (`models/graph_schema.py`).** Dropped `StudyNode.modality`. Both
+  `StudyNode` and `DatasetNode` now carry `assay` (EFO term ID, required) +
+  `molecular_layer` (`MolecularLayer`, default `UNKNOWN`). `DatasetNode.assay`
+  changed meaning from free-text name → grounded EFO ID.
+- **Normalizer (`normalize/cellxgene.py`).** Per dataset: pick the grounded EFO
+  assay ID from Census's payload (`ontology_summary.assays`, matched to the
+  dominant `modality` *name*) and derive `molecular_layer` from it; `StudyNode`
+  built at the end with the **most-frequent** dataset assay as its representative
+  + that assay's layer. Layer derivation memoised per assay ID; only `EFO:` IDs
+  are walked (an ungrounded `unknown` assay → `UNKNOWN`, no network call).
+- **Decision — assay taken from payload, not re-resolved.** CELLxGENE ships the
+  EFO assay ID; Census is authoritative for its own data and re-resolving via OLS
+  could drift, so the resolver is used *only* for the lineage walk. Free-text
+  sources (GEO/PRIDE) will resolve the assay string first. (ARCHITECTURE §5.)
+- **Decision — `molecular_layer` matching: exact labels → substring keywords.**
+  Probing live EFO exposed the PR 4 gotcha for real: `EFO:0009922` (10x 3' v3, the
+  bulk of CELLxGENE) never reaches `RNA assay` — it sits under `…transcription
+  profiling`/`library preparation`; ATAC/ChIP/methylation/WGS all collapse to a
+  generic `DNA assay` ancestor. Rewrote `ontology/layers.py` to ordered,
+  case-insensitive **substring keywords** (transcriptome/epigenome/proteome/
+  metabolome before the broad GENOME `DNA` signals), validated against live EFO.
+  Bare mass-spec and multi-omic terms stay `UNKNOWN` by design (genuinely
+  ambiguous). The marked integration test now asserts exact layers (10x + scRNA →
+  TRANSCRIPTOME, ATAC → EPIGENOME) and **passed live** this session.
+- **New contract `OntologyService`** (`ontology/base.py`, exported) = `TermResolver`
+  + `molecular_layer`; the normalizer depends on it so offline fakes inject both
+  methods. `base.py` now imports `MolecularLayer` (models is a leaf; no cycle).
+- **Gates green incl. hermetic no-`.env` run:** ruff check, ruff format --check
+  (40 files), mypy (24 files), **133 unit tests**. Live OLS integration suite (6)
+  also passed. No dep changes; no new mypy exemptions.
+- **Next session:** PR 5 (GEO extraction agent) — first LLM/Azure source; needs
+  credentials. Wire the GEO free-text facets through the existing resolver and
+  supply the agent as its LLM-fallback callback. If Azure creds are missing, stop
+  at the integration boundary and log the blocker.
+
 ### 2026-06-25 — PR 4: Ontology resolver (stage + organism wiring)
 
 - Branch `pr4-ontology-resolver` off `main` (64ed4f4).

src/parce/models/graph_schema.py

@@ -77,9 +77,21 @@ class StudyNode(BaseModel):
         ...,
         description="Provenance of the study record (e.g. 'CELLxGENE', 'GEO', 'PRIDE').",
     )
-    modality: str = Field(
+    assay: str = Field(
         ...,
-        description="High-level assay modality of the study (e.g. 'scRNA-seq', 'proteomics').",
+        description=(
+            "EFO term ID of the study's representative (dominant) assay "
+            "(e.g. 'EFO:0009922' for 10x 3' v3); 'unknown' if ungrounded. Replaces "
+            "the former free-text 'modality' string — see docs/ARCHITECTURE.md §5."
+        ),
+    )
+    molecular_layer: MolecularLayer = Field(
+        default=MolecularLayer.UNKNOWN,
+        description=(
+            "Coarse molecular readout, derived from the assay's EFO 'is-a' "
+            "lineage (never re-strung from text). 'unknown' when the lineage "
+            "reaches no anchor."
+        ),
     )
 
 
@@ -95,7 +107,18 @@ class DatasetNode(BaseModel):
     )
     assay: str = Field(
         ...,
-        description="Specific assay/technology (e.g. \"10x 3' v3\", 'Smart-seq2').",
+        description=(
+            "EFO term ID of the dataset's assay/technology (e.g. 'EFO:0009922' for "
+            "10x 3' v3); 'unknown' if ungrounded. Fine-grained: every 10x chemistry "
+            "is its own term."
+        ),
+    )
+    molecular_layer: MolecularLayer = Field(
+        default=MolecularLayer.UNKNOWN,
+        description=(
+            "Coarse molecular readout derived from the assay's EFO 'is-a' lineage "
+            "(see docs/ARCHITECTURE.md §5). 'unknown' when no anchor is reached."
+        ),
     )
     cell_count: int = Field(..., description="Number of cells (or rows) in the dataset.")