PR 5: GEO extraction agent (vertical slice)

.env.example

@@ -8,3 +8,9 @@ AZURE_AI_PROJECT_ENDPOINT=https://<your-resource>.services.ai.azure.com/api/proj
 # The model deployment name from your Foundry project's model catalog.
 # Examples: mistral-large, gpt-4o, DeepSeek-R1, Meta-Llama-3-70B
 AZURE_AI_MODEL_DEPLOYMENT_NAME=gpt-4o
+
+# Optional NCBI E-utilities / GEO courtesy parameters. NCBI asks callers to
+# identify themselves; supplying these raises rate limits. Both are optional —
+# GEO fetches work without them.
+# NCBI_EMAIL=you@example.com
+# NCBI_API_KEY=<your-ncbi-api-key>

docs/ARCHITECTURE.md

@@ -82,7 +82,14 @@ modality is wanted before proteomics.)
 - **The LLM is boxed in.** It lives only inside unstructured normalizers and is
   constrained by `response_format` to fill canonical fields — it cannot emit
   prose. This keeps the system testable and reproducible, and makes the agent's
-  output directly comparable to the deterministic path.
+  output directly comparable to the deterministic path. Concretely (PR 5), a
+  normalizer reaches the LLM only through the narrow **synchronous**
+  `StructuredExtractor` contract (`parce.agent.base`): `extract(instructions,
+  content, response_model) -> response_model`. The real implementation
+  (`parce.agent.extraction.AzureExtractionAgent`) is the *only* Azure-touching
+  module; it bridges the async `agent-framework` API to that sync seam internally,
+  so normalizers stay synchronous and are unit-tested by injecting a deterministic
+  fake extractor (no Azure in CI).
 - **Ontology resolution is a shared stage, not per-adapter.** All sources must
   land on the same IDs or the graph won't link. Deterministic resolvers are
   tried first (OLS, text2term); the LLM is a fallback for ambiguous strings.
@@ -110,7 +117,11 @@ Source-agnostic nodes (Pydantic v2, `extra="forbid"`). Implemented in PR 2 in
   `perturbation`, `timepoint`, `subject`, `organism`. (Reintroduced; the prior
   schema was dataset-level only.) All covariates are optional — different
   sources populate different subsets. Linked to its dataset/study via a typed
-  edge (e.g. `HAS_SAMPLE`).
+  edge (e.g. `HAS_SAMPLE`). *(First populated by GEO in PR 5: one `SampleNode` per
+  `GSM`, with `organism`/`data_uri` read deterministically from structured SOFT
+  fields and `condition`/`perturbation`/`timepoint`/`subject` extracted by the LLM
+  from `characteristics_ch1`. CELLxGENE still emits none — Census is dataset-level,
+  §7.)*
 - `BiologicalEntityNode` — `entity_type` ∈ {Disease, Tissue, Species,
   Perturbation, Assay}, `ontology_id`, `name`. **CellType is intentionally
   absent.**
@@ -119,6 +130,16 @@ Source-agnostic nodes (Pydantic v2, `extra="forbid"`). Implemented in PR 2 in
   `MEASURED_WITH` (Dataset→Assay), `STUDIES` (Study→Species), etc. `relation_type`
   is a free `str` for now (the vocabulary still grows as GEO/PRIDE land); it may
   become a `StrEnum` once the set stabilizes.
+  - *(Decision, PR 5: a source's design-context and `HAS_SAMPLE` edges originate
+    at whichever node is the natural containment root for that source. CELLxGENE is
+    dataset-centric (Census ships datasets), so its edges originate at the
+    `DatasetNode`. **GEO has no distinct dataset artifact** — a series *is* the
+    study, its data lives in per-sample supplementary files — so GEO emits no
+    `DatasetNode` and its `HAS_TISSUE`/`HAS_CONDITION`/`MEASURED_WITH`/`HAS_SAMPLE`
+    edges originate at the `StudyNode`, with `assay`/`molecular_layer` carried on
+    the study. This is deliberate: cross-source linking flows through shared entity
+    `ontology_id` **targets**, not the originating node, so a mixed origin does not
+    break the merge (PR 6).)*
 
 Cross-source links are *emergent*: two studies share an edge target
 (`ontology_id`) rather than any source-specific key.

docs/ROADMAP.md

@@ -8,22 +8,19 @@ protocol and [ARCHITECTURE.md](ARCHITECTURE.md) for the design.
 
 ## ▶ Next up
 
-**PR 5 — GEO extraction agent (vertical slice).** First source whose metadata is
-*unstructured* free text, so the first to use the LLM. Add a GEO adapter
-(NCBI E-utilities / GEOparse) emitting a `RawRecord`, and an **Azure extraction
-normalizer** that fills the canonical schema via `response_format` (structured
-output only — never prose). Extract sample-level **design** covariates from
-`characteristics_ch1` (`condition`, `perturbation`, `timepoint`, `subject`,
-`organism`) into `SampleNode`s — never data-inferred annotations. Ground the
-extracted free-text facets through the **existing `OntologyResolver`** (organism
-→ NCBITaxon, assay → EFO + `molecular_layer`, tissue → UBERON, disease → MONDO),
-and **supply the agent as the resolver's LLM-fallback callback** for strings OLS
-can't map (the hook already exists, default off). Mark live tests
-`@pytest.mark.integration`; keep unit tests offline by mocking the Azure client.
-**Remove the `parce.agent.*` mypy exemption** in `pyproject.toml` once the agent
-moves to the normalizer interface. **Blocker risk:** needs Azure creds + an
-`az login` session; if absent, build/unit-test the deterministic scaffolding and
-log the integration boundary as a blocker rather than working around it.
+**PR 6 — Cross-source KG merge.** Merge per-study subgraphs from *different*
+sources (CELLxGENE + GEO) into one knowledge graph, deduped by ontology entity ID,
+with provenance preserved on edges. The shared-entity machinery already exists:
+both normalizers register `BiologicalEntityNode`s keyed by `ontology_id` and emit
+edges whose **targets** are those IDs (the originating node differs by source — see
+ARCHITECTURE §4 — but the merge keys on targets). Build the merger in `graph/`
+(reserved for exactly this since PR 3), take a list of `KnowledgeGraphOutput`
+subgraphs → one merged graph, dedup entities by `ontology_id`, keep all
+study/dataset/sample nodes, and carry source provenance so a shared entity records
+which studies touch it. **Assert a cross-source edge exists in tests** — e.g. a
+CELLxGENE study and a GEO study that both touch `UBERON:0002048` (lung) or a shared
+`MONDO:` disease become connected through that one entity node. Offline unit tests
+only (assemble two canned subgraphs and merge); no network.
 
 ---
 
@@ -64,14 +61,26 @@ Each PR is one branch, one focused scope, green CI, and a roadmap update.
   exact EFO labels to ordered substring keywords** after validating against live
   EFO (the 10x family never reaches `RNA assay`); ambiguous lineages (bare
   mass-spec, multi-omic terms) stay `UNKNOWN` by design.
-- [ ] **PR 5 — GEO extraction agent (vertical slice).** GEO adapter
-  (E-utilities/GEOparse) + Azure extraction normalizer emitting the canonical
-  schema via `response_format`; extract sample covariates from
-  `characteristics_ch1`. Integration test (marked). This is the agent's real
-  job; remove the `parce.agent.*` mypy exemption. *(Next up — see top of file.)*
+- [x] **PR 5 — GEO extraction agent (vertical slice).** Deterministic `GeoAdapter`
+  (`sources/geo.py`): fetches GEO Series+Sample SOFT text from the GEO accession
+  endpoint, parses it (no GEOparse dep), carries `characteristics_ch1` **verbatim**
+  in the `RawRecord`. Agent-backed `GeoNormalizer` (`normalize/geo.py`): an LLM
+  (boxed behind the narrow sync `StructuredExtractor` seam, `agent/base.py`) fills
+  the `GeoExtraction` schema via `response_format` — design covariates only, no
+  field for any data-inferred annotation. `SampleNode`s now populated (organism +
+  data_uri read deterministically from structured SOFT fields; condition/
+  perturbation/timepoint/subject from the LLM). Facets grounded through the existing
+  `OntologyResolver`; the agent is **wired as the resolver's LLM fallback**
+  (`make_ontology_fallback`, opt-in). The concrete Azure agent
+  (`agent/extraction.py`) bridges the async `agent-framework` API to the sync seam.
+  **`parce.agent.*` mypy exemption removed** — the whole of `src/parce` is now
+  type-checked. **Blocker:** live Azure extraction round-trip unverified (no
+  `AZURE_AI_PROJECT_ENDPOINT` in the headless env); deterministic GEO fetch/parse
+  verified live. *(GEO keyword `discover` via Entrez deferred to backlog — adapter
+  `discover` is identity on a `GSEnnnnn`, mirroring CELLxGENE's DOI identity.)*
 - [ ] **PR 6 — Cross-source KG merge.** Merge CELLxGENE + GEO into one graph
   linked through shared ontology entities; dedup; provenance on edges. Assert a
-  cross-source edge exists in tests.
+  cross-source edge exists in tests. *(Next up — see top of file.)*
 - [ ] **PR 7 — PRIDE proteomics adapter.** Second modality; prove the interface
   is modality-general. Adapter + extraction normalizer + integration test.
 - [ ] **PR 8 — KG export for modeling.** Serialize per-study context + sample
@@ -85,6 +94,9 @@ Each PR is one branch, one focused scope, green CI, and a roadmap update.
 - Graph database backend (Neo4j) vs. flat JSON export — revisit at PR 8.
 - Discovery agent: given a research theme, propose seed DOIs/accessions across
   repositories.
+- GEO keyword `discover` via Entrez `esearch`+`esummary` (the adapter's `discover`
+  is currently the identity on a `GSEnnnnn` accession). Pairs with the discovery
+  agent above.
 
 ---
 
@@ -93,6 +105,63 @@ Each PR is one branch, one focused scope, green CI, and a roadmap update.
 Newest first. One entry per working session: what changed, decisions made, and
 what the next session should know. Keep entries short and factual.
 
+### 2026-06-28 — PR 5: GEO extraction agent (vertical slice)
+
+- Branch `pr5-geo-extraction-agent` off **`origin/main`** (4dcd5b7).
+- **Stale-base catch (heeded the memory hazard):** the routine worktree's local
+  `main` was `64ed4f4`, two merges behind `origin/main` (PR 4 #7 + PR 4b #8). The
+  worktree's roadmap therefore showed PR 4 as "▶ Next up" — already merged.
+  `git fetch` + compare to `origin/main` caught it; rebased onto origin and did the
+  *real* next item (PR 5). Did **not** rebuild PR 4.
+- **New files.** `sources/geo.py` (`GeoAdapter` + SOFT parser), `normalize/geo.py`
+  (`GeoNormalizer` + `GeoExtraction`/`SampleExtraction` schemas), `agent/base.py`
+  (`StructuredExtractor` Protocol), `agent/extraction.py` (`AzureExtractionAgent` +
+  `make_ontology_fallback`). Tests: `test_geo_adapter.py`, `test_geo_normalize.py`,
+  `test_geo_integration.py` (marked).
+- **Design decisions (rationale):**
+  - **Deterministic vs LLM split.** GEO ships some fields structured (per-sample
+    `organism`, `supplementary_file`) — those are read straight from the record; the
+    LLM only parses the genuinely free-text `characteristics_ch1` into design
+    covariates and reads study-level assay/tissue/disease from the prose. Follows
+    "could a deterministic step do this? then do it" (CLAUDE.md).
+  - **Sample set is the record's, not the LLM's.** One `SampleNode` per real `GSM`;
+    the extraction is matched in by `sample_id`, so a dropped/hallucinated sample
+    can't change graph shape. Extraction failure degrades to samples-without-
+    covariates (logged), never a crash.
+  - **No `DatasetNode` for GEO.** A series *is* the study (data is per-sample suppl
+    files), so `assay`/`molecular_layer` live on `StudyNode` and design-context +
+    `HAS_SAMPLE` edges originate at the study. Merge (PR 6) keys on entity
+    `ontology_id` **targets**, so the differing origin vs CELLxGENE is fine.
+    Recorded in ARCHITECTURE §4.
+  - **No GEOparse dependency.** The fields needed are a handful of `!`-keys in SOFT
+    text; a ~40-line parser keeps deps minimal and the parse unit-testable. No dep
+    changes, so `uv.lock` untouched.
+  - **`discover` = identity on a `GSEnnnnn`** (mirrors CELLxGENE's DOI identity);
+    Entrez keyword search → backlog.
+  - **LLM boxed behind a sync `StructuredExtractor` seam** (`agent/base.py`); the
+    async `agent-framework` bridge lives only in `agent/extraction.py`. Normalizers
+    stay sync + offline-testable with a fake extractor. The agent is also wired as
+    the resolver's LLM fallback (`make_ontology_fallback`, accepts a result only if
+    the CURIE prefix matches the facet's ontology). ARCHITECTURE §3 updated.
+  - Added optional `ncbi_email`/`ncbi_api_key` settings (+ `.env.example`); passed
+    to the adapter, never read by it directly (keeps unit tests Settings-free).
+- **mypy:** removed the `parce.agent.*` override — **all of `src/parce` now
+  type-checked** (28 files; agent-framework/azure are untyped so the glue is `Any`
+  at the boundary, which is sound here).
+- **Gates green (hermetic — no `.env` in the worktree):** ruff check, ruff format
+  --check (47 files), mypy (28 files), **168 unit tests** (13 integration
+  deselected). Live `TestLiveGeoFetch` run against the real GEO endpoint — passes
+  (SOFT parser validated on `GSE10072`).
+- **BLOCKER (integration boundary, per protocol):** the live **Azure extraction**
+  round-trip is **unverified** — this headless env has `az login` but no
+  `AZURE_AI_PROJECT_ENDPOINT` configured (no worktree `.env`), so
+  `TestLiveGeoExtraction` skips. The Azure call shape mirrors the previously-working
+  `agent/curator.py` (`agent.run(prompt, response_format=Model)` → `result.value`).
+  **Next session with Azure creds:** run `uv run pytest -m integration
+  tests/test_geo_integration.py` to confirm the live extraction, before relying on
+  the GEO path in PR 6's cross-source merge.
+- **Next session:** PR 6 (cross-source KG merge) — see top of file.
+
 ### 2026-06-26 — PR 4b: Schema refinement (EFO assay term + stored molecular_layer)
 
 - Branch `pr4b-schema-refinement` off `main` (35a28ce, the PR 4 merge). Note: the

pyproject.toml

@@ -63,11 +63,7 @@ warn_redundant_casts = true
 warn_unused_ignores = true
 check_untyped_defs = true
 
-# The Azure agent glue is slated for rewrite as an extraction normalizer (PR 5,
-# see docs/ROADMAP.md). Until then it is exempt from type checking so CI tracks
-# the stable core. Remove this entry when the agent moves to the normalizer
-# interface. (The former ``parce.tools.*`` fetchers were migrated to
-# ``parce.sources.*`` in PR 3 and are now fully type-checked.)
-[[tool.mypy.overrides]]
-module = ["parce.agent.*"]
-ignore_errors = true
+# No module-level type-check exemptions remain. ``parce.tools.*`` was migrated to
+# ``parce.sources.*`` in PR 3; ``parce.agent.*`` moved to the StructuredExtractor
+# interface in PR 5 (the Azure glue is now a typed extraction agent). The whole of
+# ``src/parce`` is type-checked.

src/parce/agent/base.py

@@ -0,0 +1,39 @@
+"""The :class:`StructuredExtractor` contract.
+
+An extractor is the single boundary between an agent-backed normalizer and a Large
+Language Model. It does exactly one thing: take free-text metadata plus a target
+Pydantic schema and return a populated, validated instance of that schema — the
+*structured extraction* job the LLM is boxed into (docs/ARCHITECTURE.md §3). It
+never returns prose.
+
+Normalizers depend on this narrow, **synchronous** Protocol rather than on a
+concrete Azure client, so they can be driven entirely offline in unit tests by
+injecting a deterministic fake. The real implementation
+(:class:`~parce.agent.extraction.AzureExtractionAgent`) bridges to the async
+``agent-framework`` API internally; that async complexity never leaks past this
+contract.
+"""
+
+from __future__ import annotations
+
+from typing import Protocol, TypeVar, runtime_checkable
+
+from pydantic import BaseModel
+
+#: The schema an extraction call fills. Bound to ``BaseModel`` so the extractor
+#: can validate the model's output against it via ``response_format``.
+SchemaT = TypeVar("SchemaT", bound=BaseModel)
+
+
+@runtime_checkable
+class StructuredExtractor(Protocol):
+    """Fills a Pydantic schema from free text via an LLM (structured output only)."""
+
+    def extract(self, instructions: str, content: str, response_model: type[SchemaT]) -> SchemaT:
+        """Return ``response_model`` populated from ``content`` under ``instructions``.
+
+        ``instructions`` is the system prompt (what to extract and the design-only
+        constraints); ``content`` is the free-text metadata to read. The result is
+        a validated ``response_model`` instance — never narrative text.
+        """
+        ...