Skip to content

PR 3: Source-adapter interface + deterministic CELLxGENE adapter #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mengerj merged 1 commit into from
Jun 24, 2026
Merged

PR 3: Source-adapter interface + deterministic CELLxGENE adapter #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
199 changes: 69 additions & 130 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,161 +2,100 @@

**Programmable Agent for Retrieving Contextualized Experiments**

PARCE is an agentic workflow that fetches public omics data and produces structured JSON narratives describing how the data was obtained. Each narrative interleaves human-readable descriptions with URI references to raw data files, making it suitable for training multimodal autoregressive embedding models.
PARCE harvests **public omics experiments** from heterogeneous repositories and
normalizes them into a single, ontology-grounded **knowledge graph (KG)**. The KG
is the training substrate for a downstream multi-omics autoregressive model: each
study contributes biological *context* (assay, tissue, disease, organism) and
sample-level *covariates* (condition, perturbation, timepoint, subject), plus URIs
to the raw data.

Two principles drive the design:

1. **Context is design, not outcome.** Only metadata that describes how an
experiment was *designed* is stored. Data-inferred annotations (e.g. cell type
called from expression) are excluded — they would leak the signal the
downstream model must learn.
2. **One canonical schema, many sources.** Every repository is mapped into the
same Pydantic KG schema and the same ontology IDs. Sources link to each other
*only* through shared ontology entity nodes.

See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the design and
[docs/ROADMAP.md](docs/ROADMAP.md) for the plan and current status.

## Architecture

```
User / CLI
│
▼
main.py ──► agent/curator.py ──► AzureAIAgentClient
│
┌─────────┴─────────┐
│ Azure AI Foundry │
│ (any model) │
└─────────┬─────────┘
│
┌───────────┼───────────┐
▼ │ ▼
tool_call: │ structured output:
geo_fetcher.py │ ExperimentNarrative
(tools/) │ (models/)
│ │
▼ │
metadata JSON ─────┘
```

The agent uses **Azure AI Foundry** as its model gateway. Any model deployed in your Foundry project -- Mistral, GPT-4o, DeepSeek, Llama, etc. -- can be used by changing a single environment variable (`AZURE_AI_MODEL_DEPLOYMENT_NAME`). Tools, prompts, and Pydantic output schemas remain unchanged.

### Directory Structure
The pipeline is **deterministic by default**. Each repository has a `SourceAdapter`
(all network IO) that emits source-shaped `RawRecord`s, and a `Normalizer` that
maps a `RawRecord` into canonical KG nodes. An LLM is used for exactly one job —
structured extraction from *unstructured* free-text metadata (e.g. GEO, PRIDE),
constrained to the canonical schema via `response_format`. Structured,
ontology-grounded sources (e.g. CELLxGENE) have **no LLM in their path**.

```
parce/
├── pyproject.toml # Dependencies and build config
├── .env.example # Template for required env vars
├── src/
│ └── parce/
│ ├── main.py # Entry point
│ ├── agent/
│ │ ├── curator.py # Agent factory (AzureAIAgentClient)
│ │ └── prompts.py # System prompts
│ ├── tools/
│ │ └── geo_fetcher.py # GEO metadata fetcher (stub)
│ ├── models/
│ │ └── narrative.py # Pydantic output schemas
│ └── config/
│ └── settings.py # Env-based configuration
├── data_pipelines/ # Future: Spark / ADLS scripts
├── data/ # Local test data (gitignored)
└── tests/
└── test_models.py
per source → SourceAdapter: discover(query) -> [ref]
fetch(ref) -> RawRecord
│ (source-shaped)
per source → Normalizer: RawRecord -> canonical KG nodes
• structured source -> deterministic map
• unstructured -> Azure extraction agent (PR 5+)
│ (canonical nodes w/ free-text terms)
shared → OntologyResolver: text -> UBERON/MONDO/EFO/... (PR 4)
│ (ontology-grounded nodes)
shared → GraphBuilder/Merger: assemble + merge into one KG (PR 6)
```

**Key design decisions:**

- **src-layout** prevents accidental imports from the project root.
- **agent/** is decoupled from **tools/** so new data sources (e.g. TCellAtlas, SRA) are added as new tool files without touching orchestration logic.
- **models/** schemas serve double duty: structured output for the agent (`response_format`) and validation for downstream consumers.
- **data_pipelines/** lives at the repo root because Spark jobs are submitted independently from the Python package.

## Prerequisites

- Python 3.11+
- [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) installed
- An [Azure AI Foundry](https://learn.microsoft.com/en-us/azure/ai-foundry/what-is-ai-foundry) project with at least one model deployed (e.g. Mistral Large, GPT-4o, DeepSeek-R1)

## Local Setup
The canonical schema (`models/graph_schema.py`) is the contract: the deterministic
path and the future agent path emit the *same* Pydantic models, so everything
downstream is source-agnostic.

### 1. Authenticate with Azure
### Directory structure

```bash
az login
```

This lets `AzureCliCredential` obtain tokens for your Foundry project without managing API keys.

### 2. Configure environment

```bash
cp .env.example .env
src/parce/
models/ # Canonical Pydantic KG schema + RawRecord (boundary model)
sources/ # One adapter per repository: discover() + fetch() -> RawRecord
normalize/ # RawRecord -> canonical nodes (deterministic OR agent-backed)
agent/ # Azure extraction agent (structured output only; PR 5+)
graph/ # Cross-source KG assembly + merge (PR 6)
config/ # pydantic-settings configuration
main.py # CLI entry point / orchestrator
tests/ # Unit tests (offline) + integration tests (marked)
docs/ # ARCHITECTURE.md, ROADMAP.md, AGENT_SESSION_PROMPT.md
```

Edit `.env` with your Foundry project endpoint and deployment name:

```
AZURE_AI_PROJECT_ENDPOINT=https://<your-resource>.services.ai.azure.com/api/projects/<project-id>
AZURE_AI_MODEL_DEPLOYMENT_NAME=mistral-large
```
Implemented so far: the canonical schema, the `SourceAdapter`/`Normalizer`
protocols, and the deterministic **CELLxGENE** adapter + normalizer.

The project endpoint is found in your Azure AI Foundry project settings page.
## Setup

### 3. Install
PARCE is [uv](https://docs.astral.sh/uv/)-managed.

```bash
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -e ".[dev]"
uv sync --extra dev # install (incl. dev tools)
uv run parce # run the CLI (CELLxGENE path; deterministic)
```

> **Note:** The `agent-framework` packages are currently in pre-release. If installation fails, run:
> ```bash
> pip install agent-framework agent-framework-azure-ai --pre
> pip install -e ".[dev]"
> ```
`parce` fetches the datasets for the default collection DOI from CELLxGENE Census,
normalizes them into the canonical KG, and writes `data/graphs/output.json`. This
path needs only network access — **no Azure credentials**.

### 4. Run the agent
Azure AI Foundry credentials are required only for the extraction agent (GEO/PRIDE,
PR 5+) and for the integration test suite. Copy `.env.example` to `.env` and fill
in your Foundry endpoint and deployment name, then `az login`.

```bash
parce
# or
python -m parce.main
```
## Tests & quality gates

The agent will fetch mock metadata for GSE164378 and return a structured `ExperimentNarrative` JSON.

### 5. Run tests
CI runs four gates; all four must pass locally before opening a PR:

```bash
pytest
uv run ruff check .
uv run ruff format --check .
uv run mypy src/parce
uv run pytest -m "not integration" # unit tests (offline, hermetic)
```

## Switching Models

Because PARCE uses Azure AI Foundry as a model gateway, swapping the underlying LLM is a one-line change in `.env`:
Live/credentialed tests carry the `integration` marker and are excluded from CI:

```bash
# Mistral
AZURE_AI_MODEL_DEPLOYMENT_NAME=mistral-large

# OpenAI
AZURE_AI_MODEL_DEPLOYMENT_NAME=gpt-4o

# DeepSeek
AZURE_AI_MODEL_DEPLOYMENT_NAME=DeepSeek-R1

# Meta Llama
AZURE_AI_MODEL_DEPLOYMENT_NAME=Meta-Llama-3-70B
uv run pytest -m integration # needs Azure / Census / network
```

The deployment name must match a model you have deployed in your Foundry project's model catalog. No code changes are required -- all providers produce a standard `Agent` with the same interface.

For models **not** in the Azure AI Foundry catalog (e.g. Anthropic Claude), the Microsoft Agent Framework provides dedicated providers (`AnthropicChatClient`) with an identical agent interface.

## Future Roadmap

### Data Engineering Pipelines (`data_pipelines/`)

- **FASTQ-to-Parquet** conversion using PySpark on Azure Databricks
- **Azure Data Lake Storage Gen2** integration for persisting narratives and raw genomic data at scale
- Batch orchestration for processing large T-cell transcriptomics datasets

### Additional Data Sources

- **TCellAtlas** fetcher tool
- **SRA** direct metadata fetcher
- **CellxGene** Census integration

### Embedding Model Training

The structured narratives produced by PARCE will serve as training data for a multimodal autoregressive embedding model that jointly learns from experimental metadata and raw omics signals.
85 changes: 68 additions & 17 deletions docs/ROADMAP.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,14 @@ protocol and [ARCHITECTURE.md](ARCHITECTURE.md) for the design.

## ▶ Next up

**PR 3 — Source-adapter interface + cheap CELLxGENE adapter.** Define
`SourceAdapter` / `Normalizer` protocols in `sources/` + `normalize/`. Refactor
CELLxGENE into a deterministic adapter. **Remove the LLM/Azure narrative path
entirely** (delete `agent/prompts.py` narrative role, `models/narrative.py`, the
narrative `NarrativeOutput` schema in `models/graph_schema.py`, and step 2 of
`main.py` — `_build_narrative_prompt`, the agent call, and the now-unused
`narrative` variable). Drop cell-type extraction. Remove the `parce.tools.*`
mypy exemption as those modules move under `sources/`.
**PR 4 — Ontology resolver.** Shared `ontology/` stage (see ARCHITECTURE §5). Pin
the **facet → ontology registry** as a constant (EFO, UBERON, MONDO, NCBITaxon,
ChEBI, PSI-MS, EDAM). Implement free-text → term resolution (OLS4 REST +
text2term/Zooma, on-disk cache; LLM fallback) and the **`molecular_layer`
derivation** (walk EFO `is-a` ancestors to anchor classes). Decide the anchor set
+ the no-anchor default. Wire into normalizers — start by replacing the hardcoded
`_ORGANISM_ONTOLOGY` map in `normalize/cellxgene.py`. Resolve IDs at runtime via
OLS — do not hardcode term IDs.

---

Expand All @@ -29,20 +29,21 @@ Each PR is one branch, one focused scope, green CI, and a roadmap update.
- [x] **PR 2 — Canonical KG schema.** Source-agnostic nodes/edges; add
`SampleNode` with design covariates; drop `experimental_narrative` and
`CellType`. Migrate builder + tests.
- [ ] **PR 3 — Source-adapter interface + cheap CELLxGENE adapter.** Define
`SourceAdapter` / `Normalizer` protocols in `sources/` + `normalize/`. Refactor
CELLxGENE into a deterministic adapter. **Remove the LLM/Azure narrative path
entirely** (delete `agent/prompts.py` narrative role, `models/narrative.py`,
the `NarrativeOutput` schema, and the step-2 block in `main.py`). Drop
cell-type extraction. Remove the `parce.tools.*` mypy exemption as those
modules move under `sources/`. *(Next up.)*
- [x] **PR 3 — Source-adapter interface + cheap CELLxGENE adapter.** Defined
`SourceAdapter` / `Normalizer` protocols in `sources/` + `normalize/`. Refactored
CELLxGENE into a deterministic adapter + normalizer. **Removed the LLM/Azure
narrative path entirely** (deleted `agent/curator.py`, `agent/prompts.py`,
`models/narrative.py`, the `NarrativeOutput` schema, and the step-2 block in
`main.py`). Dropped cell-type extraction. Removed the `parce.tools.*` mypy
exemption as those modules moved under `sources/`.
- [ ] **PR 4 — Ontology resolver.** Shared `ontology/` stage (see ARCHITECTURE
§5). Pin the **facet → ontology registry** as a constant (EFO, UBERON, MONDO,
NCBITaxon, ChEBI, PSI-MS, EDAM). Implement free-text → term resolution
(OLS4 REST + text2term/Zooma, on-disk cache; LLM fallback) and the
**`molecular_layer` derivation** (walk EFO `is-a` ancestors to anchor classes).
Decide the anchor set + the no-anchor default. Wire into normalizers. Resolve
IDs at runtime via OLS — do not hardcode term IDs.
Decide the anchor set + the no-anchor default. Wire into normalizers (replace
the hardcoded `_ORGANISM_ONTOLOGY` map in `normalize/cellxgene.py`). Resolve
IDs at runtime via OLS — do not hardcode term IDs. *(Next up.)*
- [ ] **PR 5 — GEO extraction agent (vertical slice).** GEO adapter
(E-utilities/GEOparse) + Azure extraction normalizer emitting the canonical
schema via `response_format`; extract sample covariates from
Expand Down Expand Up @@ -72,6 +73,56 @@ Each PR is one branch, one focused scope, green CI, and a roadmap update.
Newest first. One entry per working session: what changed, decisions made, and
what the next session should know. Keep entries short and factual.

### 2026-06-24 — PR 3: Source-adapter interface + CELLxGENE adapter

- Branch `pr3-source-adapter-interface` off `main` (69d1f68).
- **New contracts.** `models/raw_record.py` adds `RawRecord` (source-shaped:
`source`, `study_id`, `title`, free-form `payload`) — the boundary object
between adapters and normalizers. `sources/base.py` defines the `SourceAdapter`
Protocol (`source_name`, `discover(query) -> [ref]`, `fetch(ref) -> RawRecord`);
`normalize/base.py` defines the `Normalizer` Protocol
(`normalize(record) -> KnowledgeGraphOutput`). Both `@runtime_checkable`.
- **CELLxGENE migrated to a deterministic adapter + normalizer:**
- `tools/cellxgene_fetcher.py` → `sources/cellxgene.py` (`CellxgeneAdapter` +
the `fetch_cellxgene_datasets` core fn). **Cell-type extraction dropped**: the
`cell_type*` Census columns and the `cell_types` summary key are gone (reading
a data-inferred annotation is leakage even before it hits the graph).
- `tools/ncbi_fetcher.py` → `sources/publication.py` (`fetch_paper_metadata`,
EuropePMC). The adapter's `fetch` gathers the publication title here because
Census exposes dataset titles + DOI but not the publication title.
- `graph/builder.py` → `normalize/cellxgene.py` (`CellxgeneNormalizer`). It now
reads `record.source`/`title`/`payload` instead of taking a hardcoded source.
- The `@tool`-decorated wrappers (`fetch_cellxgene_data`, `fetch_paper_context`,
`fetch_geo_metadata`) were vestigial from the agent-tool-calling era and are
deleted. The whole `tools/` package and the GEO stub are removed.
- **Narrative/LLM path removed entirely:** deleted `agent/curator.py`,
`agent/prompts.py`, `models/narrative.py` (and `test_models.py`), and the
`NarrativeOutput` schema. `main.py` is now deterministic and **synchronous**:
`discover → fetch → normalize → write`. (PR 5 reintroduces async + the agent.)
- Decisions (rationale):
- **Per-study assembly is the Normalizer's job; `graph/` is reserved for the
PR 6 cross-source merger.** Matches ARCHITECTURE §3's split. `graph/__init__.py`
and `agent/__init__.py` are kept as documented placeholders.
- **Organism→NCBITaxon map (`_ORGANISM_ONTOLOGY`) lives in the normalizer, not
the adapter.** The adapter emits the raw organism string; string→ID mapping is
normalization and becomes the PR 4 OntologyResolver's job.
- **`discover` is the identity on a DOI for CELLxGENE** (a collection = a DOI).
Keyword collection search is backlog.
- **Azure-coupled retry helpers removed from `main.py`** with the LLM call (their
only caller). CELLxGENE/EuropePMC fetches currently have no retry wrapper —
see follow-up below. CLAUDE.md still references "helpers in `main.py`"; left as
is since PR 5 reintroduces retry infra for the agent.
- **mypy:** removed the `parce.tools.*` exemption (modules migrated + now fully
type-checked); `parce.agent.*` exemption stays for PR 5. mypy checks 16 files.
- **Gates green locally, incl. hermetic run with `.env` moved aside:** ruff check,
ruff format --check (24 files), mypy, **47 unit tests**. No dep changes; the
`agent-framework`/`azure-*` deps stay (PR 5 needs them).
- **Follow-up for a later session:** wire bounded-retry/backoff into the source
adapters' network calls (Census, EuropePMC) — resilience regressed when the
Azure-only retry helpers were removed.
- **Next session:** PR 4 (ontology resolver). First integration point: replace
`_ORGANISM_ONTOLOGY` in `normalize/cellxgene.py`.

### 2026-06-23 — PR 2: Canonical KG schema

- Branch `pr2-canonical-kg-schema` off `main`.
Expand Down
11 changes: 6 additions & 5 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,10 +63,11 @@ warn_redundant_casts = true
warn_unused_ignores = true
check_untyped_defs = true

# The IO-boundary modules (source fetchers, Azure agent glue) are slated for
# rewrite in the source-adapter refactor (see docs/ROADMAP.md). Until then they
# are exempt from type checking so CI tracks the stable core. Remove an entry
# here as each module is migrated to the adapter interface.
# The Azure agent glue is slated for rewrite as an extraction normalizer (PR 5,
# see docs/ROADMAP.md). Until then it is exempt from type checking so CI tracks
# the stable core. Remove this entry when the agent moves to the normalizer
# interface. (The former ``parce.tools.*`` fetchers were migrated to
# ``parce.sources.*`` in PR 3 and are now fully type-checked.)
[[tool.mypy.overrides]]
module = ["parce.tools.*", "parce.agent.*"]
module = ["parce.agent.*"]
ignore_errors = true
7 changes: 7 additions & 0 deletions src/parce/agent/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
"""Azure extraction agent (structured output only).

Reserved for the GEO/PRIDE extraction normalizers (PR 5+): an LLM constrained by
``response_format`` to emit the canonical KG schema from free-text metadata. The
legacy narrative agent that once lived here was removed in PR 3 — the LLM never
writes prose. See docs/ARCHITECTURE.md §3.
"""
Loading
Loading