mengerj · mengerj · Jun 23, 2026 · Jun 23, 2026 · Jun 23, 2026 · Jun 23, 2026
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -0,0 +1,44 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+# Cancel superseded runs on the same ref to save CI minutes.
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  quality:
+    name: lint • types • tests (py${{ matrix.python-version }})
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.11", "3.13"]
+    steps:
+      - uses: actions/checkout@v5
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v6
+        with:
+          enable-cache: true
+
+      - name: Sync dependencies (locked)
+        run: uv sync --extra dev --locked
+
+      - name: Ruff lint
+        run: uv run ruff check .
+
+      - name: Ruff format check
+        run: uv run ruff format --check .
+
+      - name: Mypy
+        run: uv run mypy src/parce
+
+      # Unit tests only. Integration tests hit live Azure / Census and are
+      # excluded here; run them locally with `uv run pytest -m integration`.
+      - name: Pytest (unit)
+        run: uv run pytest -m "not integration"
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -0,0 +1,118 @@
+# CLAUDE.md
+
+Guidance for Claude Code (and humans) working in this repository. Read this
+first, then [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the design and
+[docs/ROADMAP.md](docs/ROADMAP.md) for what to do next.
+
+## What PARCE is
+
+PARCE harvests **public omics experiments** from heterogeneous repositories and
+normalizes them into a single, ontology-grounded **knowledge graph (KG)**. The
+KG is the training substrate for a downstream multi-omics autoregressive model:
+each study contributes biological *context* (assay, tissue, disease, organism)
+and sample-level *covariates* (condition, perturbation, timepoint, subject),
+plus URIs to the raw data that an OQAE later tokenizes.
+
+Two non-negotiable principles drive the design:
+
+1. **Context is design, not outcome.** Only store metadata that describes how an
+   experiment was *designed* — assay, tissue, disease, organism, perturbation.
+   Never store data-*inferred* annotations (e.g. cell type called from
+   expression). They leak the very signal the downstream model must learn.
+2. **One canonical schema, many sources.** Every repository — structured or not —
+   is mapped into the same Pydantic KG schema and the same ontology IDs. Sources
+   link to each other *only* through shared ontology entity nodes.
+
+## Where intelligence lives
+
+The pipeline is **deterministic by default**. An LLM (Azure AI Foundry agent) is
+used for exactly one job: **structured extraction and normalization of
+free-text metadata** from unstructured sources (e.g. GEO, PRIDE). It is *never*
+used to write prose/narrative, and it is constrained to emit the canonical
+Pydantic schema via `response_format`. If a source already provides structured,
+ontology-grounded metadata (e.g. CELLxGENE), there is **no LLM in its path**.
+
+When you reach for the LLM, ask: "could a deterministic API call or an ontology
+lookup do this?" If yes, do that instead.
+
+## Project layout (target)
+
+```
+src/parce/
+  models/      # Canonical Pydantic KG schema (nodes, edges, sample covariates)
+  sources/     # One adapter per repository: discover() + fetch() -> RawRecord
+  normalize/   # RawRecord -> canonical nodes (deterministic OR agent-backed)
+  ontology/    # Shared free-text -> ontology-ID resolver (+ cache)
+  agent/       # Azure extraction agent (structured output only)
+  graph/       # Assemble + merge canonical nodes into one KG
+  config/      # pydantic-settings configuration
+  main.py      # CLI entry point / orchestrator
+tests/         # Unit tests (offline) + integration tests (marked)
+docs/          # ARCHITECTURE.md, ROADMAP.md, AGENT_SESSION_PROMPT.md
+data_pipelines/  # Future: Spark/ADLS batch jobs
+```
+
+Dependency direction is one-way: `sources`, `normalize`, `agent`, `graph` all
+depend on `models`; `models` depends on nothing. No import cycles.
+
+## Coding conventions
+
+- **Python ≥ 3.11**, `src`-layout, `from __future__ import annotations` in every
+  module. Full type hints on public functions.
+- **Pydantic v2** for all data at boundaries; KG models set
+  `model_config = ConfigDict(extra="forbid")`.
+- **Tooling (all enforced in CI):**
+  - `ruff check .` — lint (rules: E, F, I, UP, B, SIM, C4, RUF).
+  - `ruff format .` — formatting (line length 100). Never hand-format.
+  - `mypy src/parce` — types. The stable core is checked; IO modules under
+    `parce.tools.*` / `parce.agent.*` are temporarily exempt via
+    `pyproject.toml` overrides. **Remove a module's exemption when you migrate
+    it to the adapter interface** — do not add new exemptions.
+  - `pytest -m "not integration"` — unit tests, must stay offline.
+- **Tests:** unit tests must not touch the network, Azure, or Census — mock
+  them. Anything that needs live credentials or downloads is marked
+  `@pytest.mark.integration` and excluded from CI. **Unit tests must not depend
+  on a local `.env`** — no unit test may construct `Settings()` unmocked.
+  Verify hermeticity before reporting green: run the unit suite with `.env`
+  moved aside (`mv .env .env.bak && uv run pytest -m "not integration"; mv
+  .env.bak .env`), which reproduces the CI runner exactly.
+- **Config & secrets:** all config via `pydantic-settings` (`config/settings.py`)
+  and `.env`. Never hardcode endpoints or commit secrets; update `.env.example`
+  when you add a setting. `data/` is gitignored.
+- **Logging, not printing,** inside library code (stdlib `logging`). `print` is
+  for CLI user-facing summaries only.
+- **Resilience:** network calls to external repos/LLMs use bounded retries with
+  jittered backoff (see the existing helpers in `main.py`).
+
+## Environment & commands
+
+uv-managed. Common commands:
+
+```bash
+uv sync --extra dev                 # install (incl. dev tools)
+uv run parce                        # run the CLI
+uv run ruff check . && uv run ruff format --check .
+uv run mypy src/parce
+uv run pytest -m "not integration"  # unit tests (CI gate)
+uv run pytest -m integration        # live tests (needs Azure/Census)
+uv lock                             # refresh lockfile after changing deps
+```
+
+Before opening a PR, all four gates (ruff check, ruff format --check, mypy,
+pytest unit) must pass locally — they are exactly what CI runs.
+
+## Working agreement (every session)
+
+This repo is built across many short, fresh Claude Code sessions. To stay
+coherent:
+
+1. **Start** by reading `CLAUDE.md`, `docs/ARCHITECTURE.md`, and
+   `docs/ROADMAP.md`. The roadmap's **"Next up"** marker is your task.
+2. **Scope** one roadmap item per session/PR. Work on a feature branch, never
+   commit directly to `main`.
+3. **Finish** by updating `docs/ROADMAP.md`: tick the completed checklist items,
+   append a dated entry to the **Session Log**, and move the **"Next up"**
+   marker. The next session relies entirely on this — leave it accurate.
+4. Keep `ARCHITECTURE.md` in sync if you change a design decision; record *why*.
+
+Full session protocol: [docs/AGENT_SESSION_PROMPT.md](docs/AGENT_SESSION_PROMPT.md).
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
@@ -0,0 +1,141 @@
+# PARCE Architecture
+
+> Status: design baseline for the `restructure-context-metadata` line of work.
+> This document is the source of truth for *why* the code is shaped the way it
+> is. Update it (with rationale) whenever a design decision changes.
+
+## 1. Goal
+
+Produce a single, ontology-grounded **knowledge graph** of public omics
+experiments spanning **multiple modalities**, suitable as training data for a
+downstream multi-omics autoregressive model. For each study the KG records:
+
+- **Biological context** (study-level): assay/technology, tissue, disease,
+  organism — the *design* variables.
+- **Sample-level covariates**: condition, perturbation, timepoint, subject, and
+  a URI to the raw data for that sample.
+
+The downstream model treats samples within a study as a **context-conditioned
+set** (not a sequence), and links studies across modalities through **shared
+context**. Two consequences shape this repo:
+
+- **Context must be design, not data-inferred outcome.** Cell type, cluster
+  labels, etc. are excluded — they are downstream of the raw signal and would
+  leak the learning target.
+- **Cross-study/cross-modality linking flows through shared ontology nodes.** A
+  bulk RNA-seq study and a single-cell study that both touch `MONDO:0005061`
+  become connected in the graph. This is the data-layer realization of the
+  model's "shared context space."
+
+## 2. The modality / structure gradient
+
+We deliberately ingest sources spanning a gradient of metadata structure, so the
+extraction logic has a real but tractable job and the "one KG, many sources"
+claim is proven early.
+
+| Tier | Source | Modality | Metadata | LLM in path? |
+|------|--------|----------|----------|--------------|
+| Anchor | **CELLxGENE Census** | scRNA-seq | structured, ontology-grounded | No (deterministic) |
+| Extraction start | **GEO** (NCBI) | bulk RNA-seq (+ ATAC/ChIP) | semi-structured free text (`characteristics_ch1`) | Yes (extraction agent) |
+| Cross-modality | **PRIDE / ProteomeXchange** | proteomics (intensities) | free-text + partial SDRF | Yes (extraction agent) |
+
+GEO is the first extraction target because (a) its metadata is the canonical
+messy case, and (b) it overlaps biologically with CELLxGENE, so cross-source KG
+edges appear with only two sources connected. PRIDE then proves modality
+generality. (ENCODE is a clean structured alternative if a deterministic second
+modality is wanted before proteomics.)
+
+## 3. Layered design
+
+```
+                 ┌─────────────────────────────────────────────┐
+   per source →  │  SourceAdapter:  discover(query) -> [ref]    │
+                 │                  fetch(ref)      -> RawRecord │
+                 └───────────────────────┬─────────────────────┘
+                                         │ RawRecord (source-shaped)
+                 ┌───────────────────────▼─────────────────────┐
+   per source →  │  Normalizer:  RawRecord -> canonical nodes   │
+                 │   • structured source -> deterministic map   │
+                 │   • unstructured      -> Azure extraction    │
+                 │                          agent (response_     │
+                 │                          format = schema)    │
+                 └───────────────────────┬─────────────────────┘
+                                         │ canonical nodes w/ free-text terms
+                 ┌───────────────────────▼─────────────────────┐
+   shared     →  │  OntologyResolver: text -> UBERON/MONDO/...  │
+                 │   deterministic (OLS/text2term) + cache,     │
+                 │   LLM only as fallback for hard cases        │
+                 └───────────────────────┬─────────────────────┘
+                                         │ ontology-grounded nodes
+                 ┌───────────────────────▼─────────────────────┐
+   shared     →  │  GraphBuilder/Merger: assemble + merge into  │
+                 │  one KG; entities deduped by ontology ID     │
+                 └─────────────────────────────────────────────┘
+```
+
+### Why this shape
+
+- **The canonical schema is the contract.** The deterministic path and the
+  agent path emit the *same* Pydantic models. Per-source variation collapses to
+  "which adapter + is the normalizer deterministic or agent-backed". Everything
+  downstream is identical and source-agnostic.
+- **The LLM is boxed in.** It lives only inside unstructured normalizers and is
+  constrained by `response_format` to fill canonical fields — it cannot emit
+  prose. This keeps the system testable and reproducible, and makes the agent's
+  output directly comparable to the deterministic path.
+- **Ontology resolution is a shared stage, not per-adapter.** All sources must
+  land on the same IDs or the graph won't link. Deterministic resolvers are
+  tried first (OLS, text2term); the LLM is a fallback for ambiguous strings.
+
+## 4. Canonical KG schema (target)
+
+Source-agnostic nodes (Pydantic v2, `extra="forbid"`):
+
+- `StudyNode` — `study_id` (DOI/accession), `title`, `source` (provenance),
+  `modality`. *(No `experimental_narrative`; that field is removed.)*
+- `DatasetNode` — `dataset_id`, `data_uri`, `assay`, `cell_count`/size,
+  parent study.
+- `SampleNode` — `sample_id`, `data_uri`, and **design covariates**: `condition`,
+  `perturbation`, `timepoint`, `subject`, `organism`. (Reintroduced; the prior
+  schema was dataset-level only.)
+- `BiologicalEntityNode` — `entity_type` ∈ {Disease, Tissue, Species,
+  Perturbation, Assay}, `ontology_id`, `name`. **CellType is intentionally
+  absent.**
+- `GraphEdge` — typed, directed: `EXTRACTED_FROM`, `MEASURED_WITH`,
+  `HAS_CONDITION`, `STUDIES`, `HAS_SAMPLE`, etc.
+
+Cross-source links are *emergent*: two studies share an edge target
+(`ontology_id`) rather than any source-specific key.
+
+## 5. Coding style & architecture choices
+
+- **Language/runtime:** Python ≥ 3.11, `src`-layout, `from __future__ import
+  annotations` everywhere, full type hints on public APIs.
+- **Data modeling:** Pydantic v2 at every boundary; KG models forbid extra
+  fields. Schemas double as agent `response_format` and downstream validation.
+- **Determinism boundary:** library code is pure/deterministic except inside
+  `normalize` (for unstructured sources) and `agent`. Network IO is isolated in
+  `sources`/`agent`/`ontology` so the core is unit-testable offline.
+- **Interfaces over inheritance:** `SourceAdapter` and `Normalizer` are small
+  `Protocol`/ABC contracts. Adding a source = new adapter + normalizer, no edits
+  to orchestration or graph code.
+- **Errors & resilience:** external calls use bounded retries with jittered
+  exponential backoff; transient vs. terminal errors are distinguished.
+- **Config:** `pydantic-settings` + `.env`; never hardcode endpoints/secrets.
+- **Logging:** stdlib `logging` in libraries; `print` only for CLI summaries.
+- **Quality gates (CI-enforced):** `ruff check`, `ruff format --check`,
+  `mypy src/parce`, `pytest -m "not integration"`. See `pyproject.toml`.
+- **Tests:** offline unit tests by default; live/credentialed tests carry the
+  `integration` marker and are excluded from CI.
+
+## 6. Open questions (track, don't silently decide)
+
+- **Sample granularity for CELLxGENE.** Census is per-cell/dataset, not
+  per-sample in the GEO sense. Defer mapping cxg to `SampleNode` until needed;
+  keep it dataset-level for now.
+- **Ontology resolver dependency.** text2term vs. a thin OLS REST client —
+  decide when implementing `ontology/` (PR4); prefer the lighter dependency.
+- **Graph persistence/export format** for the modeling step (per-study context +
+  sample manifest + URIs). Specified in a later PR.
+- **Multi-omics is core, not optional.** CELLxGENE alone cannot carry the
+  vision; PRIDE (and beyond) is required, not a nice-to-have.
diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md
@@ -0,0 +1,82 @@
+# PARCE Roadmap
+
+Living plan. **Each session reads this to find its task and updates it before
+finishing.** See [AGENT_SESSION_PROMPT.md](AGENT_SESSION_PROMPT.md) for the
+protocol and [ARCHITECTURE.md](ARCHITECTURE.md) for the design.
+
+---
+
+## ▶ Next up
+
+**PR 2 — Canonical KG schema refactor.** Introduce the source-agnostic schema
+(`StudyNode`, `DatasetNode`, `SampleNode`, `BiologicalEntityNode`, `GraphEdge`),
+reintroduce sample-level covariates, and **remove the narrative field and the
+CellType entity**. Migrate `graph/builder.py` and tests. No new source yet.
+
+---
+
+## PR sequence
+
+Each PR is one branch, one focused scope, green CI, and a roadmap update.
+
+- [x] **PR 1 — Foundations & tooling.** CLAUDE.md, docs (architecture, roadmap,
+  session prompt), GitHub Actions CI (ruff, mypy, pytest), ruff rule set + mypy
+  config in `pyproject.toml`, code formatted to baseline. No behavior change.
+- [ ] **PR 2 — Canonical KG schema.** Source-agnostic nodes/edges; add
+  `SampleNode` with design covariates; drop `experimental_narrative` and
+  `CellType`. Migrate builder + tests. *(Next up.)*
+- [ ] **PR 3 — Source-adapter interface + cheap CELLxGENE adapter.** Define
+  `SourceAdapter` / `Normalizer` protocols in `sources/` + `normalize/`. Refactor
+  CELLxGENE into a deterministic adapter. **Remove the LLM/Azure narrative path
+  entirely** (delete `agent/prompts.py` narrative role, `models/narrative.py`,
+  the step-2 block in `main.py`). Drop cell-type extraction. Remove the
+  `parce.tools.*` mypy exemption as those modules move under `sources/`.
+- [ ] **PR 4 — Ontology resolver.** Shared `ontology/` stage: free-text →
+  UBERON/MONDO/assay IDs (deterministic OLS/text2term + on-disk cache), LLM
+  fallback for hard cases. Wire into normalizers.
+- [ ] **PR 5 — GEO extraction agent (vertical slice).** GEO adapter
+  (E-utilities/GEOparse) + Azure extraction normalizer emitting the canonical
+  schema via `response_format`; extract sample covariates from
+  `characteristics_ch1`. Integration test (marked). This is the agent's real
+  job; remove the `parce.agent.*` mypy exemption.
+- [ ] **PR 6 — Cross-source KG merge.** Merge CELLxGENE + GEO into one graph
+  linked through shared ontology entities; dedup; provenance on edges. Assert a
+  cross-source edge exists in tests.
+- [ ] **PR 7 — PRIDE proteomics adapter.** Second modality; prove the interface
+  is modality-general. Adapter + extraction normalizer + integration test.
+- [ ] **PR 8 — KG export for modeling.** Serialize per-study context + sample
+  manifest + data URIs in the form the downstream OQAE/model consumes.
+
+## Backlog / ideas
+
+- ENCODE adapter (clean structured epigenomics) if a deterministic second
+  modality is wanted.
+- Imaging modality (IDR / Human Protein Atlas) — needs a different OQAE encoder.
+- Graph database backend (Neo4j) vs. flat JSON export — revisit at PR 8.
+- Discovery agent: given a research theme, propose seed DOIs/accessions across
+  repositories.
+
+---
+
+## Session Log
+
+Newest first. One entry per working session: what changed, decisions made, and
+what the next session should know. Keep entries short and factual.
+
+### 2026-06-23 — PR 1: Foundations & tooling
+- Branch `restructure-context-metadata` off `main` (post-cxg-merge).
+- Decision: the LLM is repurposed from **narrative writing** to **structured
+  extraction from unstructured metadata**; Azure is kept to serve that (and as a
+  learning goal). Deterministic-by-default everywhere else.
+- Decision: multi-modality is core. Source gradient chosen: CELLxGENE (anchor,
+  no LLM) → GEO (first extraction target) → PRIDE (cross-modality proof).
+- Decision: context = design variables only; **cell type excluded** (data-
+  inferred → leakage). Sample-level covariates to be reintroduced into the KG.
+- Added: `CLAUDE.md`, `docs/ARCHITECTURE.md`, `docs/ROADMAP.md`,
+  `docs/AGENT_SESSION_PROMPT.md`, `.github/workflows/ci.yml`.
+- Tooling: ruff rule set (E,F,I,UP,B,SIM,C4,RUF) + `ruff format`; mypy with
+  pydantic plugin, `src/parce` checked, `parce.tools.*`/`parce.agent.*`
+  temporarily exempt (remove on migration). Formatted all files.
+- Gates green locally: ruff check, ruff format --check, mypy, 43 unit tests.
+- **Next session:** PR 2 (canonical KG schema). No code behavior changed yet;
+  `main.py` still runs the old narrative pipeline until PR 3.