decomposer: baseline wrappers (FActScore, MedScore, VeriScore) + evaluation

decomposer/README.md

@@ -1,9 +1,72 @@
 # AMFV Decomposer
 
-Breaks long-form text — answers, documents, model outputs, reasoning traces —
-into atomic, independently verifiable medical claims for the
-[verifier](../verifier/README.md), part of the
-[Agentic Medical Fact Verifier](../README.md).
+Breaks long-form text into atomic, independently verifiable medical claims.
+Part of the [Agentic Medical Fact Verifier](../README.md).
 
-Workspace member (`amfv-decomposer`). Training experiments live in
-[`../training`](../training/README.md).
+## Setup
+
+```bash
+uv sync
+```
+
+## Local evaluation
+
+```bash
+# Sanity check (5 records)
+python evaluate.py --data ../datasets/AskDocs.demo.jsonl --max-records 5
+
+# Full run
+python evaluate.py --data ../datasets/AskDocs.jsonl
+
+# With Qwen3 thinking enabled
+python evaluate.py --data ../datasets/AskDocs.jsonl --enable-thinking
+```
+
+## SLURM evaluation
+
+> Pass all mounts as one comma-separated `--container-mounts` — multiple flags override rather than merge in Pyxis.
+
+> Before the first run on a new home directory, create the log directory — SLURM opens the log file before the job script runs: `mkdir -p ~/amfv/decomposer/slurm`
+
+```bash
+# No-think (default)
+sbatch \
+    --container-mounts=/data/aymane.ouraq:/data/aymane.ouraq,/data/hf_cache:/data/hf_cache \
+    --gpus-per-task=1 \
+    decomposer/run_eval.sh \
+    --model Qwen/Qwen3-8B --tp 1 \
+    --data /admin/home/aymane.ouraq/amfv/datasets/AskDocs.jsonl
+
+# With thinking
+sbatch \
+    --container-mounts=/data/aymane.ouraq:/data/aymane.ouraq,/data/hf_cache:/data/hf_cache \
+    --gpus-per-task=1 \
+    decomposer/run_eval.sh \
+    --model Qwen/Qwen3-8B --tp 1 \
+    --data /admin/home/aymane.ouraq/amfv/datasets/AskDocs.jsonl \
+    --enable-thinking
+
+# VeriScore original (Mistral-7B PEFT)
+sbatch decomposer/run_eval_mistral.sh
+```
+
+`--gpus-per-task` must equal `--tp`. Extra vLLM server flags can be passed via the
+`VLLM_EXTRA_ARGS` env var (e.g. `VLLM_EXTRA_ARGS="--gpu-memory-utilization 0.95"`).
+
+Results land in `results/<model>[-think]/<dataset>/<timestamp>/`. Each
+`<method>_<dataset>.json` keeps the raw generations alongside parsed claims, so
+parsing changes can be re-scored offline without re-running inference.
+
+## Tests
+
+```bash
+python3 -m pytest tests/
+```
+
+## Reference numbers (MedScore paper, GPT-4o-mini, AskDocsAI)
+
+| Method    | Claims/Response | Claims/Sentence | 0-claim rate |
+|-----------|----------------|----------------|-------------|
+| FActScore | 28.60          | 4.24           | 0%          |
+| MedScore  | 13.62          | 2.02           | 0%          |
+| VeriScore | 3.87           | 0.57           | 14.67%      |

decomposer/amfv_decomposer/__init__.py

@@ -1 +1,12 @@
 """Medical claim decomposition component for AMFV."""
+
+from .base import BaseDecomposer
+from .baselines import FActScoreDecomposer, MedScoreDecomposer, VeriScoreDecomposer, VeriScoreOriginalDecomposer
+
+__all__ = [
+    "BaseDecomposer",
+    "FActScoreDecomposer",
+    "MedScoreDecomposer",
+    "VeriScoreDecomposer",
+    "VeriScoreOriginalDecomposer",
+]

decomposer/amfv_decomposer/base.py

@@ -0,0 +1,164 @@
+"""Base decomposer interface and shared text utilities."""
+
+from __future__ import annotations
+
+import re
+from abc import ABC, abstractmethod
+from typing import NamedTuple
+
+_nlp = None
+_SKIP_PREFIXES = (
+    "facts:",
+    "here are",
+    "here is",
+    "independent facts",
+    "the following",
+    "note:",
+    "based on",
+)
+_NO_CLAIM_PHRASES = frozenset({
+    "no verifiable claim.",
+    "no verifiable claim",
+    "none",
+    "n/a",
+    "no claims",
+    "no facts",
+})
+_LIST_MARKER_RE = re.compile(r"^(?:[-•*]|\d+[.)])\s+")
+_BOLD_LINE_RE = re.compile(r"^\*\*(.+)\*\*$")
+
+
+def _get_nlp():
+    global _nlp
+    if _nlp is None:
+        import spacy
+        try:
+            _nlp = spacy.load("en_core_web_sm", disable=["ner", "tagger", "lemmatizer"])
+        except OSError:
+            raise RuntimeError(
+                "spaCy model not found. Run: python -m spacy download en_core_web_sm"
+            )
+    return _nlp
+
+
+def split_sentences(text: str) -> list[str]:
+    """Split text into sentences using spaCy."""
+    nlp = _get_nlp()
+    doc = nlp(text)
+    return [s.text.strip() for s in doc.sents if s.text.strip()]
+
+
+def parse_claims(text: str) -> list[str]:
+    """Parse model output into clean claim strings, stripping list markers and noise.
+
+    Filters formatting noise observed in real runs: preamble lines
+    ("Here is the breakdown..."), markdown header lines ("**Independent Facts:**"),
+    and any line ending in a colon.
+    """
+    claims = []
+    for line in text.split("\n"):
+        line = _LIST_MARKER_RE.sub("", line.strip()).strip()
+        bold = _BOLD_LINE_RE.match(line)
+        if bold:
+            line = bold.group(1).strip()
+        if not line or line.endswith(":"):
+            continue
+        lower = line.lower()
+        if any(lower.startswith(p) for p in _SKIP_PREFIXES):
+            continue
+        if lower in _NO_CLAIM_PHRASES:
+            continue
+        claims.append(line)
+    return claims
+
+
+class RecordResult(NamedTuple):
+    """Decomposition output for a single record."""
+
+    claims: list[str]
+    raw_outputs: list[str]  # one generation per request, kept for offline re-scoring
+    n_sentences: int
+
+
+class BaseDecomposer(ABC):
+    """Shared decomposition pipeline; subclasses only build prompts.
+
+    The pipeline is: split sentences -> build_requests -> one flat generate
+    call across all records -> parse_output per generation. Batching at this
+    level keeps the inference client's concurrency cap as the only fan-out.
+    """
+
+    # "vllm": build_requests returns chat-message lists for the vLLM server.
+    # "hf": build_requests returns prompt strings for a local pipeline
+    #       loaded from `model_id`.
+    backend: str = "vllm"
+    model_id: str | None = None
+    # Record field that carries context (e.g. "question").
+    # None means the method is context-free by design (FActScore).
+    default_context_key: str | None = None
+    # When True, claims are deduplicated across the record in first-seen order
+    # (VeriScore: adjacent windows re-extract the same facts and the original
+    # counts each only once per response).
+    dedup_record: bool = False
+
+    @abstractmethod
+    def build_requests(self, text: str, sentences: list[str], context: str) -> list:
+        """Return one request payload per LLM call (typically one per sentence)."""
+
+    def parse_output(self, raw: str) -> list[str]:
+        """Parse one raw generation into claims. Override for non-default formats."""
+        return parse_claims(raw)
+
+    def postprocess_record(self, claims: list[str]) -> list[str]:
+        """Record-level cleanup applied to the flattened claims of one record."""
+        if self.dedup_record:
+            return list(dict.fromkeys(claims))
+        return claims
+
+    def decompose(self, text: str, context: str = "") -> list[str]:
+        """Decompose a single text into claims (library convenience API)."""
+        sentences = split_sentences(text)
+        if not sentences:
+            return []
+        outputs = self._generate(self.build_requests(text, sentences, context))
+        return self.postprocess_record([c for out in outputs for c in self.parse_output(out)])
+
+    def decompose_batch(
+        self,
+        records: list[dict],
+        text_key: str = "response",
+        context_key: str | None = None,
+    ) -> list[RecordResult]:
+        """Decompose all records in one flat generate call."""
+        # Explicit argument wins; fall back to the class-level default.
+        effective_key = context_key if context_key is not None else self.default_context_key
+
+        sentences_per_record: list[list[str]] = []
+        requests_per_record: list[list] = []
+        for r in records:
+            context = r.get(effective_key, "") if effective_key else ""
+            sentences = split_sentences(r[text_key])
+            sentences_per_record.append(sentences)
+            requests_per_record.append(self.build_requests(r[text_key], sentences, context))
+
+        flat_outputs = self._generate([req for reqs in requests_per_record for req in reqs])
+
+        results = []
+        pos = 0
+        for sentences, requests in zip(sentences_per_record, requests_per_record):
+            raw = flat_outputs[pos:pos + len(requests)]
+            pos += len(requests)
+            claims = self.postprocess_record([c for out in raw for c in self.parse_output(out)])
+            results.append(RecordResult(claims, raw, len(sentences)))
+        return results
+
+    def _generate(self, requests: list) -> list[str]:
+        if not requests:
+            return []
+        if self.backend == "hf":
+            if self.model_id is None:
+                raise ValueError(f"{type(self).__name__}: hf backend requires model_id")
+            from .hf_client import hf_generate
+            return hf_generate(requests, model_id=self.model_id)
+        from .vllm_client import chat_generate
+        return chat_generate(requests)

decomposer/amfv_decomposer/baselines/__init__.py

@@ -0,0 +1,11 @@
+from .factscore import FActScoreDecomposer
+from .medscore import MedScoreDecomposer
+from .veriscore import VeriScoreDecomposer
+from .veriscore_original import VeriScoreOriginalDecomposer
+
+__all__ = [
+    "FActScoreDecomposer",
+    "MedScoreDecomposer",
+    "VeriScoreDecomposer",
+    "VeriScoreOriginalDecomposer",
+]

decomposer/amfv_decomposer/baselines/factscore.py

@@ -0,0 +1,96 @@
+"""FActScore baseline: sentence-by-sentence decomposition, no context."""
+
+from __future__ import annotations
+
+from ..base import BaseDecomposer
+
+# Original FActScore few-shot prompt (biography-domain examples).
+# Source: Min et al. 2023, via the MedScore reference repo (medscore/prompts.py).
+# Note: the original repo retrieves an 8th demo per sentence via BM25 and adds
+# bio-specific post-processing and a "- "-only output parser. We instead follow
+# MedScore's frozen 8-demo prompt (its paper is our reference) and reuse the
+# shared splitter/parser so baselines differ only in their prompts.
+_PROMPT = """\
+Please breakdown the following sentence into independent facts: He made his acting debut in the film The Moon is the Sun's Dream (1992), and continued to appear in small and supporting roles throughout the 1990s.
+- He made his acting debut in the film.
+- He made his acting debut in The Moon is the Sun's Dream.
+- The Moon is the Sun's Dream is a film.
+- The Moon is the Sun's Dream was released in 1992.
+- After his acting debut, he appeared in small and supporting roles.
+- After his acting debut, he appeared in small and supporting roles throughout the 1990s.
+
+Please breakdown the following sentence into independent facts: He is also a successful producer and engineer, having worked with a wide variety of artists, including Willie Nelson, Tim McGraw, and Taylor Swift.
+- He is successful.
+- He is a producer.
+- He is an engineer.
+- He has worked with a wide variety of artists.
+- Willie Nelson is an artist.
+- He has worked with Willie Nelson.
+- Tim McGraw is an artist.
+- He has worked with Tim McGraw.
+- Taylor Swift is an artist.
+- He has worked with Taylor Swift.
+
+Please breakdown the following sentence into independent facts: In 1963, Collins became one of the third group of astronauts selected by NASA and he served as the back-up Command Module Pilot for the Gemini 7 mission.
+- Collins became an astronaut.
+- Collins became one of the third group of astronauts.
+- Collins became one of the third group of astronauts selected.
+- Collins became one of the third group of astronauts selected by NASA.
+- Collins became one of the third group of astronauts selected by NASA in 1963.
+- He served as the Command Module Pilot.
+- He served as the back-up Command Module Pilot.
+- He served as the Command Module Pilot for the Gemini 7 mission.
+
+Please breakdown the following sentence into independent facts: In addition to his acting roles, Bateman has written and directed two short films and is currently in development on his feature debut.
+- Bateman has acting roles.
+- Bateman has written two short films.
+- Bateman has directed two short films.
+- Bateman has written and directed two short films.
+- Bateman is currently in development on his feature debut.
+
+Please breakdown the following sentence into independent facts: Michael Collins (born October 31, 1930) is a retired American astronaut and test pilot who was the Command Module Pilot for the Apollo 11 mission in 1969.
+- Michael Collins was born on October 31, 1930.
+- Michael Collins is retired.
+- Michael Collins is an American.
+- Michael Collins was an astronaut.
+- Michael Collins was a test pilot.
+- Michael Collins was the Command Module Pilot.
+- Michael Collins was the Command Module Pilot for the Apollo 11 mission.
+- Michael Collins was the Command Module Pilot for the Apollo 11 mission in 1969.
+
+Please breakdown the following sentence into independent facts: He was an American composer, conductor, and musical director.
+- He was an American.
+- He was a composer.
+- He was a conductor.
+- He was a musical director.
+
+Please breakdown the following sentence into independent facts: She currently stars in the romantic comedy series, Love and Destiny, which premiered in 2019.
+- She currently stars in Love and Destiny.
+- Love and Destiny is a romantic comedy series.
+- Love and Destiny premiered in 2019.
+
+Please breakdown the following sentence into independent facts: During his professional career, McCoy played for the Broncos, the San Diego Chargers, the Minnesota Vikings, and the Jacksonville Jaguars.
+- McCoy played for the Broncos.
+- McCoy played for the Broncos during his professional career.
+- McCoy played for the San Diego Chargers.
+- McCoy played for the San Diego Chargers during his professional career.
+- McCoy played for the Minnesota Vikings.
+- McCoy played for the Minnesota Vikings during his professional career.
+- McCoy played for the Jacksonville Jaguars.
+- McCoy played for the Jacksonville Jaguars during his professional career.
+
+"""
+
+
+class FActScoreDecomposer(BaseDecomposer):
+    """Sentence-by-sentence decomposition with no context or verifiability filter."""
+
+    def build_requests(self, text: str, sentences: list[str], context: str) -> list:
+        """One user message per sentence; context is unused by design."""
+        return [
+            [{
+                "role": "user",
+                "content": f"{_PROMPT}Please breakdown the following sentence into independent facts: {sent}\n",
+            }]
+            for sent in sentences
+        ]