Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 60 additions & 14 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,8 @@ ml-mcp/
│ │ ├── staging.py # Staging dir, manifest, atomic writes, source_id mapping
│ │ ├── completeness.py # List/table row coverage check for extracted pages
│ │ ├── label_vocabulary.py # Closed node-label set and the rewrite that enforces it
│ │ ├── relationship_vocabulary.py # Closed relationship-type set and canonical rewrites
│ │ ├── deterministic_topology.py # One type and direction for category->item edges
│ │ ├── canonical_nodes.py # Canonical merge keys (one node per real entity)
│ │ ├── title_sanity.py # Titles are entity names: no enumerators, no fragments
│ │ ├── ingestion_guardrails.py # Generated Cypher may only MERGE the page's own entities
Expand Down Expand Up @@ -194,7 +196,7 @@ ml-mcp/
│ ├── test_grader_stability_benchmark.py # Replay harness: same rows, N grader verdicts
│ ├── test_question_analysis.py # Question-literal detection, phrase extraction
│ ├── test_llm_determinism_config.py # Both models pinned to temperature 0
│ ├── test_graph_schema_config.py # Closed label set stays internally consistent
│ ├── test_graph_schema_config.py # Closed graph vocabularies stay internally consistent
│ ├── test_schema_visibility.py # Bookkeeping labels never reach the Cypher prompt
│ ├── test_logging_config.py # LOG_LEVEL resolution and root-logger setup
│ ├── test_no_print_in_src.py # src/ logs instead of printing (AST walk)
Expand Down Expand Up @@ -365,11 +367,12 @@ entities, or it is refused (see *Ingestion Cypher Can Only MERGE the Page's Own

### Ingestion Extraction Quality

Three deterministic passes run over the model's output in `llm_cypher_generation`, in this
order. Each exists because the prompt alone only gets it right most of the time, and "most of
the time" is what silently corrupts a knowledge graph.
Five deterministic passes run over the model's output in `llm_cypher_generation`, in this
execution order: completeness → vocabularies → title sanity → topology → canonical keys.
Each exists because the prompt alone only gets it right most of the time, and "most of the
time" is what silently corrupts a knowledge graph. The sections below are grouped by concern.

**1. Completeness (`completeness.py`).** The prompt requires every list/table row to become its
**Completeness (`completeness.py`).** The prompt requires every list/table row to become its
own node. Rows are also counted from the source page and checked against the generated nodes: a
row counts as covered when one *single* node both carries most of the row's wording and has a
`title` that lines up with the row — in either direction, since a title may be the row's short
Expand Down Expand Up @@ -420,14 +423,55 @@ competencies as nodes while R1–R3 kept theirs only as text inside a category n
`Jakie są kompetencje pożądane dla naukowca R2?` answered "Nie wiem". Both pages are regression
cases in `tests/data_pipeline/test_completeness.py`.

**2. Closed label set (`label_vocabulary.py`).** `graph_schema.node_labels` in
**Closed extraction vocabularies (`label_vocabulary.py`, `relationship_vocabulary.py`).**
`graph_schema.node_labels` in
`graph_config.yaml` is the only vocabulary; `graph_schema.label_aliases` maps known drift
(`Program`→`StudyProgram`, `CriterionItem`→`Criterion`, `Holiday`→`DayOff`, …) and anything still
unrecognised becomes `graph_schema.fallback_label`. Matching ignores case and diacritics. Only
node labels are rewritten — relationship types and quoted values are left verbatim, since a
wrong relationship type is visible in the traversal while a wrong label silently hides the node.

**3. Canonical merge keys (`canonical_nodes.py`).** A node MERGE keys on `key`, a normalized form
unrecognised becomes `graph_schema.fallback_label`.

Relationship names are canonicalized too. This changed after #104: the same page could produce
different relationship types across runs (`HAS_CRITERION` in one run, `REQUIRES`/`RECOMMENDS` in
another), so "the wrong type is visible in traversal" stopped being a useful safety argument —
the type itself was not stable. `graph_schema.relationship_aliases` maps known drift
(`WORKS_IN`→`EMPLOYED_BY`, …), and unknown types fall back to
`graph_schema.fallback_relationship_type` during rewrite. Prompt payloads list only allowed
relationship names (excluding the fallback), so the generic type is a safety net for ingestion,
not a preferred modeling choice.

Relationship matching ignores case, diacritics, underscores, hyphens and spaces
(`works-at`, `works_at`, `WORKSAT` all normalize together), and quoted values are left untouched.
Before adding a new relationship alias, inspect the live graph first:
`MATCH ()-[r]->() RETURN type(r) AS t, count(*) AS n ORDER BY n DESC`.

**Category -> item edges (`deterministic_topology.py`).** The same page attached R2's
competencies by `HAS_CRITERION` on one run and by `REQUIRES`/`RECOMMENDS` on the next,
sometimes pointing from the item to the category. The topology pass enforces one shape:
category -> item over configured pairs (`graph_schema.category_item_labels`). The pair decides
which edges those are, not their type: an item the model hung under its category by
`HAS_SUBCOMPETENCY` is rewritten, where counting only four types once left that edge and added
a second one beside it.

A criterion edge is always `HAS_CRITERION`. A competency edge takes the nearest qualifier above
the item's title (`graph_schema.relationship_qualifier_rules`), or the page's single qualifier
when the title cannot be found. Qualifiers are read from heading lines only — short, not a list
row, not ending a sentence — because policy pages say "wymagane dokumenty" in prose and that
turned criterion edges into `REQUIRES`. The R1–R4 page title still counts: it is a heading, and
it sits above the stage headings rather than between a stage and its items.

A `Topic` end takes the other half of its pair only when the edge points the category's way: a
`Topic` pointing at an item is its category, one a category points at is its item. A `Topic`
pointing at a category is the group heading above it, and relabelling it inverted the
hierarchy. A node carrying another pair's item label takes its category's item label in either
direction, since an item label never names a heading: on the real graph R2's competencies were
`Criterion` under a `CompetencyCategory`, a pair neither configured pair matched, so the pass
used to leave R2 — the example this issue opens with — exactly as the model wrote it. A node
linked to categories of two different pairs keeps its label. An unlinked item is attached to the nearest category title above its row when the
pair matches. Statements are rewritten in place or appended, never dropped, so one that also
binds a node keeps binding it.
`tests/data_pipeline/test_extraction_reproducibility.py` checks both node-label drift and
edge-shape drift converge to one signature.

**Canonical merge keys (`canonical_nodes.py`).** A node MERGE keys on `key`, a normalized form
of the title (case folded, diacritics folded, punctuation dropped), never on `title + context`.

A trailing bracket is dropped from the key **only when it abbreviates the title** — `(CBE)`,
Expand Down Expand Up @@ -470,7 +514,7 @@ nothing downstream can reintroduce the shape. `pipeline.py` is its only caller
`scripts/populate_graph.py` builds its own driver and runs its own MERGE blocks, and nothing in
the repo stores generated Cypher or replays it.

**4. Title sanity (`title_sanity.py`).** A title is the entity's name. What extraction kept
**Title sanity (`title_sanity.py`).** A title is the entity's name. What extraction kept
writing was the line the name was copied from: `a) doswiadczenie w kierowaniu i pracy w
zespolach naukowych.`, `Odbyte szkolenia:`, `Udzial w`, `zagranicznych` (issue #79).

Expand Down Expand Up @@ -936,7 +980,9 @@ the model gets. `tests/test_schema_visibility.py` builds its input by calling th
fails there instead of quietly letting the bookkeeping back in.

The pipeline's own `reflect_on_schema` still reads the unfiltered schema; it feeds the
extraction model, not this one, and `label_vocabulary` bounds what it can produce.
extraction model, not this one, and `label_vocabulary` with `relationship_vocabulary` bound what
it can produce. Existing edges written before relationship canonicalization remain as-is until
the graph is re-ingested.

### Abstention Is a Retrieval Decision

Expand Down Expand Up @@ -1323,6 +1369,6 @@ fix for that is a second key, not a second attempt.

9. **The containerised pipeline does not run the schedule** — `Dockerfile.prefect` installs from `uv.lock` (Prefect 3.6.11), so the version mismatch this used to warn about is gone. What is missing is the deployment: the image's `CMD` is only `prefect server start`, nothing invokes `serve_refresh` (`uv run prefect-refresh`), so the cron added in #51 exists on a developer machine and not in Docker. `compose.prefect.yml` is also its own stack with no `neo4j` service and no link to `mcp_network`, so the pipeline has no route to the graph. See #54.

10. **Graph schema** — `graph_schema` in `graph_config.yaml` enumerates 27 node labels and 32 relationship types. The label set is closed and enforced at ingestion (see *Ingestion Extraction Quality*); adding a label means editing the config and running `just generate-models`. Retrieval still reads the live schema from Neo4j, which may also contain labels written before the set was enforced until the dedup pass relabels them.
10. **Graph schema** — `graph_schema` in `graph_config.yaml` enumerates 27 node labels and 33 relationship types (including the fallback `RELATED_TO`). Node labels and relationship types are closed and enforced at ingestion (see *Ingestion Extraction Quality*); changing either set means editing config and running `just generate-models`. Retrieval still reads the live schema from Neo4j, which may contain legacy labels/relationship types written before canonicalization until the graph is re-ingested.

11. **Pipeline CLI exit code** — `data_pipeline_flow()` returns a `PipelineOutcome`, so the `prefect_pipeline` script must point at `src.data_pipeline.cli:prefect_pipeline_main`. Wiring it straight to the flow makes the generated wrapper call `sys.exit(PipelineOutcome(...))`, which exits `1` after a successful run. Page failures are not fatal, so `PipelineOutcome.failed` carries them: the wrapper then exits non-zero and `refresh_sources_flow` raises.
10 changes: 10 additions & 0 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,3 +77,13 @@ and #108, not captured from the production graph. With `NEO4J_URI`, `NEO4J_USER`
`NEO4J_PASSWORD` pointing at a real graph, `--capture` replaces each case's rows with what the
full-text search returns and drops `expected`, whose indices named the old rows. Replaying
needs only an LLM key; `--only <case id>` limits a run to one case.

# Extraction reproducibility (#104)

`run_extraction_reproducibility.py` runs the real extraction model on one page several times and
compares the label and relationship-type counts of each run. It exits non-zero when the runs
disagree. Save the page's extracted text to a file first:

```bash
uv run python -m benchmarks.run_extraction_reproducibility page.txt --runs 3
```
54 changes: 54 additions & 0 deletions benchmarks/run_extraction_reproducibility.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
from __future__ import annotations

import argparse
import re
from collections import Counter
from pathlib import Path

from prefect.logging import disable_run_logger

from src.data_pipeline.flows.llm_cypher_generation import generate_cypher_queries

LABEL_RE = re.compile(r"\(\s*[A-Za-z_]\w*\s*:\s*(?P<label>[A-Za-z_]\w*)")
REL_TYPE_RE = re.compile(r"\[:(?P<relationship_type>[A-Z_]+)\]")


def shape_signature(cypher: str) -> tuple[list[tuple[str, int]], list[tuple[str, int]]]:
labels = Counter(match.group("label") for match in LABEL_RE.finditer(cypher))
types = Counter(match.group("relationship_type") for match in REL_TYPE_RE.finditer(cypher))
return sorted(labels.items()), sorted(types.items())


def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser()
parser.add_argument("page_file", type=Path)
parser.add_argument("--runs", type=int, default=3)
return parser.parse_args()


def main() -> int:
args = parse_args()
if args.runs < 1:
raise ValueError("--runs must be >= 1")

page_text = args.page_file.read_text(encoding="utf-8")
signatures: list[tuple[list[tuple[str, int]], list[tuple[str, int]]]] = []
for _ in range(args.runs):
with disable_run_logger():
cypher = generate_cypher_queries.fn(page_text)
signatures.append(shape_signature(cypher))

print("run\tlabels\trelationships")
for index, (labels, relationships) in enumerate(signatures, start=1):
print(f"{index}\t{labels}\t{relationships}")

unique = {repr(signature) for signature in signatures}
if len(unique) != 1:
print("reproducibility check failed: signatures differ across runs")
return 1
print("reproducibility check passed: all signatures match")
return 0


if __name__ == "__main__":
raise SystemExit(main())
49 changes: 40 additions & 9 deletions graph_config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -214,8 +214,6 @@ graph_schema:
- invented: "Article"
canonical: "Document"

# Preferred relationship types. Unlike labels these are not rewritten, because a wrong
# relationship type is visible in the traversal while a wrong label silently hides the node.
relationship_types:
- "PART_OF"
- "BELONGS_TO"
Expand Down Expand Up @@ -249,6 +247,41 @@ graph_schema:
- "CONSIDERS"
- "ASSESSES"
- "DEFINED_IN"
- "RELATED_TO"

relationship_aliases:
- invented: "WORKS_IN"
canonical: "EMPLOYED_BY"
- invented: "WORKS_AT"
canonical: "EMPLOYED_BY"
- invented: "IS_PART_OF"
canonical: "PART_OF"
- invented: "SUGGESTS"
canonical: "RECOMMENDS"
- invented: "NEEDS"
canonical: "REQUIRES"

fallback_relationship_type: "RELATED_TO"

category_item_labels:
- category: "CompetencyCategory"
item: "Competency"
- category: "CriterionCategory"
item: "Criterion"

relationship_qualifier_rules:
- relationship_type: "RECOMMENDS"
token_stems:
- "pozadan"
words:
- "pozadane"
- relationship_type: "REQUIRES"
token_stems:
- "wymagan"
- "niezbedn"
words:
- "wymagane"
- "niezbedne"

database:
name: "neo4j"
Expand Down Expand Up @@ -319,7 +352,7 @@ prompts:
ALLOWED NODE LABELS - use these and nothing else:
{node_labels}

PREFERRED RELATIONSHIP TYPES - reuse one of these whenever it fits:
ALLOWED RELATIONSHIP TYPES - use only these names:
{relationship_types}

1. COMPLETENESS - this is the most important rule:
Expand Down Expand Up @@ -374,7 +407,7 @@ prompts:

5. FOR RELATIONSHIPS:
- MERGE between existing node variables
- Use UPPER_SNAKE_CASE; prefer a type from PREFERRED RELATIONSHIP TYPES
- Use UPPER_SNAKE_CASE and choose only from ALLOWED RELATIONSHIP TYPES
- Direction matters (A->B is not B->A)

EXAMPLE OUTPUT:
Expand All @@ -400,7 +433,7 @@ prompts:
ALLOWED NODE LABELS - use these and nothing else:
{node_labels}

PREFERRED RELATIONSHIP TYPES - reuse one of these whenever it fits:
ALLOWED RELATIONSHIP TYPES - use only these names:
{relationship_types}

RULES:
Expand Down Expand Up @@ -489,10 +522,8 @@ prompts:
WRONG, matches nothing:
WHERE toLower(g.title) CONTAINS toLower('jakie sa kryteria doboru kandydatki lub kandydata')
- "Jakie są kompetencje pożądane dla naukowca R2?" names the entity by its code "R2", so
filter on the code, not on the words around it, and collect the items under it. The
question also says which items it wants: "pożądane" is the RECOMMENDS relationship and
"niezbędne"/"wymagane" is REQUIRES, so traverse only that type, never a union of types,
which mixes the two lists into one:
filter on the code, not on the words around it, and collect the items under it.
{relationship_qualifier_guidance}
MATCH (c:CompetencyCategory)-[r:RECOMMENDS]->(i)
WHERE toLower(c.title) CONTAINS toLower('r2')
RETURN c.title, type(r), collect(i.title) AS items
Expand Down
22 changes: 21 additions & 1 deletion src/config/config_models.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# generated by datamodel-codegen:
# filename: graph_config.yaml
# timestamp: 2026-09-16T09:54:18+00:00
# timestamp: 2026-09-26T09:41:51+00:00

from __future__ import annotations

Expand Down Expand Up @@ -86,11 +86,31 @@ class LabelAliase(BaseModel):
canonical: str


class RelationshipAliase(BaseModel):
invented: str
canonical: str


class CategoryItemLabel(BaseModel):
category: str
item: str


class RelationshipQualifierRule(BaseModel):
relationship_type: str
token_stems: list[str]
words: list[str]


class GraphSchema(BaseModel):
fallback_label: str
node_labels: list[str]
label_aliases: list[LabelAliase]
relationship_types: list[str]
relationship_aliases: list[RelationshipAliase]
fallback_relationship_type: str
category_item_labels: list[CategoryItemLabel]
relationship_qualifier_rules: list[RelationshipQualifierRule]


class Database(BaseModel):
Expand Down
44 changes: 44 additions & 0 deletions src/config/relationship_qualifiers.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
from src.config.config_models import GraphSchema
from src.text_normalization import normalize_search_text


def get_relationship_qualifier_rules(
schema: GraphSchema,
) -> tuple[tuple[str, tuple[str, ...]], ...]:
"""Return qualifier->relationship mapping from config.

Args:
schema: ``graph_schema`` section of loaded config

Returns:
Tuple of ``(relationship_type, token_stems)``
"""
rules: list[tuple[str, tuple[str, ...]]] = []
for rule in schema.relationship_qualifier_rules:
rules.append(
(
rule.relationship_type,
tuple(normalize_search_text(stem).strip() for stem in rule.token_stems),
)
)
return tuple(rules)


def render_relationship_qualifier_guidance(schema: GraphSchema) -> str:
"""Render one prompt sentence describing qualifier->relationship mapping.

Args:
schema: ``graph_schema`` section of loaded config

Returns:
One sentence suitable for ``prompts.cypher_search``
"""
mappings: list[str] = []
for rule in schema.relationship_qualifier_rules:
words = ", ".join(f'"{word}"' for word in rule.words)
mappings.append(f"{words} -> {rule.relationship_type}")
return (
"The question also says which items it wants: qualifiers map as "
f"{'; '.join(mappings)}; traverse only that type, never a union of types, "
"which mixes the two lists into one:"
)
Loading
Loading