Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# Build context for docker/Dockerfile.mcp and docker/Dockerfile.prefect (both build from the
# repo root). The Dockerfiles COPY only pyproject.toml, uv.lock, graph_config.yaml and parts of
# src/, so none of this reaches an image anyway; the point is not shipping a local virtualenv,
# secrets or graph dumps to the build daemon on every build (#98).

# Secrets. .history/ is an editor plugin's local history and holds copies of .env.
.env
.env.*
.envrc
.history/
*.pem
*.key

# Local environments and build output
.venv/
venv/
venc/
env/
envs/
dist/
build/
*.egg-info/

# Data: graph dumps, staged source documents, the Prefect Postgres volume
dumps/
staging/
data/

# Caches and bytecode (a host's .pyc would otherwise be copied along with src/)
**/__pycache__/
**/*.py[cod]
.pytest_cache/
.ruff_cache/
.mypy_cache/
.cache/
.coverage
.coverage.*
htmlcov/
.prefect/
.prefect-test/
.tmp_prefect_home/

# Nothing an image needs
.git/
.github/
.claude/
.idea/
.vscode/
docs/
tests/
benchmarks/
29 changes: 25 additions & 4 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ Docker network.

### Install Dependencies
```bash
uv sync
uv sync # installs the server deps plus the `pipeline` and `dev` groups (see Gotcha 8)
# or for initial setup:
just setup # runs uv sync + generates Pydantic models
```
Expand Down Expand Up @@ -208,10 +208,11 @@ ml-mcp/
│ ├── compose.stack.yml # neo4j + mcp-server, no host ports
│ ├── compose.dev.yml # Override: republish the ports on 127.0.0.1
│ ├── compose.prefect.yml # Data pipeline only
│ ├── Dockerfile.mcp # MCP server image
│ └── Dockerfile.prefect # Data pipeline image
│ ├── Dockerfile.mcp # MCP server image ([project] deps only)
│ └── Dockerfile.prefect # Data pipeline image (+ `pipeline` group)
├── .dockerignore # Keeps .venv, .env, dumps/, staging/ out of the build context
├── graph_config.yaml # Master config: LLM settings, graph schema, prompts
├── pyproject.toml # Dependencies, ruff config, entry points
├── pyproject.toml # Server deps, `pipeline`/`dev` groups, ruff config, entry points
├── justfile # All dev commands
└── .env.example # Environment variable template
```
Expand Down Expand Up @@ -1367,6 +1368,26 @@ fix for that is a second key, not a second attempt.

8. **Docker multi-stage builds** — `Dockerfile.mcp` uses `ghcr.io/astral-sh/uv:python3.12` as builder then copy to `python:3.12-slim`. This keeps images small.

**Each image installs only its own dependencies (#98).** `[project].dependencies` is what the
MCP server and the `kg` client import, and it is all `Dockerfile.mcp` installs
(`uv sync --no-default-groups`). Prefect, PyMuPDF, pytesseract, python-docx, bs4, Pillow
and `langchain-community` are the `pipeline` dependency group, which `Dockerfile.prefect`
adds with `--group pipeline`; ruff, datamodel-code-generator and pytest are `dev` and go
into neither image. `[tool.uv] default-groups` keeps a plain `uv sync` installing all
three, so local work and CI see no difference.

The cost is one rule: a package the server or client needs at runtime belongs in
`[project]`, never in a group. The unit tests cannot catch a mistake there, since they run
with every group installed; the CI `integration` job does, because the server imports its
whole module tree before `/health` can answer. `langchain` is in `[project]` for that
reason even though nothing in `src/mcp_server` imports it — `langfuse.langchain` does, and
refuses to load without it. `fastmcp` is pinned `<3`, as its own startup banner asks.

`.dockerignore` keeps `.venv`, `.env`, `.history/` (editor snapshots of `.env`), `dumps/`,
`staging/` and host `__pycache__` out of the build context. The Dockerfiles `COPY` named
paths only, so none of it reached an image before either; it was shipped to the build
daemon on every build, `.venv` alone being ~500 MB.

9. **The containerised pipeline does not run the schedule** — `Dockerfile.prefect` installs from `uv.lock` (Prefect 3.6.11), so the version mismatch this used to warn about is gone. What is missing is the deployment: the image's `CMD` is only `prefect server start`, nothing invokes `serve_refresh` (`uv run prefect-refresh`), so the cron added in #51 exists on a developer machine and not in Docker. `compose.prefect.yml` is also its own stack with no `neo4j` service and no link to `mcp_network`, so the pipeline has no route to the graph. See #54.

10. **Graph schema** — `graph_schema` in `graph_config.yaml` enumerates 27 node labels and 33 relationship types (including the fallback `RELATED_TO`). Node labels and relationship types are closed and enforced at ingestion (see *Ingestion Extraction Quality*); changing either set means editing config and running `just generate-models`. Retrieval still reads the live schema from Neo4j, which may contain legacy labels/relationship types written before canonicalization until the graph is re-ingested.
Expand Down
3 changes: 2 additions & 1 deletion docker/Dockerfile.mcp
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@ ENV UV_NO_CACHE=1

COPY pyproject.toml uv.lock ./

RUN uv sync --locked --no-dev --compile-bytecode
# Only [project] dependencies: no Prefect, PyMuPDF or dev tools in the serving image (#98).
RUN uv sync --locked --no-default-groups --compile-bytecode

FROM python:3.12-slim-bookworm

Expand Down
2 changes: 1 addition & 1 deletion docker/Dockerfile.prefect
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ RUN apt-get update \
&& rm -rf /var/lib/apt/lists/*

COPY pyproject.toml uv.lock ./
RUN uv sync --locked --no-dev --compile-bytecode
RUN uv sync --locked --no-default-groups --group pipeline --compile-bytecode

FROM python:3.12-slim-bookworm
WORKDIR /app
Expand Down
35 changes: 24 additions & 11 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,27 +4,21 @@ version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.11"
# What the MCP server and the kg client import, and nothing else: this is all that goes into
# the serving image. The pipeline's own libraries are the `pipeline` group below (#98).
dependencies = [
"beautifulsoup4>=4.15.0",
"datamodel-code-generator>=0.27.2",
"dotenv>=0.9.9",
"fastmcp>=2.12.4",
"httpx>=0.28.0",
# fastmcp's own startup banner says to pin < 3 in production and upgrade deliberately.
"fastmcp>=2.12.4,<3",
# Nothing in src/mcp_server imports it, but langfuse.langchain.CallbackHandler does.
"langchain>=0.3.27",
"langchain-community>=0.3.31",
"langchain-google-genai>=2.0.0",
"langchain-neo4j>=0.5.0",
"langchain-openai>=0.3.35",
"langfuse>=3.6.2",
"langgraph>=0.6.10",
"pillow>=12.3.0",
"prefect>=3.6.7",
"pydantic>=2.10.0",
"pymupdf>=1.28.2",
"pytesseract>=0.3.13",
"python-docx>=1.2.0",
"pyyaml>=6.0.0",
"ruff>=0.14.1",
]

[project.scripts]
Expand Down Expand Up @@ -67,7 +61,26 @@ omit = ["src/scripts/*"]
fail_under = 50

[dependency-groups]
# Ingestion and acquisition (src/data_pipeline, src/scripts). Dockerfile.prefect installs it,
# Dockerfile.mcp does not.
pipeline = [
"beautifulsoup4>=4.15.0",
"httpx>=0.28.0",
"langchain-community>=0.3.31",
"pillow>=12.3.0",
"prefect>=3.6.7",
"pymupdf>=1.28.2",
"pytesseract>=0.3.13",
"python-docx>=1.2.0",
]
dev = [
"datamodel-code-generator>=0.27.2",
"pytest>=9.1.1",
"pytest-cov>=7.1.0",
"ruff>=0.14.1",
]

[tool.uv]
# A plain `uv sync` still installs everything, so local work and CI are unchanged. The images
# opt out with --no-default-groups and name the groups they need.
default-groups = ["dev", "pipeline"]
54 changes: 29 additions & 25 deletions uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading