diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 0000000..8092b32 --- /dev/null +++ b/.dockerignore @@ -0,0 +1,51 @@ +# Build context for docker/Dockerfile.mcp and docker/Dockerfile.prefect (both build from the +# repo root). The Dockerfiles COPY only pyproject.toml, uv.lock, graph_config.yaml and parts of +# src/, so none of this reaches an image anyway; the point is not shipping a local virtualenv, +# secrets or graph dumps to the build daemon on every build (#98). + +# Secrets. .history/ is an editor plugin's local history and holds copies of .env. +.env +.env.* +.envrc +.history/ +*.pem +*.key + +# Local environments and build output +.venv/ +venv/ +venc/ +env/ +envs/ +dist/ +build/ +*.egg-info/ + +# Data: graph dumps, staged source documents, the Prefect Postgres volume +dumps/ +staging/ +data/ + +# Caches and bytecode (a host's .pyc would otherwise be copied along with src/) +**/__pycache__/ +**/*.py[cod] +.pytest_cache/ +.ruff_cache/ +.mypy_cache/ +.cache/ +.coverage +.coverage.* +htmlcov/ +.prefect/ +.prefect-test/ +.tmp_prefect_home/ + +# Nothing an image needs +.git/ +.github/ +.claude/ +.idea/ +.vscode/ +docs/ +tests/ +benchmarks/ diff --git a/CLAUDE.md b/CLAUDE.md index c0a1ec6..4a41a5f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -40,7 +40,7 @@ Docker network. ### Install Dependencies ```bash -uv sync +uv sync # installs the server deps plus the `pipeline` and `dev` groups (see Gotcha 8) # or for initial setup: just setup # runs uv sync + generates Pydantic models ``` @@ -208,10 +208,11 @@ ml-mcp/ │ ├── compose.stack.yml # neo4j + mcp-server, no host ports │ ├── compose.dev.yml # Override: republish the ports on 127.0.0.1 │ ├── compose.prefect.yml # Data pipeline only -│ ├── Dockerfile.mcp # MCP server image -│ └── Dockerfile.prefect # Data pipeline image +│ ├── Dockerfile.mcp # MCP server image ([project] deps only) +│ └── Dockerfile.prefect # Data pipeline image (+ `pipeline` group) +├── .dockerignore # Keeps .venv, .env, dumps/, staging/ out of the build context ├── graph_config.yaml # Master config: LLM settings, graph schema, prompts -├── pyproject.toml # Dependencies, ruff config, entry points +├── pyproject.toml # Server deps, `pipeline`/`dev` groups, ruff config, entry points ├── justfile # All dev commands └── .env.example # Environment variable template ``` @@ -1367,6 +1368,26 @@ fix for that is a second key, not a second attempt. 8. **Docker multi-stage builds** — `Dockerfile.mcp` uses `ghcr.io/astral-sh/uv:python3.12` as builder then copy to `python:3.12-slim`. This keeps images small. + **Each image installs only its own dependencies (#98).** `[project].dependencies` is what the + MCP server and the `kg` client import, and it is all `Dockerfile.mcp` installs + (`uv sync --no-default-groups`). Prefect, PyMuPDF, pytesseract, python-docx, bs4, Pillow + and `langchain-community` are the `pipeline` dependency group, which `Dockerfile.prefect` + adds with `--group pipeline`; ruff, datamodel-code-generator and pytest are `dev` and go + into neither image. `[tool.uv] default-groups` keeps a plain `uv sync` installing all + three, so local work and CI see no difference. + + The cost is one rule: a package the server or client needs at runtime belongs in + `[project]`, never in a group. The unit tests cannot catch a mistake there, since they run + with every group installed; the CI `integration` job does, because the server imports its + whole module tree before `/health` can answer. `langchain` is in `[project]` for that + reason even though nothing in `src/mcp_server` imports it — `langfuse.langchain` does, and + refuses to load without it. `fastmcp` is pinned `<3`, as its own startup banner asks. + + `.dockerignore` keeps `.venv`, `.env`, `.history/` (editor snapshots of `.env`), `dumps/`, + `staging/` and host `__pycache__` out of the build context. The Dockerfiles `COPY` named + paths only, so none of it reached an image before either; it was shipped to the build + daemon on every build, `.venv` alone being ~500 MB. + 9. **The containerised pipeline does not run the schedule** — `Dockerfile.prefect` installs from `uv.lock` (Prefect 3.6.11), so the version mismatch this used to warn about is gone. What is missing is the deployment: the image's `CMD` is only `prefect server start`, nothing invokes `serve_refresh` (`uv run prefect-refresh`), so the cron added in #51 exists on a developer machine and not in Docker. `compose.prefect.yml` is also its own stack with no `neo4j` service and no link to `mcp_network`, so the pipeline has no route to the graph. See #54. 10. **Graph schema** — `graph_schema` in `graph_config.yaml` enumerates 27 node labels and 33 relationship types (including the fallback `RELATED_TO`). Node labels and relationship types are closed and enforced at ingestion (see *Ingestion Extraction Quality*); changing either set means editing config and running `just generate-models`. Retrieval still reads the live schema from Neo4j, which may contain legacy labels/relationship types written before canonicalization until the graph is re-ingested. diff --git a/docker/Dockerfile.mcp b/docker/Dockerfile.mcp index e4c36e7..8d8fd94 100644 --- a/docker/Dockerfile.mcp +++ b/docker/Dockerfile.mcp @@ -7,7 +7,8 @@ ENV UV_NO_CACHE=1 COPY pyproject.toml uv.lock ./ -RUN uv sync --locked --no-dev --compile-bytecode +# Only [project] dependencies: no Prefect, PyMuPDF or dev tools in the serving image (#98). +RUN uv sync --locked --no-default-groups --compile-bytecode FROM python:3.12-slim-bookworm diff --git a/docker/Dockerfile.prefect b/docker/Dockerfile.prefect index 1ba1b9c..eb66090 100644 --- a/docker/Dockerfile.prefect +++ b/docker/Dockerfile.prefect @@ -9,7 +9,7 @@ RUN apt-get update \ && rm -rf /var/lib/apt/lists/* COPY pyproject.toml uv.lock ./ -RUN uv sync --locked --no-dev --compile-bytecode +RUN uv sync --locked --no-default-groups --group pipeline --compile-bytecode FROM python:3.12-slim-bookworm WORKDIR /app diff --git a/pyproject.toml b/pyproject.toml index db26782..b3e0fdf 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,27 +4,21 @@ version = "0.1.0" description = "Add your description here" readme = "README.md" requires-python = ">=3.11" +# What the MCP server and the kg client import, and nothing else: this is all that goes into +# the serving image. The pipeline's own libraries are the `pipeline` group below (#98). dependencies = [ - "beautifulsoup4>=4.15.0", - "datamodel-code-generator>=0.27.2", "dotenv>=0.9.9", - "fastmcp>=2.12.4", - "httpx>=0.28.0", + # fastmcp's own startup banner says to pin < 3 in production and upgrade deliberately. + "fastmcp>=2.12.4,<3", + # Nothing in src/mcp_server imports it, but langfuse.langchain.CallbackHandler does. "langchain>=0.3.27", - "langchain-community>=0.3.31", "langchain-google-genai>=2.0.0", "langchain-neo4j>=0.5.0", "langchain-openai>=0.3.35", "langfuse>=3.6.2", "langgraph>=0.6.10", - "pillow>=12.3.0", - "prefect>=3.6.7", "pydantic>=2.10.0", - "pymupdf>=1.28.2", - "pytesseract>=0.3.13", - "python-docx>=1.2.0", "pyyaml>=6.0.0", - "ruff>=0.14.1", ] [project.scripts] @@ -67,7 +61,26 @@ omit = ["src/scripts/*"] fail_under = 50 [dependency-groups] +# Ingestion and acquisition (src/data_pipeline, src/scripts). Dockerfile.prefect installs it, +# Dockerfile.mcp does not. +pipeline = [ + "beautifulsoup4>=4.15.0", + "httpx>=0.28.0", + "langchain-community>=0.3.31", + "pillow>=12.3.0", + "prefect>=3.6.7", + "pymupdf>=1.28.2", + "pytesseract>=0.3.13", + "python-docx>=1.2.0", +] dev = [ + "datamodel-code-generator>=0.27.2", "pytest>=9.1.1", "pytest-cov>=7.1.0", + "ruff>=0.14.1", ] + +[tool.uv] +# A plain `uv sync` still installs everything, so local work and CI are unchanged. The images +# opt out with --no-default-groups and name the groups they need. +default-groups = ["dev", "pipeline"] diff --git a/uv.lock b/uv.lock index 4df544c..81b1296 100644 --- a/uv.lock +++ b/uv.lock @@ -2715,8 +2715,8 @@ name = "pendulum" version = "3.1.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "python-dateutil" }, - { name = "tzdata" }, + { name = "python-dateutil", marker = "python_full_version < '3.13'" }, + { name = "tzdata", marker = "python_full_version < '3.13'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/23/7c/009c12b86c7cc6c403aec80f8a4308598dfc5995e5c523a5491faaa3952e/pendulum-3.1.0.tar.gz", hash = "sha256:66f96303560f41d097bee7d2dc98ffca716fbb3a832c4b3062034c2d45865015", size = 85930, upload-time = "2025-04-19T14:30:01.675Z" } wheels = [ @@ -4095,62 +4095,66 @@ name = "solvro-mcp" version = "0.1.0" source = { editable = "." } dependencies = [ - { name = "beautifulsoup4" }, - { name = "datamodel-code-generator" }, { name = "dotenv" }, { name = "fastmcp" }, - { name = "httpx" }, { name = "langchain" }, - { name = "langchain-community" }, { name = "langchain-google-genai" }, { name = "langchain-neo4j" }, { name = "langchain-openai" }, { name = "langfuse" }, { name = "langgraph" }, - { name = "pillow" }, - { name = "prefect" }, { name = "pydantic" }, - { name = "pymupdf" }, - { name = "pytesseract" }, - { name = "python-docx" }, { name = "pyyaml" }, - { name = "ruff" }, ] [package.dev-dependencies] dev = [ + { name = "datamodel-code-generator" }, { name = "pytest" }, { name = "pytest-cov" }, + { name = "ruff" }, +] +pipeline = [ + { name = "beautifulsoup4" }, + { name = "httpx" }, + { name = "langchain-community" }, + { name = "pillow" }, + { name = "prefect" }, + { name = "pymupdf" }, + { name = "pytesseract" }, + { name = "python-docx" }, ] [package.metadata] requires-dist = [ - { name = "beautifulsoup4", specifier = ">=4.15.0" }, - { name = "datamodel-code-generator", specifier = ">=0.27.2" }, { name = "dotenv", specifier = ">=0.9.9" }, - { name = "fastmcp", specifier = ">=2.12.4" }, - { name = "httpx", specifier = ">=0.28.0" }, + { name = "fastmcp", specifier = ">=2.12.4,<3" }, { name = "langchain", specifier = ">=0.3.27" }, - { name = "langchain-community", specifier = ">=0.3.31" }, { name = "langchain-google-genai", specifier = ">=2.0.0" }, { name = "langchain-neo4j", specifier = ">=0.5.0" }, { name = "langchain-openai", specifier = ">=0.3.35" }, { name = "langfuse", specifier = ">=3.6.2" }, { name = "langgraph", specifier = ">=0.6.10" }, - { name = "pillow", specifier = ">=12.3.0" }, - { name = "prefect", specifier = ">=3.6.7" }, { name = "pydantic", specifier = ">=2.10.0" }, - { name = "pymupdf", specifier = ">=1.28.2" }, - { name = "pytesseract", specifier = ">=0.3.13" }, - { name = "python-docx", specifier = ">=1.2.0" }, { name = "pyyaml", specifier = ">=6.0.0" }, - { name = "ruff", specifier = ">=0.14.1" }, ] [package.metadata.requires-dev] dev = [ + { name = "datamodel-code-generator", specifier = ">=0.27.2" }, { name = "pytest", specifier = ">=9.1.1" }, { name = "pytest-cov", specifier = ">=7.1.0" }, + { name = "ruff", specifier = ">=0.14.1" }, +] +pipeline = [ + { name = "beautifulsoup4", specifier = ">=4.15.0" }, + { name = "httpx", specifier = ">=0.28.0" }, + { name = "langchain-community", specifier = ">=0.3.31" }, + { name = "pillow", specifier = ">=12.3.0" }, + { name = "prefect", specifier = ">=3.6.7" }, + { name = "pymupdf", specifier = ">=1.28.2" }, + { name = "pytesseract", specifier = ">=0.3.13" }, + { name = "python-docx", specifier = ">=1.2.0" }, ] [[package]] @@ -4580,8 +4584,8 @@ name = "whenever" version = "0.9.5" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "tzdata", marker = "sys_platform == 'win32'" }, - { name = "tzlocal", marker = "sys_platform != 'darwin' and sys_platform != 'linux'" }, + { name = "tzdata", marker = "python_full_version >= '3.13' and sys_platform == 'win32'" }, + { name = "tzlocal", marker = "python_full_version >= '3.13' and sys_platform != 'darwin' and sys_platform != 'linux'" }, ] sdist = { url = "https://files.pythonhosted.org/packages/cd/9a/6b12d5708a703010877bec0efc5172ae8a851516abc13cd4543ce4d02a20/whenever-0.9.5.tar.gz", hash = "sha256:9d8f2fbc70acdab98a99b81a2ac594ebd4cc68d5b3506b990729a5f0b04d0083", size = 259436, upload-time = "2026-01-11T19:47:51.608Z" } wheels = [