devflow

A multi-agent coding pipeline built on LangGraph: Claude supervises, worker models write code and tests, and nothing reaches your repo unless the tests pass inside a sandbox.

Leer en español

devflow "create a todo package with a JSON store and a CLI, with tests" --repo ./myapp --apply

[mode claude] supervisor: claude-opus-5-5 | coder / tester: claude-sonnet-5-5 / claude-haiku-4-5
[graphrag] 0 nodes, 0 edges
[supervisor] -> coder: Empty project: have the coder create the whole package (three closely related modules) in one turn.
[coder] todo/__init__.py, todo/models.py, todo/store.py, todo/cli.py, todo/__main__.py
[supervisor] -> tester: The package is delivered and has no tests yet.
[tester] python -m pytest tests/ -q -> OK
[supervisor] -> FINISH: 81 tests pass. Nothing left to do.

Tests: PASS
9 file(s) written

(Log translated from Spanish, the language the prompts are written in.)

I built this to understand how agentic systems work under the hood, by assembling one from its parts instead of using a finished product. Each component below is there to solve a specific problem I ran into, and the What I learned section is honest about where the approach works and where it does not.

Architecture

flowchart LR
    S([START]) --> R[retriever<br/><i>GraphRAG</i>]
    R --> SUP{supervisor<br/><i>Claude Opus</i>}
    SUP -- coder --> C[coder<br/><i>Ollama or Sonnet</i>]
    SUP -- tester --> T[tester<br/><i>Ollama or Haiku</i>]
    T <--> SB[(sandbox<br/><i>Podman / E2B</i>)]
    C --> SUP
    T --> SUP
    SUP -- FINISH --> E([END])

Loading

This is the Supervisor pattern: workers never talk to each other. Every worker returns to a central node that reads the full state (task, code, test output) and decides what happens next. When tests fail, the supervisor reads the real output, works out whether the bug is in the code or in the test, and sends a targeted fix to the right worker.

Component	File	What it does here	Why it is needed
LangGraph	graph.py, state.py	State graph with conditional routing and typed shared state	The loop length is not known in advance; the route is decided at runtime
Claude	llm_claude.py, agents.py	Supervisor: plans, routes, diagnoses failures	Few calls, high leverage: this is where a strong model pays off
Ollama	llm_local.py	Serves a local coding model on a consumer GPU	High-volume generation at zero marginal cost, code stays on the machine
Outlines	llm_local.py	Constrains the local model's output to a Pydantic schema	A 14B model breaks free-form JSON often; constrained decoding makes that impossible
GraphRAG	codegraph.py	Builds a code graph (files, symbols, imports, calls) and retrieves along its edges	The relevant context for code is what a function calls and who calls it, not just text that looks similar
Podman / E2B	sandbox.py	Runs generated tests in a disposable container or microVM	Both the code and the shell commands were written by an LLM
LangSmith	env vars only	Traces every node, prompt and model call	Without traces, a failed run tells you nothing about which agent got it wrong

Two modes

Mode	Supervisor	Coder	Tester	Cost
local	Claude Opus	qwen2.5-coder:14b	qwen2.5-coder:14b	Only the supervisor's turns
claude	Claude Opus	Claude Sonnet	Claude Haiku	Every call

The graph is identical in both; only the worker backend changes (--mode local|claude). The supervisor's prompt adapts: with local workers it splits the task into small pieces, with Claude workers it batches related files into one turn. Retrieved context grows from 12k to 60k characters in claude mode.

Design decisions

• Hand-built graph instead of a prebuilt supervisor. The routing is about 30 lines in graph.py, and writing it out is what makes the control flow inspectable and testable.
• Structured output everywhere. The supervisor returns a Decision (next + instructions), the coder a CodeChange, the tester a TestPlan. Routing never depends on parsing free text.
• One local model for both workers. Two different models would swap in and out of 12 GB of VRAM on every turn.
• A code graph instead of a document GraphRAG library. Document-oriented GraphRAG uses an LLM to extract entities, which is slow on local hardware and unnecessary for code: a parser gives exact nodes and edges for free. Python is parsed with ast; JS/TS with regular expressions. Seeds are ranked with BM25, then expanded one hop.
• Call edges only for unambiguous names. Without type analysis, a name defined twice would produce an invented edge, so those are skipped.
• Two guard rails against loops. A hard cap on supervisor turns, and a token cap on the local model (constrained decoding can otherwise generate forever).
• Generated paths are sanitised. Absolute paths and .. are dropped, so a model cannot write outside the repo.
• Changes are gated on tests. With --apply, files are written to the repo only if the sandboxed tests pass; otherwise they go to runs/<timestamp>/ for inspection.

What I learned

Measured on an RX 6700 XT (12 GB VRAM), Ryzen 5 7600, 32 GB RAM:

Task	Mode	Supervisor turns	Result	Time
One function (slugify) + tests	local	4	14 tests pass, one test file had to be regenerated	1 min 43 s
Package with 5 modules + tests	claude	3	81 tests pass, no correction turns	3 min 31 s

These are single runs on different tasks, so they show how each mode behaves rather than a controlled comparison.

• The supervisor is what makes weak workers usable. The local model produced a corrupted line and left Markdown fences inside generated files. In both cases Opus caught it by reading the code or the test output and routed a precise fix. The fences are now stripped deterministically, which is cheaper than spending a supervisor turn.
• Constrained decoding guarantees shape, not termination. One run hung for over ten minutes with the model looping inside a JSON string. A token cap is mandatory.
• Context size is a VRAM budget. At a 16k context the model spilled 8% onto the CPU; at 10k it fits entirely on the GPU.
• A 14B model is the ceiling on 12 GB, and it is a real limit. It handles one bounded function well and struggles beyond that.
• For day-to-day coding, a mature coding agent is better than this pipeline. Each handoff between agents loses context, and the workers see a retrieved summary rather than the real repository. The value of this project is the architecture: the same pieces are what you need to build an AI feature where no finished product exists, such as document processing, support triage, or data analysis that executes generated code.

Limitations

• Retrieval is lexical (BM25): a task written in different words from the code retrieves poorly. Embeddings are the next step.
• Only Python and JS/TS are indexed.
• Workers return whole files, so large files are out of reach for the local model.
• Each run is independent: there is no task queue or memory between runs.
• The Podman sandbox has network access (tests need to install dependencies) and shares the host kernel. E2B gives stronger isolation.
• Browser testing with Playwright is not implemented.
• Prompts and code comments are in Spanish.

Setup

Requires Python 3.12, uv, Ollama, Podman, and Claude Code (claude) logged in.

git clone https://github.com/MguevaraJ/devflow && cd devflow
uv sync
ollama pull qwen2.5-coder:14b
cp .env.example .env    # add the LangSmith key (optional)

Claude calls go through claude -p (Claude Code's non-interactive mode) by default, so no API key is needed. Set SUPERVISOR_BACKEND=api and ANTHROPIC_API_KEY to use the Anthropic API instead.

Usage

uv run devflow "task" --repo PATH                 # output goes to runs/<timestamp>/
uv run devflow "task" --repo PATH --apply         # write to the repo if tests pass
uv run devflow "task" --repo PATH --mode claude   # Claude workers instead of Ollama
uv run devflow "task" --repo PATH --show-context  # only print what GraphRAG retrieves
uv run pytest                                     # pipeline tests with mocked LLMs

All models, the sandbox backend, context size and turn limit are configurable in .env.example.

License

MIT