sprite-forge-mcp

English · 日本語

Type an idea → get a transparent, game-ready character sprite. Same engine, drivable by you (WebUI) or your AI agent (MCP).

_{One character bible, generated from a single base sprite — a consistent turnaround + expressions + actions + alt costumes. And every sprite comes out as clean, transparent RGBA:}

sprite-forge is a local ComfyUI studio that turns a text idea into a transparent RGBA sprite, a full character bible, a character LoRA, and then that character in any pose — with hard-won production rules baked into quality gates, so broken assets get caught before they reach your game.

One Python backend, two faces: a step-flow WebUI for humans and a FastMCP server for agents (Claude / Codex / any MCP client) — both calling the same backend/services.py, so the gates can't be bypassed from either side. Heavy generation runs on a remote ComfyUI GPU box; the backend is a thin orchestrator that does the deterministic, gate-keeping work.

MCP = Model Context Protocol — the open standard that lets agents like Claude Code plug capabilities into an AI.

Status: runs end-to-end on a real RTX 5090 + Mac setup — the matte, character-bible, and character-LoRA pipelines are all working. MIT. Needs a CUDA GPU running ComfyUI (see Requirements).

Why

Making character sprites for a small game with image models is a special kind of pain, and I hit all of it shipping assets for my own RPG:

• Chroma-key black leaks — the "closed black" trapped between an arm and a ribbon survives a naive flood-fill; one accepted sprite had 200k+ leaked pixels.
• Damage-version drift — the -damaged version comes back re-posed and a few pixels off, so it no longer lines up with the base in-game and the CSS layout breaks. "Don't move the original by a single dot."
• Style drift — drop the retro-pixel phrase and the model slides into glossy anime.
• Manual touch-ups never work — repainting cloth/skin/outline by hand had a success rate of zero.
• The version explosion — "which one looks right" is subjective, so one sprite ground out to v39.

So sprite-forge doesn't just call a model — it turns each of those scars into a gate. Outputs are forced to RGBA with transparent corners; damaged variants are machine-checked to match the base bbox within ≤1 px; the mandatory style phrase is auto-injected; there is no hand-paint tool (you point with masks/dots, the model regenerates); and nothing reaches your game folder without passing the adopt gate. A broken asset can't silently reach your game — it fails a gate instead.

Why not just use ComfyUI directly?

You already have ComfyUI — that's the engine here. sprite-forge adds the parts ComfyUI doesn't:

• Two faces, one logic — a guided WebUI and an MCP server for agents, sharing one gated backend. Your AI agent can run the whole pipeline end to end; you don't hand-wire node graphs.
• A character pipeline, not one-off images — idea → sprite → a consistent bible → LoRA → that character in any pose, as a single flow.
• Production gates — transparent-corner RGBA, ≤1 px damage-version alignment, mandatory style, audited adopt. Raw ComfyUI will happily hand you a broken sprite; this won't.
• Reproducible, no hand-paint — you point with masks/dots and regenerate; no manual pixel rescue.

Architecture

flowchart LR
    H["🧑‍🎨 Human<br/>step-flow WebUI"] -->|HTTP / SSE| B
    A["🤖 Agent<br/>Claude / Codex · MCP"] -->|/mcp · 15 self-documenting tools| B
    subgraph control["control machine — Mac / Linux / Windows"]
      B["sprite-forge backend<br/>FastAPI + FastMCP → shared services.py<br/><b>quality gates live here</b>"]
    end
    subgraph gpu["GPU box — RTX-class, ~30 GB VRAM"]
      C["ComfyUI<br/>SDXL · Qwen-Image-Edit · ControlNet · LoRA"]
    end
    B -->|HTTP + WebSocket, LAN| C
    B -.->|adopt — gated, opt-in| G["your game's sprites/"]

Loading

The backend can run on the same machine as ComfyUI or a different one (it just needs SPRITEFORGE_COMFY_URL). It does no local inference — that is by design.

The pipeline — three stages, each one stops for correction

① samples   create_character → add_samples / remove_sample / set_caption → look at samples.png
② LoRA      train_character_lora (minutes, only when asked) → preview_character (seconds, any words, any seed)
③ bible     generate_character_bible (23 panels, ~3 min) → redraw_panel (one panel by words) → generate_from_bible
   styles   create_style → add_style_samples → train_style_lora   (a saved look; stack it on any character, or draw with it alone)

A character is one folder (.cache/characters/<name>/): sample pictures with captions, the LoRA trained on them, its bible and its per-panel corrections. A style is the same thing for a look (.cache/styles/<name>/). The product never describes a style in words: LoRAs trained on pictures carry character and rendering style; every generation is Anima + those LoRAs with content tags only. Nothing retrains unless you call train_*_lora; a correction costs seconds, not the whole run.

1. Samples — create_character(name, char_desc, attr?, trigger?, lora_name?) (char_desc names the subject: she/he/they; lora_name adopts an existing LoRA), add_samples(name, images, captions?) (comma-separated paths / URLs / data URLs; captions |-separated, e.g. the outfit of each picture so outfits are learned apart), remove_sample, set_caption, character_info, list_characters.
2. LoRA — train_character_lora(name, steps?) on fox (sd-scripts, bf16), then preview_character(name, tags?, seed?, count?, style?) to check identity and look before spending on the sheet.
3. Bible — generate_character_bible(name, seed?, style?): turnaround, leotard body reference, six expressions, three actions, three costumes, chibi, wardrobe items → aligned PNG sheet + self-contained HTML + panels. list_bible_panels · redraw_panel(name, panel, tags?, seed?, avoid?) fixes one panel (old one kept under history/; the correction sticks for the next sheet). generate_from_bible(name, prompt, style?) draws new pictures.
4. Styles — create_style(name, note?), add_style_samples(name, images, captions?), train_style_lora(name, steps?), style_info, list_styles, delete_style. set_character_style(name, style, strength?) makes a character wear a look (character from picture A, look from picture B). generate_image(prompt, style, …) draws anything in that look alone.

Also: generate_sprite(prompt, lora_name?, pose_image?) (Anima + ToonOut → RGBA sprites with measurements), refine_image(image, prompt, lora_name, denoise?) (img2img redraw with a LoRA), and the JoyAI reference-picture edit make_mask (SAM 3.1) → generate_variant (base pixels restored outside the mask). make_transparent (ToonOut) and pixelize (Pillow) finish sprites. Pictures come in as a cache path, an http(s) URL, a data: URL, or the WebUI upload (POST /api/upload).

Requirements

This is a remote-GPU orchestrator — it does not generate anything by itself. You need:

• A CUDA GPU running ComfyUI 0.34+ reachable over HTTP (RTX 5090 class; the JoyAI int8 edit model and Anima co-load).
• The model set: Anima Base/Turbo (+ qwen_3_06b_base, qwen_image_vae), Anima-Control-Pose, JoyAI-Image-Edit-Plus (+ qwen3vl_8b_joyimage_edit_plus, wan_2.1_vae), ToonOut via ComfyUI-RMBG, SAM 3.1 (native).
• Python 3.13 and uv for the backend; sd-scripts (Anima) on the GPU box for LoRA training.

Quickstart

git clone https://github.com/kitepon/sprite-forge-mcp.git && cd sprite-forge-mcp
uv sync
cp .env.example .env            # SPRITEFORGE_COMFY_URL → your ComfyUI, SPRITEFORGE_BOX_SSH → the GPU box
.venv/bin/uvicorn backend.app:app --host 0.0.0.0 --port 8765

Or on the server: docker compose up -d --build (port 8766 → 8765, .cache as a volume).

• WebUI → http://<host>:8765/ · health → curl <host>:8765/api/gpu
• MCP endpoint → http://<host>:8765/mcp/ (streamable HTTP; register it in your agent CLI)

MCP tools (the agent face)

MCP and REST call the same Services functions; defaults live only in those signatures.

Group	Tools
Status	gpu_status · list_loras · list_jobs · job_status
Sprites	generate_base · generate_sprite · make_transparent · pixelize
Character	create_character · add_samples · remove_sample · set_caption · set_character_style · character_info · list_characters
LoRA	train_character_lora · train_status · preview_character · refine_image
Bible	generate_character_bible · bible_status · list_bible_panels · redraw_panel · generate_from_bible
Style	create_style · add_style_samples · train_style_lora · style_info · list_styles · delete_style · generate_image
Variants	make_mask · generate_variant

Design principles (the "scars")

• Measure, don't gate. Corner alpha, bbox, canvas and bbox deltas are reported as numbers; nothing is rejected for you.
• The look lives in pictures, not in prompts. No style phrase is baked into the product; presets are bundles of pictures.
• Wait for the GPU, don't time it out. A job waits as long as ComfyUI holds the prompt; it fails only when the prompt is gone.
• No hand-paint. You point (mask / point / line); the model regenerates. Fixes are re-generation, never manual rescue.
• No silent fallback. Failures surface with the stage and reason; nothing is quietly "fixed."
• Adopt is explicit and irreversible — it writes into your game project only when you ask.

Documentation

• INSTALL.md — setup, optional features, honest limitations
• docs/00_overview.md — canonical documentation map
• docs/models.md — model sources, placement, required ComfyUI nodes
• AGENTS.md — canonical project instructions and current design policy
• docs/ — design docs (context, research, architecture, output contract, …)

License

MIT. Model weights are not included — download them yourself (see docs/models.md); each model carries its own license.