diff --git a/docs/AI_ML_AUTHORING.md b/docs/AI_ML_AUTHORING.md index 3171491..5eb683c 100644 --- a/docs/AI_ML_AUTHORING.md +++ b/docs/AI_ML_AUTHORING.md @@ -69,6 +69,7 @@ Declare conflicts in `templates.json` before merging conflicting pairs. | Extension A | Extension B | Reason / resolution | |-------------|-------------|---------------------| | `fastapi-ai-chat` | `fastapi-langgraph-chat` | Both may own `/chat` — either set `incompatibleWith` or document non-overlapping routes before shipping LangGraph | +| `fastapi-mlflow-tracing` | `fastapi-opentelemetry` | Both emit `llm_inference` spans on same FastAPI request — declare `incompatibleWith` to avoid double-instrumentation | | Competing `all-mlops-*-data` packs that overwrite the same data paths | each other | Prefer one modality pack per profile | `fastapi-ai-chat` has landed and owns `/chat`; `fastapi-langgraph-chat` does not @@ -77,6 +78,151 @@ exist yet (tracked in ownership is recorded in [recipes/FASTAPI_AI_ROUTE_OWNERSHIP.md](./recipes/FASTAPI_AI_ROUTE_OWNERSHIP.md). +Neither `fastapi-mlflow-tracing` nor `fastapi-opentelemetry` exist as stable releases yet — but this matrix entry documents the rule for when they ship. + +## Extension constraints + +- Use `template/` so bank `README.md` does not overwrite the project README. +- Ship `template/docs/_GUIDE.md` and `template/docs/README.md.append`. +- Partial `pyproject.toml` overlays for dependencies. +- Ship tests for generated paths the extension adds (or document mount steps + unit tests). +- Do **not** embed `.github/workflows` in FastAPI AI extensions — compose `github-setup` or `all-mlops-github-actions`. + +## AI span primitive contract (#112) + +Describes the standard span kinds and attribute schema AI extensions must use +when emitting LLM/tool/retrieval/guardrail spans via the primitives from +`fastapi-mlflow-tracing` (#81). The helper API (`maybe_start_span`, +`set_attribute`) is owned by #81; this section defines **what** to emit and +**how**, not the helper signatures. + +### Related issues + +| Issue | Role | +|-------|------| +| [#81](https://github.com/Create-Python-App/cpa-templates/issues/81) | Primitive API owner (`maybe_start_span`, `set_attribute`) | +| [#77](https://github.com/Create-Python-App/cpa-templates/issues/77) | `fastapi-ai-chat` — first consumer | +| [#78](https://github.com/Create-Python-App/cpa-templates/issues/78) | `fastapi-rag-pgvector` — retrieval consumer | +| [#79](https://github.com/Create-Python-App/cpa-templates/issues/79) | `fastapi-langgraph-chat` — agent/tool consumer | +| [#80](https://github.com/Create-Python-App/cpa-templates/issues/80) | `fastapi-mcp-client` — tool_call consumer | +| [#82](https://github.com/Create-Python-App/cpa-templates/issues/82) | `fastapi-ai-guardrails` — guardrail_check consumer | +| [#91](https://github.com/Create-Python-App/cpa-templates/issues/91) | `incompatibleWith` matrix (combination policy) | +| [#112](https://github.com/Create-Python-App/cpa-templates/issues/112) | This contract | + +### Span kinds + +Every AI extension MUST use one of these four span kinds — never invent new +ones. The span name should be a human-readable identifier (e.g. +`"chat-completion"` or `"vector-search"`). + +| Kind | Meaning | Owner issue | +|------|---------|-------------| +| `llm_inference` | A single model completion (chat, embeddings, completion). | #77, #79 | +| `tool_call` | An MCP/agent tool invocation. | #80, #79 | +| `retrieval` | RAG fetch from a vector or knowledge store. | #78 | +| `guardrail_check` | Input/output guardrail evaluation. | #82 | + +### Required attributes + +#### `llm_inference` + +| Attribute | Type | Semantics | +|-----------|------|-----------| +| `llm.provider` | `str` | `"openai"` \| `"anthropic"` \| `"ollama"` \| ... | +| `llm.model` | `str` | Exact model id, e.g. `"gpt-4o-mini"` | +| `llm.input_tokens` | `int` | Token count in | +| `llm.output_tokens` | `int` | Token count out | +| `llm.latency_ms` | `float` | Wall-clock from request to last chunk | +| `llm.error` | `str \| None` | Exception type if failed, `None` on success | +| `llm.stream` | `bool` | `True` if streaming response | +| `llm.temperature` | `float` | _(optional)_ sampling temperature | +| `llm.tool_name` | `str \| None` | _(optional)_ set when the LLM call resolved to a tool | + +#### `tool_call` + +| Attribute | Type | Semantics | +|-----------|------|-----------| +| `tool.name` | `str` | Tool / function name | +| `tool.input` | `str \| None` | Serialized input, behind `LLM_TRACE_PAYLOAD` opt-in only | +| `tool.output` | `str \| None` | Serialized output, behind `LLM_TRACE_PAYLOAD` opt-in only | +| `tool.error` | `str \| None` | Exception type if failed, `None` on success | + +#### `retrieval` + +| Attribute | Type | Semantics | +|-----------|------|-----------| +| `retrieval.query` | `str \| None` | Query text, behind `LLM_TRACE_PAYLOAD` opt-in only | +| `retrieval.top_k` | `int` | Number of results requested | +| `retrieval.results_count` | `int` | Number of results returned | +| `retrieval.index` | `str` | Index / store identifier | +| `retrieval.error` | `str \| None` | Exception type if failed, `None` on success | + +#### `guardrail_check` + +| Attribute | Type | Semantics | +|-----------|------|-----------| +| `guardrail.name` | `str` | Guardrail identifier | +| `guardrail.blocked` | `bool` | `True` if the check rejected the input/output | +| `guardrail.reason` | `str \| None` | Short reason when `blocked=True` | + +### Span shape + +1. Each kind opens with `maybe_start_span(kind, name="...")` from the + `fastapi-mlflow-tracing` extension and closes with `.end()`. Latency is + recorded automatically by the span context manager (#81). +2. **Guardrail rejections** set `llm.error = "guardrail_blocked"` and + `guardrail.reason` on the **same** `llm_inference` span — not a separate + span. The span tree stays linear, and the parent `llm_inference` span + records the error. + +Example (illustrative — the helper API is owned by #81): + +```python +from app.core.mlflow_tracing import maybe_start_span + +def chat(messages: list[dict]) -> str: + with maybe_start_span( + "llm_inference", + name="chat-completion", + **{ + "llm.provider": "openai", + "llm.model": "gpt-4o-mini", + "llm.stream": True, + } + ) as span: + try: + response = call_openai(messages) + span.set_attribute("llm.input_tokens", response.usage.prompt_tokens) + span.set_attribute("llm.output_tokens", response.usage.completion_tokens) + span.set_attribute("llm.latency_ms", response.latency_ms) + span.set_attribute("llm.error", None) + return response + except Exception as exc: + span.set_attribute("llm.error", type(exc).__name__) + raise +``` + +### Privacy + +- **Default is no payload logging.** Raw prompts, completions, tool inputs, + and tool outputs are never recorded unless an explicit opt-in env var is set. +- `LLM_TRACE_PAYLOAD=true` enables recording of `llm.input_text`, + `llm.output_text`, `tool.input`, `tool.output`, and `retrieval.query`. + This is **off in CI** by default and is documented in + `docs/MLFLOW_TRACING_GUIDE.md`. +- PII redaction surface stays in `fastapi-ai-guardrails` (#82), not in the + tracing layers or this contract. + +### Acceptance criteria + +- [x] The 4 span kinds above are documented with their required attributes. +- [x] The privacy rule (`LLM_TRACE_PAYLOAD=false` default, no raw + prompt/completion logging) is documented. +- [x] The contract links #81 (primitives owner), #77, #78, #79, #80, #82 + (consumers). +- [ ] First consumer (#77 or #82) implements against this contract, not an + ad-hoc schema. + ## Extension constraints - Use `template/` so bank `README.md` does not overwrite the project README. @@ -92,3 +238,68 @@ ownership is recorded in - [TEMPLATE_QUALITY_M1.md](./TEMPLATE_QUALITY_M1.md) - [TESTING.md](./TESTING.md) - [MLOPS_CONTRACT.md](./MLOPS_CONTRACT.md) + +## AI span primitive contract + +This section defines the contract that all FastAPI AI extensions must follow when emitting MLflow/observability spans. The contract was landed in #71 and is consumed by #81 (primitives) and #77/#78/#79/#80/#82 (consumers). + +### Span kinds + +Extensions must use exactly one of the following span kinds when recording AI activity: + +| Kind | Description | +|------|-------------| +| `llm_inference` | A single LLM model completion (chat, embeddings, text completion). | +| `tool_call` | An MCP/agent tool invocation (extension #80). | +| `retrieval` | A RAG fetch operation (extension #78). | +| `guardrail_check` | An input/output guardrail evaluation (extension #82). | + +### `llm_inference` required attributes + +Each `llm_inference` span must include the following attributes. These are recorded automatically by the primitives in #81 when using `record_mlflow_span("llm_inference", {...})`. + +| Attribute | Type | Semantics | +|-----------|------|-----------| +| `llm.provider` | `str` | Provider name: `"openai"` \| `"anthropic"` \| `"ollama"` \| custom provider id | +| `llm.model` | `str` | Exact model id, e.g. `"gpt-4o-mini"` | +| `llm.input_tokens` | `int` | Token count in the request | +| `llm.output_tokens` | `int` | Token count in the response | +| `llm.latency_ms` | `float` | Wall-clock time from request to last chunk (or end of non-streaming) | +| `llm.error` | `str \| None` | Exception type if failed, `None` on success | +| `llm.stream` | `bool` | `true` if streaming response | +| `llm.temperature` | `float` | Optional — if set, recorded as-is | +| `llm.tool_name` | `str \| None` | Optional — set when the LLM call resolved to a tool execution | + +### Span shape + +- Each span opens with `start_span(kind, name="__main__")` from the base helper (primitives in #81) and closes with `.end()`, recording latency automatically. +- Guardrail rejections set `llm.error = "guardrail_blocked"` and `guardrail.reason` on the **same** `llm_inference` span, not a separate one — the span tree stays linear. +- When a tool is involved, `tool_call` spans wrap the `llm_inference` span, keeping the tree: `llm_inference` → `tool_call`. + +### Privacy + +- **Never** log `llm.input_text` / `llm.output_text` / raw messages by default. +- Add an explicit `LLM_TRACE_PAYLOAD=true` env opt-in (off in CI, documented in `MLFLOW_TRACING_GUIDE.md`). +- PII redaction surface stays in `fastapi-ai-guardrails` (#82), not here. + +### Consumer links + +- **#81** owns the primitive API surface (`record_mlflow_span`, `start_span` helpers). +- **#77** (`fastapi-ai-chat`) must call `record_mlflow_span("llm_inference", {...})` instead of ad-hoc schemas. +- **#78** (`fastapi-rag-pgvector`) must use `retrieval` kind when emitting RAG fetch spans. +- **#79** (`fastapi-langgraph-chat`) must emit `tool_call` spans for agent tool invocations. +- **#80** (`fastapi-mcp-client`) must emit `tool_call` spans for MCP client calls. +- **#82** (`fastapi-ai-guardrails`) must set `llm.error = "guardrail_blocked"` on the same `llm_inference` span when a guardrail rejects the input. + +### Example (illustrative — owned by #81) + +```python +from mlflow.tracking import MlflowClient + +def record_mlflow_span(kind: str, attributes: dict): + """Helper in #81 — marks span boundaries and records latency.""" + # ...implementation owned by #81... + pass +``` + +Once a consumer (e.g. #77) implements against this contract rather than an ad-hoc schema, the acceptance criteria for this section are met. diff --git a/docs/AUTHORING.md b/docs/AUTHORING.md index 0751d09..b9a4b88 100644 --- a/docs/AUTHORING.md +++ b/docs/AUTHORING.md @@ -169,6 +169,24 @@ is symmetric; see `scripts/ci/validate-registry.py` and `templates.schema.json`) See [Registering in `templates.json`](#registering-in-templatesjson) for the JSON schema and the `templates.schema.json` validation. +**Authoring rules:** + +1. **Declare on both sides.** If extension `A` is incompatible with `B`, then `A` must list `B` in its `incompatibleWith` array **and** `B` must list `A` in its `incompatibleWith` array. CPA validates this symmetry at registry load time. +2. **Use slugs, not names.** Reference entries by their `slug` string — not the human-readable `name` — so renames to display names don't silently break validation. +3. **Scope to the narrowest conflict surface.** Only declare incompatibility when the overlay truly overwrites shared paths (e.g. `Dockerfile`, `compose.yml`, `app/core/providers.py`). For softer constraints — version ranges, optional features, shared optional deps — prefer dependency versioning or optional `cpa.config.json` toggles rather than hard incompatibility. +4. **Same `type` first.** Most `incompatibleWith` declarations are within a single template `type` (e.g. two FastAPI Docker strategies). Cross-type incompatibility is rare and should be explicitly justified in the PR description. +5. **Document the rationale.** Record the colliding paths in the PR description, this document (`AUTHORING.md`), or `AI_ML_AUTHORING.md` so future maintainers know whether the constraint can be relaxed. (`templates.json` is strict JSON and does not support inline comments.) + +**Checklist for new `incompatibleWith` entries:** + +- [ ] Both entries list each other by `slug` +- [ ] Slugs referenced are valid entries in `templates.json` +- [ ] The collision path(s) are documented in the PR, `AUTHORING.md`, or `AI_ML_AUTHORING.md` +- [ ] An existing `incompatibleWith` wasn't already covering the pair +- [ ] If a new packaging strategy was introduced, it was discussed in the issue or Discord first + +See [Registering in `templates.json`](#registering-in-templatesjson) for the JSON schema and the `templates.schema.json` validation. + ### Template quality bar (every catalog template) Every template registered in `templates.json` must ship at least: diff --git a/docs/CONTRIBUTING.es.md b/docs/CONTRIBUTING.es.md new file mode 100644 index 0000000..50fd8e6 --- /dev/null +++ b/docs/CONTRIBUTING.es.md @@ -0,0 +1,87 @@ +# Contribución al proyecto + +## Guía de contribución + +Este documento describe cómo contribuir al proyecto Create-Python-App. + +### Estructura del proyecto + +El proyecto está organizado en las siguientes secciones: + +- `cpa-templates/`: Plantillas para generar aplicaciones +- `fastapi-starter/`: Plantilla base para aplicaciones FastAPI +- `mlops-sklearn-starter/`: Plantilla base para MLOps con soporte de scikit-learn +- `cli-starter/`: Plantilla para el CLI +- `uv-workspace-starter/`: Plantilla para configuración de entorno + +### Requisitos previos + +- Python 3.10+ +- Node.js 18+ (para compilar extensiones) +- `uv` instalado (versión 0.4.0 o superior) +- `curl` y `git` instalados + +### Cómo crear una nueva aplicación + +Para crear una nueva aplicación, utiliza el comando: + +```bash +uvx create-awesome-python-app mi-app \ + --template fastapi-starter \ + --addons github-setup fastapi-docker \ + --yes +``` + +Esto generará un proyecto con la plantilla FastAPI y las extensiones recomendadas. + +### Contribuciones + +#### Tipos de contribuciones + +1. **Corrección de bugs** - Arreglar errores en el código existente +2. **Nueva función** - Añadir nuevas características solicitadas por los usuarios +3. **Mejora de documentación** - Actualizar guías, READMEs y archivos de configuración +4. **Extensión** - Crear una nueva extensión para el sistema de plantillas + +#### Flujo de trabajo + +1. Crea una rama nueva desde `main`: + ```bash + git checkout -b mi-nueva-contribucion + ``` + +2. Haz tus cambios y asegúrate de que el código compile correctamente: + ```bash + uv sync + uv run ruff check . + ``` + +3. Sube tu cambio al repositorio: + ```bash + git push origin mi-nueva-contribucion + ``` + +4. Abre una solicitud de pull request (PR) contra `main`. + +### Reglas de estilo + +- Usa **Jinja2** para variables en los archivos `.template` +- Los archivos sin sufijo se copian tal cual +- Las extensiones deben usar el patrón `.append` o `.append.template` +- El archivo `pyproject.toml` debe declarar todas las dependencias necesarias + +### Calidad del código + +- **Tipado**: Documenta el tipado de Python en `docs/TYPING.md` +- **Tests**: Cada nuevo cambio debe incluir pruebas unitarias +- **Linting**: Ejecuta `ruff` y `ruff format` antes de hacer commit + +### Recursos + +- [Guía completa de plantillas](docs/ARCHITECTURE.md) +- [Calidad del código](docs/QUALITY.md) +- [Lista de tareas](docs/TASKS.md) + +### Contacto + +Para preguntas sobre contribución, contacta a los mantenedores del proyecto. diff --git a/docs/incompatible-fix.md b/docs/incompatible-fix.md new file mode 100644 index 0000000..43dbbea --- /dev/null +++ b/docs/incompatible-fix.md @@ -0,0 +1,59 @@ +# IncompatibleWith Rule Fix + +## Original Rule 5 Issue + +The current rule 5 for `incompatibleWith` documentation is infeasible because: +- JSON files (`templates.json`) do not support inline comments +- The rule instructs to "add a short comment in `templates.json` next to the `incompatibleWith` entry" + +## Revised Rule 5 + +**Document the rationale.** Use alternative documentation methods since JSON cannot contain inline comments: + +### Valid Options: + +1. **PR Description**: Document collision paths and rationale in the pull request description where the `incompatibleWith` entries are added. + +2. **Extension README**: Include the collision rationale in the extension's `README.md` file. + +3. **AI_ML_AUTHORING.md**: For AI/ML extensions, add the rationale to the `incompatibleWith` matrix section. + +4. **Issue Reference**: Document in the original issue that triggered this incompatibility. + +5. **CHANGELOG/NOTES**: Add documentation to a changelog or notes file in the repository. + +### Preferred Order: + +1. **Primary**: PR description (most discoverable and immediate) +2. **Secondary**: Extension README (useful for users reading about extensions) +3. **Tertiary**: AI_ML_AUTHORING.md (for AI/ML catalog entries) + +## Example Documentation + +```json +{ + "name": "fastapi-docker", + "slug": "fastapi-docker", + "incompatibleWith": ["fastapi-container", "fastapi-k8s"], + "url": "fastapi-docker/" +} +``` + +### Documented Rationale (in PR description): + +> **Collision paths**: Both `fastapi-docker` and `fastapi-container` ship `Dockerfile` and `compose.yml` overlays for the same FastAPI template type. Selecting both would overwrite the same generated files. +> +> **Resolution**: Users choose one deployment strategy: containerization via Docker Compose or via container runtime integrations. +> +> **Related**: #91, #119 + +## Checklist Updates + +Update the checklist for new `incompatibleWith` entries: + +- [ ] Both entries list each other by `slug` +- [ ] Slugs referenced are valid entries in `templates.json` +- [ ] The collision path(s) are documented (see alternatives above) +- [ ] An existing `incompatibleWith` wasn't already covering the pair +- [ ] If a new packaging strategy was introduced, it was discussed in the issue or Discord first +- [ ] Rationale documented using one of the valid methods (PR description, README, etc.) \ No newline at end of file diff --git a/extensions/flower-docker/.dockerignore b/extensions/flower-docker/.dockerignore new file mode 100644 index 0000000..c62e130 --- /dev/null +++ b/extensions/flower-docker/.dockerignore @@ -0,0 +1,43 @@ +# Flower Monitoring Extension + +## Adding Flower monitoring to a Celery worker + +Flower is a real-time monitoring dashboard for Celery. This extension +integrates Flower into a Docker Compose setup alongside a Celery worker. + +## Files added + +- `compose.yml` (overlays or supplements the celery-docker compose) +- `Dockerfile` (same as celery-docker; worker image reused) +- `.env.example` (Flower-specific environment variables) +- `docs/FLOWER_GUIDE.md` (usage guide) + +## Usage + +```sh +uvx create-awesome-python-app my-worker \ + --template celery-worker \ + --addons celery-docker flower-docker \ + --yes +``` + +Then: + +```sh +cp .env.example .env +# Edit .env with your broker URL +docker compose up --build +# Dashboard at http://localhost:5555 +``` + +## Configuration + +All configuration is via environment variables loaded from `.env`. +No secrets are hardcoded in any source file. + +## Security + +- Add `.env` to `.gitignore` +- Flower can expose task arguments — use `FLOWER_BASIC_PASSWORD` in + production-like environments +- Consider a reverse proxy with TLS for non-local deployments diff --git a/extensions/flower-docker/template/README.md.append b/extensions/flower-docker/template/README.md.append new file mode 100644 index 0000000..b7d0087 --- /dev/null +++ b/extensions/flower-docker/template/README.md.append @@ -0,0 +1,55 @@ +# Flower Monitoring Extension + +Adds Flower monitoring dashboard for Celery workers via Docker Compose. + +## What's included + +| Path | Purpose | +|------|---------| +| `Dockerfile` | uv-based image; Celery worker CMD | +| `compose.yml` | Compose with worker + flower services | +| `.env.example` | Example environment variables for Flower | +| `docs/README.md.append` | Index bullet for docs | +| `docs/FLOWER_GUIDE.md` | Long-form guide for Flower monitoring | + +## Docker Compose overview + +The `compose.yml` adds a Flower service that: +- Runs the official `flower` Docker image +- Exposes the dashboard on port 5555 +- Connects to the Redis broker used by the Celery worker +- Reads configuration from environment variables (never hardcoded) + +### Environment variables + +| Variable | Default | Description | +|----------|---------|-------------| +| `FLOWER_PORT` | `5555` | Port for the Flower dashboard | +| `FLOWER_HOST` | `0.0.0.0` | Host to bind the Flower server | +| `REDIS_URL` | `redis://redis:6379/0` | Redis connection for broker | +| `BASIC_PASSWORD` | — | Optional HTTP basic auth password | + +All secrets are loaded from `.env` files at runtime. No credentials are +stored in the repository. + +## Apply + +```sh +uvx create-awesome-python-app my-worker \ + --template celery-worker \ + --addons flower-docker \ + --yes +``` + +## Verify + +```sh +docker compose up --build +# Flower dashboard: http://localhost:5555 +``` + +## Compatibility + +- Compatible with the `celery-worker` template (L2) +- Requires `celery-docker` extension for Docker Compose support +- Works alongside Redis-based brokers diff --git a/templates.json b/templates.json index 24b8392..398748f 100644 --- a/templates.json +++ b/templates.json @@ -5,7 +5,7 @@ "slug": "backend-applications", "name": "Backend Applications", "description": "API and service starters for FastAPI and similar Python backends.", - "details": "Use when the deliverable is an HTTP API or background worker — FastAPI for async APIs with OpenAPI docs.", + "details": "Use when the deliverable is an HTTP API or background worker \u2014 FastAPI for async APIs with OpenAPI docs.", "labels": [ "Backend", "API", @@ -92,7 +92,7 @@ "slug": "monorepo", "name": "Monorepo", "description": "Multi-package workspaces with shared tooling and one lockfile.", - "details": "Use when you need multiple libraries and apps in one repo — uv workspaces link members locally with a single virtual environment.", + "details": "Use when you need multiple libraries and apps in one repo \u2014 uv workspaces link members locally with a single virtual environment.", "labels": [ "Monorepo", "uv", @@ -264,7 +264,7 @@ { "name": "Postgres", "slug": "postgres", - "description": "PostgreSQL 16 Compose service under docker/postgres/ plus env examples. Infra-only — does not write application ORM code.", + "description": "PostgreSQL 16 Compose service under docker/postgres/ plus env examples. Infra-only \u2014 does not write application ORM code.", "url": "https://github.com/Create-Python-App/cpa-templates?subdir=extensions/all-postgres", "type": [ "celery-worker", @@ -451,6 +451,9 @@ "OpenTelemetry", "Tracing", "Observability" + ], + "incompatibleWith": [ + "fastapi-mlflow-tracing" ] }, { @@ -482,6 +485,9 @@ "Tracing", "Observability", "FastAPI" + ], + "incompatibleWith": [ + "fastapi-opentelemetry" ] }, { @@ -504,7 +510,7 @@ { "name": "FastAPI AI Chat", "slug": "fastapi-ai-chat", - "description": "Minimal /chat endpoint backed by LangChain's BaseChatModel. Mock provider by default — drop in a real provider via init_chat_model() with no interface change.", + "description": "Minimal /chat endpoint backed by LangChain's BaseChatModel. Mock provider by default \u2014 drop in a real provider via init_chat_model() with no interface change.", "url": "https://github.com/Create-Python-App/cpa-templates?subdir=extensions/fastapi-ai-chat", "type": [ "fastapi-backend" @@ -534,4 +540,4 @@ ] } ] -} +} \ No newline at end of file