-
Notifications
You must be signed in to change notification settings - Fork 7
fix: AI/ML CI profile and documentation updates #191
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
43a8b9e
6f0ddfb
6a88888
4276e06
af501d6
8893fea
cbd1996
897ef3f
ca058d8
7e95427
35447e5
872cd31
492bfba
576ffef
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -69,6 +69,7 @@ Declare conflicts in `templates.json` before merging conflicting pairs. | |
| | Extension A | Extension B | Reason / resolution | | ||
| |-------------|-------------|---------------------| | ||
| | `fastapi-ai-chat` | `fastapi-langgraph-chat` | Both may own `/chat` — either set `incompatibleWith` or document non-overlapping routes before shipping LangGraph | | ||
| | `fastapi-mlflow-tracing` | `fastapi-opentelemetry` | Both emit `llm_inference` spans on same FastAPI request — declare `incompatibleWith` to avoid double-instrumentation | | ||
| | Competing `all-mlops-*-data` packs that overwrite the same data paths | each other | Prefer one modality pack per profile | | ||
|
|
||
| `fastapi-ai-chat` has landed and owns `/chat`; `fastapi-langgraph-chat` does not | ||
|
|
@@ -77,6 +78,151 @@ exist yet (tracked in | |
| ownership is recorded in | ||
| [recipes/FASTAPI_AI_ROUTE_OWNERSHIP.md](./recipes/FASTAPI_AI_ROUTE_OWNERSHIP.md). | ||
|
|
||
| Neither `fastapi-mlflow-tracing` nor `fastapi-opentelemetry` exist as stable releases yet — but this matrix entry documents the rule for when they ship. | ||
|
|
||
| ## Extension constraints | ||
|
|
||
| - Use `template/` so bank `README.md` does not overwrite the project README. | ||
| - Ship `template/docs/<TOPIC>_GUIDE.md` and `template/docs/README.md.append`. | ||
| - Partial `pyproject.toml` overlays for dependencies. | ||
| - Ship tests for generated paths the extension adds (or document mount steps + unit tests). | ||
| - Do **not** embed `.github/workflows` in FastAPI AI extensions — compose `github-setup` or `all-mlops-github-actions`. | ||
|
|
||
| ## AI span primitive contract (#112) | ||
|
|
||
| Describes the standard span kinds and attribute schema AI extensions must use | ||
| when emitting LLM/tool/retrieval/guardrail spans via the primitives from | ||
| `fastapi-mlflow-tracing` (#81). The helper API (`maybe_start_span`, | ||
| `set_attribute`) is owned by #81; this section defines **what** to emit and | ||
| **how**, not the helper signatures. | ||
|
|
||
| ### Related issues | ||
|
|
||
| | Issue | Role | | ||
| |-------|------| | ||
| | [#81](https://github.com/Create-Python-App/cpa-templates/issues/81) | Primitive API owner (`maybe_start_span`, `set_attribute`) | | ||
| | [#77](https://github.com/Create-Python-App/cpa-templates/issues/77) | `fastapi-ai-chat` — first consumer | | ||
| | [#78](https://github.com/Create-Python-App/cpa-templates/issues/78) | `fastapi-rag-pgvector` — retrieval consumer | | ||
| | [#79](https://github.com/Create-Python-App/cpa-templates/issues/79) | `fastapi-langgraph-chat` — agent/tool consumer | | ||
| | [#80](https://github.com/Create-Python-App/cpa-templates/issues/80) | `fastapi-mcp-client` — tool_call consumer | | ||
| | [#82](https://github.com/Create-Python-App/cpa-templates/issues/82) | `fastapi-ai-guardrails` — guardrail_check consumer | | ||
| | [#91](https://github.com/Create-Python-App/cpa-templates/issues/91) | `incompatibleWith` matrix (combination policy) | | ||
| | [#112](https://github.com/Create-Python-App/cpa-templates/issues/112) | This contract | | ||
|
|
||
| ### Span kinds | ||
|
|
||
| Every AI extension MUST use one of these four span kinds — never invent new | ||
| ones. The span name should be a human-readable identifier (e.g. | ||
| `"chat-completion"` or `"vector-search"`). | ||
|
|
||
| | Kind | Meaning | Owner issue | | ||
| |------|---------|-------------| | ||
| | `llm_inference` | A single model completion (chat, embeddings, completion). | #77, #79 | | ||
| | `tool_call` | An MCP/agent tool invocation. | #80, #79 | | ||
| | `retrieval` | RAG fetch from a vector or knowledge store. | #78 | | ||
| | `guardrail_check` | Input/output guardrail evaluation. | #82 | | ||
|
|
||
| ### Required attributes | ||
|
|
||
| #### `llm_inference` | ||
|
|
||
| | Attribute | Type | Semantics | | ||
| |-----------|------|-----------| | ||
| | `llm.provider` | `str` | `"openai"` \| `"anthropic"` \| `"ollama"` \| ... | | ||
| | `llm.model` | `str` | Exact model id, e.g. `"gpt-4o-mini"` | | ||
| | `llm.input_tokens` | `int` | Token count in | | ||
| | `llm.output_tokens` | `int` | Token count out | | ||
| | `llm.latency_ms` | `float` | Wall-clock from request to last chunk | | ||
| | `llm.error` | `str \| None` | Exception type if failed, `None` on success | | ||
| | `llm.stream` | `bool` | `True` if streaming response | | ||
| | `llm.temperature` | `float` | _(optional)_ sampling temperature | | ||
| | `llm.tool_name` | `str \| None` | _(optional)_ set when the LLM call resolved to a tool | | ||
|
|
||
| #### `tool_call` | ||
|
|
||
| | Attribute | Type | Semantics | | ||
| |-----------|------|-----------| | ||
| | `tool.name` | `str` | Tool / function name | | ||
| | `tool.input` | `str \| None` | Serialized input, behind `LLM_TRACE_PAYLOAD` opt-in only | | ||
| | `tool.output` | `str \| None` | Serialized output, behind `LLM_TRACE_PAYLOAD` opt-in only | | ||
| | `tool.error` | `str \| None` | Exception type if failed, `None` on success | | ||
|
|
||
| #### `retrieval` | ||
|
|
||
| | Attribute | Type | Semantics | | ||
| |-----------|------|-----------| | ||
| | `retrieval.query` | `str \| None` | Query text, behind `LLM_TRACE_PAYLOAD` opt-in only | | ||
| | `retrieval.top_k` | `int` | Number of results requested | | ||
| | `retrieval.results_count` | `int` | Number of results returned | | ||
| | `retrieval.index` | `str` | Index / store identifier | | ||
| | `retrieval.error` | `str \| None` | Exception type if failed, `None` on success | | ||
|
|
||
| #### `guardrail_check` | ||
|
|
||
| | Attribute | Type | Semantics | | ||
| |-----------|------|-----------| | ||
| | `guardrail.name` | `str` | Guardrail identifier | | ||
| | `guardrail.blocked` | `bool` | `True` if the check rejected the input/output | | ||
| | `guardrail.reason` | `str \| None` | Short reason when `blocked=True` | | ||
|
|
||
| ### Span shape | ||
|
|
||
| 1. Each kind opens with `maybe_start_span(kind, name="...")` from the | ||
| `fastapi-mlflow-tracing` extension and closes with `.end()`. Latency is | ||
| recorded automatically by the span context manager (#81). | ||
| 2. **Guardrail rejections** set `llm.error = "guardrail_blocked"` and | ||
| `guardrail.reason` on the **same** `llm_inference` span — not a separate | ||
| span. The span tree stays linear, and the parent `llm_inference` span | ||
| records the error. | ||
|
|
||
| Example (illustrative — the helper API is owned by #81): | ||
|
|
||
| ```python | ||
| from app.core.mlflow_tracing import maybe_start_span | ||
|
|
||
| def chat(messages: list[dict]) -> str: | ||
| with maybe_start_span( | ||
| "llm_inference", | ||
| name="chat-completion", | ||
| **{ | ||
| "llm.provider": "openai", | ||
| "llm.model": "gpt-4o-mini", | ||
| "llm.stream": True, | ||
| } | ||
| ) as span: | ||
| try: | ||
| response = call_openai(messages) | ||
| span.set_attribute("llm.input_tokens", response.usage.prompt_tokens) | ||
| span.set_attribute("llm.output_tokens", response.usage.completion_tokens) | ||
| span.set_attribute("llm.latency_ms", response.latency_ms) | ||
| span.set_attribute("llm.error", None) | ||
| return response | ||
| except Exception as exc: | ||
| span.set_attribute("llm.error", type(exc).__name__) | ||
| raise | ||
| ``` | ||
|
|
||
| ### Privacy | ||
|
|
||
| - **Default is no payload logging.** Raw prompts, completions, tool inputs, | ||
| and tool outputs are never recorded unless an explicit opt-in env var is set. | ||
| - `LLM_TRACE_PAYLOAD=true` enables recording of `llm.input_text`, | ||
| `llm.output_text`, `tool.input`, `tool.output`, and `retrieval.query`. | ||
| This is **off in CI** by default and is documented in | ||
| `docs/MLFLOW_TRACING_GUIDE.md`. | ||
| - PII redaction surface stays in `fastapi-ai-guardrails` (#82), not in the | ||
| tracing layers or this contract. | ||
|
|
||
| ### Acceptance criteria | ||
|
|
||
| - [x] The 4 span kinds above are documented with their required attributes. | ||
| - [x] The privacy rule (`LLM_TRACE_PAYLOAD=false` default, no raw | ||
| prompt/completion logging) is documented. | ||
| - [x] The contract links #81 (primitives owner), #77, #78, #79, #80, #82 | ||
| (consumers). | ||
| - [ ] First consumer (#77 or #82) implements against this contract, not an | ||
| ad-hoc schema. | ||
|
|
||
| ## Extension constraints | ||
|
|
||
| - Use `template/` so bank `README.md` does not overwrite the project README. | ||
|
|
@@ -92,3 +238,68 @@ ownership is recorded in | |
| - [TEMPLATE_QUALITY_M1.md](./TEMPLATE_QUALITY_M1.md) | ||
| - [TESTING.md](./TESTING.md) | ||
| - [MLOPS_CONTRACT.md](./MLOPS_CONTRACT.md) | ||
|
|
||
| ## AI span primitive contract | ||
|
|
||
| This section defines the contract that all FastAPI AI extensions must follow when emitting MLflow/observability spans. The contract was landed in #71 and is consumed by #81 (primitives) and #77/#78/#79/#80/#82 (consumers). | ||
|
|
||
| ### Span kinds | ||
|
|
||
| Extensions must use exactly one of the following span kinds when recording AI activity: | ||
|
|
||
| | Kind | Description | | ||
| |------|-------------| | ||
| | `llm_inference` | A single LLM model completion (chat, embeddings, text completion). | | ||
| | `tool_call` | An MCP/agent tool invocation (extension #80). | | ||
| | `retrieval` | A RAG fetch operation (extension #78). | | ||
| | `guardrail_check` | An input/output guardrail evaluation (extension #82). | | ||
|
|
||
| ### `llm_inference` required attributes | ||
|
|
||
| Each `llm_inference` span must include the following attributes. These are recorded automatically by the primitives in #81 when using `record_mlflow_span("llm_inference", {...})`. | ||
|
|
||
| | Attribute | Type | Semantics | | ||
| |-----------|------|-----------| | ||
| | `llm.provider` | `str` | Provider name: `"openai"` \| `"anthropic"` \| `"ollama"` \| custom provider id | | ||
| | `llm.model` | `str` | Exact model id, e.g. `"gpt-4o-mini"` | | ||
| | `llm.input_tokens` | `int` | Token count in the request | | ||
| | `llm.output_tokens` | `int` | Token count in the response | | ||
| | `llm.latency_ms` | `float` | Wall-clock time from request to last chunk (or end of non-streaming) | | ||
| | `llm.error` | `str \| None` | Exception type if failed, `None` on success | | ||
| | `llm.stream` | `bool` | `true` if streaming response | | ||
| | `llm.temperature` | `float` | Optional — if set, recorded as-is | | ||
| | `llm.tool_name` | `str \| None` | Optional — set when the LLM call resolved to a tool execution | | ||
|
|
||
| ### Span shape | ||
|
|
||
| - Each span opens with `start_span(kind, name="__main__")` from the base helper (primitives in #81) and closes with `.end()`, recording latency automatically. | ||
| - Guardrail rejections set `llm.error = "guardrail_blocked"` and `guardrail.reason` on the **same** `llm_inference` span, not a separate one — the span tree stays linear. | ||
| - When a tool is involved, `tool_call` spans wrap the `llm_inference` span, keeping the tree: `llm_inference` → `tool_call`. | ||
|
|
||
| ### Privacy | ||
|
|
||
| - **Never** log `llm.input_text` / `llm.output_text` / raw messages by default. | ||
| - Add an explicit `LLM_TRACE_PAYLOAD=true` env opt-in (off in CI, documented in `MLFLOW_TRACING_GUIDE.md`). | ||
| - PII redaction surface stays in `fastapi-ai-guardrails` (#82), not here. | ||
|
|
||
| ### Consumer links | ||
|
|
||
| - **#81** owns the primitive API surface (`record_mlflow_span`, `start_span` helpers). | ||
| - **#77** (`fastapi-ai-chat`) must call `record_mlflow_span("llm_inference", {...})` instead of ad-hoc schemas. | ||
| - **#78** (`fastapi-rag-pgvector`) must use `retrieval` kind when emitting RAG fetch spans. | ||
| - **#79** (`fastapi-langgraph-chat`) must emit `tool_call` spans for agent tool invocations. | ||
| - **#80** (`fastapi-mcp-client`) must emit `tool_call` spans for MCP client calls. | ||
| - **#82** (`fastapi-ai-guardrails`) must set `llm.error = "guardrail_blocked"` on the same `llm_inference` span when a guardrail rejects the input. | ||
|
|
||
| ### Example (illustrative — owned by #81) | ||
|
|
||
| ```python | ||
| from mlflow.tracking import MlflowClient | ||
|
|
||
| def record_mlflow_span(kind: str, attributes: dict): | ||
| """Helper in #81 — marks span boundaries and records latency.""" | ||
| # ...implementation owned by #81... | ||
| pass | ||
| ``` | ||
|
|
||
| Once a consumer (e.g. #77) implements against this contract rather than an ad-hoc schema, the acceptance criteria for this section are met. | ||
|
Comment on lines
+242
to
+305
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win Remove superseded documentation sections instead of keeping competing copies. Keep one canonical contract and one canonical incompatibility rule set.
🧰 Tools🪛 LanguageTool[grammar] ~242-~242: Use a hyphen to join words. (QB_NEW_EN_HYPHEN) 📍 Affects 2 files
🤖 Prompt for AI Agents |
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -169,6 +169,24 @@ is symmetric; see `scripts/ci/validate-registry.py` and `templates.schema.json`) | |
|
|
||
| See [Registering in `templates.json`](#registering-in-templatesjson) for the JSON schema and the `templates.schema.json` validation. | ||
|
|
||
| **Authoring rules:** | ||
|
|
||
| 1. **Declare on both sides.** If extension `A` is incompatible with `B`, then `A` must list `B` in its `incompatibleWith` array **and** `B` must list `A` in its `incompatibleWith` array. CPA validates this symmetry at registry load time. | ||
| 2. **Use slugs, not names.** Reference entries by their `slug` string — not the human-readable `name` — so renames to display names don't silently break validation. | ||
| 3. **Scope to the narrowest conflict surface.** Only declare incompatibility when the overlay truly overwrites shared paths (e.g. `Dockerfile`, `compose.yml`, `app/core/providers.py`). For softer constraints — version ranges, optional features, shared optional deps — prefer dependency versioning or optional `cpa.config.json` toggles rather than hard incompatibility. | ||
| 4. **Same `type` first.** Most `incompatibleWith` declarations are within a single template `type` (e.g. two FastAPI Docker strategies). Cross-type incompatibility is rare and should be explicitly justified in the PR description. | ||
| 5. **Document the rationale.** Record the colliding paths in the PR description, this document (`AUTHORING.md`), or `AI_ML_AUTHORING.md` so future maintainers know whether the constraint can be relaxed. (`templates.json` is strict JSON and does not support inline comments.) | ||
|
Comment on lines
+176
to
+178
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win Allow documented behavioral incompatibilities. The new 🤖 Prompt for AI Agents |
||
|
|
||
| **Checklist for new `incompatibleWith` entries:** | ||
|
|
||
| - [ ] Both entries list each other by `slug` | ||
| - [ ] Slugs referenced are valid entries in `templates.json` | ||
| - [ ] The collision path(s) are documented in the PR, `AUTHORING.md`, or `AI_ML_AUTHORING.md` | ||
| - [ ] An existing `incompatibleWith` wasn't already covering the pair | ||
| - [ ] If a new packaging strategy was introduced, it was discussed in the issue or Discord first | ||
|
|
||
| See [Registering in `templates.json`](#registering-in-templatesjson) for the JSON schema and the `templates.schema.json` validation. | ||
|
|
||
| ### Template quality bar (every catalog template) | ||
|
|
||
| Every template registered in `templates.json` must ship at least: | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| # Contribución al proyecto | ||
|
|
||
| ## Guía de contribución | ||
|
|
||
| Este documento describe cómo contribuir al proyecto Create-Python-App. | ||
|
|
||
| ### Estructura del proyecto | ||
|
|
||
| El proyecto está organizado en las siguientes secciones: | ||
|
|
||
| - `cpa-templates/`: Plantillas para generar aplicaciones | ||
| - `fastapi-starter/`: Plantilla base para aplicaciones FastAPI | ||
| - `mlops-sklearn-starter/`: Plantilla base para MLOps con soporte de scikit-learn | ||
| - `cli-starter/`: Plantilla para el CLI | ||
| - `uv-workspace-starter/`: Plantilla para configuración de entorno | ||
|
|
||
| ### Requisitos previos | ||
|
|
||
| - Python 3.10+ | ||
| - Node.js 18+ (para compilar extensiones) | ||
| - `uv` instalado (versión 0.4.0 o superior) | ||
| - `curl` y `git` instalados | ||
|
|
||
| ### Cómo crear una nueva aplicación | ||
|
|
||
| Para crear una nueva aplicación, utiliza el comando: | ||
|
|
||
| ```bash | ||
| uvx create-awesome-python-app mi-app \ | ||
| --template fastapi-starter \ | ||
| --addons github-setup fastapi-docker \ | ||
| --yes | ||
| ``` | ||
|
|
||
| Esto generará un proyecto con la plantilla FastAPI y las extensiones recomendadas. | ||
|
|
||
| ### Contribuciones | ||
|
|
||
| #### Tipos de contribuciones | ||
|
|
||
| 1. **Corrección de bugs** - Arreglar errores en el código existente | ||
| 2. **Nueva función** - Añadir nuevas características solicitadas por los usuarios | ||
| 3. **Mejora de documentación** - Actualizar guías, READMEs y archivos de configuración | ||
| 4. **Extensión** - Crear una nueva extensión para el sistema de plantillas | ||
|
|
||
| #### Flujo de trabajo | ||
|
|
||
| 1. Crea una rama nueva desde `main`: | ||
| ```bash | ||
| git checkout -b mi-nueva-contribucion | ||
| ``` | ||
|
|
||
| 2. Haz tus cambios y asegúrate de que el código compile correctamente: | ||
| ```bash | ||
| uv sync | ||
| uv run ruff check . | ||
| ``` | ||
|
Comment on lines
+53
to
+57
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win Incluye la validación local requerida. El flujo no prueba una plantilla o extensión generada. Añade la generación local con 🤖 Prompt for AI AgentsSource: Coding guidelines |
||
|
|
||
| 3. Sube tu cambio al repositorio: | ||
| ```bash | ||
| git push origin mi-nueva-contribucion | ||
| ``` | ||
|
|
||
| 4. Abre una solicitud de pull request (PR) contra `main`. | ||
|
|
||
| ### Reglas de estilo | ||
|
|
||
| - Usa **Jinja2** para variables en los archivos `.template` | ||
| - Los archivos sin sufijo se copian tal cual | ||
| - Las extensiones deben usar el patrón `.append` o `.append.template` | ||
| - El archivo `pyproject.toml` debe declarar todas las dependencias necesarias | ||
|
|
||
| ### Calidad del código | ||
|
|
||
| - **Tipado**: Documenta el tipado de Python en `docs/TYPING.md` | ||
| - **Tests**: Cada nuevo cambio debe incluir pruebas unitarias | ||
| - **Linting**: Ejecuta `ruff` y `ruff format` antes de hacer commit | ||
|
|
||
| ### Recursos | ||
|
|
||
| - [Guía completa de plantillas](docs/ARCHITECTURE.md) | ||
| - [Calidad del código](docs/QUALITY.md) | ||
| - [Lista de tareas](docs/TASKS.md) | ||
|
Comment on lines
+81
to
+83
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win Corrige las rutas relativas de los recursos. Desde 🤖 Prompt for AI Agents |
||
|
|
||
| ### Contacto | ||
|
|
||
| Para preguntas sobre contribución, contacta a los mantenedores del proyecto. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Clarify the
guardrail_checkspan rule.The document defines
guardrail_checkas a span kind, then requires a rejected guardrail to writellm.errorandguardrail.reasononllm_inferenceinstead of a separate span. State when consumers emitguardrail_check, when they annotatellm_inference, and whether both spans can exist. Otherwise, the same workflow can produce incompatible span trees.🤖 Prompt for AI Agents